GPU kernel performance modeling / Spark-H3

How Spark-H3 Works + Speedup Lab

Explore Spark-H3’s reblocking and reweighting ideas step-by-step, then project their real impact on DiT-layer and end-to-end video latency.

Self-contained interactive HTML · no external libraries
Sparse layer speedup
1.75×
57.0 / 100 ms
Step-averaged / modeled speedup
1.35×
Across warmup and fallback layers
Attention-only ceiling
2.50×
Amdahl limit without overhead
Reblock + routing overhead
5.0 ms
Per optimized layer (amortized)

The idea behind Spark-H3

MiniMax-H3 · adaptive block sparse attention
01 / Exact attention branch
Spark-Reblock

Consecutive video positions are not necessarily similar in attention space, especially in noisy denoising steps. Reblock partitions tokens by learned Q/K affinity so a fixed Top-K block budget captures more important attention interactions.

Original Q / K→Landmark tree→Coherent 64-token blocks→Top-K exact
02 / Compressed attention branch
Spark-Reweight

Skipping the other key blocks entirely would discard their attention mass. Spark-Reweight creates weighted K/V summaries for the unselected tail and corrects their softmax mass with a log-bias, instead of treating a simple block mean as exact.

Unselected K/V→Weighted summary→Log-mass correction→Merge output
Put together: Spark-H3 computes the most relevant blocks exactly and covers the rest using a cheaper corrected approximation. Conditioning text/video/audio K/V sinks remain exact. In the reference integration, tokens form 64-wide leaf blocks, typical Top-K video-block density is 10%, and dense warmup steps keep early evaluations exact.

Interactive reblock & sparse-attention walkthrough

16 toy tokens · 4-token blocks; real integration uses 64-token blocks
Walkthrough / step-by-step
Better block recallReordering increases the chance that a single representative reflects the exact attention value of its tokens.
Two paths, one outputSelected blocks use dense QK/PV; skipped blocks contribute through reweighted K/V summaries.
Speedup has a priceLandmark construction, token permutation, Top-K routing, and output restoration all add overhead.
Published ablation · Spark-Reblock only

Does grouping actually improve selection?

At the same 10% exact video-block budget, the authors report an increase in attention-mass recall for their fixed-layout BSA baseline.

Fixed blocks
68.66%
Attention-mass recall
With reblocking
81.54%
+12.88 percentage points
Where this fits in one denoising step

Repeated work within DiT

New layer Q/K/V→Reblock*→Route / Reweight→Attention

Q/K representations change across layers and denoising evaluations. By default, eligible layers rebuild layouts. An implementation can reuse a layout across a small group of nearby layers; cross-step reuse is a separate hypothesis requiring validation.

*The Reblock reuse across layers slider below models how amortizing this work affects effective latency. Reusing a permutation does not reuse Q/K/V or exact Top-K routes.

01 / Tunable assumptions

Explore the what-if scenario

Assume a baseline layer takes 100 ms. All percentages and overheads are relative to this illustrative baseline, not Spark-H3 measurements.

Other ops = GEMMs, norms, residuals, FFN, etc.
Kernel-only speedup before overheads
Feature grouping, permutation construction, rearrangement
Approximate amortization: reblock cost ÷ reused layers
Proxy scoring, selecting key blocks, routing metadata
Non-amortized Q/K/V and output layout work
Fast presets:

Layer time breakdown

100 ms reference layer

What happens inside the optimized layer?

Bar widths use the same time scale. The remainder is the amount of time saved.

BF16 baseline layer100 ms
Optimized layer57.0 ms
Attention Unchanged Reblock Route Other
You save 43.0 ms per eligible layer.

5× versus 10× — same baseline

The benefit of faster attention diminishes as other layer operations dominate.

The bars are normalized to the original 100 ms baseline, using your current overhead configuration.

Amdahl projection curve

Attention speedup 1× to 20×

Attention-kernel speedup vs. layer speedup

Solid line: current overhead. Dashed line: ideal zero-overhead. The vertical marker follows your selected kernel speedup.

Including overhead Zero-overhead ideal Current selection

Reference attention speedups

Computed using the selected attention share and your currently configured overhead.

KernelAttn timeLayer timeLayer speedup
Theoretical ceiling: speeding up attention infinitely cannot eliminate the rest of the layer. The limit is 1 / (1 − attention fraction), ignoring overhead.

Denoising-step and full-model projection

Extended Amdahl model

Account for dense warmup and fallback

Even a 2× sparse-layer improvement translates into less when some denoising steps or layers still use BF16 dense attention.

Across the model layers, excluding permanently dense layers
VAE, text encoder, loading etc. as fraction of baseline full runtime
Denoising timeline
Dense warmup Sparse-enabled step

Spark-H3 reblock reuse opportunities

Approximate sensitivity

How often must layouts be rebuilt?

Assume the reblock process takes the same cost per invocation. Increasing the number of layers sharing a permutation amortizes that cost but may affect block quality.

Reuse intervalReblock / layerSparse layerFull-run speedup

Layer reuse vs. cross-step reuse

Reusing layout across layers: skips some landmark and permutation construction. Must check Q/K feature changes; Top-K can still be recalculated.
Reusing layout across denoising steps: could save more setup work, but the feature structure changes with denoising. Needs dedicated correctness and visual-quality validation.
Important: A reused token permutation is not the same as reusing selected attention blocks or Q/K/V values.

Research interpretation

For GPU kernel and end-to-end profiling

Measure each overhead independently

Record baseline attention, sparse attention, reblocking, Top-K proxy + selection, layout copies and residual layer ops with GPU events. Avoid double counting asynchronous work.

T_new = T_other + T_sparse + T_reblock + T_route + T_layout

Compare end-to-end, not only kernel TFLOPS

Check realized layer and video speedups, selected-block recall / attention mass, and video quality. Higher sparse-kernel throughput can be offset by reblocking overhead or lower utilization.

S_full = T_baseline_full / T_optimized_full
Default example: Attention is 60% of layer time. A 5× attention-only speedup gives 1.92× ideal layer speedup. With 5 ms of overhead it becomes 1.75×; with dense warmup steps and fallback layers, full-run gains are smaller. These are illustrative values, not measured Spark-H3 throughput.