Consecutive video positions are not necessarily similar in attention space, especially in noisy denoising steps. Reblock partitions tokens by learned Q/K affinity so a fixed Top-K block budget captures more important attention interactions.
How Spark-H3 Works + Speedup Lab
Explore Spark-H3’s reblocking and reweighting ideas step-by-step, then project their real impact on DiT-layer and end-to-end video latency.
Skipping the other key blocks entirely would discard their attention mass. Spark-Reweight creates weighted K/V summaries for the unselected tail and corrects their softmax mass with a log-bias, instead of treating a simple block mean as exact.
Does grouping actually improve selection?
At the same 10% exact video-block budget, the authors report an increase in attention-mass recall for their fixed-layout BSA baseline.
Source: Spark-H3 technical blog. These are published ablation results, not measurements from this simulator.
Repeated work within DiT
Q/K representations change across layers and denoising evaluations. By default, eligible layers rebuild layouts. An implementation can reuse a layout across a small group of nearby layers; cross-step reuse is a separate hypothesis requiring validation.
*The Reblock reuse across layers slider below models how amortizing this work affects effective latency. Reusing a permutation does not reuse Q/K/V or exact Top-K routes.
Explore the what-if scenario
Assume a baseline layer takes 100 ms. All percentages and overheads are relative to this illustrative baseline, not Spark-H3 measurements.
What happens inside the optimized layer?
Bar widths use the same time scale. The remainder is the amount of time saved.
5× versus 10× — same baseline
The benefit of faster attention diminishes as other layer operations dominate.
Attention-kernel speedup vs. layer speedup
Solid line: current overhead. Dashed line: ideal zero-overhead. The vertical marker follows your selected kernel speedup.
Reference attention speedups
Computed using the selected attention share and your currently configured overhead.
| Kernel | Attn time | Layer time | Layer speedup |
|---|
Account for dense warmup and fallback
Even a 2× sparse-layer improvement translates into less when some denoising steps or layers still use BF16 dense attention.
How often must layouts be rebuilt?
Assume the reblock process takes the same cost per invocation. Increasing the number of layers sharing a permutation amortizes that cost but may affect block quality.
| Reuse interval | Reblock / layer | Sparse layer | Full-run speedup |
|---|
Layer reuse vs. cross-step reuse
Measure each overhead independently
Record baseline attention, sparse attention, reblocking, Top-K proxy + selection, layout copies and residual layer ops with GPU events. Avoid double counting asynchronous work.
Compare end-to-end, not only kernel TFLOPS
Check realized layer and video speedups, selected-block recall / attention mass, and video quality. Higher sparse-kernel throughput can be offset by reblocking overhead or lower utilization.