00 TRIAGE: from Q/K/V to an error-aware attention kernel
Predict the damage of approximating each attention tile, then spend compute only where the predicted error warrants it.
One Q block attends to four K/V tiles, each with four keys.
Attention mass, key variance, and K–V covariance. No dense QK required.
Ẽ = m̃ · (J + R) · √wₕ for the currently selected tile.
Compare predicted error reduction per compute cost to price π.
Merge all tile contributions via one shared online softmax.
Replace pairs of K/V vectors with virtual mean K/V tokens. Keep approximate attention contribution.
Keep all four tokens. Use the high NVFP4 components for QK and PV matrix products.
Keep all tokens, add high/low cross-products to recover accuracy on sensitive tiles.
—
01 Tile routing map
Select a tile to inspect its attention error. The tier responds to π, Q, K, V, and head importance.
02 Attention distribution
Actual token probabilities vs. approximated probabilities, for both queries. A tile contains 4 keys.
03 Where does error come from?
Selected tile —Paper estimates are computed from cluster statistics, not from measuring dense attention.
04 Controlled error attribution
Measured from dense referenceFor just the selected tile, keep all other tiles exact. Replace K, V, or both with cluster means.
The actual K-only and V-only output changes may cancel or reinforce each other. J and R instead estimate the magnitudes of the two mechanisms, so they do not need to match these exact percentages.
05 Dense vs. approximate output
Q1||Oapprox − Odense||₂ / ||Odense||₂.06 Router decision
—The router trades predicted error reduction against π. Four tiles in this toy Q row means Nₜ = 4.
07 Q / K / V numbers for selected tile
Dense reference + skipped-tile counterfactual| Token | K [x, y] | V [x, y] | K̄ [x, y] | V̄ [x, y] | QK/√d | p dense | p skip |
|---|
ln(2). This has the same softmax result as replacing both individual keys and values by that cluster mean.08 Reading the equations
Paper §5.2 ↗K variance measures the missing exponential weight from mean-key replacement (softmax-normalizer effect). If K is identical within clusters, J = 0.
K–V covariance measures the shift in the value favored by attention. V variation alone is not sufficient: K and V must be correlated within the cluster for this first-order term.
Mass is estimated from two corrected low-cost views. This demo computes both from the numeric Q/K/V and also displays the exact mass for comparison.
Output head energy supplies a relative head-level multiplier. The score is a heuristic, not an exact per-tile L₂ error or an error bound.