INTERACTIVE KERNEL NOTEBOOK / PAPER EXPLAINER

TRIAGE / Attention Error Lab

Find out why an attention tile loses accuracy. Explore the real effect of approximating K, V, or both, then see how TRIAGE predicts error to choose Skip · Skim · Refine.

↗ arXiv 2610.35706
Router-predicted tile error
—
Selected tile · heuristic score
Measured isolated Skip error
—
Selected tile vs. dense · relative L₂
Measured routed output error
—
All tiles with current tiers · relative L₂
Illustrative compute budget
—
Cost: Skip 0 / Skim 1 / Refine 3
The big picture

00  TRIAGE: from Q/K/V to an error-aware attention kernel

Predict the damage of approximating each attention tile, then spend compute only where the predicted error warrants it.

Live with the controls at left
01 · INPUT
Q · 2×2K/V · 16×2
Split into tiles

One Q block attends to four K/V tiles, each with four keys.

02 · CHEAP STATISTICS
Mass m̃JR
Estimate sensitivity

Attention mass, key variance, and K–V covariance. No dense QK required.

03 · PREDICTION
—
Estimate Skip error

Ẽ = m̃ · (J + R) · √wₕ for the currently selected tile.

04 · ROUTER
—
Choose a precision tier

Compare predicted error reduction per compute cost to price π.

05 · FUSED EXECUTION
—
One attention output

Merge all tile contributions via one shared online softmax.

1. Each K/V tile takes its own path · click to inspect
2. The three possible execution paths · selected tile highlighted
SKIP
Cheapest · mean approximation
→

Replace pairs of K/V vectors with virtual mean K/V tokens. Keep approximate attention contribution.

SKIM
Low cost · low precision
→

Keep all four tokens. Use the high NVFP4 components for QK and PV matrix products.

REFINE
Highest cost · residual correction
→

Keep all tokens, add high/low cross-products to recover accuracy on sensitive tiles.

Why this tile gets this path

—

—
This flow explains the paper’s high-level routing and execution. The small numerical demo uses educational low-bit approximations, not the authors’ production CuTe/NVFP4 kernel. A predicted score is not the same as measured output error.

01  Tile routing map

Select a tile to inspect its attention error. The tier responds to π, Q, K, V, and head importance.

—
Skip · cluster meanSkim · low-bitRefine · residual correction

02  Attention distribution

Actual token probabilities vs. approximated probabilities, for both queries. A tile contains 4 keys.

Darker → lower attention; brighter → higher attention
—

03  Where does error come from?

Selected tile —

Paper estimates are computed from cluster statistics, not from measuring dense attention.

TRIAGE predicted Skip error
Ẽ = — × (— + —) × √—
—
Actual mass · selected Q
—
Computed from dense QK
Router mass · entire Q block
—
Estimated, no dense QK

04  Controlled error attribution

Measured from dense reference

For just the selected tile, keep all other tiles exact. Replace K, V, or both with cluster means.

Vector interactions, not additive scalar errors
ΔKV = ΔK + ΔV + interaction
—

The actual K-only and V-only output changes may cancel or reinforce each other. J and R instead estimate the magnitudes of the two mechanisms, so they do not need to match these exact percentages.

05  Dense vs. approximate output

Q1
The full routed output combines all four tiles. The K-only, V-only, and isolated Skip outputs change only the selected tile. Error always means ||Oapprox − Odense||₂ / ||Odense||₂.

06  Router decision

—

The router trades predicted error reduction against π. Four tiles in this toy Q row means Nₜ = 4.

Skipcluster averages
Skim1× low-bit work
Refine3× low-bit work
Price πcost threshold
—

07  Q / K / V numbers for selected tile

Dense reference + skipped-tile counterfactual
—
TokenK [x, y]V [x, y]K̄ [x, y]V̄ [x, y]QK/√dp densep skip
Skip does not drop the tile. Each two-key cluster is represented by one mean key and mean value with a virtual logit correction of ln(2). This has the same softmax result as replacing both individual keys and values by that cluster mean.

08  Reading the equations

Paper §5.2 ↗
J = 1 − exp(−||q||² · ⟨tr ΣK⟩ / (2d²))

K variance measures the missing exponential weight from mean-key replacement (softmax-normalizer effect). If K is identical within clusters, J = 0.

R = ||q|| / d · ⟨||ΣKV||F / ||V̄||⟩

K–V covariance measures the shift in the value favored by attention. V variation alone is not sufficient: K and V must be correlated within the cluster for this first-order term.

m̃ = ½ (m̃ from Q̄·K + m̃ from Q·K̄)

Mass is estimated from two corrected low-cost views. This demo computes both from the numeric Q/K/V and also displays the exact mass for comparison.

Ẽ = m̃ · (J + R) · √wₕ

Output head energy supplies a relative head-level multiplier. The score is a heuristic, not an exact per-tile L₂ error or an error bound.


Route: Refine if (0.22 × Nₜ × Ẽ / 2) ≥ π; else Skim if (0.78 × Nₜ × Ẽ) ≥ π; otherwise Skip.