Inside FlashAttention
Follow one query tile through three K/V tiles. Watch QK MMA → score scaling → row max → two exponentials → denominator → PV MMA → accumulation → output, with real values, tensor shapes, and the matching pseudocode.
Bq=2 · Bk=2 · D=2GPU-sized example: Bq=64 · Bk=64 · D=128Q: [2,2] · K/V: [6,2]Non-causal (unmasked) example · exact attention04 / New learning section
Block-by-block kernel execution
One query tile, three KV blocks, and every operation as its own click-to-inspect step. Read the original nested pseudocode from top to bottom. Operation lines are colored by LOAD, MMA, MAX, EXP, ROW SUM, ACCUMULATE, and NORMALIZE. Click a line to inspect its values for the selected KV tile. No playback or separate KV-block timelines.
The code is one Q tile's complete loop; choose KV tile 1, 2, or 3 above to inspect a different loop iteration. Colors are algorithmic groups, not hardware execution times.
Load inputs
Checking against direct full-softmax reference…
Teaching model: non-causal toy tensor shapes with JavaScript FP64 arithmetic; no fully masked rows. The two matrix multiplications are shown as MMA stages, not a cycle-accurate GPU pipeline. Values are rounded only for display.
01 / Step through the kernel
Step 1 / 28Load Q, K, V
...