Interactive kernel walkthrough · numerical simulator

Inside FlashAttention

Follow one query tile through three K/V tiles. Watch QK MMA → score scaling → row max → two exponentials → denominator → PV MMA → accumulation → output, with real values, tensor shapes, and the matching pseudocode.

Teaching example: Bq=2 · Bk=2 · D=2GPU-sized example: Bq=64 · Bk=64 · D=128Q: [2,2] · K/V: [6,2]Non-causal (unmasked) example · exact attention

04 / New learning section

Block-by-block kernel execution

One query tile, three KV blocks, and every operation as its own click-to-inspect step. Read the original nested pseudocode from top to bottom. Operation lines are colored by LOAD, MMA, MAX, EXP, ROW SUM, ACCUMULATE, and NORMALIZE. Click a line to inspect its values for the selected KV tile. No playback or separate KV-block timelines.

40 manual operations · single query head
1 / 40
FlashAttention pseudocodeSame loop and indentation · select a line to inspect
Load & initQK MMAMaxEXPRow sumPV MMARescale & addNormalize

The code is one Q tile's complete loop; choose KV tile 1, 2, or 3 above to inspect a different loop iteration. Colors are algorithmic groups, not hardware execution times.

Load inputs

Checking against direct full-softmax reference…

Teaching model: non-causal toy tensor shapes with JavaScript FP64 arithmetic; no fully masked rows. The two matrix multiplications are shown as MMA stages, not a cycle-accurate GPU pipeline. Values are rounded only for display.

01 / Step through the kernel

Step 1 / 28
Keyboard: ← →
KV tile 1 / 3 · Stage 1 / 9

Load Q, K, V

K_tile, V_tile ← HBM

...

02 / Tensor values

Each cell displays its numerical value
Color highlights the tensors used or produced in the current step. Values are rounded for display; calculations keep full JavaScript floating-point precision. In a real kernel, score and weight tiles stay on-chip rather than becoming full HBM matrices.

03 / Running softmax state

Persists across KV tiles

State entering this tile

State after the complete tile

Here m and l are length-Bq vectors; acc is [Bq, D]. The final-state table appears after the current tile completes. The separate new-max card shows the candidate maximum before running_max is committed at the Accumulate step.