Kernel Performance Lab

Attention Roofline Explorer

Non-causal, fused prefill attention · 75,600-token × 40-head × d128 examples · interactive HBM roofline · no external dependencies

GPU ridge point
2,000
FLOPs / HBM byte
Tiled HBM intensity
124.1
FLOPs / HBM byte
Ideal HBM intensity
2,048
FLOPs / HBM byte
Tiled roofline ceiling
496.5
TFLOP/s (upper bound)

Three dtype examples · BF16 output

Same sequence length, head dimension, tile size, head counts, and GPU specifications

N = 75,600 · Hq/Hkv = 40/40 · d = 128 · Bq = 128
Q/K/V storageTiled AI
FLOPs/B
Ideal AI
FLOPs/B
Tiled ceiling
TFLOP/s
Roofline regimePreset

The comparison keeps the same entered GPU TFLOP/s for all three dtypes; hardware throughput for BF16, FP8 and MXFP4 may differ. Tiled HBM assumes repeated K/V traffic, while the ideal model assumes one HBM read per Q/K/V element.

GPU Roofline Model

Log–log axes · HBM bandwidth roof · model predictions

Tiled HBM Ideal HBM GPU roof

Tiled attention · repeated KV reads

Matrix FLOPs (QKᵀ + PV)
—
HBM traffic
—
Roofline performance
—
Bandwidth lower-bound time
—
Compute lower-bound time
—
Predicted bottleneck
—

Ideal HBM reuse · tensor read/write once

Matrix FLOPs (QKᵀ + PV)
—
HBM traffic
—
Roofline performance
—
Bandwidth lower-bound time
—
Compute lower-bound time
—
Predicted bottleneck
—

How the model is calculated

Matrix arithmetic (non-causal)

FLOPs = 4 × Hq × N² × d

Ridge point

AI_ridge = GPU_TFLOPs / HBM_TBps

Ideal HBM bytes

N × d × [Hq(sQ + sO) + Hkv(sK + sV)]

Tiled HBM bytes

N × d × [Hq(sQ + sO) + ceil(N/Bq) × (Hq/r)(sK + sV)]

N = shared Q/KV sequence length; d = head dimension; s* = stored bytes per element, including modeled MX scale metadata; r = modeled KV sharing across query heads (1 by default, or Hq/Hkv if enabled). The tiled estimate assumes KV blocks are re-fetched from HBM for each query tile per sharing group; actual L2 reuse can make HBM AI significantly higher. Head dimension cancels from AI in this simplified model but still changes FLOPs, bytes, and predicted time. In practice, GPU utilization, softmax, and on-chip memory effects can reduce achieved performance far below the roof.