How BlockSelector works
Watch 64 tokens become four 16-token mean vectors, produce a 4×4 similarity matrix, collapse into one FP32 block score, and finally become a sparse KV-block selection. All numbers are computed from reproducible synthetic Q/K tensors in your browser.
Start with Q and K
Pick a query block and a key block to inspect their 64-token representations.
The current scorer pools 16-token groups instead of computing all 4096 Q×K token interactions.
Inside one 64 × 64 block pair
Q0 and K5 · four pooled vectors each
Query tokens [64, 32]
Key tokens [64, 32]
Sixteen similarities → one block score
Actual dot products from the pooled tensors above (D=32).
Block scores → Top-k selection
4 query blocks × 24 key blocks; each cell is one LogSumExp score.
Upstream's _fused_topk uses 16–32 rounds of a Triton threshold search, then compacts selected block IDs. This page ranks scores exactly for illustration; tie/threshold behavior in the GPU implementation can differ in pathological cases.
BlockSelection contract
The adapter consumes indices and counts; it does not redo block selection.
[0, 1, 4, 8, ...]counts: int32 [1, 1, 4][10, 10, 10, 10]Source ↔ visualization
The source branch is a draft of PR #8335 (not yet merged at creation). It supports CUDA, FP16/BF16 Q/K, a protected KV prefix, and native MHA/MQA/GQA mapping.
Important interpretation
The score is an approximation of Q-block relevance to a KV block. It is not the exact probability mass from full 64×64 token attention. Top-k uses the score rank rather than a Softmax probability.
The output has only KV-block IDs and counts. V is not used by this selector; the selected-key attention provider consumes Q/K/V afterward.
The data here are synthetic but every pooled mean, dot product, LogSumExp, budget and output array is recomputed on interaction. No CDN, tracking, external scripts or build dependencies.