Compressed Sparse Attention · ratio m = 4

How DeepSeek compresses 4 tokens into 1 KV entry

The model keeps every token in the residual stream. What shrinks is the historical KV-cache sequence dimension: one compressed entry is emitted every four tokens, using overlapping, learned, per-feature gated pooling.

Most precise mental model:
“4 → 1” describes the number of stored compressed KV positions—not a merger of four text tokens, not a reduction in query count, and not ordinary average pooling.

N → N/4

compressed KV positions

8 → 1

candidate vectors per output

The full compression path

Click a step to focus it. Shapes use the released Flash configuration: hidden size 4096 and compressed KV width 512.

Prefill view

Token hidden states

All original token states remain available to the Transformer.

x: [B, N, 4096]

→

Two KV + gate branches

Each token produces branch A, branch B, and matching feature-wise gate logits.

kv, score: [B, N, 1024]

→

Group with stride 4

Four tokens form each current block; branch B from the previous block is brought forward.

[B, N/4, 8, 512]

→

Per-feature softmax pool

For every output feature, eight logits normalize into eight weights, then weighted vectors are summed.

softmax over candidate axis

→

Normalize + RoPE + cache

The result is normalized, rotary position is applied, then it is stored as one compressed KV entry.

C: [B, N/4, 512]

What is—and is not—compressed

This distinction prevents the most common misunderstanding.

✕ Not merged

The four input tokens do not become one vocabulary token. Their residual hidden states, query vectors, and token-level outputs remain separate.

✓ Compressed

The historical key/value representation stored for attention is reduced from N positions to approximately N/4.

Why one output uses eight candidates

Choose a compressed entry. Pink tokens feed its previous-block branch; cyan tokens feed its current-block branch.

Overlap only when ratio = 4

C₀ · one 512-D KV entry

Per-channel gated pooling

The gate is not one scalar per token. Each of the 512 output features gets its own eight-way softmax. Move the feature slider to see an illustrative weight pattern.

Illustrative weights

Output feature k
0 … 511

k = 137

These bars are a deterministic teaching example, not extracted model weights. The real logits come from wgate(x) + positional bias and differ by token, layer, entry, and feature.

Tensor-shape walkthrough

The sequence length shrinks only at the final pooling stage.

Input

[B, N, 4096]

One hidden vector per original token.

KV / gate projections

[B, N, 1024]

1024 = 2 branches × 512 features.

Group four tokens

[B, N/4, 4, 1024]

The group axis identifies offsets 0–3.

Overlap transform

[B, N/4, 8, 512]

Previous B branch + current A branch, with 8 candidates total.

Weighted reduction

[B, N/4, 512]

Softmax and sum over the candidate axis.

Try a sequence length

Use a multiple of four to see the prefill shape exactly.

16input token positions

4compressed positions

32candidate uses across outputs*

*Counts eight candidate slots per compressed output, including the padded previous branch for the first entry. Unique source tokens are not reduced to this count because adjacent entries overlap.

Prefill and decode use the same rule

Prefill reshapes a full sequence. Decode buffers tokens and emits a new compressed entry every fourth token.

Shape operationspseudocode

Meaningshape trace

This is concise pseudocode mapped to the released implementation, not a verbatim copy. In the actual code, compression arithmetic is promoted to FP32 before normalization and cache quantization.

Where the compressed entry goes next

CSA combines exact recent context with sparsely selected compressed history.

Core attention path

Recent exact KV

Sliding-window cache keeps 128 recent token-level KV entries.

Compressed history

A separate 128-D indexer compressor scores historical compressed positions.

→

Top-k selection

For CSA layers, the released Flash configuration selects up to 512 compressed positions per query.

→

Shared-KV multi-query attention

64 query heads attend to a shared 512-D KV entry for each selected position.

candidates_for_query = recent_128_exact_KV + topk_512_compressed_KV → sparse_attn(Q, shared_KV)

Released Flash configuration

Values below come from the model configuration and released inference implementation.

Parameter	Value	Why it matters here
Hidden size	4096	Width of each input token hidden state.
Attention / compressed KV width	512	One compressed KV entry has 512 features.
RoPE sub-dimension	64	RoPE is applied after pooling to the rotary slice.
CSA compression ratio	4	One output position is emitted per four input positions.
HCA compression ratio	128	Other interleaved layers compress more aggressively and use dense attention.
Sliding window	128	Preserves recent token-level detail.
Indexer heads × width	64 × 128	Used only to score compressed positions.
Indexer top-k	512	Maximum selected compressed historical positions per query.
KV heads	1 shared KV	Queries are multi-head; compressed K/V is shared.

One-sentence takeaway

Every four tokens advance the compressed cache by one position, but that position is an overlapping learned mixture of eight branch-specific candidate vectors, with a separate softmax for every one of its 512 features.

Sources

Primary paper and released DeepSeek-V4-Flash implementation.

DeepSeek-V4 technical report, §2.3.1 and equations 9–12: arxiv.org/pdf/2606.19348
Released compressor, indexer, and attention code: DeepSeek-V4-Flash/inference/model.py
Released model configuration: DeepSeek-V4-Flash/config.json