Attention matrix view:
make only frame-0 query rows dense

This visual explains the debugging intervention: keep sparse attention for most video tokens, but override the sparse mask for the query rows belonging to the first video frame. Those rows attend to every key/value token densely.

Dense: keep all K/V columns Sparse kept block Dropped block Frame-0 query rows

1. Token packing: rows are Q, columns are K/V

In video diffusion self-attention, tokens are usually flattened into one sequence. A common mental model is:

Text
prompt/global
F0
first frame
F1
frame 1
F2
frame 2
F3
frame 3
F4
frame 4
F5
frame 5
F6
frame 6
F7
frame 7
sequence = [text tokens] + [frame0 tokens] + [frame1 tokens] + ... + [frame7 tokens]

attention scores = Q @ Kᵀ
one row = one query token attending over all K/V tokens

2. What “frame-0 query rows dense” means

Only the rows whose output belongs to frame 0 are forced dense. The columns remain all K/V tokens: text, frame 0, frame 1, … frame 7.

This checks whether the first-frame artifact comes from corrupted attention outputs for frame-0 tokens. If the bright spots disappear, the sparse mask for frame-0 query rows is the likely failure point. 

for q_token in frame0_tokens:
    output[q_token] = dense_attention(Q[q_token], all_K, all_V)

for q_token in frame1_to_frame7_tokens:
    output[q_token] = sparse_attention(Q[q_token], selected_K, selected_V)

3. Conceptual attention matrix

Rows are query groups. Columns are key/value groups. The green-labeled Q frame0 row is the only visual row group forced to be dense across all columns.

4. Block-level mask override

Real sparse kernels usually operate on blocks, not individual tokens. If a query block overlaps frame 0, mark the whole query block dense across all key blocks. 

visual_start = text_len
tokens_per_frame = latent_H * latent_W

frame0_q_start = visual_start
frame0_q_end   = visual_start + tokens_per_frame

first_q_block = frame0_q_start // q_block_size
last_q_block  = (frame0_q_end - 1) // q_block_size

# Force all K/V blocks to be selected for frame-0 query blocks.
mask[:, :, first_q_block:last_q_block + 1, :] = True

5. Why this is a good diagnostic

1Minimal changeOnly frame-0 query rows change from sparse to dense. Other frames still use the sparse policy.
2Artifact localizationIf frame-0 bright spots vanish, the error is located in frame-0 attention output.
3Production guardYou can keep most speedup and add a small first-frame dense fallback for short videos.

6. Interactive block-mask example

Adjust the example shape. The blue horizontal stripe shows the query blocks that overlap frame 0 and are forced dense across all key blocks.

forced dense for frame-0 query block sparse selected block dropped block block overlaps frame 0

Before global topk sparse

Frame 0 gets the same sparse ratio as every other frame. For short videos, one dropped key block can represent a large fraction of available temporal/context information. 

mask[:, :, :, :] = sparse_topk_mask

After frame-0 rows protected

Only the first-frame query blocks are overwritten to dense. This protects the temporal boundary frame while preserving sparse attention elsewhere. 

mask[:, :, frame0_q_blocks, :] = True

7. How to read the result

If the first-frame bright spots disappear: the sparse mask is harming frame-0 query outputs. Add an adaptive guard: dense first-frame rows for short sequences, higher topk for short videos, or always keep text/global blocks.

Note: actual token order may differ by model, especially with temporal patches, joint text-image attention, Hilbert permutation, or custom packing. The core idea remains the same: identify the token range whose output maps to temporal position 0, then override the sparse mask for those query rows or query blocks.