Cosmos 3 — How the Generator Uses Reasoner Context

A mechanism-level visualization of how the Reasoner (AR tower) and Generator (DM tower) cooperate inside Cosmos 3. The core idea is that the generator does not simply read a final text answer from the reasoner. Instead, it denoises noisy target tokens while attending to the reasoner’s hidden token context, which acts as a semantic and physical anchor.

Reasoner = scene/task understanding
Generator = diffusion denoising
Conditioning via joint attention
Main information flow: Reasoner → Generator

1) Big-picture flow

Input

Prompt + observed image/video/action history

→

Reasoner / AR tower

Builds hidden understanding of the world, task, object relations, motion, and constraints.

→

Generator / DM tower

Starts from noisy target latents and repeatedly denoises them using reasoner context.

→

Output

Generated image / video / audio / action rollout

The generator uses the reasoner as context, not as a detached post-processor.

2) What each side is responsible for

Reasoner learns / represents

  • what objects are present
  • where they are
  • which objects matter for the goal
  • what action or future state is desired
  • what motion or interaction is plausible

Generator learns / represents

  • how to turn noisy latents into outputs
  • how to keep objects visually consistent
  • how to produce smooth motion across time
  • how to generate actions or future observations that match the context

3) Token-level view: where the context lives

Cosmos 3 conceptually has an AR subsequence and a DM subsequence. The AR side holds the context and reasoning state; the DM side holds the noisy target tokens to be generated.

Sequence sketch

Prompt
Obs frame 1
Obs frame 2
AR token 1
AR token 2
AR token 3
Noise
DM token 1
DM token 2
DM token 3

Observed context and reasoning tokens sit on the AR side; generation targets sit on the DM side.

AR tokens

Hold scene semantics, task understanding, temporal context, and high-level physical cues.

DM tokens

Represent noisy target latents for future image/video/audio/action outputs.

Key idea

Generator queries repeatedly read AR context during denoising.

4) Attention access pattern

The reasoner side remains the stable context. The generator side can look back to that context while denoising. The table below is a simplified mental model.

Previous AR
Current AR
AR context
DM tokens
AR query
Yes
causal/self path
No extra future access
No / limited in this mental model
DM query
Yes
Yes
Main conditioning source
Yes

Simplified interpretation: the generator can attend to reasoner context while producing/denoising target tokens.

5) What happens during one diffusion step

1
Noisy DM input
The generator starts from noisy future image/video/audio/action latent tokens. These are incomplete and ambiguous on their own.
2
Read AR context
DM queries attend to AR tokens that encode the task, objects, relations, observed history, and likely future constraints.
3
Predict cleaner latents
Using this context, the generator predicts a less noisy version of the target tokens.
4
Repeat
Over many diffusion steps, the output evolves from noise → coarse structure → objects/motion → final coherent output.
DM query at step t: “I am generating this future patch / future action token. What should I become?” Reads from AR context: - prompt meaning - observed scene tokens - object identities - spatial relations - temporal history - task / action goal - plausible physical behavior

6) Why this helps semantic consistency

Because the generator repeatedly reads the same AR context, it is less likely to drift away from the intended objects, goal, or scene layout.

Example prompt: “Pick up the red cup and place it next to the bowl” Reasoner context may encode: - red cup = target object - bowl = reference destination - cup is on the left side of the table - action sequence: grasp → lift → move → place

During denoising, the generator can keep asking: Which object is the target? Where should it end up? What scene elements must remain consistent?

  • same red cup across frames
  • same bowl across frames
  • consistent table layout
  • task-aligned motion direction

7) Why this helps physical consistency

The reasoner side can anchor the generator with motion and interaction constraints learned from multimodal world data.

  • objects should remain in plausible places unless acted upon
  • motion should be temporally smooth
  • contact should occur before an object moves with the hand/robot
  • scene changes should follow the requested action causally
  • future observations should remain compatible with prior observations

This is learned physical consistency, not an explicit symbolic or rigid-body physics solver. The model learns these patterns from training data and uses the reasoner context to stabilize generation.

8) Layer-by-layer mental model

For each transformer layer: AR path: update AR tokens preserve / refine scene and task understanding DM path: update noisy DM tokens attend to AR hidden states produce better denoising features

So the generator is not conditioned only once at the beginning. It is conditioned repeatedly across layers and across diffusion timesteps.

Across diffusion time: t = T (very noisy) ↓ use AR context rough structure ↓ use AR context again objects / motion emerge ↓ use AR context again final coherent output

9) Simple intuition

Reasoner = planner / interpreter

Understands what is happening, what matters, and what should happen next.

Generator = renderer / rollout maker

Actually synthesizes the future observation or action sequence.

Together

“Understand first, then generate while staying anchored to that understanding.”

11) Generation inference mental model

Yes — during generation inference, the most useful mental model is: the Reasoner first builds hidden/context representations from the prompt and observed multimodal inputs, and then the Generator uses that reasoner context repeatedly during each diffusion denoising step.

Conceptual inference flow: 1. Reasoner processes: - prompt - observed image / video / action history 2. Reasoner produces hidden context states C 3. Generator initializes noisy target latents x_T 4. For each denoise step t = T ... 1: x_{t-1} = Generator( noisy target = x_t, timestep = t, conditioning context = C ) 5. Final output = x_0
Important nuance: The reasoner usually does NOT need to emit an explicit final text answer first. Instead, what matters most is its hidden representation / token context. The generator attends to that context repeatedly across diffusion steps.

What the Reasoner provides

  • object identities
  • scene layout and spatial relations
  • task / intent understanding
  • temporal history from observed frames
  • high-level physical plausibility cues

How the Generator uses it

  • reads the same context while denoising noisy target tokens
  • uses it again at the next denoise step
  • keeps object identity and motion more consistent
  • aligns generation with the original goal and observations
Simple pseudocode: C = Reasoner(prompt, observed_frames, actions, ...) x_T = noise for t in range(T, 0, -1): x_t = Generator(x_t, timestep=t, context=C) return x_0

In practice, whether the reasoner context is recomputed every denoise step or cached/reused is an implementation detail. The conceptual behavior is that the generator is conditioned on the reasoner context at every denoising step.

11b) Visual: generation inference mental model

This diagram shows the core idea visually: the reasoner first builds a stable hidden context, and the generator then reuses that same context while denoising from xT down to x0.

Input context

Prompt
Obs frame
Video history
Action history
↓

Reasoner pass

Build hidden context C:

objects
layout
goal
history
physics cues
↓

Generator denoising loop

Start from noisy target latents and denoise step by step while attending to context C.

Step t = T

Input: xT (very noisy)

Read context C

Output: coarse structure

Step t = T-1

Input: less noisy latents

Read context C again

Output: better objects / motion

... repeat ...

Each step reuses the same reasoner context

Noise → structure → coherent output

↓

Final result

Generated video / image / audio / action rollout x0

Context reuse view

Context C
Produced by Reasoner once conceptually from the input context
↘
Denoise step T
Generator attends to C
↘
Denoise step T-1
Generator attends to C again
↘
Denoise step T-2
Generator attends to C again
↘
...
Same conditioning idea across the whole denoising chain

Implementation detail: a runtime may cache this context rather than recomputing it every step.

Compact pseudocode: C = Reasoner(prompt, observed_frames, actions, ...) x_T = noise for t in range(T, 0, -1): x_{t-1} = Generator(x_t, timestep=t, context=C) return x_0

10) One-sentence summary

The Generator keeps generation physically and semantically consistent by repeatedly attending to the Reasoner’s hidden token context during every transformer layer and diffusion denoising step.

This visualization is a simplified explanatory view based on the previously discussed Cosmos 3 model description, where the reasoner provides context and the generator uses that context for guided diffusion generation.