Language/text
Vision understanding tokens
Diffusion visual tokens
Audio
Action
Noisy target tokens
Cosmos 3 Super is best understood as a two-path Mixture-of-Transformers model: an autoregressive reasoner builds a discrete understanding of the scene, and a diffusion generator denoises continuous world tokens for image, video, audio, and action outputs.
Vision-language pathway. Causal attention, next-token text output.
Diffusion pathway. Full attention, iterative denoising of continuous tokens.
Two parameter paths inside one unified token layout.
Position encoding for spatial and temporal structure across modalities.
Generator sees reasoner context; reasoner remains self-contained.
Own re-drawn mental model of the Cosmos 3 Super dataflow, not a copy of NVIDIA's figure.
The two towers are complementary rather than redundant.
| Part | What it consumes | How attention works | What it produces | Typical use |
|---|---|---|---|---|
| Reasoner tower | Text plus vision tokens from images/videos. | Causal self-attention on the AR subsequence. It behaves like a VLM/LLM and predicts the next text token. | Text: captions, structured reasoning, localization, action plans, physical plausibility judgment. | Understanding-only tasks, VLM API, prompt upsampling, planning, scene diagnosis. |
| Generator tower | Text/vision/action context plus noisy image/video/audio/action tokens. | Full attention over its diffusion tokens and the AR context. It repeatedly denoises target tokens. | Image, video, synchronized sound, future rollouts, robot/agent action chunks. | Text-to-image/video, image-to-video, forward dynamics, inverse dynamics, policy generation. |
| MoT connection | One packed sequence: AR first, DM second. | AR cannot look into DM. DM can look back into AR. This makes guidance one-way: reasoner -> generator. | A single model surface that can act like VLM, video generator, simulator, or action model. | Avoids stitching several separate models and pipelines for physical AI workflows. |
Why the generator can use the reasoner's understanding without corrupting autoregressive text reasoning.
The reasoner builds a compact textual/visual understanding: objects, motion, intent, constraints, and physical context. The generator then uses that AR context as conditioning while denoising the target world tokens. Because DM attends to AR but AR does not attend to DM, generation is guided by reasoning while text reasoning stays a clean next-token process.
Same checkpoint, different runtime surfaces.
Text goes through the tokenizer. Image/video for understanding goes through a ViT into the AR subsequence. Generation targets or conditions are represented as DM tokens by VAE/audio/action encoders.
For understanding-only tasks, this path is enough: it decodes text using standard next-token autoregressive generation.
For generation tasks, noisy DM target tokens are appended. The generator tower attends to AR context and all DM tokens, then predicts denoised tokens.
After diffusion steps, decoders map clean tokens back to pixels, video frames, audio streams, or action vectors.
| Reasoner-only mode | text/image/video -> text captioning, 2D grounding, temporal localization, physical reasoning. |
| Generator mode | text/image/video/action -> image/video/audio/action text-to-video, image-to-video, forward dynamics, inverse dynamics, policy rollouts. |
| Prompt upsampling | A reasoner/planner can expand short prompts into dense structured prompts before diffusion generation, improving fidelity and physical detail. |
Different applications mostly change what appears in AR vs DM, not the backbone.
AR only. Text and optional ViT image/video tokens are decoded autoregressively. Diffusion parameters are not activated.
AR prompt + noisy DM target. The prompt lives in AR. The generator denoises visual tokens into an image or video.
AR text + clean visual condition + noisy video target. The input image constrains identity/layout; diffusion predicts motion.
State + action -> future observation. Clean action/context condition the DM target video rollout.
Observed video -> action. The model infers the action trajectory that explains the visual transition.
Instruction + observation -> action chunk + rollout. Useful for robot policy learning and embodiment-specific control.
Why the split helps instead of forcing one transformer behavior onto everything.
Large-scale image-text and video-text learning gives the model visual-language grounding. Later supervised tuning specializes it for Physical AI: temporal events, 2D grounding, robot task planning, physical plausibility, and action reasoning.
The diffusion pathway learns to synthesize and simulate continuous modalities. It is trained on image/video/audio/action data and learns to denoise target tokens while conditioned on AR context.
Public descriptions indicate both AR and DM paths share the same layer template and multimodal positional scheme, while using separate parameter sets inside transformer layers. This lets the generator inherit semantic/world knowledge but optimize for denoising.
The reasoner path resembles LLM/VLM inference with causal attention and KV cache. The generator path resembles diffusion inference: repeated denoising steps, full attention over dense multimodal tokens, and heavier video-token memory pressure.
Main public references behind this visualization.