FP32 tensors, TF32 math, Tensor Core speed

Why Tensor Cores use TF32 instead of true FP32

TF32 is a Tensor Core compute mode for FP32 matrix math. It keeps FP32-like dynamic range, reduces input mantissa precision, and accumulates into FP32. The goal is much higher GEMM/conv throughput with acceptable accuracy for many deep-learning workloads.

Same FP32 exponent range Lower input precision FP32 tensors stay FP32 in memory Mainly matmul / convolution

B200 numbers to keep straight

Regular FP32 / CUDA-core spec

75 TFLOPS

Full FP32 path; not the TF32 Tensor Core path.

TF32 Tensor Core spec

~1.1 / 2.2 PFLOPS

Dense / sparse. 2.2 PFLOPS = 2200 TFLOPS.

Measured in the B200 microbenchmark paper

964.5 TFLOPS

TF32 Tensor Core throughput, ~96.5% of theoretical peak.

1) FP32 vs TF32 bit layout

FP32 and TF32 both use an 8-bit exponent. That is why TF32 has a similar value range to FP32. The precision difference comes from the mantissa/fraction field.

FP32 1 sign + 8 exponent + 23 fraction bits

S 1

Exponent 8

Fraction / mantissa 23

Higher precision. More expensive multipliers. Usually lower peak throughput for matrix-heavy AI workloads.

TF32 1 sign + 8 exponent + 10 mantissa bits

S 1

Exponent 8

Mantissa 10

dropped / rounded lower bits

Same exponent width as FP32, but less mantissa precision. NVIDIA/PyTorch describe TF32 as rounding FP32 inputs to 10 mantissa bits and accumulating in FP32.

2) What actually happens during a TF32 matmul?

The tensor can still be torch.float32. TF32 is not normally a storage dtype exposed to your model; it is an internal math mode for selected FP32 matmul/conv kernels.

FP32 tensorA and B are stored as float32 in memory.

Round inputsBefore multiply, input values are rounded to TF32 precision.

Tensor Core MMAMatrix multiply runs on Tensor Cores at much higher throughput.

FP32 accumulateProducts are accumulated with FP32 precision.

FP32 outputOutput tensor remains float32 unless your framework casts it.

Important: TF32 changes the internal multiply precision, not the tensor storage format. Other non-matmul operations generally remain FP32.

Why NVIDIA added TF32

Deep learning spends most of its time in GEMM and convolution. Tensor Cores are specialized matrix engines. To pack many matrix multipliers into the GPU power/area budget, the input multiply precision must be smaller than full FP32.

TF32 is a compromise: keep FP32 dynamic range, reduce mantissa precision, and use Tensor Cores.

Why not just use true FP32?

Full FP32 multipliers require much wider significand hardware. That increases area, power, and datapath cost. For massive matrix throughput, NVIDIA gets far more operations per cycle by using reduced-precision inputs such as TF32, BF16, FP16, FP8, and FP4.

So true FP32 is still available, but it is not the fastest AI matrix path.

When TF32 can be wrong choice

TF32 is not bit-equivalent to FP32. It can change numerical results, especially for workloads that are sensitive to small differences: scientific/HPC solvers, strict regression tests, deterministic debugging, or algorithms with ill-conditioned matrices.

For those cases, use true IEEE FP32 and verify kernels with profiler/SASS counters.

3) Throughput intuition on B200

Using the public HGX B200 spec: individual B200 GPU FP32 is 75 TFLOPS, while TF32 Tensor Core is listed as 2.2 PFLOPS with sparsity. Dense TF32 is half of sparse, about 1.1 PFLOPS.

FP32 CUDA core

75 TFLOPS

TF32 TC dense

1100 TFLOPS

TF32 TC sparse

2200 TFLOPS

This is why a plain torch.float32 GEMM can look much faster when TF32 is enabled: it may use Tensor Cores rather than the full FP32 CUDA-core path.

4) B200 paper caveat: “FP32 Tensor Core” is not the same as FP32 CUDA core

The B200 microbenchmarking paper reports Tensor Core throughput by precision. Its table shows FP32 = 482.0 TFLOPS and TF32 = 964.5 TFLOPS for Tensor Core microbenchmarks.

Path	Meaning	B200 number
FP32 spec	Regular FP32 throughput, usually CUDA-core/SIMT path.	75 TFLOPS
FP32 Tensor Core row	Paper’s Tensor Core precision benchmark, not the same as the normal FP32 spec row.	482.0 TFLOPS
TF32 Tensor Core row	TF32 Tensor Core benchmark.	964.5 TFLOPS

Do not mix these labels: product “FP32 75 TFLOPS” and paper “FP32 Tensor Core 482 TFLOPS” refer to different hardware paths/benchmark definitions.

5) PyTorch controls

Use the new precision API when available. It explicitly asks PyTorch to use IEEE FP32 internally instead of TF32 for CUDA matmul/cuDNN paths.

# PyTorch 2.9+ style import torch torch.backends.fp32_precision = "ieee" torch.backends.cuda.matmul.fp32_precision = "ieee" torch.backends.cudnn.fp32_precision = "ieee"

Older API, still commonly used:

torch.backends.cuda.matmul.allow_tf32 = False torch.backends.cudnn.allow_tf32 = False torch.set_float32_matmul_precision("highest")

6) Benchmarking checklist

Goal	Use	Verify
Measure ~75 TFLOPS FP32	Disable TF32; use true FP32/SIMT kernel or cuBLAS IEEE FP32 path.	Nsight/SASS shows FFMA-like FP32 path, not Tensor Core MMA.
Measure ~1.1 PFLOPS TF32 dense	Enable TF32; use large FP32 GEMM.	Tensor Core instructions/counters active.
Measure ~2.2 PFLOPS TF32 sparse	Use supported 2:4 structured sparsity Tensor Core path.	Sparse Tensor Core path active.

Sources used

NVIDIA Technical Blog, “Accelerating AI Training with NVIDIA TF32 Tensor Cores”: TF32 is a Tensor Core mode, rounds FP32 inputs to TF32, accumulates in FP32, and uses 8 exponent + 10 mantissa bits.
PyTorch CUDA semantics documentation: TF32 controls, behavior on float32 matmul/convolution, old and new APIs.
NVIDIA Blackwell / HGX B200 datasheet: individual HGX B200 GPU specifications, including FP32 75 TFLOPS and TF32 Tensor Core 2.2 PFLOPS with sparsity.
Jarmusch & Chandrasekaran, “Microbenchmarking NVIDIA’s Blackwell Architecture”: B200 dual-die / 148 SM description and Tensor Core throughput table.