Comparing INT4 and NVFP4 Palettes on Real Gradient Tensors

Four-bit training quantizes every number to one of 16 values. NVFP4’s menu is {0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6}, with one scale factor per block of 16 elements. Those levels are spaced like a float: fine near zero, coarse at the top. The standard pipeline also applies a random Hadamard rotation before quantizing, which spreads outlier energy across a block and pulls the per-coordinate distribution toward a bell curve.

Both of those exist to handle outliers, which made me suspect they were paying for the same thing twice. So I treated the 16-value menu as a design variable and searched for better ones. Plain evenly-spaced INT4 beat NVFP4 on Gaussian blocks at both rounding modes. On my heavy-tailed stand-in for unrotated data it lost by a factor of 2.3. That looked like a clean conditional, so I wrote it as one: the uniform grid wins after the rotation, because the rotation has already removed the outliers the float spacing exists for.

Then I ran the comparison on 45 real gradient tensors instead of synthetic ones. Uniform INT4 wins on 44 of 45 rotated, and on 41 of 45 unrotated. The condition I attached to the claim was an artifact of how I generated fake data, and the reason is a single number I hadn’t looked at.

(This is the second thing I’ve written about the same study. The noise-floor post covers the scale rule and why two rented GPUs couldn’t see it; that argument isn’t repeated here. The code is not public, so this post carries the numbers instead.)

The two corners, and what the palette does to them

Every measurement below scores a palette at two extremes of the same trade-off. The unbiased corner scales each block by its absmax, so nothing clips, and rounds stochastically. Its error is computed exactly as E[(y-l)(u-y)] over the interval each value lands in. The biased corner sweeps 19 candidate scales per block, rounds to nearest, keeps the best, and applies the per-block reconstruction scale that minimizes error. Values above the top level clip, so it is biased, but its error is much lower. Gradient training wants the unbiased corner and the forward pass wants the low-error one, which is why both matter.

On synthetic blocks (12,000 for fitting, 30,000 held out, normalized MSE):

Uniform INT4 is 19.2% lower error at the unbiased corner and 9.3% lower at the biased one, with no format machinery at all. Optimizing the palette properly gets 28.4% and 23.6%, and the two optimizations converge on nearly the same grid (0, 0.098, 0.206, 0.319, 0.438, 0.588, 0.774, 1.000 for the unbiased objective), which is a useful thing to know: a future format could pick one palette without committing to a rounding philosophy.

Nothing here escapes the underlying trade-off, though. The unbiased-to-biased ratio sits between 2.18x and 2.80x for every grid I tested, including the ones optimized specifically for one corner. Palette design moves the level of the error; it does not buy an unbiased estimator at biased-corner cost. That trade is a property of having 16 values, not of NVIDIA’s choice of which 16.

And then the row that produced the wrong claim. Repeating the measurement on blocks where 5% of values are contaminated with noise at ten times the scale, my stand-in for what unrotated gradients look like:

Uniform INT4 loses by 2.26x at both corners. The story wrote itself: the float spacing is dynamic range, dynamic range is what you need when a block holds an outlier, and the rotation is what removes outliers. Hence “after the rotation”.

45 real tensors disagree

The real test is real gradients. I trained a small transformer (3M parameters, byte-level, three layers) and captured every 2-D weight-gradient tensor at steps 20, 60 and 120, giving 45 tensors at a training loss of 2.79. Then I scored both palettes on each tensor twice: once raw, once through the study’s own 16x16 random Hadamard.

Uniform INT4 wins in every cell. Worse for my framing, it wins more without the rotation than with it (22.1% against 17.5% at the unbiased corner), so the rotation is not what enables the result; it very slightly shaves it. The rotation moves every number in these tables by under 4% in either direction, which for a transform I had cast as the precondition is close to a no-op.

The number I should have looked at first

Excess kurtosis is the standard measure of how outlier-heavy a distribution is (0 is Gaussian). Measured on the same 45 tensors, two ways:

Real gradients are extremely outlier-heavy, exactly as the literature says: one tensor has excess kurtosis of 255. But a quantizer with a per-16-element scale never sees the whole tensor. It sees one block at a time, divided by that block’s own absmax, and by that measure real gradients are slightly flatter than a Gaussian at -0.19, whether you rotate them or not.

The outliers in a real gradient live between blocks, not within them. A block containing a huge value gets a huge scale, and the per-block scale has already absorbed the problem before any level spacing is consulted. My heavy-tailed stand-in contaminated 5% of values independently, which scatters roughly one outlier into every block of 16 and leaves nothing for the scale to absorb. That is a within-block problem, and within-block spread is exactly what exponential level spacing is for. So the stand-in did not model unrotated gradients. It modelled a distribution whose outliers sit at the one granularity where NVFP4’s palette earns its keep. Getting the tail weight roughly right was not enough, because what mattered was the tail’s position relative to the scale granularity.

Once blocks are that well behaved, NVFP4’s spacing is straightforwardly a handicap, and it is easy to see where the error is. Decomposing the unbiased corner’s error on Gaussian blocks by which interval each value fell into:

Stochastic rounding error inside an interval scales with the square of its width, so NVFP4’s widest interval turns 19.3% of the values into 49.4% of the error. Its widest gap is 4.00 times its narrowest. Uniform INT4’s is 1.00 by construction, and even the coordinate-descent palettes only stretch to 2.31x; nothing the search found wanted spacing as aggressive as a float’s.

What I can’t claim from this

Every number above is quantization error, not training loss, and the earlier post is the reason I am careful about that: the same study showed that differences of this size do not surface in loss until batches reach one to five million tokens, well past anything I have run. Uniform INT4 has a 17 to 22% error advantage on real gradient blocks and an entirely unmeasured effect on final loss. Nobody should switch formats on the strength of a proxy.

The model is also tiny (3M parameters on a byte-level corpus, 45 tensors, one rotation seed), and 1 to 7 of those 45 tensors go the other way depending on the corner, so this is a distribution-level result rather than a universal one. My harness also gives both palettes an exact floating-point block scale, where real NVFP4 stores that scale in FP8 E4M3 alongside a per-tensor FP32 scale. That compares palettes under identical conditions, not two shipped formats. And NVFP4 has Blackwell tensor cores behind it while an INT4 gradient path does not, so a 19% error reduction is not a 19% anything else. I measured no wall clock.

What generalizes

The finding I would keep is that scale granularity mattered more than level placement, and I have it twice now from unrelated work. A separate experiment in the same repository, quantizing a token embedding table, found that int4 with one scale per tensor costs 2.5 points of top-1 accuracy while int4 with one scale per row costs 0.3, for 16 KB of extra scales (1.7% more bytes). Same lesson from the other direction: given a per-block scale fine enough, the levels can be evenly spaced, and given one that is not, no palette saves you.

The methodological half is the part I will actually carry forward. Replacing real data with a synthetic distribution means choosing a tail weight, and I knew that; what I did not think about is that it also silently chooses where the tails act relative to the scale granularity. My stand-in matched real gradients on the statistic I checked and inverted the result anyway. The check that caught it was one line of kurtosis on the blocks my quantizer actually consumes, and I could have run it before the synthetic study rather than after.