Error Feedback, Gradient Compression, and Why Adam Breaks It

Error feedback is the oldest trick in lossy numerics wearing a machine-learning hat. You compress something, you keep the part you threw away, and you add the leftover back into the next thing you compress. Kahan summation does it for floating-point addition; delta-sigma converters do it for audio. In gradient compression it does something specific and valuable: each individual step stays biased, but the bias cancels across steps, so the sum of what the optimizer actually applied converges to the sum of the true gradients.

That guarantee is why error feedback shows up in every low-precision training recipe, including the one I wrote. It is also why I never questioned it. The guarantee is real, it is provable, and it holds under SGD.

It holds under SGD because SGD is linear in the gradient. Adam is not, and it turns out that is not a footnote. Composed naively with Adam, error feedback in my measurements is not merely useless, it is worse than skipping the correction entirely: 1.9 times further from the optimum than plain quantization, and 2.1 times further than not quantizing at all. Then the fix I published for it, when I finally ran the ablation that isolates it, turned out to be measuring something other than what I said it was.

The code for this study isn’t public, so this post carries the numbers instead. A companion write-up (“My gradient quantizer won on 45 of 45 tensors”) covers the scale rule and the noise floor this project mostly existed to measure. The correction below revises one paragraph of it, and I’ll come to that.

The harness, and why it’s deliberately crude

At four bits, everything in this project landed inside seed noise, which is a separate story with its own post. So this audit runs where the effect is large enough to see: a 3-level quantizer (values snap to {-1, 0, +1} times one scale per block of 16) with a hard clip at 0.35 of the block maximum. That setting throws away a lot. Measured on Gaussian blocks, it has a relative MSE of 28.2% and clips 29.7% of all values, against roughly 0.7% relative MSE for NVFP4. It is about 40 times cruder than the format I actually care about, and that is the point: error feedback exists to clean up exactly this kind of systematic clipping bias, so a harsh clip is the setting where feedback should shine hardest.

The problem is convex least squares, 64 inputs by 32 outputs, 4,096 samples, minibatches of 128, constant learning rate, 4,000 steps. The metric is squared distance to the exact least-squares optimum (available in closed form, so there is no ambiguity about where the run should end up), averaged over the last 1,000 steps to read steady state rather than a transient, over 4 seeds. Every ordering I claim below holds at three sigma of the across-seed spread.

Convex, tiny, and coarse. Hold that thought for the caveats section, because it constrains what the result licenses.

Under SGD it is textbook

Feedback puts the quantized run back on the full-precision trajectory to three digits: 2.31e-5 against 2.33e-5. This is what the theory promises and it is satisfying to watch.

The quantized arm apparently beating full precision is a confound rather than a win. Hard clipping shrinks the effective step size, and on a constant-learning-rate convex problem a smaller step means a tighter steady-state ball around the optimum. It is the same reason a learning-rate sweep on this kind of objective always “prefers” the smallest rate offered. That is why every comparison here stays inside one optimizer, never across.

Under Adam it is a landmine

Same quantizer, same clip, same seeds, same metric. Only the update rule changes.

Quantization alone costs about 8% in distance, which is unremarkable. Adding the correction costs 94% more than not adding it, and lands 2.1 times further out than the full-precision run. On steady-state loss it is +20.5% against full precision, where plain quantization was +1.4%. Read as excess distance over full precision, which is the quantity a correction exists to shrink, quantization’s cost goes from 7.2e-6 to 9.74e-5: the one intervention whose entire purpose is to remove quantization’s bias made quantization 13.5 times more damaging.

The mechanism has two halves, and both come from the same source: Adam’s update is m / sqrt(v), so the gradient’s magnitude is divided out.

The telescoping argument dies. Error feedback’s guarantee is a statement about a sum: residual added at step t cancels residual removed at step t-1, so what you applied over the whole run equals what you should have applied. That cancellation requires the map from gradient to parameter change to be linear. Adam’s is not, so the residual is not re-applied with the same weight it was withheld. It is re-applied divided by whatever sqrt(v) happens to be several steps later, which no longer has any reason to match.

And the residual poisons the preconditioner. The quantity fed into Adam is gradient + residual, and v is an average of its square. The carried residual is not signal; it is accumulated rounding error, and it inflates v most in exactly the coordinates where the quantizer is struggling most. Adam responds by shrinking the step in those coordinates. So feedback pushes the correction into the numerator while simultaneously telling the optimizer to distrust the coordinates it is correcting. That is why the two effects compound into something worse than doing nothing, rather than merely cancelling to nothing.

The fix, and the ablation I should have run first

The standard repair is to freeze v after a warmup period, which is what 1-bit Adam (Tang et al., 2021) does for compressed gradient communication. A frozen second moment is a fixed diagonal preconditioner, and a fixed preconditioner is a linear map, so error feedback’s assumption is restored. I froze v at step 500 of 4,000, measured 1.67e-5, wrote “repairs it completely,” and moved on.

That number should have stopped me. 1.67e-5 is not a repair to the full-precision level of 8.86e-5; it is five times better than the full-precision run. No fix to a quantization defect can beat not quantizing. Something else was in the measurement, and I published the claim without isolating it.

So here is the missing two-by-two. Two arms nobody had run: freeze v with no quantization at all, and quantize with frozen v but no feedback.

Freezing v is worth 4.3 times on this problem with no quantization in the run at all. That, and not the repair of anything, is where the 1.67e-5 comes from: the frozen-v arms all sit together between 1.67e-5 and 2.04e-5, quantized or not, corrected or not. And the two arms that separate feedback’s contribution (1.88e-5 without it, 1.67e-5 with it) differ by 2.1e-6 against a three-sigma threshold of 4.5e-6, so on this problem, once v is frozen, error feedback’s measurable contribution is nothing at all.

The honest reading is narrower than what I published and slightly more interesting. Freezing v does not restore error feedback’s validity so much as remove the conditions under which feedback was hurting. Feedback stops being actively harmful, and stops doing anything either. Meanwhile most of the headline gain is a different optimizer being better suited to a constant-learning-rate convex objective, which is not a result about quantization and will not necessarily survive contact with a non-stationary pretraining loss.

The correction to the earlier write-up: it reported this arm as error feedback being repaired, in a table whose other rows are fine. The landmine is unaffected; the repair row is confounded, and the deployable rule changes shape. Not “error feedback needs frozen v under Adam” but “gradient-space error feedback under vanilla Adam is harmful, and freezing v removes the harm without demonstrating any benefit from the feedback.”

One smaller correction while I’m here. The earlier post published 1.78e-4 for the Adam feedback arm, and re-running the same script twice for this post gave 1.86e-4 both times, against an across-seed spread of 2.4e-5. Every ordering is unchanged, but 1.86e-4 is the number I can reproduce and 1.78e-4 is the number I published.

What this does not establish

The setting is coarse on purpose, and the caveats are proportional to that.

The quantizer’s 28.2% relative MSE is about 40 times NVFP4’s, so the magnitude here says nothing about the magnitude at four bits, where the harm should be proportionally smaller. That prediction has weak support from a rented B200 run in the same project: the feedback arm finished +0.01% against bf16, indistinguishable from noise. It was silently running the broken configuration and nothing bad visibly happened, which is not reassurance so much as an illustration that at FP4 the effect is below the floor in either direction.

The objective is convex least squares with a constant learning rate, which is where the frozen-v gain lives and probably where it dies. A fixed preconditioner reduces steady-state jitter on a stationary problem; a real pretraining run has a schedule, a loss landscape that keeps moving, and gradient statistics that shift under it. I would not carry the 4.3 times over, and the ablation cannot tell me what replaces it.

What does travel is the ordering, which is a claim about algebra rather than about scale: under an optimizer that is nonlinear in the gradient, the correction can have the wrong sign, and nothing in the setup will warn you.

What generalizes

A correction inherits the algebra of whatever consumes it. Error feedback is correct, its proof is fine, and its assumption is stated plainly in the papers that introduce it; I simply carried it across a boundary where it stops holding, because it kept being described as a property of the compressor and it is actually a property of the compressor and the optimizer together. My GPU runs had it bolted onto AdamW the whole time.

The part I’ll actually remember is smaller and more embarrassing. When a fix outperforms the thing it was supposed to restore, the fix is doing something you haven’t measured. 1.67e-5 against a full-precision 8.86e-5 was five times too good, sitting in my own table, and I read it as a strong result instead of an anomaly. The ablation that explained it is two extra arms of an existing script and ran in a couple of minutes on a laptop. The expensive part was never the experiment; it was being pleased with the number.