Tagged training
7 write-ups.
Training Among Us Bots: The Crew That Stopped Fixing the Reactor
Ten thousand games into a self-play run, my Among Us impostors found that sabotaging the reactor beat the crew. The crew now weighted its next task 4.1 times as much as the fix, and fixed 0 of 7 sabotages. Paying for each fix flipped that to 7 times toward the fix, and 30 of 34.
Why NEAT Stopped Forming Species When the Starting Genome Grew
The same NEAT code that split an XOR population into 13 species in ten generations kept a Monopoly population in exactly one, because a 180-gene starting genome shrank every compatibility distance to a median of 0.368 against a threshold of 3. Counting only the genes added since the start moved that median to 6.379.
Just Train More: Measuring the Exchange Rate
The obvious objection to a zero-parameter cache beating a small transformer is that the transformer is undertrained. Sixteen times the training data moved the crossover 6.6x, which is document length times 1.60 per doubling: about as much as letting the cache read the rest of the file it was already sitting in.
Was TinyStories the Domain or the Vocabulary?
TinyStories is famous for a ~1,500-word vocabulary, so I capped mine at 1,500 and scored every arm on identical positions. It came out 4 points worse at top-1 than a 4,000-word cap, and more than a third of its output was the unknown-word token. The transferable lever was the domain, not the vocabulary.
Error Feedback, Gradient Compression, and Why Adam Breaks It
Error feedback makes a biased gradient compressor unbiased over time, and under SGD it restored the full-precision trajectory to three digits. Under Adam it landed 1.9 times further from the optimum than no correction at all, and the fix I published helps just as much with no quantization in the run.
A Better FP4 Gradient Quantizer That Training Couldn't Notice
A per-block scale that cuts FP4 gradient-quantization error on 45 of 45 tensors, 14% against the published state of the art, two rented-GPU training runs that landed within noise anyway, and the measurement that explains both: the gap only shows at million-token batches, 35 to 643 times larger than anything I ran.
Comparing INT4 and NVFP4 Palettes on Real Gradient Tensors
A palette study on synthetic blocks said evenly-spaced INT4 beats NVIDIA's FP4 grid once a Hadamard rotation gaussianizes the data, and loses by 2.3x without it. On real gradient tensors it wins either way, 41 of 45 without the rotation, because my stand-in for unrotated data put the outliers in the wrong place.