Tagged llm
12 write-ups.
Choosing Video Frames by Content Instead of by a Clock
At an equal frame budget, uniform sampling missed 9 of 28 shots across five real videos and spent 12 of its 40 frames re-photographing things it had already seen. Content-aware selection missed none and wasted two; here is the measure where it still comes out level, and the one video where uniform sampling wins.
Ranking Language Models by How Well They Spot Liars
I built a benchmark to rank language models on spotting liars, then ran 522 games of Mafia through it. The same model scored -0.400 and +0.078 in one run, 52 games apart. Day 1 accusations came in at -0.005 against chance on n=2,103, and the whole thing reproduces in 11 seconds with no API keys.
An AI Capture-the-Flag Tournament: What the Scoreboard Counted
A preliminary report concluded that models under 3B parameters cannot chain exploits. A later tournament with much larger models put an 8B model last: the engine credited 2 of its 279 logged captures, because a per-turn marker had counted each of the 232 times it read bonus flags off its own machine.
Checking a Cost Model Against a Stranger's Config File
I bounded an ambiguity in a config file at 2.4% and called it immaterial. The model's own header settled it months later, and correcting the assumption pushed a gate from 1.1% off to 3.4% off, outside the tolerance it had been passing.
Why a Small Transformer Can't Copy a Word It Hasn't Seen
An 11.9M-parameter transformer scores 83% strict on the domains it trained on and 0/8 on a single unseen noun. Three experiments to find out why, one of which corrected the diagnosis I had already written down, and a zero-parameter mechanism that does the copying 20 times out of 20.
How a Dedup Pass Deleted My Training Curriculum
A fine-tuning pipeline that weighted its best examples 6x by duplicating them to 94,022 lines, followed by a dedup pass that handed the trainer 50,145, each exactly once. Two retrains were rolled back for regressing before I found that the graded curriculum had never reached the trainer at all.
Error Feedback, Gradient Compression, and Why Adam Breaks It
Error feedback makes a biased gradient compressor unbiased over time, and under SGD it restored the full-precision trajectory to three digits. Under Adam it landed 1.9 times further from the optimum than no correction at all, and the fix I published helps just as much with no quantization in the run.
A Better FP4 Gradient Quantizer That Training Couldn't Notice
A per-block scale that cuts FP4 gradient-quantization error on 45 of 45 tensors, 14% against the published state of the art, two rented-GPU training runs that landed within noise anyway, and the measurement that explains both: the gap only shows at million-token batches, 35 to 643 times larger than anything I ran.
What a Code Completer's Eval Never Measures: Ghost Text
A hybrid n-gram and 2.48M-parameter code completer that gets 54.6% of tokens right, tuned dial by dial against its own held-out eval. Then I scored the one feature that emits more than a single token and it came back at 0.070, because the eval hands the model a true prefix and the feature hands it its own output.
Comparing INT4 and NVFP4 Palettes on Real Gradient Tensors
A palette study on synthetic blocks said evenly-spaced INT4 beats NVIDIA's FP4 grid once a Hadamard rotation gaussianizes the data, and loses by 2.3x without it. On real gradient tensors it wins either way, 41 of 45 without the rotation, because my stand-in for unrotated data put the outliers in the wrong place.
An App Generator That Verifies Everything Except Its Parser
A natural-language app generator whose verification sweep is exhaustive over everything except the part that reads English. The composer scores 100% on 810 cells built from feature sets; the parser that has to produce those feature sets scores 0.70, and nothing prints that number next to the other one.
Predicting the Speed of a 276B Model Streamed From an SSD
A from-config cost model for streaming a mixture-of-experts model off an SSD, validated byte-for-byte against a runtime I didn't write. Then the SSD benchmark turned out to be measuring RAM, and the real run missed the speed prediction by 23×, for the one reason the model had declared it couldn't account for.