Tagged measurement
13 write-ups.
Choosing Video Frames by Content Instead of by a Clock
At an equal frame budget, uniform sampling missed 9 of 28 shots across five real videos and spent 12 of its 40 frames re-photographing things it had already seen. Content-aware selection missed none and wasted two; here is the measure where it still comes out level, and the one video where uniform sampling wins.
Ranking Language Models by How Well They Spot Liars
I built a benchmark to rank language models on spotting liars, then ran 522 games of Mafia through it. The same model scored -0.400 and +0.078 in one run, 52 games apart. Day 1 accusations came in at -0.005 against chance on n=2,103, and the whole thing reproduces in 11 seconds with no API keys.
An AI Capture-the-Flag Tournament: What the Scoreboard Counted
A preliminary report concluded that models under 3B parameters cannot chain exploits. A later tournament with much larger models put an 8B model last: the engine credited 2 of its 279 logged captures, because a per-turn marker had counted each of the 232 times it read bonus flags off its own machine.
Checking a Cost Model Against a Stranger's Config File
I bounded an ambiguity in a config file at 2.4% and called it immaterial. The model's own header settled it months later, and correcting the assumption pushed a gate from 1.1% off to 3.4% off, outside the tolerance it had been passing.
Why a Small Transformer Can't Copy a Word It Hasn't Seen
An 11.9M-parameter transformer scores 83% strict on the domains it trained on and 0/8 on a single unseen noun. Three experiments to find out why, one of which corrected the diagnosis I had already written down, and a zero-parameter mechanism that does the copying 20 times out of 20.
How a Dedup Pass Deleted My Training Curriculum
A fine-tuning pipeline that weighted its best examples 6x by duplicating them to 94,022 lines, followed by a dedup pass that handed the trainer 50,145, each exactly once. Two retrains were rolled back for regressing before I found that the graded curriculum had never reached the trainer at all.
Error Feedback, Gradient Compression, and Why Adam Breaks It
Error feedback makes a biased gradient compressor unbiased over time, and under SGD it restored the full-precision trajectory to three digits. Under Adam it landed 1.9 times further from the optimum than no correction at all, and the fix I published helps just as much with no quantization in the run.
A Better FP4 Gradient Quantizer That Training Couldn't Notice
A per-block scale that cuts FP4 gradient-quantization error on 45 of 45 tensors, 14% against the published state of the art, two rented-GPU training runs that landed within noise anyway, and the measurement that explains both: the gap only shows at million-token batches, 35 to 643 times larger than anything I ran.
What a Code Completer's Eval Never Measures: Ghost Text
A hybrid n-gram and 2.48M-parameter code completer that gets 54.6% of tokens right, tuned dial by dial against its own held-out eval. Then I scored the one feature that emits more than a single token and it came back at 0.070, because the eval hands the model a true prefix and the feature hands it its own output.
Comparing INT4 and NVFP4 Palettes on Real Gradient Tensors
A palette study on synthetic blocks said evenly-spaced INT4 beats NVIDIA's FP4 grid once a Hadamard rotation gaussianizes the data, and loses by 2.3x without it. On real gradient tensors it wins either way, 41 of 45 without the rotation, because my stand-in for unrotated data put the outliers in the wrong place.
An App Generator That Verifies Everything Except Its Parser
A natural-language app generator whose verification sweep is exhaustive over everything except the part that reads English. The composer scores 100% on 810 cells built from feature sets; the parser that has to produce those feature sets scores 0.70, and nothing prints that number next to the other one.
What a Hibernated Browser Tab Actually Costs
My simulated tab scheduler priced a hibernated tab at 32 KB; real WebKit charges 39 MB. It claimed 7.9–11.5x less memory than an unmanaged browser and delivered 1.4–2.6x. That one number, wrong by three orders of magnitude, explains the entire gap.
Predicting the Speed of a 276B Model Streamed From an SSD
A from-config cost model for streaming a mixture-of-experts model off an SSD, validated byte-for-byte against a runtime I didn't write. Then the SSD benchmark turned out to be measuring RAM, and the real run missed the speed prediction by 23×, for the one reason the model had declared it couldn't account for.