Writing
Mostly post-mortems: a design claim, the measurement that contradicted it, and what the gap turned out to be.
Choosing Video Frames by Content Instead of by a Clock
· measurement, llm, computer-vision, python · 5 min read
At an equal frame budget, uniform sampling missed 9 of 28 shots across five real videos and spent 12 of its 40 frames re-photographing things it had already seen. Content-aware selection missed none and wasted two; here is the measure where it still comes out level, and the one video where uniform sampling wins.
Ranking Language Models by How Well They Spot Liars
· llm, measurement, benchmarks, statistics · 11 min read
I built a benchmark to rank language models on spotting liars, then ran 522 games of Mafia through it. The same model scored -0.400 and +0.078 in one run, 52 games apart. Day 1 accusations came in at -0.005 against chance on n=2,103, and the whole thing reproduces in 11 seconds with no API keys.
An AI Capture-the-Flag Tournament: What the Scoreboard Counted
· llm, measurement, security, benchmarks · 8 min read
A preliminary report concluded that models under 3B parameters cannot chain exploits. A later tournament with much larger models put an 8B model last: the engine credited 2 of its 279 logged captures, because a per-turn marker had counted each of the 232 times it read bonus flags off its own machine.
Checking a Cost Model Against a Stranger's Config File
· llm, measurement, mixture-of-experts · 7 min read
I bounded an ambiguity in a config file at 2.4% and called it immaterial. The model's own header settled it months later, and correcting the assumption pushed a gate from 1.1% off to 3.4% off, outside the tolerance it had been passing.
Why a Small Transformer Can't Copy a Word It Hasn't Seen
· measurement, code-generation, llm, tokenization · 13 min read
An 11.9M-parameter transformer scores 83% strict on the domains it trained on and 0/8 on a single unseen noun. Three experiments to find out why, one of which corrected the diagnosis I had already written down, and a zero-parameter mechanism that does the copying 20 times out of 20.
How a Dedup Pass Deleted My Training Curriculum
· llm, measurement, fine-tuning, security · 9 min read
A fine-tuning pipeline that weighted its best examples 6x by duplicating them to 94,022 lines, followed by a dedup pass that handed the trainer 50,145, each exactly once. Two retrains were rolled back for regressing before I found that the graded curriculum had never reached the trainer at all.
Error Feedback, Gradient Compression, and Why Adam Breaks It
· llm, measurement, quantization, training, optimizers · 10 min read
Error feedback makes a biased gradient compressor unbiased over time, and under SGD it restored the full-precision trajectory to three digits. Under Adam it landed 1.9 times further from the optimum than no correction at all, and the fix I published helps just as much with no quantization in the run.
A Better FP4 Gradient Quantizer That Training Couldn't Notice
· llm, measurement, quantization, training · 9 min read
A per-block scale that cuts FP4 gradient-quantization error on 45 of 45 tensors, 14% against the published state of the art, two rented-GPU training runs that landed within noise anyway, and the measurement that explains both: the gap only shows at million-token batches, 35 to 643 times larger than anything I ran.
What a Code Completer's Eval Never Measures: Ghost Text
· llm, measurement, code-completion, python · 11 min read
A hybrid n-gram and 2.48M-parameter code completer that gets 54.6% of tokens right, tuned dial by dial against its own held-out eval. Then I scored the one feature that emits more than a single token and it came back at 0.070, because the eval hands the model a true prefix and the feature hands it its own output.
Comparing INT4 and NVFP4 Palettes on Real Gradient Tensors
· llm, quantization, training, measurement · 9 min read
A palette study on synthetic blocks said evenly-spaced INT4 beats NVIDIA's FP4 grid once a Hadamard rotation gaussianizes the data, and loses by 2.3x without it. On real gradient tensors it wins either way, 41 of 45 without the rotation, because my stand-in for unrotated data put the outliers in the wrong place.
An App Generator That Verifies Everything Except Its Parser
· measurement, code-generation, verification, llm · 14 min read
A natural-language app generator whose verification sweep is exhaustive over everything except the part that reads English. The composer scores 100% on 810 cells built from feature sets; the parser that has to produce those feature sets scores 0.70, and nothing prints that number next to the other one.
Reimplementing Enough of Kubernetes to Fool kubectl
· kubernetes, node, protocols, api-compatibility · 8 min read
Reimplementing enough of the Kubernetes API that the real kubectl can't tell the difference. The surprise wasn't the resource schemas; it was that kubectl delegates its own output formatting to the server, and that a Quantity sent as a plain string decodes as zero on the client rather than erroring.
What a Hibernated Browser Tab Actually Costs
· memory, swift, webkit, measurement · 12 min read
My simulated tab scheduler priced a hibernated tab at 32 KB; real WebKit charges 39 MB. It claimed 7.9–11.5x less memory than an unmanaged browser and delivered 1.4–2.6x. That one number, wrong by three orders of magnitude, explains the entire gap.
Predicting the Speed of a 276B Model Streamed From an SSD
· llm, measurement, macos, ssd, mixture-of-experts · 10 min read
A from-config cost model for streaming a mixture-of-experts model off an SSD, validated byte-for-byte against a runtime I didn't write. Then the SSD benchmark turned out to be measuring RAM, and the real run missed the speed prediction by 23×, for the one reason the model had declared it couldn't account for.
Rate Limiting a Crawler Across Node Cluster Workers
· concurrency, node, mongodb, crawlers · 7 min read
A per-process rate limiter under Node's cluster module enforced a perfect five-second delay while the host on the other end received a hundred requests at once. Fixing it took an atomic claim, and two more bugs on the way.