Checking a Cost Model Against a Stranger's Config File

I wrote a cost model for streaming a mixture-of-experts model off an SSD, and the part I was proudest of was that it gated itself twice against artifacts I had nothing to do with. Gate 1 reproduced a third-party runtime’s on-disk container byte-for-byte. Gate 2 predicted the size of a model conversion published by different people using a different tool, and came within 1.1% against a 2% tolerance.

The experiment behind that post has a limitations section I now want to revisit. It listed an ambiguity in the model’s config.json, bounded the damage, and moved on:

dense_mlp_idx=2 is ambiguous: one dense layer, or layers 0 and 1. This takes one, which moves bytes/token by 2.4%. Immaterial to the conclusion, unresolved regardless.

The bound was right. The assumption was wrong. Correcting it took Gate 2 from 1.1% off to 3.4% off, which is outside the tolerance it had been passing, and the reason is more interesting than the arithmetic.

The header states the fact outright

A config file describes a model’s architecture in the vocabulary of the framework that trained it; reading one therefore means inferring what a field name meant to its author. dense_mlp_idx=2 could reasonably be “the layer at index 2 is dense” or “layers below index 2 are dense”, and those give 41 and 40 mixture-of-experts layers respectively.

I eventually downloaded the model in GGUF form (the container format llama.cpp uses) to run it. GGUF puts a block of key/value metadata at the front of the file, and that metadata is written by the conversion script after it has already resolved every question of this kind. So the answer was sitting in a file on my own disk:

inkling.block_count               = 42
inkling.dense_block_count         = 2
inkling.expert_count              = 256
inkling.expert_used_count         = 6
inkling.expert_feed_forward_length = 2048
inkling.expert_shared_count       = 2

dense_block_count = 2. Forty mixture-of-experts layers, not forty-one.

Reading it took about forty lines of Python. The parser was wrong the first time in a way worth repeating: GGUF stores arrays of strings length-prefixed, and I read only the first few elements without consuming the rest. Every subsequent offset was then shifted, and the parse failed later with an invalid-looking type code rather than at the array that caused it. If you write one of these, consume every element even when you intend to display four.

What the correction does

The expert size does not change, so the per-token cost moves purely through the layer count:

The model now under-predicts the published file by 5.27 GB, and the script does what I built it to do: it prints GATE 2: FAIL and refuses to emit any token-rate estimate at all. That refusal was written months earlier for exactly this case; it is the only reason I noticed, rather than quietly publishing a slightly different number.

Two errors, pointing opposite ways

Here’s the mechanism, and it’s the whole point of the post.

Gate 2 predicts a whole container: routed experts, shared experts, attention, embeddings, dense layers. The 1.1% agreement was the sum of two errors that happened to have opposite signs.

Error one, mine. An extra mixture-of-experts layer added 256 experts of 14,155,776 bytes each, inflating the prediction by 3.62 GB of routed experts; net of the dense layer it displaced, +3.54 GB.

Error two, structural and disclosed. The model does not attempt the vision and audio encoders, the multi-token-prediction module, the RMS norms, or the fact that the published conversion keeps non-expert tensors at 8 bits rather than 4. With the layer count correct, those account for -5.27 GB.

Add +3.54 to -5.27 and you land 1.73 GB low, which reads as 1.1% and looks like a model that understands the format. Remove the inflation and the deflation stands alone at 5.27 GB, or 3.4%. The residual was always that large. The agreement was partly luck, and a tolerance was never evidence that the two sides matched for the same reason.

I want to be precise about what this does and does not undermine, because “a gate failed” is easy to over-read.

Gate 1 is untouched and still byte-exact. It compares computed integers against a published layout with no tolerance at all: expert stride 1,769,472, per-layer blob 452,984,832, 256 experts, 40 layers. Equality has no room for two errors to meet inside it; that is the argument for preferring an exact gate to a tolerance wherever the domain offers one.

The load-bearing number got better, not worse. The whole model exists to produce one figure: bytes streamed per token. That figure depends on the expert stride (Gate 1 verifies it exactly), the number of experts routed per token, and the mixture-of-experts layer count. The header now confirms the last two outright, so 3.397 GB/token rests on three measured quantities where 3.482 rested on two measured and one inferred.

So a gate went from pass to fail while the number I actually care about became better grounded. That is only a contradiction if you think a container-size prediction and a per-token-cost prediction are the same claim. They share inputs and they fail differently, and this is the case that separates them.

Two independent checks agree with the corrected figure. A separate script that estimates per-token bytes from published file sizes rather than from architecture gives 3.405 GB against 3.397 computed directly, 0.2% apart. And when I finally ran the model, the kernel’s page-in counters put actual disk traffic between 1.80 and 3.96 GB per generated token, a bracket that contains it.

What I’d take from this

The rule I had been operating under is that a gate passing within tolerance means the model is right. It doesn’t. It means the sum of every error sits inside the tolerance, and a sum can be small because the terms are small, or because they cancel. Those two situations look identical from outside and behave completely differently when you fix something: one improves, the other exposes what the cancellation was hiding.

There’s a cheaper habit buried in this, too. I spent real effort inferring what dense_mlp_idx meant, wrote a careful bound around my uncertainty, and shipped the inference. The artifact stated the fact outright in a header I could read in forty lines. When you find yourself bounding an inference about someone else’s format, check first whether some downstream tool has already resolved it and written the answer down, because conversion scripts have to answer these questions to do their job at all.

And the disclosed limitation earns its keep again. The last post’s failure was attributable because the caveat had been written in advance; this one was detectable because the tolerance and the refusal-to-proceed were written in advance. I keep finding that the value isn’t in the estimate, which gets superseded. It’s in having decided, before the measurement, what would count as being wrong.

The code for this one isn’t public. It’s a research repo with no release, and I’d rather link nothing than link something nobody can run.