Predicting the Speed of a 276B Model Streamed From an SSD

A mixture-of-experts model only activates a few of its experts per token, which means (unlike a dense model, where every forward pass touches every weight) you can leave the weights on disk and read in just the ones each token routes to. That turns “does this model fit in RAM” into “how fast is your SSD,” and it puts models far larger than your machine nominally supports within reach, at some token rate.

I wanted to know what that rate actually is. The question that started it was concrete: can Inkling-Small (276B total parameters, 12B active) run on a 24 GB Mac mini?

The answer wanted to be arithmetic rather than a guess. This is what happened when the arithmetic met the machine.

Act 1: a prediction is worthless unless it can be wrong

I wrote a ~250-line cost model with no dependencies that downloads no weights. Every input is a config.json or manifest.json; every output is a byte count derived from it. It models MLX affine group quantization the way it’s actually stored on disk: per quantized projection, the weight bytes plus one bf16 scale and one bf16 bias per group of 64 values along the input dimension.

The obvious failure mode is a model tuned until it agrees with the one thing you checked it against. So it’s gated twice, against artifacts I had nothing to do with.

Gate 1: reproduce a real container byte-for-byte. Swiftlet is a third-party Swift/Metal runtime that stores MoE weights in its own on-disk format. From a model’s config.json alone, predict the layout it publishes. These are exact integers, so the gate demands equality, not closeness:

All four exact. The full container total came within 0.137%, the residual being the tokenizer and chat template, which the model doesn’t attempt.

Gate 2: predict a stranger’s conversion. Gate 1 could still pass by being fitted to the one container it was checked against. So: predict the on-disk size of a different model, converted by different people using a different tool. Predicted 151.8 GB against a published 153.5 GB, 1.1% off, against a 2% tolerance. The residual is unmodelled and deliberately not fudged: vision and audio encoders, RMS norms, 8-bit router gates, safetensors headers.

Correction, 2026-08-14. This gate no longer passes, and the 1.1% was two errors cancelling. The model assumed one dense-MLP layer where the published model’s own header states two, so it counted 41 mixture-of-experts layers instead of 40. Correcting that drops the prediction to 148.23 GB, which is 3.4% off and outside the 2% tolerance. The extra layer had been inflating the prediction by 3.54 GB while the genuinely unmodelled components listed above deflated it by 5.27 GB, so the two nearly cancelled and the residual was always larger than 1.1% suggested. Gate 1 above is unaffected and still exact. The follow-up post works through it.

Both gates run on every invocation, and the script refuses to print any token-rate estimate if either one fails. That mattered later.

The interesting structural result was that total parameter count is nearly irrelevant. What sets the per-token cost is experts_per_token × layers × expert_size. And the useful finding was a cliff rather than a gradient: the fraction of each model’s expert set that fits in 24 GB of RAM:

Correction, 2026-08-14. That last figure read 3.482 GB when this post went up, for the layer-count reason described above. The corrected 3.397 GB raises the predicted token rate quoted later in this post by a factor of 1.025, which is far too small to affect the 23× conclusion. The Qwen row is unaffected.

That 93.7% explains something I’d otherwise have taken at face value. Swiftlet’s README reports its decode loop is “dispatch bound, not IO bound,” which sounds like a claim about kernel quality. On a 24 GB machine, after warmup, almost all of the 35B’s experts are resident; it is barely streaming at all. The streaming machinery is what lets it start in a couple of gigabytes, not what it lives in. Past the cliff, no amount of kernel work helps.

Now I just needed a GB/s number to turn bytes per token into tokens per second.

Act 2: the benchmark was measuring RAM

An earlier experiment of mine had priced disk reads at “a nominal 100 µs” and said so in its own limitations; the latency was an assumption, not an observation. So before trusting any rate, I measured the SSD.

The first version reported 11–24 GB/s on a device that does about 7.

The cause is a specific and easy misreading of a macOS API. fcntl(fd, F_NOCACHE, 1) stops the kernel from caching future reads on that descriptor. It does not evict pages that are already resident. I was reading a file I had just written, so it was entirely in the unified buffer cache. The flag was set, it did exactly what it says, and the benchmark measured memory bandwidth.

What makes this the interesting half is that no flag fixes it. The fix has to be structural: write a file 2.5× larger than physical RAM (160 GiB on this machine) and sample only from its first quarter, whose pages the later 120 GiB of writes necessarily evicted.

I also stopped trusting myself and added two gates that run every time:

• Content gate: every timed read must contain a magic header written at that offset. This catches short reads, misalignment, and filesystem tricks returning zeros at implausible speed. It immediately caught a page-alignment bug: 8 of 384 reads verified.
• Cache gate, read the same offsets twice. With caching genuinely off, the two passes must agree. If pass 2 is much faster, the cache is serving reads and the run is void. It came out at 1.10×.

The real numbers, in the access pattern a router actually produces (top-k random experts inside one layer’s blob, then the next layer) were 3.54 GB/s issued serially and 7.82 GB/s with six reads in flight.

That 2.2× is the single biggest lever in the whole exercise. Issuing the top-k expert reads one at a time leaves half the device idle. For Inkling-Small it is the difference between roughly 1.1 and 2.5 tok/s: between unusable and marginal.

A separate measurement at 4 KB found something worth designing against: the mean read is 154.1 µs, but p99 is 757.6 µs, 5.8× the median. A mean-based estimate is structurally blind to that, and any paging design needs a deadline policy that answers from whatever is resident rather than waiting on the tail.

The generalizable version of this act: when a mechanism is supposed to disable an optimization, verify the optimization is actually off rather than trusting the switch. I had a flag that did what it promised, and a benchmark that was wrong by 3×.

Act 3: running it, and missing by 23×

Prediction, with the measured SSD: 1.46–3.22 tok/s.

Getting to a real number was mostly an exercise in discovering that published weights are not evidence that anything can load them. Four plausible runtimes:

Three of those four have published quantized weights and no working loader. The GGUF ladder exists because its author built it against their own unmerged branch.

It runs. On an M1 Max, CPU-only, streaming 127.4 GB from SSD via mmap:

The bytes side held. 14,516,912 page-ins over 18 prompt plus 15 generated tokens brackets the predicted 2.926 GB/token: 3.96 GB if every page-in is charged to the generated tokens, 1.80 GB if spread across all 33. The batched prompt routes to far more than six experts per layer, so the true figure sits inside that bracket. Config arithmetic and kernel page-in counters are entirely independent methods, and they agree. That’s the thing the experiment existed to get.

The rate side missed by 23×. And here is the part I actually care about.

Act 1’s write-up carried this in its limitations, written months before any of this ran: compute is not modelled at all; every rate here is an IO ceiling, the real rate is the minimum of IO and compute, and this experiment cannot tell you which binds.

That is precisely what happened. The only configuration that completes a forward pass is CPU-only: every Metal configuration dies with an out-of-memory error, because ~125 GB of experts leak onto a GPU with a 51.5 GB working set, and the flag that’s supposed to keep them off it doesn’t work for this architecture. So 12B active parameters get multiplied on 8 CPU threads with no GPU, and that sets 15.7 seconds per token. The SSD was never the constraint.

The IO ceiling was never wrong. It was just not the binding term in the only configuration available, and the run that works is therefore also the run that cannot test the ceiling. The 24 GB mini’s predicted rate remains unmeasured, and I’d rather say that than quote 0.064 tok/s as though it answered the question.

The part that was worth the whole exercise

There’s an operational footnote I’ll keep short: a process streaming an oversized mmap’d model can become genuinely unkillable, sitting in uninterruptible-exiting state for 40 minutes after SIGKILL, because it cannot finish unwinding 127 GB of mapped memory while blocked in the page-fault path. pkill reports success. Only ps -o stat= shows otherwise. On unified memory this starves everything else on the machine, which is a good reason not to do this on a box something else depends on.

But the thing I’d take away is about the caveat.

The estimate was not the valuable output. Estimates get superseded. What made this worth doing is that the failure was attributable: the prediction missed for a reason that had been written down, in advance, as the thing the model could not account for. I didn’t have to reconstruct an explanation after the fact, and I didn’t get to choose a flattering one.

That is the difference between a model that taught me something and a number that happened to be wrong. A caveat written before the measurement is a prediction about your own blind spot; a caveat written after it is an excuse. They read identically in the final write-up, which is exactly why the order matters.

I’ve now had this pattern three times in a row. A crawler whose rate limiter enforced a perfect delay while sending a hundred simultaneous requests. A browser whose hibernated tabs cost three orders of magnitude more than I’d modelled. And a cost model that was right about every byte and wrong about the clock. In all three the arithmetic was fine and the boundary of the arithmetic was where the truth was hiding, so the useful discipline isn’t being more careful inside the model, it’s being explicit about where the model stops.