Measuring the Wrong Process for Eight Months

In What a Hibernated Browser Tab Actually Costs I reported that Kestrel’s memory ladder held 121.7 MB against an unmanaged browser’s 319.9 MB on a set of real websites, and that it never went over its budget. Both numbers were wrong. Re-measured, the same run holds 345.9 MB against an unmanaged 255.3 MB, and it is over budget in 85% of samples. The ladder did not do less good than I published; on that workload it did harm, and the published figures reported it as the best result in the post.

The correction is worth more than the original finding, because what broke was not the design. It was the instrument.

The number that gave it away

Kestrel reported a Jira board as costing 52 MB. The process rendering that page held 511 MB. The 52 MB was real, and it belonged to a completely different application’s WebContent process that had been running for eight days.

I did not catch this with a test. I caught it with a screen recording, watching the same figure sit unchanged before and after loading an entirely different page. That is a temporal observation, and no single assertion or screenshot contains one. You have to see two moments next to each other and notice they agree when they should not.

Three mistakes, stacked

Every memory figure in the earlier post was a per-tab attributed sum: one process id per tab, claimed by diffing ps output around tab creation, then added up. Three things were wrong with that, and each one hid the next.

ps lists every WebContent process on the machine. They are XPC services parented to launchd with identical command lines, so there is no parent and no client tag to filter on. Nothing filtered at all, which made any content process on the system a candidate, including ones belonging to Safari or to an Electron app someone left open. The single sound discriminator available is age: a process older than the browser cannot belong to it.

Tab creation picked .first on an unordered set. WebKit spawns several content processes at once (five, on a measured launch), and the diff-at-creation trick assumed it would see exactly one new pid. Whichever one it happened to grab, the tab reported for the rest of its life.

Nothing distinguished “cannot measure” from “costs nothing.” Both rendered as 0 MB. A live page the browser had lost track of was therefore counted as free by the scheduler: never in the total, never a candidate for demotion.

Why it survived eight months

The number was always plausible. 52 MB for a page is unremarkable; so is 115, and so is 399. A wrong number inside a believable range is invisible in a way that a crash is not, and every test written against it passed, because the tests asked the browser what it thought rather than checking the browser against the machine.

That is the same failure this project had already recorded once. An earlier round of work improved what about:memory reported while returning nothing to the OS, and I wrote it down at the time. I then built a benchmark with the identical shape and did not recognise it for eight months. Writing a lesson down is not the same as having learned it.

The corrected numbers

The benchmark now records both the attributed sum and the browser’s real footprint, which is every WebContent process younger than the run, summed. That second figure needs no attribution at all, so it cannot fail in this way. Heavy pages, 800 MB budget, 40 events:

The 100% row is the one that matters. With every tab live and every process owned by a tab, the two methods agree to within 1 MB, which means attribution became exact once the web view named its own process. So the 25 to 32% gap under the demoting policies is not measurement error. It is real memory sitting in processes that outlived the tabs they belonged to.

That splits a question the project could not previously answer. Of the gap, roughly 0% is mis-attribution and all of it is processes orphaned by demotion. Demoting a tab releases its web view; WebKit keeps the process; nobody owns that memory, and nobody was counting it.

Per event, the shape is unmistakable once the two columns sit side by side:

policy=none     event 12:  attributed 1049   measured 1089   procs 10
policy=kestrel  event 12:  attributed  428   measured  668   procs 10
policy=kestrel  event 36:  attributed  749   measured  989   procs 13

The error is not uniform, and it flatters exactly the policies the benchmark exists to evaluate. With no policy running, every process belongs to a live tab and attribution captures effectively all of it. Start demoting and the processes stop being attributed to anything, so the reported number falls while the memory does not. A measurement whose error scales with the treatment is not a noisy measurement; it is a measurement that manufactures the result.

Corrected against the unmanaged run, discard-LRU comes to 975 / 802 = 1.22x and Kestrel to 975 / 788 = 1.24x. Across three runs Kestrel measured 1.28x, 1.28x and 1.24x, and discard-LRU 1.21x, 1.20x and 1.22x. The “0% over budget” claim does not survive at all: measured, Kestrel is over budget in 48% of samples and discard-LRU in 62%. The scheduler was demoting until its own accounting said it was under budget, which turns out not to be the same thing as being under budget.

The light pages inverted

The published post’s worst-looking table was the light-pages run, and I described it there as a loss. It was considerably worse than a loss. Eleven mixed-weight real sites, 150 MB budget, 40 events:

Attribution accounted for 100% of the unmanaged run and only 25% and 36% of the managed ones. The policies were reporting roughly a quarter of what they actually held, and the error was largest precisely where the design looked best.

This is not a new failure mode. It is this design’s own predicted one, finally measured. DESIGN.md states the feasibility rule as budget > live working set + (39 MB x parked tabs). Eleven tabs at the 39 MB COLD floor need 429 MB before a single page is displayed, and this run was handed 150 MB, which is 2.9x below its own floor. The scheduler says so in its logs: gave_up fires 5 times for discard-LRU and 7 times for Kestrel.

What the design predicted below the floor was degradation toward discard-LRU. What actually happens is degradation below doing nothing. A demotion leaves the old WebContent process alive, the restore spawns another, and 45 demotions with 18 restores over 40 events churn processes faster than WebKit reclaims them. Measured peak reached 530 MB for Kestrel and 650 MB for discard-LRU, against 309 MB for the run that managed nothing at all, on the workload whose entire premise was that the pages were light.

What survives

The ladder’s own physics were measured correctly the whole time. Re-measured with the web view naming its own process, six tabs, same synthetic page:

LIVE    128.0 MB
WARM    106.0 MB   (83% of live)
COLD     39.0 MB   (30% of live)
restore COLD -> LIVE: 87 ms

Identical to the published figures, to the decimal. Two warnings the re-measurement was watching for did not fire: the pid diff was never ambiguous in that run (one tab at a time, 3.5 s apart, exactly one WebContent process each), and navigating to about:blank did not move the page to a different process, which was the way the 39 MB COLD figure could have been some abandoned process’s footprint rather than the parked tab’s. The feasibility floor built on 39 MB stands unchanged, which is why it was able to predict the inversion above.

The comparative case also came out stronger than the old numbers suggested. Kestrel holds less real memory than discard-LRU (788 MB against 802 MB) while destroying fewer tabs (6 against 8) and spending far less time over budget (48% against 62%). On the attributed figures it looked marginally worse on memory and better only on state loss.

So the honest summary is workload-dependent, and one side of it is negative:

The ladder helps when the budget is reachable and hurts when it is not, because below the floor the process churn costs more than the parked pages save. The old numbers hid that completely. They reported 2.63x better on precisely the run where the design was 1.35x worse.

What I changed, and what I did not

WKWebView._webProcessIdentifier is private API, verified on macOS 15.5, and it is now asked at every point where a tab’s process is established, with age-filtered ps diffing as a fallback. The whole-browser total is measured independently by summing the run’s own processes, which needs no attribution and therefore cannot fail this way. Where the two disagree by more than 20%, the interface says so rather than picking one.

The scheduler itself is unfixed, and a tweak is the wrong response. A browser that cannot meet its budget should refuse the budget: surface the floor, name the number of tabs it can hold, and stop demoting, rather than thrash against a target it can prove is unreachable. That is recorded in BROKEN.md and not implemented.

The part that generalises

Both times this project measured itself wrong, the instrument reported on the thing it was part of, and both times it reported success. Tier-down improved about:memory while returning nothing to the operating system. Per-tab attribution improved the benchmark’s total while the memory stayed in processes nobody counted. The fix was the same on both occasions: measure from outside the thing being measured, where the number cannot be produced by the component whose behaviour is in question.

The tell is worth naming, because it is available before you know anything is wrong. Ask what your measurement would report if the feature did nothing at all, and then ask what it would report if the feature worked perfectly. If the second answer can be produced by the component simply losing track of its own work, the instrument is not measuring the feature. Mine could, and for eight months it did.

The full account of the bug is in DEBUGGING.md, and the corrected tables with their per-event traces are at the end of RESULTS-ENGINE.md. Kestrel is 10,746 lines of Swift across 45 files on top of WKWebView; the measurements here are all on one machine, an M1 Max running macOS 15.5, Swift 6.1.2 and system WebKit.