What a Code Completer's Eval Never Measures: Ghost Text

pycomplete is a code completer I built out of a research repo’s own findings, and it works. Index numpy and it reports this, on 24 files it has never seen:

indexed 441 files / 2,065,825 tokens in 6.9s; evaluating on 24 held-out files
alpha=0.45 cache_beta=0.5  positions: 2000  top-1: 0.546  top-5: 0.739  (11.1 ms)

54.6% of the time, the token you were about to type is the one at the top of the list. That is from a 2.48M-parameter transformer (2 layers, d=128, a 256-token window, 10 MB on disk) blended with an n-gram model over your repo and a cache over the buffer above your cursor, all of it on a laptop CPU in 11 milliseconds.

Every dial in it was set by that number. The transformer’s mixture weight alpha=0.45, the cache’s cache_beta=0.5, the order-5 count tables, the choice of a 256-token window over 64 and over 1024: each one won a sweep against held-out next-token prediction. That is a lot of measurement, and I trusted it.

Then I scored the feature people actually look at, and it came back at 0.070.

The thing that was never measured

pycomplete emits two different kinds of output from one mixture. suggest() returns the autocomplete list: one token, judged top-1 or top-5, which is exactly what the eval scores. suggest_multi() returns ghost text, up to 8 tokens of greyed-out continuation that you accept or reject as a single thing.

Nothing had ever scored the second one. Not because it was hard; because the eval had a unit and that unit was the token. So I measured 4-token ghosts by exact match end to end, which is what accepting ghost text actually commits you to:

Right one time in fourteen. That is not a reason to remove the feature, since a wrong ghost costs one keystroke to ignore, but it is the whole distance between “usually right” and “usually wrong,” and the tool had shipped it since the day it was built.

The 0.070 and the 0.546 are both correct, and they stop being in tension once you see what separates them. The eval’s inner loop is one line:

sugg = [w for w, _ in comp.suggest(toks[:t], k=5)]

toks[:t] is the true prefix of a real file. Ghost text is one line too:

w = top[0][0]
out.append(w)
toks.append(w)

It appends its own guess and asks again. So the eval measures the model reading text a human wrote, while the feature runs the model reading text the model wrote. Those are different distributions, and every weight in the tool was fitted on the first one.

The cliff, and where it is

Whether that mattered was a measurable question rather than a rhetorical one. The experiment replaces the last N tokens before the cursor with tokens the shipped system generated (its own output, not synthetic noise), leaves everything earlier true, and asks each component for the same true next token.

At n=400 held-out positions, with 64 tokens of tail at stake:

The collapse is entirely at the first step (shipped -0.367, 14.8 SE), and everything past f=0.25 is a plateau where the arms sit inside each other’s noise. Degradation is not gradual, so there is no gentle region to tune in.

I re-ran that ladder while writing this, against a much smaller repo index, and it is worth saying what survived. The cliff did, and harder: the shipped mixture goes 0.435 to 0.098 across the same first step, a 4.4x collapse at 13.6 SE, then the same flat plateau to 0.068. Absolute heights move with the index, which the experiment already warns about, so the cliff is the thing to quote and the multiple is not.

One row did not survive. The original run found the repo-counts arm essentially immune to self-generated text (-0.018, 1.1 SE, not significant), on the reasoning that generated code still looks enough like code for short n-gram matches to keep working. On my index it drops -0.103 at 5.9 SE, which is not immunity under any reading. So the per-component immunity is unreplicated and looks index-dependent. What reproduced is the cliff, the plateau, and the ordering at f=0. That is the part the rest of this post rests on, and I would rather flag the row that moved than quote a table as though all of it held.

The axis label matters more than the curve here, and I had it wrong at first. I assumed f was the fraction of the buffer that is model-generated. It is not: each component receives the whole file prefix, mean 11,441 tokens in this corpus, so f=1.00 means “the 64 tokens immediately before the cursor are generated,” about 0.6% of what the component can see. f=0.25 is sixteen tokens.

That makes the result stronger than the label I gave it. Sixteen wrong tokens at the end of an eleven-thousand-token buffer cut accuracy by more than half, and sixteen tokens is a short ghost.

The component that prefers noise

The interesting row is not the cliff, which every component falls off. It is the control I added to ask whether the model’s own output carries any signal at all, or whether any corruption is equally fatal: replace the tail with tokens drawn from the unigram distribution instead. Paired on identical positions, reporting the McNemar split because these arms agree about most cases:

Three components would much rather read the model’s output than random tokens. The cache would rather read noise. Its own z sits exactly on the p=0.05 boundary, so taken alone that is a coin-flip claim; what the finding rests on is that it is the sole sign reversal among four components measured on the same positions, while the other three point the other way decisively. A larger paired run is queued, and until it lands the direction is supported and the magnitude is not.

My first explanation for it was wrong, and the correction is the useful part. I said the cache was following false n-gram matches in its own table. It cannot be that: the table is 99.4% true text, since 64 generated tokens are appended to an 11k-token real buffer. Bounding the buffer so each component sees only the last 64 tokens makes the table genuinely all generated, and that reverses the effect (13:0 for self, z=3.33).

So the hazard is not generated text sitting in the cache. It is generated text used to query a cache built from real text. A plausible-but-wrong query matches real entries and pulls back a confident wrong continuation; a random query matches nothing, the cache backs off to short context, and it stays harmless at 0.188 across the entire sweep. Remove the true table and plausible text beats noise, exactly as it does for everything else.

Ghost text is that setting precisely: a query the model wrote, run against an index of code humans wrote.

One more row is worth keeping for anyone building a mixture. Under random contamination the shipped blend is the worst arm on the board (0.040, against the cache alone at 0.188 and the transformer alone at 0.090). Combining two degraded signals with weights fitted on clean input is more fragile than either signal by itself, and no teacher-forced measurement can show you that.

What this does not say

It does not say alpha=0.45 and cache_beta=0.5 are wrong. Past the cliff no weighting is distinguishable from another, so the shipped values are not harmful; they are simply not evidence about ghost text, in either direction. What it does do is bound every mixture number in the project, including the 0.546 at the top of this post, to the f=0 column.

A control also failed before any of this was readable, and it failed for the right reason. I described the repo-counts arm as one that “does not read the buffer,” so its curve should have been flat. An n-gram model conditions on the preceding tokens, which are exactly what gets contaminated; its spread came out at 0.125 against a 0.02 threshold and the harness refused to print the table. No component here is immune. The real control is a context-free unigram arm, whose spread across the whole sweep is 0.000 because it ignores context by construction. The level it sits at depends on the corpus (0.000 in the run above, 0.035 in mine), and flatness rather than level is what makes the other rows comparable.

What generalises

This is the second time the same instrument told me what I wanted to hear.

The first time, the eval walked each held-out file from its start, so buffers never got past a few hundred tokens and the long-buffer buckets were always empty. I had a research result suggesting the transformer’s weight should fall as the open file grows, and the eval agreed there was nothing there, because it could not reach a file long enough to disagree. Adding --stride N, so one file contributes cursor positions across its whole length, moved the numbers a lot: at 1,517 strided positions the tool scores 0.506 top-1 rather than 0.546, and swapping the transformer’s window from 64 to 256 tokens is worth +0.050 top-1 (paired CI [+0.033, +0.067]) on that set against +0.013 on the short one. A longer window pays where there is more to read, which is the regime the old eval had never entered.

Both blind spots have the same shape, and neither one was a bug. Nothing computed a wrong number. In each case the harness was sound and its reachable set did not contain the question, so the measurement came back agreeable and specific and irrelevant. That is much harder to catch than a wrong answer, because a wrong answer at least disagrees with something.

It is also a different failure from the one I wrote up in the mixture-of-experts streaming post, which came out of the same research repo. There, the model declared in advance that it did not account for compute, and the prediction then missed by 23x for exactly that reason; the boundary was written down and the measurement walked across it. Here nothing was declared, because the gap was not in the model at all. It was in the set of questions the instrument could be pointed at.

The rule I would write down from it: state the unit of correctness and the input distribution before quoting an accuracy, then check that your harness can actually produce both. pycomplete’s eval scores tokens on human text; its ghost text emits symbols over its own output. One number, 0.546, was measured very carefully in a place the feature never goes.

The research repo isn’t public, so none of this is one git clone away. The figures come from the project’s own harnesses (pycomplete eval, eval_ghost.py, eval_contamination.py), and I re-ran all three today rather than quoting the record. 0.546/0.739 reproduces exactly. The single-token-versus-ghost gap comes back as 0.323 against 0.073 on an index built fresh from a different corpus, the same 4.4x by a different route. The contamination ladder reproduces its cliff and loses one of its rows, which is recorded above rather than tidied away.