Why a Small Transformer Can't Copy a Word It Hasn't Seen

I have a small transformer that turns a one-line spec into a working web app. It is 11.9M parameters, 6 layers, d=384, trained on 4,176 generated programs. Given “a support ticket system with marking a ticket closed, a stats page, searching tickets and creating and viewing tickets” it writes a 1,158-token Python file that compiles, serves HTTP, implements search, stats and toggle, and does not implement comments, edit, delete or category. On held-out feature combinations it does that 5 times out of 6.

Ask it for a book catalogue and it writes an appointment booker.

I built this arm to justify a specific claim, and the claim turned out to be wrong. This post is that retraction, plus the three experiments it took to find out why the model fails, one of which corrected a diagnosis I had already written down and believed.

(I have written about this project once before, on how its verification sweep reports on itself. That post is about the symbolic half of the same system; this one is about the learned half and does not depend on it. The code is in a private research repo, so there is no link. Every figure below comes from a results file or a command I ran, and I say which.)

The claim I was defending

The shipped tool does not use the model at all. It uses compositional synthesis: a parser turns your request into an entity schema plus a feature set, and hand-written emitters assemble exactly that program. It is exact within its grammar and about 2 ms on a CPU.

Its cost is human labour. Each program kind is 160 to 287 lines of hand-written emitter, each language is another 104 to 231, and there is no transfer between them. I wrote in two separate experiment write-ups that this is “precisely the labour a learned generator would amortise.” A model, the argument went, would learn the mapping once and cover new cases for free.

For a long time that was untestable, because the model scored 0%.

0% was the plumbing, four times over

Five experiments in a row scored 0/6 compile. The diagnosis I kept reaching for was capacity or training time, and it was wrong every time. Training to 3.5× lower loss left the failure distribution identical.

What actually fixed it was two one-line changes to the data pipeline:

1. Word-split docstrings. Every app in the corpus gets a spec-derived docstring, and docstrings were tokenised atomically, so every held-out app’s docstring was <unk> by construction. 4,141 of 5,953 vocabulary entries were one-off docstrings. Splitting them into words took the vocabulary to 1,817.
2. An <eos> token. Training apps had nothing marking the end, so nothing taught the model to stop, and every previous arm’s output was mechanically truncated at the 3,200-token cap.

Same architecture, same data, same optimizer. The results file (021-pipeline-fixes/fixed_combo.json) records what happened:

Strict means the requested features work and the unrequested ones are absent. The model does both. Dev loss improved as well, 0.034 to 0.026, from deleting 4,100 vocabulary entries: entries that can only ever be copied are pure liability, since they consume capacity and are unusable on held-out input.

So the generator worked, and the claim became testable.

The test, and the retraction

The library domain (entity book, table books) was held out of the corpus before any training. If the amortisation argument were right, this is where it would show: a new noun costs the symbolic system a schema synthesiser of about 15 lines, and should cost the model nothing at all.

From 025-generator-transfer/fixed_domain.json, on 8 held-out requests:

Zero. Not a degradation, a floor.

I checked what it wrote instead by reading the eight generated files rather than trusting the summary. Every one names a table it was trained on:

000_library_basic.py      contacts
001_library_delete.py     tickets
002_library_edit.py       bookings
003_library_search.py     contacts
004_library_toggle.py     bookings   (with `tickets` leaking in elsewhere)
005_library_comments.py   bookings + requests
006_library_stats.py      bookings
007_library_category.py   bookings + requests

bookings for books is the tell: the model reaches for the lexically nearest entity it was trained on. The apps are internally consistent for the wrong entity, which is why 7 of 8 still compile while 0 of 8 are right.

Here is the part that made me stop defending the argument. This is the top of 006_library_stats.py, unedited:

# Spec:a simple book lending app offering a stats page and creating and viewing books
...
"""A simple address book app offering a stats page and creating and viewing bookings"""
PORT = int(os.environ.get('PORT', '8806'))
DB = os.environ.get('DB', 'bookings.db')
con.execute('CREATE TABLE IF NOT EXISTS bookings (id INTEGER PRIMARY KEY AUTOINCREMENT, guest TEXT, details TEXT)')

Line 1 is the prompt. The model then copies nine consecutive words of it verbatim (“offering a stats page and creating and viewing”), swaps book lending for address book, swaps books for bookings, and gives the table a guest column. The word it needed was in its context, on line 1, and I confirmed that across all eight files: the string books appears exactly once in each, always on line 1, never in the code.

So the model can copy a nine-word span and cannot copy the one word that decides correctness. The amortisation claim is dead: on the axis where a learned model was supposed to win, it scores 0/8 while 15 lines of symbolic schema synthesis score 10/10.

The diagnosis I got wrong

I wrote down the obvious conclusion: this is a copying problem, and the remedy is pointer attention or byte-level entity tokens.

Before building any of that, a 30-line probe against the checkpoint (029-entity-generalization/probe_vocab.py) checked whether copying was even the binding constraint. It was not.

books is not in the 1,817-token vocabulary. It never occurs in training, so it is <unk> on the way in and unreachable on the way out. Every SQL statement naming it is a single atomic string token, so 'INSERT INTO books (...)' is also one out-of-vocabulary token. Only 3.9% of a held-out app’s tokens are OOV, and they are exactly the load-bearing ones.

No mechanism defined over that vocabulary could have produced the right program, pointer or otherwise. “It substitutes a trained entity” was a symptom of an unreachable output slot, not evidence about copying. That is the fifth representation defect in this line of work, and all five were invisible in the loss curve.

So: make the noun expressible. Spell the entity at character level (<w> b o o k s </w>) and split literals at the entity boundary so the template fragments stay in vocabulary. Vocabulary went down, 1,817 to 1,623, and dev loss was unchanged at 0.027.

It still scored 0/8 on fidelity. But it failed in a new and much more informative way. At epoch 1 the model visibly attempted the character channel:

booooooooooookings

Starting b, o, o, then degenerating into a repetition loop. By epoch 9 it wrote bookings cleanly, with no character attempt at all. More training made it abandon the channel it had started to use.

That is not a capability failure. It is the objective working as specified. Every training app’s entity was one of nine in-vocabulary nouns, so the entity was always predictable from the app’s own internal consistency, and spelling a novel noun was never required to be right. Predicting the familiar sequence b-o-o-k-i-n-g-s is simply lower loss. The model was never asked to copy, so it did not learn to.

Making it expressible was necessary and not sufficient. It also cost real quality: in-domain compile fell from 6/6 to 1/6, because spelling an identifier as eight tokens gives eight chances to break it.

Randomise the nouns, and the substitution disappears

The fix the diagnosis implies is to change the data, not the model: draw a fresh entity noun for every app, so the mapping cannot be memorised and copying out of the spec is the only way to be right.

I regenerated that corpus while writing this. It takes 9 seconds and reproduces byte-for-byte against the tracked manifest:

600 training nouns, 10 held-out nouns, 128 feature combos
wrote 4240 apps ({'train': 4200, 'test_combo': 20, 'test_entity': 20}), 0 skipped

The ten held-out nouns include book, so the number stays comparable rather than becoming an easier case, and the script asserts no held-out noun leaks into training.

Trained on that corpus and decoded greedily from the spec alone, substitution is gone: 0/10 generations write a trained noun, against 8/8 before. The model now attempts the novel noun. And the new failure names the next blocker precisely:

books      -> bs
sprockets  -> sckets
lectures   -> letes
harvests   -> heves

It gets the first character right and the ending right and loses the middle. Spelling a symbol character by character needs state for how far through it you are, and nothing in the architecture carries that. The blocker was never vocabulary, and after the randomised corpus it is no longer distribution; it is positional.

The zero-parameter mechanism that just does it

Meanwhile a different line of work in the same repo had a document cache: a count-based model over the text currently being written, with no parameters and no training.

Copying a noun out of the prompt is exactly what such a cache is for, and nobody had put the two together. The measurement walks each held-out program left to right and, at every position whose target is part of the entity noun, asks each system to predict it. I re-ran it (cache_copy.py, 63 seconds):

WHOLE-NOUN accuracy where the program FIRST names it:
  corpus   0/20 apps
  cache    20/20 apps
  mix      17/20 apps

system       spec (floor)    in the code      first in code
corpus             0.243          0.254             0.100
cache              0.662          0.970             1.000
mix                0.719          0.978             1.000

corpus-model errors that were a TRAINED noun's character: 3960

A cache with zero parameters gets the noun right 20 times out of 20. A count model trained on all 4,200 apps gets it right 0 times out of 20, and its errors are a trained noun’s characters 3,960 times. The corpus arm is the control that makes this mean something: it fails the same way the neural model did, and it is not a data-scale problem, because 4,200 apps is the whole corpus and the score is zero.

The information was always extractable from the request. What was missing was a mechanism that emits what it just read.

A sweep over the mixing weight is worth putting next to that, because it is the kind of result that flips on the metric you pick. Per-position accuracy peaks at a blend (0.978 at β=0.70) while whole-noun correctness is monotone and peaks at pure cache (20/20 at β=1.00). Blend when the unit of correctness is a token; do not blend when it is a multi-token symbol that is wrong if any character is.

The honest catch, and it is a big one: all of that is teacher-forced. Under free generation the cache’s best weight collapses from about 1.0 to about 0.3, because it begins reading the model’s own output rather than true text. Pushed further, it amplifies whatever was just emitted: the gated variant runs degeneracy from 0.34 to 0.92 and takes “no table at all” from 1 of 10 to 10 of 10. The component that wins on prediction is the one whose input degrades under generation.

What did survive was constraining decoding to spell one of the spans the request actually contains: a trie over the user’s own words, again with zero parameters. With the cache at low weight, that takes table hit from 0/10 to 10/10 and purity from 0.00 to 1.00.

The obvious objection is that masking to “spell one of the two nouns in the request” leaves so little freedom that a model which learned nothing would look correct. That was tested by injecting a second, equally novel, equally legal noun into each request: 9 of 10 chose the requested noun, 1 chose the distractor, against a chance rate of 0.5 (binomial p = 0.011). The constraint supplies legal spellings; the model supplies which one.

It fixes the entity, not the program. Whole-app validity is untouched, and a line like PORT = int(os.environ.get('PORT', deleting them in the same outputs is a fair reminder of that.

What generalises

Three things, and the first one is the one I would actually carry anywhere.

A 30-line probe beat five experiments of intuition, twice. “Needs more capacity” and “needs more training” were wrong at every stage, and the loss curve never once indicated the real defect. Both diagnoses that mattered came from cheap instruments: a teacher-forced per-position probe that localised two pipeline bugs to single token positions, and a vocabulary probe that showed the target string was unreachable before anyone built a copy mechanism for it. Five separate representation defects, all invisible in the loss.

A model will not learn a capability its training data never requires. The character channel was available, expressible, and visibly attempted at epoch 1, and training removed it, because being right never depended on it. That is not the model failing to generalise; that is me failing to specify. The fix was a corpus change, and it worked on the first try.

Check whether the failing thing is even representable before designing the fix. I had written “this is a copying problem, use pointer attention” into my notes as a conclusion. It was a plausible, standard, entirely wrong remedy for a slot that was <unk> on the way in and unreachable on the way out. The probe cost 30 lines and saved building the wrong mechanism.

The arm still does not do what I built it to do. Inside domains it has seen it writes correct software at 83%, which is a genuinely stronger result than I expected from 11.9M parameters, and it is compositional generalisation over 2⁷ feature subsets with no combination-specific code. It just is not the labour-amortising property that would have justified preferring it, and the thing that finally did the copying had no parameters at all.