An App Generator That Verifies Everything Except Its Parser

appgen is a tool I built to answer a narrow question: can you get from a sentence to a running, verified application without a language model doing the generating? You type "I want a support desk system with priorities, comments, search and closing tickets" and it writes a single dependency-free Python file, starts it on a private port, exercises every feature you asked for over real HTTP, and only then shows it to you. It takes about 2 ms of CPU and no GPU.

It works. That is the uncomfortable part, because the way it reports that it works is misleading, and I wrote the reporting.

The tool’s README says the verification sweep covers “all 810 domain × feature × family cells,” and that this “is why the 100% claims above are trustworthy.” Both halves of that are true in a narrow sense and wrong in the sense a reader will take. This post is about the gap, which I found by asking a question I should have asked much earlier: what is a cell, exactly?

The code lives in a private research repo, so there is no link to it here. Every figure below comes from a file or a command I ran, and I have named which.

What a cell is made of

The sweep lives in experiments/028-relations/exhaustive.py, and the whole thing turns on a six-line generator:

def cells():
    for dom in DOMAINS:
        yield dom, frozenset()
        for f in FEATURES:
            yield dom, frozenset([f])
        yield dom, frozenset(FEATURES)

DOMAINS has 10 entries (blog, booking, contacts, events, expenses, inventory, library, recipes, support_desk, todo) and FEATURES has 7 (category, comments, delete, edit, search, stats, toggle). So each domain contributes one empty set, seven single-feature sets and one full set: 9 configurations, 90 cells. Nine families of emitter (single-file unstyled, single-file styled, api, cli, project, relational, node_web, node_api, c_cli) multiply that to 810. I checked the arithmetic rather than trusting the README, and 810 is right.

Now look at what a cell is. It is a tuple of a dictionary key and a frozen set of strings. A cell is not a request. Nothing in cells() produces English, and nothing downstream of it consumes English. The emitters are handed a schema and a feature set and asked to produce a program.

The sweep is explicit about this where it would otherwise be tempting to use the real entry point. In sweep_project, at line 96:

# build_project parses a request; drive the emitters directly instead

I wrote that comment for a good reason (a sweep should test one thing, and build_project would drag the parser into an emitter test), and it is the right call for the sweep. The problem is what happened to the claim afterwards. appgen is a tool you drive with a sentence, and its verification is exhaustive over everything downstream of that sentence while touching the sentence zero times. When the README says 810 cells are why the 100% is trustworthy, the 100% it is vouching for belongs to the composer, and a user’s request has to survive a different component first.

The component that reads English, measured on its own

That component is a hand-written keyword planner, and to its credit I did measure it, in a separate experiment (041) built specifically to ask whether a learned model should replace it.

Measuring an intent classifier fairly is harder than it looks, because I wrote the test phrasings, so a random train/test split leaves near-duplicates on both sides and everything scores well. Instead the corpus is grouped into eight phrasing families (imperative, desiderative, interrogative, polite, declarative, need, terse, wordy) and whole families are held out, so a model is tested on a way of speaking it has never seen. Slot vocabularies (entities, features, languages, kinds, shapes) are disjoint across the split too, so nothing wins by memorising bug tracker. That is 146 training clauses and 32 test clauses per fold, 8 folds, over 7 operations.

I re-ran compare.py while writing this. It reproduces:

The rules win, decisively, against a Naive Bayes classifier (0.24, barely above the always-guess-the-most-common floor of 0.29) and against nearest-neighbour retrieval (0.48). That was the experiment’s finding, and it is why the planner is still hand-written; there was nothing worth grafting on, since the hybrid (0.68) does not beat the rules alone.

But read the winning column as a user rather than as a comparison. 0.70 means that on a phrasing the rules were not written for, roughly three requests in ten are read wrong. The per-operation breakdown says which:

kind is how the tool decides whether you asked for an HTML app, a JSON API or a CLI. On held-out phrasing it gets that right 32% of the time.

I wrote a paragraph here claiming the obvious consequence: that the composer then builds the wrong kind of program, verifies it exhaustively, and reports ALL CHECKS PASSED on an app nobody asked for. Then I went and checked, and it does not follow.

Of the 17 misread kind clauses, 15 are read as update, and they all share one shape: the clause names a kind and a feature (“turn it into a web app with search”). Driven end to end against a real app, that label produces the right program every time:

The update op hands the whole sentence to the updater, where one function converts the kind and another applies the features. It is the only operation that carries both; a bare kind op carries the kind alone. So “improving” that 0.32, which means relabelling those 15 clauses, would convert the program and silently drop the feature asked for in the same breath. The labels are wrong for compound clauses, not the planner.

That is worth stating plainly because it cuts against the post: an op-label metric is not a behaviour metric, and I reached for the scary reading of a low number before driving the tool. The number that describes what a user gets is 4 out of 4 on those goals, not 0.32.

What survives is narrower and still the point: 0.70 and 0.32 are real measurements of a component that the 810 cells do not touch, and neither figure appears next to the 100%. The two numbers describing this tool measure different things, and only one of them is on the front page.

Built is not the same as reachable

The second shortcoming is the one I find hardest to defend, because the repo diagnosed it, wrote a fix, and then left the original offenders alone.

Experiment 045 audited every transform in the repo against what the shipped tool could actually reach, and found six built, verified, written-up transforms that no user could ask for. Pagination, SMS, CSV and email had all been explicitly requested. Asking returned “not supported” while the implementation sat in the repo, tested and documented.

The mechanism is worth naming: reaching a feature meant wiring it at five separate sites (the grammar, the kinds table, the emit path, the verify path, the manifest stickiness, plus the agent’s tokens and its goal self-check). Exp 045 wired pagination by hand. It worked, and it did not scale; CSV was still unreachable thirty rounds later.

The same failure appears twice more, in comments I left in appgen.py itself:

Exp 034d built and verified the Node one (18/18 with the SAME oracle); it was simply never wired up, so porting an authenticated app silently removed its guard.

Three lines down, the same story for ranked search: a Node transform verified at 10/10 that “was never wired up, so porting a ranked app to JavaScript silently downgraded it to LIKE.” A verified transform that nothing dispatches to is a verified transform that does not exist, and in both of those cases the failure was silent and security-relevant.

Experiment 040 built the structural fix, and it is the right one: a feature declares its own tokens, flag, transforms, oracle and applicable cells, and generic dispatch reads a registry. Discovery is a glob over feature_*.py, because naming the modules in an import list would recreate the defect being removed; a glob cannot forget. Wiring that in cost about 120 lines once, after which SMS, email and validation each cost a single declaration file with no edit to appgen.py.

Here is what I did not do. The three transforms that motivated the whole exercise are still hand-wired. As of today appgen.py (1,586 lines) references the AUTH_KINDS, RANKED_KINDS and PAGED_KINDS tables 21 times, and the registry those tables were supposed to be replaced by holds four features:

Each is available in exactly one of the six language × kind cells the tool supports. The fix works, and it was applied to the new features rather than to the old ones, which means the class of bug that produced six unreachable transforms is still live in the three places where it did the most damage.

The sweep became a constraint on the design

This is the part I did not expect to find, and it is the strongest form of the problem.

Ranked search should probably have been an eighth feature in the grammar. It is not, and the comment explaining why sits above RANKED_WORDS:

baking an eighth feature into the grammar would change the corpus from 2^7 to 2^8 subsets and invalidate exp 028’s 810 cells

That is a verification suite dictating a design decision. Ranked search became a transform layered over the emitted app, rather than a feature the composer knows about, in order to keep a number from going stale. The reasoning is locally sensible (a 2^8 corpus would invalidate published results across a dozen experiments) and it is still the tail wagging the dog.

It also gives the 810 figure a shape worth stating plainly. 810 cells is exhaustive over domain × single feature, plus the empty and full sets. It is not exhaustive over feature combinations: 7 features admit 2^7 = 128 subsets, and the sweep visits 9 of them. The sweep’s own docstring is honest about this (“every domain x feature pair covered at least once”), and pairwise coverage is a real, defensible standard that catches interaction bugs cheaply. The README’s “all 810 cells” is the phrase that quietly promises more.

Why I trust the 810 anyway, and what that is worth

None of this makes the sweep theatre. It replaced random sampling, and it exists because of a bug that random sampling hid.

Every behavioural result in the repo before experiment 028 came from a random sample of roughly 10 cells. That sample missed contacts × category, where the generated column was named group, a SQL reserved word. Those apps compile cleanly and crash on the first query. 235 of 4,412 corpus apps (5.3%) never worked, and the defect survived from experiment 012 to experiment 028, so sixteen experiments were trained and scored on a corpus containing 5.3% crash-on-start code. Exhaustive sweeping along the axis that mattered found it immediately.

I re-ran the whole thing while writing this, all six flags, and it is green:

single-file, unstyled (corpus path): 90 cells    ALL PASS
single-file, styled (tool path):     90 cells    ALL PASS
api + cli kinds:                    180 cells    ALL PASS
multi-file projects:                 90 cells    ALL PASS
relational projects:                 90 cells    ALL PASS
node + C emitters:                  270 cells    ALL PASS

total failures: 0  (810 cells)

So the headline is not inflated. 810 of 810 really do pass, today, on my machine. That is what makes this a reporting problem rather than a correctness one, and it is why I would not remove the number. Usefully, the script also prints its own limits when run partially, labelling the run a PARTIAL sweep and naming the flags the full 810 requires. An instrument that reports its own coverage was already doing something the README was not.

So the composer is genuinely exact inside its grammar, and the grammar is genuinely narrow: three program kinds and three languages, but only 6 of the 9 cells exist (Python does web, api and cli; Node does web and api; C does cli only), because each cell is a hand-written emitter of 104 to 287 lines with no transfer between them. --project and --relations are Python-only. C stores flat records, so it drops comments, stats and category with a printed note. Coverage is exactly the union of the emitters someone wrote, and it grows only by someone writing more.

That narrowness is not a footnote to the 100%; it is the reason for it.

What generalises

The thing I would carry to another project is not “sweep exhaustively,” though the group bug argues for it. It is sharper than that:

An exhaustive sweep proves a property of the component it drives, and its credibility then transfers silently to the whole tool. 810 is a real number about the emitters. It became, in my own README, a claim about appgen. Nobody lied; the sweep simply had a much better publicist than the parser did, because the parser’s number lives in an experiment write-up and the emitters’ number lives on the tool’s front page.

The tell was available the whole time, and I had written it myself: the sweep bypasses the entry point, in a comment, on line 96. A verification suite that cannot be driven the way users drive the tool is measuring something adjacent to what you are claiming, and the honest fix is not to weaken the sweep but to give the uncovered component its own number and print it next to the other one.

For appgen specifically, the shortcomings I would fix in order: add a sweep that starts from request strings, so the parser is measured where the emitters already are; migrate AUTH_KINDS, RANKED_KINDS and PAGED_KINDS into the registry, so the unreachable-transform bug cannot recur where it already has; and stop printing 100% without naming the component it belongs to.

The correction above is the reason that list is in that order. I had the sweep second and the number’s interpretation settled. Building the sweep first is what showed the interpretation was wrong, and a request sweep would have caught a defect the 810 cells never could: the tool read a script tracker with search as a request for a command-line program, because six of its sixteen kind keywords are ordinary nouns and script was simultaneously the entity it had resolved.

The composer was never the risky part. It was just the part with a number.