Tagged code-generation
9 write-ups.
The Table Named After the Programming Language
"Build your own Blockchain in JavaScript" produced a table of javascripts. Fourteen lines that name no language fixed that class of error, and every one of the six rows it actually gained on was a different preposition from the one in that title.
A Learner That Scored Below Its Own Random Control
Growing hidden units scored 15.5 of 75 where random units of the same shape scored 17.0 and a hand-written keyword rule scored 47. In a separate protocol, handing the learner the rule's own answer as one more feature was worth a reliable +7.8 rows and still did not reach the rule that supplied it.
How a Pluralisation Rule Broke a Correct Answer
A planner answered `movies` for a request containing the word `movie`. The generator singularised that to `movy` and its own checker then rejected the result. Two candidates round-trip and the order decides, and the same two words defeat both obvious repairs.
Taking the Language Model Out of a Build Planner
Over 88 requests, with everything downstream held constant, the model's plan rescued zero rows an empty seat lost, under all four ways of scoring the run. The one reading in which it looked resolved was resting on two rows the scorer could not see.
A Ceiling That Was Measuring My Test Harness
The headroom over nine build configurations came out zero, and then zero again. The first zero meant eight of the nine were invisible to the driver; the second, from 1,584 builds through a driver that reaches all nine, means they return the same verdict on all 88 requests.
Searching for a Program Instead of Generating One
A primitive algebra searched from examples solved five of twenty-eight exercises, and every one already had a hand-written implementation in the same repo. A behavioural index recognised one function in forty-eight, and zero of eleven when handed other people's correct solutions to the same exercises.
When a Test Suite Rejects a Correct Program
Eleven of twenty-one operations never passed my frozen checks, and I wrote that up twice as a limit of the generator. A wrapper that only re-expressed each program's own stdout repaired 90 of 102 failures, changing no algorithm.
Why a Small Transformer Can't Copy a Word It Hasn't Seen
An 11.9M-parameter transformer scores 83% strict on the domains it trained on and 0/8 on a single unseen noun. Three experiments to find out why, one of which corrected the diagnosis I had already written down, and a zero-parameter mechanism that does the copying 20 times out of 20.
An App Generator That Verifies Everything Except Its Parser
A natural-language app generator whose verification sweep is exhaustive over everything except the part that reads English. The composer scores 100% on 810 cells built from feature sets; the parser that has to produce those feature sets scores 0.70, and nothing prints that number next to the other one.