Giving an Open-Weight Language Model the Vote in Among Us Meetings

Meetings are the part of Among Us that a language model looks made for. The crew is trying to find the few hidden impostors among them. When a body is found, everyone talks, states where they were, accuses someone and votes, and whoever gets the most votes is thrown off the ship. My simulator’s bots normally vote with small recurrent networks trained on thousands of meetings, which read who saw whom and who accused whom and output a suspicion score for each player. I expected that handing the meeting to a real language model, one that could argue, ask questions and catch lies, would make the crew better at finding impostors. Across six setups it never did.

The setup

The model is gpt-oss:20b, an open-weight model from OpenAI, running locally through Ollama. In a meeting every surviving player, crewmate or impostor, is a separate prompt to the same model, and its reply is that player’s speech and vote. A crewmate’s prompt contains only what that player could know: where it walked, who it saw and when, any kill or vent it witnessed, and the table of where everyone claims to have been. An impostor’s prompt also contains its real movements, marked secret, and permission to lie. Movement, tasks and kills between meetings stay with the trained bots; only the meetings change.

There were three meeting formats. In the first, players speak in rounds and may ask each other structured questions (“where were you 20 seconds ago?”). In the second, each crewmate’s prompt also includes the voting network’s guesses as an “instinct” (described below). In the third, players hold a free conversation in turns, again with the instinct.

The comparison is the same bots in the same lobby with the trained networks doing all the voting, crew and impostors alike, over 300 games. So the language model replaces both sides’ votes at once, and a worse result for the crew could come from worse crew votes or from better impostor talk; the runs below cannot tell those apart. I measured the baselines after the language-model runs, on the current version of the simulator. Five of the six language-model runs used an earlier version, which I mark. The bots come from different leagues (training runs), and each league trained its own voting network.

Eight players, two impostors

These used the bots from my second league. The network baseline did not record wrongly ejected crewmates.

On its own the language model did much worse than the networks. The crew ejected an impostor once in sixteen games and an innocent crewmate almost every game, and won 2 games of 16 where the networks’ crew wins about half. That gap is well beyond chance (one-sided Fisher exact test, p = 0.0017), with the caveat that the simulator version differed.

The “instinct” put the voting network’s numbers into each crewmate’s prompt as a line of text. It opens “Your instinct (a trained model reading everything YOU saw, plus the accusations so far; it picks the right impostor about 70% of the time, so weigh it with the facts):” and then gives each other player’s percentage chance of being an impostor. The 70% in that line was my rough estimate when I wrote the prompt, not a measurement. With the instinct and the rounds format, the crew won 8 of 16, indistinguishable from the networks (p = 0.54). If the model mostly voted the way the instinct line pointed, that is the result you would expect; I did not measure how often it did.

Fifteen players, three impostors

In these lobbies the crew wins more often under both kinds of voting, and the ordering held. All three runs use the conversation format with the instinct.

The league 6 row is the only like-for-like comparison: same bots, same simulator version, same settings, run on the same day. Sixteen of 20 against 280 of 300 is a one-sided Fisher exact p of 0.052, and the 95% interval on 16 of 20 is 58% to 92%. On its own, that row is suggestive rather than settled. The league 4 gap is far larger (p = 0.0025), but between that run and its baseline I changed how meetings treat a running sabotage and what impostors know when they shut doors, so I would not lean on it.

Counting across all six setups: in one the language model was clearly worse than the networks (eight players, no instinct). In the five with the instinct it matched the networks once and fell short four times, by margins that are individually too small or too confounded to be conclusive. I have no setup in which it did better.

What it costs

A 15-player game with gpt-oss meetings took 524 seconds of wall-clock time on average in the league 6 run. Nearly all of that was about 4 meetings’ worth of turns, at one model call per player per turn. The networks’ votes need no model calls at all. That would be a fair price for better votes, and the votes were not better.

Why I think it happens

This part is my reading rather than a measurement. The networks have an unfair advantage that is also the point: they were trained on the outcomes of thousands of meetings in this simulator, so their suspicion scores are calibrated against who actually turned out to be an impostor here. The language model brings general knowledge of how deception sounds. An impostor here does not sound like anything; it names a room, and the only way to catch the lie is to check that claim against who could have seen that room. The prompts contain what each player saw. Nothing in the results suggests the model made that check reliably, and with the network’s numbers in the prompt it did no better than the network.

An earlier benchmark of mine, which ranked language models on spotting liars in Mafia, found first-day accusations at chance. There, on the first day, nothing about the liars is knowable yet. Here the evidence is in each crewmate’s prompt, which makes falling short of a small network harder to excuse.

The simulator, the meeting prompts and the measurement scripts were written by Claude Code at my direction. The code is not public. Every figure above comes from the result files the runs wrote, and the significance tests are recomputed from those counts.