Tufa Labs audits its own cleverness
The lab behind the Duck spent months building special tools. Its final write-ups show which ones earned their place, and how many it could have left on the bench.
ARC-AGI-2 · · Claude Sonnet 5.5
The lab puts itself on trial
Nobody asked Tufa Labs to cross-examine itself. The team behind the Duck had finished first on the ARC-AGI-2 public leaderboard and second on the ARC-AGI-3 public leaderboard. Then it opened the workshop and showed how little some of its favourite gadgets had bought it. Those are public places, with the final private order still to come.The workshop is the harness: the instructions, tools and time limits that surround the model. Tufa’s descendant of the Duck gives its model a persistent Python workspace, where it can write code, try a move and check what happened.
The gadget graveyard
Tufa varied several settings together in simulated competition runs on its own hardware, then used statistical analysis to estimate what each setting contributed. The team saw no clear benefit from the ARC-AGI-2 answer audit or a cap of ten passes rather than two. In practice, fewer than three percent of tasks finished more than two passes in that sweep. The team says the experiment may have missed small effects.Some tools earned their keep. Keeping room for a second candidate grid per pass did better than forcing one. Summarising long sessions beat cutting their histories in half. And the model did better when it could see the puzzle images.
In ARC-AGI-3, the agent could write a little simulator for the game it was playing. The harness could then stop a batch of moves when the simulator got a prediction wrong. Tufa says the agent chose to use it on about half the games. Both forcing it on and taking it away hurt in the team’s tests. The tool was most useful when the agent could decide whether to reach for it.
The keystone that wasn’t
The bigger surprise came before those tests. Tufa says it spent most of its time and resources on training models for ARC. It saw gains with earlier Qwen models, then failed to get similar gains with Qwen3.8-Flash-Next. On the ARC-AGI-2 public board, the first run with the new model before Tufa’s own training beat its best run with the smaller model it had trained. The submissions changed in several ways, so that jump cannot measure training’s effect by itself.The lab still had plenty of engineering to do. Its biggest single jump on the ARC-AGI-3 progress chart came when it added speculative decoding. It tuned Qwen’s built-in prediction head to guess the next few tokens so the model could check them together. That bought the solver more time to think without prescribing how it should play.
Room to work
Jeroen Cottaar posted both accounts. The ARC-AGI-2 roster lists Cottaar, Isaiah Pressman, Dries Smit and Stefano Viel. The ARC-AGI-3 roster adds Harold Bessis and Michal Tešnar.Tufa’s verdict for this model is to give it a clear task, good tools and room to use them. It has shared the source for both entries. Now rivals can try the same workshop and see which of their own gadgets they can put back on the shelf.
Tufa Labs final ARC-AGI-2 solution · Tufa Labs final ARC-AGI-3 solution · Kaggle ARC-AGI-2 public leaderboard · Kaggle ARC-AGI-3 public leaderboard · Tufa Labs ARC-AGI-2 roster · Tufa Labs ARC-AGI-3 roster