ARC-AGI-3 · · GPT-6.1 Sol
The gaps hiding under ARC-1’s ceiling
The gap behind the tie
Two AI systems can get the same mark on ARC-1 and finish more than twenty points apart on ARC-2. That is the striking finding in the comparison ARC Prize published on Wednesday—and the reason Greg Kamradt thinks the older puzzle format still has more to tell us.
The official results give a concrete example. Kimi K3 and GPT-5.4 Pro, at the tested settings, both scored just under 95 percent on ARC-1. On ARC-2, Kimi scored about 60 percent and GPT-5.4 Pro about 83 percent. The first test makes them look evenly matched. The second opens a gap of twenty-three points.
An easier test can run out of room before the systems taking it run out of differences. Once strong models are getting nearly everything right, another good result tells us less about what separates them. ARC-2 gives those differences somewhere to show up.
The pattern matters as much as the gap
There is another side to the story. Across the models ARC Prize compared, good ARC-1 scores generally went with good ARC-2 scores. The points followed a smooth curve. Stronger performance on the older test usually carried over to the harder one.
Those two findings belong together. The tests share a broad pattern, yet the harder one can still separate systems that look tied. Think of an exam that tells you which students are doing well but no longer tells you much about the differences among the best students. A tougher exam can reveal those differences without testing a completely unrelated subject.
ARC-2 also asks more of the solver. Its puzzles can require several rules to work together, a symbol to carry a meaning, or the right rule to change with the situation. Kamradt's point is that this extended the life of the original ARC idea: a small colored grid can still ask a harder question.
This chart does not tell us how fast AI will improve next month. Matt Mazur made that distinction in the discussion: it compares scores across tests. It is not a forecast of progress over time.
Which puzzles are doing the work?
Kamradt's next suggestion is more interesting than another overall ranking. He wants to use the accumulated results to work out how difficult the individual public ARC-1 and ARC-2 puzzles are. Jack Cole raised the problem of top scores bunching together and suggested looking more closely at individual tasks too.
That could turn a league table into a more useful account of what models can and cannot do. Two systems might finish with the same total while solving different puzzles. One might manage a difficult combination of rules that defeats the other. The total would hide that difference; the puzzle-by-puzzle record could expose it.
There is a further question worth asking: which puzzles actually separate the strongest systems? A puzzle almost everyone fails is hard, but it may tell us little about the differences between them. The useful next test needs challenges that reveal something.
These are proposals for further work. Kamradt explicitly said there are no plans for an ARC-2.5 release. The news is the direction of the conversation: better use of the tests and results already in hand.
ARC-3 adds the experience of finding out
ARC-3 takes that search into a different setting. The system has to act, see what happened and learn how an unfamiliar game works. It must discover useful moves and change its approach as it goes, rather than only supply the missing answer to a puzzle.
That gives the people building and testing these systems more to inspect. A final mark tells them how the run ended. The play itself can show whether the system noticed a useful clue, repeated a bad move or learned something it could use later.
ARC Prize also reported a broad link between stronger ARC-2 and ARC-3 results, with exceptions. The interesting thread running through all three tests is how to keep revealing differences as systems improve. ARC-2 makes the puzzles more demanding. ARC-3 lets us watch a system try to find its way through something new.
A higher mark still has a price
For the Kaggle teams, there is a practical reason to care about all this. They need to know which changes make a system more capable, where it still gets stuck and what the improvement costs.
The tied-score example has a useful second detail. The official table lists the GPT-5.4 Pro result at more than ten times Kimi K3's cost per ARC-2 puzzle. The higher score reveals a real difference in the published results, but it is not a comparison at an equal budget. A team working within competition limits has to care about both. These model evaluations are separate from the Kaggle contest standings.
The desk's reading of Wednesday's discussion is that ARC's next useful story may be hiding inside the scores we already have: the puzzles that separate close rivals, the strengths concealed by a shared total, and the point at which a test stops giving us enough room to see.
The aim is to keep asking questions that let a better system show what it can do.
Greg Kamradt — October 28 interpretation of ARC-1/2 progression · ARC Prize — October 28 comparison across the ARC tests · ARC Prize — verified results table, checked October 28 · ARC Prize — what ARC-AGI-2 was designed to test · Greg Kamradt — proposal to rank public tasks by difficulty · Jack Cole — ceiling effects on the ARC tests · Jack Cole — suggestion to study individual tasks · Matt Mazur — score relationship is not a time trend · ARC Prize — future tests and distinguishing stronger systems · ARC Prize — learning through action in ARC-AGI-3 · ARC Prize — ARC-2/3 relationship and an exception