GPT-6 Astra’s report card, Part 2: What does 99.9% tell us about AGI?
Two public ARC-AGI-3 examples explain the 99.9% score, the human reference and why the surrounding software matters.
By Liqing Pan7 min read
Translated from 简体中文
AI
Industry
Benchmarks
Image credit: Prompthon editorial artwork.
After seven benchmarks, we reach the result that drew the biggest question mark: 99.9%.
OpenAI launched GPT-6 Astra on September 3, 2026, accompanied by Greg Brockman’s welcome to the AGI era, as reported by Fast Company.[1][2] Part 1 examined science, mathematics, code and professional work. This article focuses on ARC-AGI-3: can a system learn to act in a new environment without being handed its instruction manual?
AGI means artificial general intelligence. Here, the question is whether competence extends to learning across unfamiliar tasks, rather than only performing well on known ones. A benchmark contributes evidence; its name does not settle that larger question.
Watch the tasks, then interpret the score. Two public examples make the test concrete. We will then unpack the human reference, the supporting software and the limits of the result.
Image credit: Prompthon editorial artwork.
Prompthon editorial overview. The percentages use different metrics and cannot be added. Human references are drawn from separate studies. The ARC score and human comparison use different settings.
A deep-learning researcher asks a harder question
François Chollet introduced the original ARC in 2019. He also created Keras, a widely used tool for building neural networks. His question was practical: is being excellent at familiar tasks the same as learning new skills efficiently?[3]
The early test used colored grids. Several before-and-after examples revealed a rule; the solver had to apply it to a new grid. You did not need a doctorate to understand the format, and an encyclopedia did not hand you the transformation.
The first competition in 2020 attracted 913 teams; the winner solved about 20% of the private test set. OpenAI’s o3-preview brought renewed attention in late 2024.[3][4] The third generation moves from static puzzles to interactive environments: take an action, observe what changes, revise your hypothesis and use what you learn in later levels.[5]
Think of trying an unfamiliar appliance with no manual. Finding the power button is one discovery; working out which controls interact and how to complete a whole task takes more experimentation. ARC-AGI-3 makes that learning process measurable within designed game environments.
First example: The controls are visible, the rules are not
The public environment s5i5 contains colored mechanisms and controls whose states change as the solver acts.[6][7]
Public example: s5i5. Which control extends a mechanism? Which rotates it? In what order should they move? The solver must infer these relationships from feedback and plan a sequence.
Image credit: ARC Prize — Astra evaluation report; official demonstration or comparison.
ARC Prize’s published animation. It illustrates a public run, not a Prompthon test or proof that every level was completed.
Pressing a button once may not even reveal what you changed. Understanding one control is not enough: an extended piece may obstruct the next move, while a rotation changes positions you had memorized. The model must connect experiments into a working account of the mechanism.
ARC Prize describes Astra keeping compact records of the colored pieces, their lengths, control locations and planned operations.[6] It resembles notes a person might keep while solving an unfamiliar mechanical puzzle. The interesting behavior is using a newly discovered rule to organize the next actions.
Second example: The route is clear, but the guards move
The tu93 environment adds guards and moving patrols to a maze-like task.[6][8] The published run below uses additional programming tools.
Public example: tu93. A path on the map is not necessarily safe right now. The solver must learn movement, combat and patrol behavior, then time its actions accordingly.
Image credit: ARC Prize — Astra evaluation report; official demonstration or comparison.
Excerpt from ARC Prize’s demonstration using PRO-LONG, which can execute custom code. Its tools differ from the 99.9% scoring setup and the human participants’ conditions.
Planning through a moving environment is like crossing a busy intersection: knowing where you want to go does not tell you when to move. A route can become invalid after the next patrol step. The solver must compare its prediction with what actually happens and revise the plan when they diverge.
In this run, Astra wrote helpers for pathfinding, combat and patrol simulation and checked their predictions against observations.[6][9] This shows what a model working with tools can attempt. It should not be presented as a controlled human–AI contest under identical conditions.
What did the human comparison measure?
ARC Prize recruited about 500 members of the public without specifically selecting puzzle experts. For each level, it used the median action count among successful human solvers as the reference.[10]
The comparison concerns actions, not reaction time or elapsed seconds. As an invented illustration, suppose the human reference is 20 actions. A model using 20 is near that reference on this level; one using 40 eventually succeeds but takes a less direct route. These example numbers are not reported test data.
There are no separate aggregate columns for ordinary people, specialists and expert teams. We have this particular tested public reference, and should not invent scores for other groups.
Image credit: OpenAI — GPT-6 Astra release; official evaluation chart.
OpenAI’s complete chart: Astra 99.9%, GPT-5.6 Sol 7.8%. These are evaluation results under specified setups, not a comparison of two casual chat sessions.
Now bring back the 99.9%
The gap between 99.9% and 7.8% is striking.[2] But ARC-AGI-3 does not simply count correct multiple-choice answers. It considers success and action efficiency relative to humans, with more weight on later levels. The project describes full marks in terms of completing the games at human-level action efficiency.[5][10]
One detail deserves equal attention: changing the software around the same model changes the score.
The Provider Adapter is a test program tailored to the model’s capabilities. “Provider” refers here to OpenAI; the adapter connects Astra to the environment. It is neither a second model nor a different question set.[6]
Suppose you have tried several controls and just discovered a useful pattern. On the next move, you want to continue from that understanding. The standard setup lets Astra preserve written notes; the adapter also carries internal reasoning state between requests and manages long conversations through compaction. Both can preserve useful information. The distinction is what can be carried forward, beyond visible notes.
The best standard result was 62.7%; the best adapter result was 99.9%. They were obtained at different reasoning settings. At max effort, the adapter scored 98.6%, while the headline 99.9% used high effort.[6]
Imagine comparing the same student working with different permitted scratch-work arrangements: the result belongs to the student and the exam conditions together. This analogy concerns continuity of work, not the adapter supplying answers. The published scores establish performance for the evaluated system, rather than for every way someone might access Astra.
Does “96% of levels” mean “96% of people”?
Image credit: ARC Prize — Astra evaluation report; official demonstration or comparison.
ARC Prize’s chart. Each point is a level. Points below the reference line indicate fewer model actions. This comparison uses the adapter at max effort.
The report says Astra used fewer actions than the human reference on 96% of levels.[6] This counts levels, not defeated people. It also compares against successful human solvers’ median action count, not the fastest human in every level.
Keep the configurations together: 99.9% is the high-effort score; the 96% comparison uses max effort, whose overall score was 98.6%. Both statements can be true without describing one identical run.
That is a meaningful reason for interest: in many of these unfamiliar environments, Astra found successful approaches without needing far more trial and error than the human reference.
A strong result, an unfinished question
The test’s designers do not declare AGI achieved. Even unfamiliar puzzles have designed mechanics and goals. Real work often begins with incomplete instructions, contradictory documents and changing priorities. People may first have to agree on what a good outcome would look like.
Imagine assigning an AI a project that has stalled for six months. It must identify missing information, decide when to ask for a decision and recover from mistakes. Some choices have several defensible answers. That is a broader challenge than a game with a definite completion state.
The ARC-AGI-3 paper explicitly states that saturating the benchmark would not prove AGI.[10] Astra’s progress deserves scrutiny and credit. Its score is neither the percentage of all real-world work it can do nor a probability that it is generally intelligent.
Across this series, the useful habit is to ask four questions: what was tested, how was it scored, who was the comparison, and under what conditions did the system work?
Image credit: Prompthon editorial artwork.
Editorial illustration: AI makes progress in tests while the wider world presents more unknowns.
Astra has made substantial progress on learning within these unfamiliar environments. How far that ability extends remains a question for further evidence.
Localized from the Chinese edition published September 8, 2026. Examples paraphrase public tasks and official demonstrations; they are not our own experiments. Figures retain their original evaluation versions and configurations. Source numbers match the Chinese edition.
