GPT-6 Astra’s report card, Part 1: What seven benchmarks actually test
Read the tasks before the scores: seven Astra benchmarks, their everyday implications and the limits of their human comparisons.
By Liqing Pan11 min read
Translated from 简体中文
AI
Industry
Benchmarks
Image credit: Prompthon editorial artwork.
On September 3, 2026, OpenAI introduced GPT-6 Astra. At the launch briefing, president Greg Brockman welcomed the arrival of the AGI era, according to Fast Company.[1] The accompanying report card ranged from advanced mathematics to software engineering and professional work.[2]
The standout was 99.9% on ARC-AGI-3, a test of learning in unfamiliar interactive environments. ARC Prize also reported fewer actions than the human reference on 96% of levels, under a different configuration.[3] Those are striking results. But before reading a report card, you need to know what was on the exam.
Read the tasks before the scores. This two-part guide explains what the tests ask, how they score success and what their human comparisons mean. Part 1 covers seven benchmarks; Part 2 examines ARC-AGI-3 and the claim of general intelligence.
1. GPQA Diamond: An internet connection is not enough
In November 2023, David Rein and colleagues introduced science multiple-choice questions designed to resist an obvious shortcut: looking up the answer. Biology, physics and chemistry each came with four options, but answering required specialist judgment.[5]
The researchers recruited both specialists and highly educated people outside the relevant field. The latter could browse freely and spent more than half an hour per question on average, yet answered only about a third correctly on the original evaluation. That is the idea behind “Google-proof”: search is allowed; search alone does not supply the reasoning.
Illustrative question, written for this article—not a GPQA item: An inhibitor competes with a substrate for the same binding site on an enzyme. Temperature and acidity remain unchanged. As more substrate is added, does the maximum reaction rate necessarily fall, or can the reaction still approach its original maximum rate?
The second answer is correct under those assumptions. Picture two groups competing for the same limited set of seats: adding more members of one group changes who gets seated without destroying the seats. The example illustrates reasoning from conditions, not GPQA’s actual difficulty. We have not reproduced its questions; the authors ask people not to repost the dataset online.[4]
Image credit: OpenAI — GPT-6 Astra release; official evaluation chart.
OpenAI’s chart: Astra 96.0%, GPT-5.6 Sol 94.6%. Multiple points represent different evaluation settings; see Section 5 for the cost axis.
Diamond contains 198 selected questions. Table 5 of the original paper reports 81.2% for domain experts and 21.9% for respondents from other fields. Those other respondents were PhDs or PhD students, not a random public sample. Selection favored questions specialists could answer and outsiders struggled with, so these figures do not constitute a fresh, controlled human–AI contest. Two experts reviewing a question separately is also not an expert team taking the test.[4]
Astra’s 96.0% improves on Sol’s 94.6%. Both are strong results. Neither score tells us whether the model can set up and carry out a laboratory experiment, which this test does not examine.
2. FrontierMath: When mathematicians write new exams
As older mathematics tests became easier for AI, Epoch AI commissioned new, unpublished problems. Elliot Glazer and colleagues introduced FrontierMath in November 2024. Some problems require specialists hours or days of work. Epoch’s retrospective describes asking three Fields Medalists and an International Mathematical Olympiad expert to assess difficulty.[7][8]
Models may compute and try different approaches, but must ultimately produce an answer that can be checked.
Public example from Appendix A.5 of the 2024 paper—not Tier 4 v2: Over a finite field with 5¹⁸ elements, how many nonzero solutions satisfy x³y + y³z + z³x = 0, counting solutions that differ only by a common nonzero scale factor as the same?[17]
A finite field is a finite collection of values with precisely defined arithmetic. The task is to count within those rules while avoiding duplicates. Think of counting distinct designs when resizing an entire design does not make a new one: finding candidates is only half the job; deciding which are equivalent matters too. That analogy explains the counting issue, not the field arithmetic. This early example was classified as relatively easy and is not evidence about which items produced Astra’s Tier 4 score.
Image credit: OpenAI — GPT-6 Astra release; official evaluation chart.
OpenAI: FrontierMath Tier 4 v2, Astra 97.6%, Sol 83.0%. These are scores on a defined problem set.
The jump from 83.0% to 97.6% is substantial. Keep the version label: Epoch corrected some problems in June 2026, so results on different versions are not automatically comparable.[6]
Mathematicians created and checked the tasks, but the documentation reviewed for the Chinese edition did not provide a comparable aggregate score for the public, experts or expert teams. “Experts may need days” is a difficulty reference, not a timed leaderboard. And 97.6% does not mean Astra has solved 97.6% of humanity’s unsolved mathematics.
3. DeepSWE: Can it fix the code without having seen the fix?
Public software repositories are useful sources of engineering tasks. They also expose issue discussions and completed patches. If training includes those materials, a high score can partly reflect prior exposure.
Wenqi Huang and colleagues introduced DeepSWE in July 2026 to address that problem. They designed 113 engineering tasks in real projects with separate checking programs. A working solution need not copy a reference patch line for line.[9][10]
Public task paraphrase: Fix cleanup in Happy DOM, a simulated browser environment. When a page closes or navigates away, unfinished requests and responses should stop appropriately, already-read content should remain usable, and timers that should no longer run must be cleared.[9]
Closing a tab looks like one action; inside the software, several jobs may still be running. It resembles closing a shop: cancel pending deliveries when appropriate, keep completed receipts, and switch off equipment without destroying completed work. The model must trace those responsibilities through real code and avoid breaking unrelated behavior.
Image credit: OpenAI — GPT-6 Astra release; official evaluation chart.
OpenAI: DeepSWE v1.1, Astra 74.1%, Sol 72.7%.
The improvement here is smaller than in mathematics. Engineers supplied reference implementations and tests, not a comparable timed human score; the paper also did not estimate actual human completion time. This measures useful engineering work, but it cannot tell us what proportion of programmers Astra outperforms.[9]
4. Terminal-Bench Science: Will it finish the research work?
A scientist may appreciate a clear explanation, but still have data to process and a broken simulation to repair. Terminal-Bench Science, led by Stanford researchers working with the Terminal-Bench team and scientists, draws tasks from those practical problems.
Its initial selection admitted 70 tasks from 920 proposals. Difficulty alone was insufficient: a task had to matter to research and have an outcome that could be checked.[11]
Example from the current public repository: Repair a program that transfers atmospheric or ocean data between spherical grids, respecting masked regions and the required conservation and range constraints.[18]
Imagine two maps dividing the same stretch of ocean into differently shaped cells. Transferring measurements is more than making the second map look smooth: the conversion must not invent or lose the quantity being tracked. Missing regions and curved boundaries make simple averaging unreliable. This example illustrates the workflow; we do not identify it as a scored item in version 0.1.
Image credit: OpenAI — GPT-6 Astra release; official evaluation chart.
OpenAI: Terminal-Bench Science 0.1, Astra 64.6%, Sol 22.4%.
The change is large: more of these specified research jobs were completed. These figures come from OpenAI’s evaluation, not our rerun. Scientists helped define and check the tasks, but the launch report did not supply a comparable cross-domain human examination score. Completing assigned research work also leaves open the question of choosing valuable new research problems independently.
5. Agents’ Last Exam: The software is open—now do the job
Introduced in June 2026 by Yiyou Sun and colleagues, with Berkeley RDI and the RDI Foundation leading the project, Agents’ Last Exam asks models to work inside professional software.[12][13][14]
Task-design example from the paper: Composite a running cheetah from one video into another competition video.[12]
Anyone who has tried replacing a video background can see the problem. A still cutout will not do: the animal moves, its pose changes and the camera moves too. The subject must be separated, motion must remain coherent and the result must fit its new surroundings. This illustrates the intended task scale, not a specific scored run shown on the release page.
Image credit: OpenAI — GPT-6 Astra release; official evaluation chart.
OpenAI: Agents’ Last Exam, Astra 59.3%, Sol 53.6%. Read both the score and the cost.
Across these official plots, the horizontal axis shows API cost and the vertical axis shows score. Several points for one model indicate evaluated settings, not several different models. OpenAI says its summary table takes the best score across reasoning efforts.[2] The release text does not explain why some other models have only one plotted point; that is not evidence that they offer only one setting.
Professional reference outputs and expert review committees help establish standards. They do not constitute an expert team competing against the model. The reviewed sources did not supply comparable aggregate human scores. And 59.3% does not mean 59.3% of professions can be handed to AI: it is a result on these tasks, judged through the work produced.
6. AutomationBench: The right email to the wrong person still fails
Updating a customer record, changing a calendar entry and notifying a colleague sound routine. They are also full of ways to be almost right. A correct amount attached to the wrong customer can be worse than leaving the record alone.
Zapier’s AutomationBench places models in simulated business tools and checks the resulting state. Every required check must pass for a task to count as fully completed.[15]
Simplified public example: Two meetings overlap in Zoom and Google Calendar. Consult a spreadsheet’s priority rules, add [RESCHEDULED] to the appropriate meeting’s title, and explain the result in the specified Slack channel with both meeting IDs.[19]
The trap is deciding which meeting to change. An external partner’s email claims higher priority, but cannot override the company’s rules. Similarly named meetings can refer to different records. In this example, the rescheduling requirement is a title marker—not an additional request to find a new time. The model must obey the actual task rather than expand it.
Image credit: OpenAI — GPT-6 Astra release; official evaluation chart.
OpenAI: AutomationBench, Astra 41.4%, Sol 18.1%.
This is a marked improvement, alongside many tasks still not fully completed. The release evaluates a harder private set, not the public practice leaderboard. No comparable aggregate score for office workers or expert teams was found in the sources reviewed for the Chinese edition. Familiar work should not tempt us to invent a human perfect score.
7. BenchCAD: It looks right, but will the bolt fit?
A mechanical part can look convincing and still fail because a hole is offset or a wall is too thick. BenchCAD asks models to reconstruct parts from multiple views using code, with industrial examples including material drawn from real standards.[16]
Public editing example: Create a cylindrical through-hole with a 20 mm radius in a motor end cap, passing through both the lower disk and the raised upper section. This is BenchCAD-Edit case 167, used here to illustrate a spatial error—not the reconstruction item behind the headline score.[20]
The paper shows failures where a hole is cut first and solid geometry added back above it. From one angle, the part appears drilled; inside, the passage is blocked. Anyone trying to pass a bolt through it would discover the difference immediately.
Image credit: OpenAI — GPT-6 Astra release; official evaluation chart.
OpenAI: BenchCAD with a Python tool, Astra 95.9%, Sol 83.3%. The metric measures shape overlap, not factory readiness.
Geometry is a meaningful achievement. Material strength, manufacturability and assembly still need separate checks. A 95.9% score is not a claim that 95.9% of generated parts can go straight into production.
The report card in context
Image credit: Prompthon editorial artwork.
Prompthon editorial overview. Percentages use different scoring rules and cannot be added. Human references come from separate studies, not one common competition; ARC configurations are explained in Part 2.
Across seven tests, progress takes different forms: answering specialist questions, repairing code, processing research data and producing professional artifacts. The gains are uneven, which is more informative than a blanket claim of superiority.
The remaining score, 99.9%, asks a different question: can a system learn how to act when the environment is unfamiliar and nobody hands it a manual? Part 2 follows two public puzzle examples, then examines the human reference and the supporting software behind the result.
Adapted from the Chinese edition published September 8, 2026. Figures describe specific versions and configurations, not a continuously updated leaderboard. Public examples and official demonstrations are not Prompthon experiments. Source numbers retain the Chinese edition’s references.
