OpenAI published findings on July 29, 2026 showing that GPT-5.6 Sol's ARC-AGI-3 result changed substantially depending on how the evaluation was run. On the public leaderboard, GPT-5.6 Sol scored 7.8%. When OpenAI re-ran the benchmark under the official harness, the score came in at 13.3%. After enabling two settings in its Responses API, the same model scored 38.3%, while using six times fewer output tokens.
The two settings are retained reasoning and compaction. Both are API configuration choices, not model changes.
What the Official Harness Was Getting Wrong
ARC-AGI-3 measures how well AI agents learn and reason through unfamiliar 2D games without explicit instructions. The official harness is intentionally generic, without tools or special features. ARC's stated rationale: a simple harness makes model shortcomings more visible and comparisons more fair.
That design created a specific problem for GPT-5.6 Sol. After each game action, all private reasoning was discarded. The model could see a record of past moves and brief accompanying notes, but not the plans, insights, or thoughts that led to them.
The harness also used a rolling truncation window. When context exceeded 175,000 characters, the oldest messages were dropped. OpenAI's own implementation used a limit of 175,000 tokens instead of characters, but this proved similar in practice because the vast majority of text is action grids, tokenized at a 1:1 ratio. The model was simultaneously losing its reasoning and, eventually, its action history.
OpenAI concluded that much of the model's confusion was not inherent to the model itself, but due to settings in the harness.
What Retained Reasoning and Compaction Actually Do
OpenAI's models are trained to think with private reasoning messages before they output replies or tool calls. Those messages are retained as part of the conversation history. This is how the models are trained, and also how they are deployed in ChatGPT and Codex.
In the Responses API, for GPT-5.6, passing the previous response ID automatically retains reasoning across tool calls and turns. With that setting active, GPT-5.6 Sol spent less time thinking before each action, because it no longer had to reinterpret the game from scratch. It could build on earlier conclusions instead of reconstructing them.
Compaction replaces rolling truncation. If a conversation grows too long, the model summarizes it and continues. OpenAI notes that rolling truncation carries two drawbacks: the model loses earlier observations and actions, and it spends much of the tasks operating with a fuller context window, which can slightly impair performance.
With compaction enabled, GPT-5.6 Sol was better able to preserve what it had learned about each game across longer runs, achieving a higher score with fewer output tokens.
The score difference on one specific puzzle game is stark. On the leaderboard for that game, no frontier model solves any level beyond the first. With retained reasoning and compaction, GPT-5.6 Sol solves all six. ARC-AGI-3 scores measure Relative Human Action Efficiency, comparing model performance to a human baseline. OpenAI estimates the average human tester scored 48% on the same tasks.
What This Means for Agent Deployments in Real Workflows
The mechanism behind this finding can apply beyond benchmark contexts. Any integration that invokes a reasoning model across multiple steps faces the same structural choice: does the agent carry its thinking forward, or does it start fresh each time?
Consider a reasoning model working through order triage, document review, support escalation, or code generation. Discarding reasoning between steps may impose a similar penalty to what OpenAI measured on ARC-AGI-3. That inference is consistent with the benchmark mechanism, but whether it extends to a specific enterprise workflow is not confirmed in OpenAI's analysis.
The token-efficiency finding matters separately. A 6x reduction in output tokens may suggest a potential cost benefit worth modeling, depending on your token pricing and call volumes. OpenAI's analysis does not discuss pricing, so teams should model their own workloads. Better configuration produced both better output and fewer tokens on the ARC-AGI-3 task set.
The announcement does not describe how these settings interact with all model versions or third-party orchestration layers. Teams using middleware, custom harnesses, or non-Responses API integrations should verify whether those paths expose equivalent controls.
Before You Attribute a Poor Result to the Model
If your AI agent is underperforming on a multi-step workflow, check three things before assuming the model is the constraint.
First, confirm whether your integration retains reasoning across turns. If your harness or orchestration layer drops the reasoning trace after each step, the model is re-starting its analysis every time.
Second, check how your integration handles context limits. Rolling truncation discards the oldest messages when context fills. If your setup works the same way, the agent is operating with an incomplete record of its own work.
Third, if you are using OpenAI's Responses API directly with GPT-5.6, verify that you are passing the previous response ID on each call. OpenAI identifies that as the mechanism that retains reasoning across tool calls and turns for GPT-5.6 specifically.
The ARC-AGI-3 result is OpenAI's own benchmark on OpenAI's own model. Independent confirmation of the same effect in enterprise workflows is not in OpenAI's analysis. The underlying mechanism is consistent with the benchmark evidence: an agent that cannot remember its reasoning will re-do that reasoning repeatedly. That is worth testing before the next production deployment review.
