OpenAI Raises GPT-5.6 Sol ARC-AGI-3 Benchmark Score 188% on New API Settings
OpenAI said enabling retained reasoning and context compaction in the Responses API raised GPT-5.6 Sol’s score on the ARC-AGI-3 benchmark by 188%, while cutting output tokens by six times. The company noted the standard testing harness discarded the model’s reasoning after each game move and dropped earlier actions as context filled, forcing the model to restart.
Benchmarks measure the model as much as the harness and settings used to run it, the company said. Developers testing frontier models should deploy these Responses API configurations to maximize performance, rather than relying on legacy chat and completions endpoints.
From the sources (15 posts)
@openaiGPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games? We investigated. The harness was not letting it remember what it had learned. We found that enablin
@thsottiauxTurns out GPT-5.6 Sol is actually SoTA on ARC-AGI-3. Just took two setting changes. You just have to allow it to reason and work over multiple context windows with the help of our canonical compaction implementation.
@sebastienbubeckRT @thsottiaux: Turns out GPT-5.6 Sol is actually SoTA on ARC-AGI-3. Just took two setting changes. You just have to allow it to reason a…
@mobav0RT @thsottiaux: Turns out GPT-5.6 Sol is actually SoTA on ARC-AGI-3. Just took two setting changes. You just have to allow it to reason a…
@samagoblin-level blog post
@thsottiauxTerrific work by @ilanbigio and @sandersted on the investigation and post. Seems like it was a lot of fun. Take care of your harnesses out there, your model will thank you.
@openaiWe implemented the harness with the Responses API and turned on: → Retained reasoning → Context compaction On the public set, GPT-5.6 Sol’s score rose 188% while using 6x fewer output tokens.
@openaiARC-AGI-3 tests how well models can learn unfamiliar 2D games without instructions. The standard harness discarded GPT-5.6 Sol’s reasoning after each move and dropped earlier actions as the context filled up. The model had to keep starting
@johnowhitakerRT @OpenAI: We implemented the harness with the Responses API and turned on: → Retained reasoning → Context compaction On the public set,…
@openaiWe hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. If you’re an API developer trying to maximize
@emollickModel + harness. We have barely begun to understand the best ways to do harness engineering. A huge amount of untapped potential even without models getting better (but models are getting better)
@openaiA benchmark score reflects the model as well as the harness and settings used to run it. For long-running agents, retaining reasoning and compacting context lets the model build on what it has already learned.
@mtsliveSITUATION UPDATE: OpenAI says GPT-5.6 Sol is state of the art on ARC-AGI-3 after two configuration changes, letting the model reason and work across multiple context windows.
@ns123abcThat’s pretty interesting ARC-AGI-3 performance is bottlenecked by context management rather than raw reasoning so letting the model retain its reasoning across context windows tripled the score Not sure if it’s “actually SoTA” since oth
@hesamationthis would actually mean GPT-5.6 Sol is SOTA on ARC-AGI-3, even beating the (currently) top model, Opus 5 (30.2% score) by a margin.