OpenAI Boosts GPT-5.6 Sol ARC-AGI-3 Score to 38.3% on New API Tweaks, Cuts Serving Costs 20%
OpenAI said enabling memory retention and context compaction in its API boosted GPT-5.6 Sol’s score on the ARC-AGI-3 benchmark to 38.3% on the public set, an 188% increase from the standard 13.3% result. The company reported the model used 6x fewer output tokens under the new configuration.
The benchmark’s standard evaluation harness intentionally drops private reasoning and trims older context to ensure fair comparisons across AI providers. Benchmark creator Francois Chollet noted that adopting general-purpose API settings is acceptable as long as the configurations are clearly reported. OpenAI also stated that GPT-5.6 Sol autonomously optimized its own production serving kernels to lower end-to-end costs by 20%.
From the sources (25 posts)
@openaiGPT-5.6 Sol has been used to solve open problems in mathematics. So why was it struggling with ARC-AGI-3, a benchmark of 2D puzzle games? We investigated. The harness was not letting it remember what it had learned. We found that enablin
@thsottiauxTurns out GPT-5.6 Sol is actually SoTA on ARC-AGI-3. Just took two setting changes. You just have to allow it to reason and work over multiple context windows with the help of our canonical compaction implementation.
@sebastienbubeckRT @thsottiaux: Turns out GPT-5.6 Sol is actually SoTA on ARC-AGI-3. Just took two setting changes. You just have to allow it to reason a…
@mobav0RT @thsottiaux: Turns out GPT-5.6 Sol is actually SoTA on ARC-AGI-3. Just took two setting changes. You just have to allow it to reason a…
@samagoblin-level blog post
@thsottiauxTerrific work by @ilanbigio and @sandersted on the investigation and post. Seems like it was a lot of fun. Take care of your harnesses out there, your model will thank you.
@openaiWe implemented the harness with the Responses API and turned on: → Retained reasoning → Context compaction On the public set, GPT-5.6 Sol’s score rose 188% while using 6x fewer output tokens.
@openaiARC-AGI-3 tests how well models can learn unfamiliar 2D games without instructions. The standard harness discarded GPT-5.6 Sol’s reasoning after each move and dropped earlier actions as the context filled up. The model had to keep starting
@johnowhitakerRT @OpenAI: We implemented the harness with the Responses API and turned on: → Retained reasoning → Context compaction On the public set,…
@openaiWe hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. If you’re an API developer trying to maximize
@emollickModel + harness. We have barely begun to understand the best ways to do harness engineering. A huge amount of untapped potential even without models getting better (but models are getting better)
@openaiA benchmark score reflects the model as well as the harness and settings used to run it. For long-running agents, retaining reasoning and compacting context lets the model build on what it has already learned.
@mtsliveSITUATION UPDATE: OpenAI says GPT-5.6 Sol is state of the art on ARC-AGI-3 after two configuration changes, letting the model reason and work across multiple context windows.
@ns123abcThat’s pretty interesting ARC-AGI-3 performance is bottlenecked by context management rather than raw reasoning so letting the model retain its reasoning across context windows tripled the score Not sure if it’s “actually SoTA” since oth
@hesamationthis would actually mean GPT-5.6 Sol is SOTA on ARC-AGI-3, even beating the (currently) top model, Opus 5 (30.2% score) by a margin.
@pvncherRT @charliermarsh: I followed this work internally and found it super interesting. The TL;DR is that if you allow GPT-5.6 Sol to retain it…
@apples_jimmyRT @thsottiaux: Turns out GPT-5.6 Sol is actually SoTA on ARC-AGI-3. Just took two setting changes. You just have to allow it to reason a…
@romainhuetRT @thsottiaux: Turns out GPT-5.6 Sol is actually SoTA on ARC-AGI-3. Just took two setting changes. You just have to allow it to reason a…
@nptacekfor anyone who wasn't aware, there is a capabilities overhang
@boazbaraktcsRT @thsottiaux: Turns out GPT-5.6 Sol is actually SoTA on ARC-AGI-3. Just took two setting changes. You just have to allow it to reason a…
@simonwGPT-5.6 found optimizations that "reduced end-to-end serving costs by 20%" for OpenAI to serve that model Presumably that's billions of dollars a month in savings at this point?
@yacinemtbRT @thsottiaux: Turns out GPT-5.6 Sol is actually SoTA on ARC-AGI-3. Just took two setting changes. You just have to allow it to reason a…
@johncooganRT @OpenAI: After deployment, we applied GPT-5.6 Sol to advance the frontier of efficiency by making itself more efficient to run. The re…
@jon_stokesRT @sandersted: On ARC-AGI-3, GPT-5.6 is dumb as dirt. But it turns out if you turn on two API settings that we use in ChatGPT and Codex,…
@rohanpaul_aiOpenAI says two API settings raised GPT-5.6 Sol’s ARC-AGI-3 public-set score from 13.3% to 38.3%. And this happened because, the ARC-AGI official evaluation kept wiping 5.6 Sol's working memory. i.e. it discarded private reasoning after ev