Cognition Launches FrontierCode to Test Whether AI Code Is Mergeable; Best Model Scores 13.4/100
Cognition introduced FrontierCode, a coding benchmark designed to test whether AI-generated code would actually be accepted into an open-source repository rather than simply pass unit tests. The company said more than 20 open-source developers worked with maintainers from 36 repositories, spending more than 40 hours on each task; FrontierCode includes Extended, Main and Diamond sets with 150, 100 and 50 tasks, respectively, and the top model scored 13.4 out of 100 on Diamond.
Cognition said FrontierCode combines unit tests, rubrics and custom verifiers to assess correctness, test quality, scope discipline, style and adherence to repository standards. The company said its quality-control process, including adversarial testing, calibration, multi-stage review and manual review by a researcher, reduces misclassification errors by 81% versus SWE-Bench Pro and is meant to address a gap in benchmarks that judge code mainly by whether it passes a pre-constructed unit test.
From the sources (11 posts)
@cognitionIntroducing FrontierCode: a coding eval that raises the bar for difficulty & quality. Each task took 40+ hrs of work by leading open-source maintainers. Models write sloppy code that works but isn’t maintainable. Our eval is first to measu
@cognition20+ world-class open-source developers built realistic coding tasks on repos they maintain. They define what “mergeable” means in their repo. What does it take to measure mergeability? We use a mix of unit tests, rubrics and novel verifier
@cognitionFrontierCode was built in close partnership with the expert maintainers of 36 flagship open-source repositories, like @smilingnosrati, CEO & Tech Lead @CeleryOrg (29k stars), and Martin McKeaveney, CTO of @Budibase (28k stars). Maintainers
@cognitionRigorous quality control is important, so we built an extensive QC pipeline with adversarial testing, calibration, and multi-stage review. Every task is manually reviewed by a Cognition researcher. This reduces both the false positive and
@cognitionTasks in the dataset have a concise problem statement with large solutions that cut across multiple files. FrontierCode’s task set is more diverse than other software engineering evals, measuring ability across a wide range of languages an
@cognitionFrontierCode has three task sets: Extended (150 tasks), Main (100 tasks) and Diamond (50 tasks). SOTA LLMs have significant room for improvement, with the top model earning a score of just 13.4/100 on our Diamond task set.
@swyxRT @cognition: Introducing FrontierCode: a coding eval that raises the bar for difficulty & quality. Each task took 40+ hrs of work by lead…
@imjaredzIntroducing the new most important coding eval set "Is this code mergeable?" The main thing evals like SWE-bench miss is that they grade on code correctness, not quality. For FrontierCode we ask if the code maintainer will actually approv
@scaling01Opus 4.8 is the best coding model out there FrontierCode by Cognition is probably the highest quality coding benchmark we have seen so far it moves beyond just using unit-testing for scoring, it also tests for regression safety, mechanica
@calebfahlgrenRT @cognition: Introducing FrontierCode: a coding eval that raises the bar for difficulty & quality. Each task took 40+ hrs of work by lead…
@scottwu46SWE-Bench style grading has been the standard for years now - you ask the agent to solve an issue and then run its code on a pre-constructed unit test. The problem is that passing a unit test is only one part of writing production-ready co