Cognition releases FrontierCode 2.0, a coding benchmark where no model passes 50%
Cognition’s FrontierCode 2.0 grades AI coding agents on whether a maintainer would merge their change. The best model, Claude Opus 5.5, scores 45.5 out of 100.
By ModelCompare ·
Cognition, which makes the Devin coding agent, released FrontierCode 2.0 on 27 October. Its 132 tasks come from open-source projects, and more than 20 of their maintainers spent over 40 hours on each one. Each task has hidden tests plus maintainer-written criteria, such as fixing every copy of a bug, not breaking a public API, staying inside the files the task needs and writing tests that catch bugs. An attempt that fails any must-pass criterion scores zero.
Claude Opus 5.5 leads with 45.5 out of 100, and 50.3% of its attempts fail at least one must-pass check, so by Cognition’s grading they wouldn’t be merged. On FrontierCode 1.1 it scored 54.6.
Top scores on four coding benchmarksEach dot is a model we list, scored out of 100. The best score on each is named, and the shaded band is the room above it.
Sources: Cognition (FrontierCode, each model at its best-scoring effort), Cursor (CursorBench) and Artificial Analysis (Terminal-Bench), as of 27 October.
On the other coding benchmarks we track, top scores are higher. On Terminal-Bench 2.1, Claude Fable 5.1 scores 91.4 and 6 models score above 83. On CursorBench 4.0, Claude Opus 5.5 leads with 57.8. FrontierCode’s top score is the lowest of any coding benchmark you can rank models by on this site.
Cognition says FrontierCode separates models more clearly. On the 8 models both benchmarks score, the spread is 31.3 points against CursorBench’s 21.0.
Cost and effort
Claude Opus 5.5 spends $12.82 per attempt. GPT-6.1 Sol scores 39.0 for $1.03 per attempt: 86% of Opus’s score at 8% of its cost. Cognition singles it out for performance relative to cost.
Cognition also notes that more effort can lower a score, because models write more code and touch more files. Claude Fable 5.1 scores 32.4 at extra-high effort and 31.7 at max, which costs $11.46 per attempt instead of $8.68.
FrontierCode scores, pass rates and costs are Cognition’s, from the FrontierCode leaderboard data as of 27 October, for the models in our catalog. The score is a weighted total of the review criteria, with zero for any attempt that fails a must-pass one. The pass rate is the share of attempts that clear every must-pass criterion. Cost is the mean spend per attempt at list API prices. Each model runs in an agent harness (Claude Code, Codex and others), at the effort where it scores best, as Cognition’s leaderboard shows it. Cognition makes coding models of its own; none of them is in our catalog.
CursorBench 4.0 scores are Cursor’s and Terminal-Bench scores Artificial Analysis’s, each at the highest effort they list. Terminal-Bench 2.1 is the earlier version AA still reports for some models. The 1.1 score, the task counts and the maintainers’ hours are from Cognition’s launch announcement.