🔍 Read the full analysis: FrontierCode: The Review Process For Pull Requests Written By AI on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis published results from 23 models tested on FrontierCode, a benchmark for whether AI-written code changes meet repository and review requirements. Claude Opus 5.5 led with a 52.2% score and passed every blocking check in 57.7% of attempts; the results indicate that human review remains necessary, while costs varied sharply among models.
Artificial Analysis published results on October 12 for 23 AI models tested on FrontierCode, a benchmark that evaluates whether an AI-written code change would meet criteria for acceptance in a real software repository. Claude Opus 5.5 topped the tested models with a 52.2% score, but passed every blocking check in just 57.7% of attempts, underscoring that the benchmark does not support treating AI-generated pull requests as ready to merge without review.
FrontierCode is designed to assess more than whether a patch makes tests pass. Its checks cover hidden tests, regressions, test quality, file scope and diff size, as well as an LLM review against criteria set by maintainers. Any failed blocking check results in a zero for that attempt. Attempts that a trajectory scanner flags for looking up a solution also receive zero, according to the source report.
In Artificial Analysis’ results, GPT-6.1 Sol ranked second at 49.1%, followed closely by Claude Fable 5.1 at 49.0%. Nine models scored from 43.2% to 52.2%. The report says nine of the benchmark’s 100 Main-set tasks were not solved by any model on any attempt. These scores are benchmark results, not evidence that the same models will perform identically across all repositories or real-world pull requests.
The evaluation used the same open-source agent harness, Stirrup, for every model, with a limit of 250 turns per attempt and three attempts per task. Its review stage used GPT-6 Luna (medium) as the judge. The benchmark’s full set contains 150 tasks, while Artificial Analysis ran the 100-task Main set.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
The results focus attention on a distinction that matters to software teams: a patch can pass tests and still be difficult or unsuitable to merge. FrontierCode’s checks include whether tests meaningfully catch regressions, whether the change stays within the requested scope, and whether it meets maintainers’ review criteria. Those are practical concerns for teams deciding how much oversight to retain as coding agents take on more work.
The top model’s 57.7% blocking-check pass rate means it failed at least one required check in the remaining attempts under this benchmark’s setup. That does not establish a universal failure rate for AI coding tools, but it does show that strong benchmark performance is not the same as dependable, unattended pull requests. The scores give developers a reason to evaluate tools on review quality and repository constraints, not only on general coding or test results.
Price is another part of the decision. Artificial Analysis reported per-task costs from $0.09 for GPT-6 Luna to $10.15 for the leading Opus configuration. The report’s cost-frontier examples show a trade-off: Opus scored 3.1 percentage points above GPT-6.1 Sol, while costing about 12.5 times as much per task. Those figures reflect the tested configurations and task setup, rather than a complete estimate of production costs.
Products Worth Considering
EcoFlow DELTA 3 Portable Power Station, 1024Wh, 1800W
Jackery Explorer 1000 v2 Portable Power Station with 200W Solar Panel
Jackery HomePower 3600 Plus Portable Power Station, 3584Wh, 3600W
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Patches
FrontierCode was developed by Cognition and open-source maintainers and announced in June 2026. Version 1.1 followed in July with rules against looking up solutions online, according to the source report. More than 20 maintainers created tasks from 36 established repositories, including Celery, Budibase, uppy and Mattermost. The report says maintainers spent more than 40 hours on each task.
The tasks are private, which is intended to reduce the chance that models encountered their exact solutions in training. FrontierCode applies six evaluation areas: hidden tests for correctness; command checks for regressions and cleanliness; reverse tests to establish that an agent’s own tests fail on the original code; adaptive tests for complex behavior; scope limits on files and diff size; and an LLM review. Blocking checks determine whether an attempt scores at all; quality checks influence scores for attempts that pass them.
Artificial Analysis’ table also shows that rankings can differ from broader intelligence evaluations. For example, the report places Claude Sonnet 5.5 fifth on FrontierCode despite ranking it second among the compared models on the Intelligence Index. It lists DeepSeek V4.1 Flash seventh on FrontierCode. This comparison is specific to the models and scores included in the report, not a claim that one benchmark predicts every coding task.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, quoted in Cognition’s announcement
Limits of the Published Results
The results cover 23 models and the 100-task Main set, not every coding model or the full 150-task benchmark. As of October 13, the source report said there were no FrontierCode results for Gemini 4 Argon, Muse Wonder, Muse Wonder Full or Mistral Large 4 Preview. Their eventual results could change comparisons among the tested systems.
The published scores also depend on the benchmark’s task selection, agent harness, attempt limits and review judge. The source material does not establish how closely these conditions match individual companies’ codebases, review practices or production workloads. Nor does it provide results from human reviewers evaluating the same patches, so the figures should not be read as a direct comparison between AI and human review.
Per-task costs are reported for the tested configurations, but the source material does not spell out every cost component or show how spending would vary with different usage patterns. It is also not clear from the reported results how the models would perform over a larger sample of tasks or under different repository constraints.
Awaiting More Model Results
The next comparison point is whether additional models receive FrontierCode results, particularly the systems missing from the October 13 report. More evaluations could clarify how stable the leading group is and whether models absent from the initial table alter the cost and performance comparisons.
For development teams, the immediate practical step is to treat these results as one benchmark signal, not a merge policy. Teams considering AI-written pull requests can compare tools on their own repositories and retain review checks suited to their code, risk tolerance and testing practices. FrontierCode’s published findings do not announce a change to the benchmark or a deployment recommendation; they provide a basis for further evaluation.
Key Questions
What does FrontierCode measure?
It evaluates whether AI-written code changes meet a set of repository and maintainer-style criteria, including correctness, regressions, test quality, change scope and an LLM review.
Which model ranked first?
Claude Opus 5.5 (max with fallback) led Artificial Analysis’ tested models with a 52.2% FrontierCode score. It passed every blocking check in 57.7% of attempts.
Does a high score mean a model can merge pull requests without review?
No. The benchmark reports model performance under its own tasks and setup. The leading model did not pass every blocking check in 42.3% of attempts, and the results do not establish that AI-written changes are safe to merge without human review.
How much did the tested models cost per task?
Artificial Analysis reported costs ranging from $0.09 to $10.15 per task among the configurations discussed. These are benchmark-specific figures, not a universal estimate of the cost of using the models in production.
Were all major models included?
No. The source report said Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode results as of October 13, 2026.
Source: ThorstenMeyerAI.com
