Does the strongest model pay off?

You can give a coding agent a model and an effort level for every task. Is the strongest pair worth its price? And does "more effort" help every model the same way?

spatz (opens in a new tab) recommends a pair of model and effort for a coding task, and I introduced it in Who picks the model?. A recommender needs a ground truth: how often does each pair really solve a task? So I collected 1,961 recorded runs of coding agents on a fixed set of tasks, in one morning, and counted the passes.

This series is about that measurement. This part shows what the passes looked like. Part 2 covers the easy tasks, part 3 the mistakes in my setup, and part 4 compares a simulation with the real data.

How the runs worked

A headless run means nobody sits in front of the agent: it gets one prompt, works alone in a fresh folder, and then hidden tests decide if the result passes. The hidden tests are the same for every run, so the pass rate is comparable.

I used 40 tasks in two sets: 24 easy ones and 16 harder ones (I call the harder set bench2). Each task ran several times for each model and effort level. In total that was 1,961 runs, with Claude models and with Codex models.

Everything below needs this warning. There are only 40 tasks, and a task that runs four times is still one task. The runs are not independent samples, and everything ran on one morning. Read the numbers as a first look, not as a ranking. A larger set with more kinds of tasks is in progress. It marks tasks I invented as synthetic, and I'm replacing them with real tasks from my own repositories.

The strongest pair is not the cheapest pass

On the 16 harder tasks, each Claude pair ran 64 times. The pass rates for Sonnet 5.5 and Opus 5.5 (cost per pass is the total spend divided by the number of passes):

Three things stand out:

• Sonnet 5.5 at high effort passed as often as Opus 5.5 at medium effort (59 of 64 runs each). It cost $0.174 per pass instead of $0.294, which is about 60% of the price.
• Opus 5.5 at high effort passed the most of the six Claude pairs. Compared with Sonnet 5.5 at medium, it gained about 10 points (94% against 84%) and cost about 2.5 times as much per pass.
• Compared with Sonnet 5.5 at high, Opus 5.5 at high gained 2 points for nearly twice the price per pass.

The 95% Wilson interval (a standard range for a pass rate) for Sonnet 5.5 at high effort is 83% to 97%. It treats every run as independent, so it doesn't capture the uncertainty from testing only 16 tasks. With 64 runs per pair, the tests did not clearly detect differences among several pairs.

Sonnet 5.5 at medium effort had the lowest cost per pass, and Opus 5.5 at high effort had the highest observed pass rate. Which one is better depends on what a failed run costs you.

Effort does not work the same everywhere

You might expect that more effort always means more passes. Here are all four models I measured, and they disagree.

On Sonnet 5.5, each step up gained about 10 points. On Opus 5.5, it rose from 84% to 94%, with most of the gain from low to medium. On gpt-6-luna, effort mattered even more: high passed twice as often as low. On gpt-6.1-sol, the rate stayed between 93% and 95% at every level. I saw no clear gain from extra effort here, while the median run grew from 56 seconds at low effort to 101 seconds at high.

The Codex runs are 36 to 44 per cell, and they had no cost data, so I compare only pass rates there.

So "turn effort up when in doubt" helped three of these models and showed no clear gain on the fourth. That suggests testing effort separately for each model.

Next in this series: Why my easy tasks barely separated Claude pass rates