Building a model recommender

What does it take to build a tool that tells your coding agent which model to use for a task? Part 1 of this series, "Who picks the model?", asked who should make that choice. Part 2, "What is a decision model?", showed a decision model, a model that answers fixed questions about a task. This part puts the pieces together.

The tool is spatz, my proof of concept. It's a command line tool. You give it a task, and it recommends a pair of model and effort level. It also learns which pairs succeed. It never runs the task, and the command line tool never switches the model. The person or the agent makes the final pick, or a Claude Code mod applies it, which part 6 covers.

Four steps and a loop

Think of a foreman in a workshop who hands out jobs. A job comes in. The foreman reads it, looks at who is in today, checks the logbook for how similar jobs went, and names the person who should do it. After the job, the inspection and the customer say whether it worked, and that goes into the logbook.

A model recommender does the same:

1. Understand the task.
2. Take the models you can use.
3. Look at past results for similar tasks.
4. Rank the models and hand over the best one.

Then a feedback loop writes the result of each task back, so step 3 knows more next time. The next sections go through each step, and then the loop.

Understand the task

The recommender first needs to know what kind of task this is. spatz asks the decision model from part 2, Jev, four questions in one request. What type of task is it, for example a bug fix, a new feature or a review? How hard is it? Is it critical? And which of the candidate models fits best?

Critical means the task touches security, money and legal rules, or stored data. Those tasks get special care, more on that below.

This step sends the task text to a third party, TypeSafe, the company behind Jev. So spatz first checks the text for secrets, like an API key or a password. When it finds one, it does not send the text and uses simple keyword rules instead. A project can also turn Jev off completely.

Take the models you can use

The recommender does not know which models you have access to. You can tell it: pass a list of models and effort levels, or set one in a config file. Without a list, spatz uses a preset for the coding agent it runs in, Claude Code or Codex. The presets come from a model catalog that is updated daily, and they list what the agent offers, not what your account may use. Either way, spatz only ranks these candidates (the pairs it may pick from) and never suggests a model outside them.

spatz sorts the candidates from cheap to expensive, using a public price list. The order matters later, because the goal is the cheapest pair that is good enough, not the cheapest pair overall.

Look at past results

Here the foreman opens the logbook. spatz sorts each task into a box made of two things: the task type and the difficulty. "Easy bug fix" is one box, "hard refactoring" is another. spatz calls this box a cell.

For each pair of model and effort, spatz keeps a success estimate per cell, built from the outcomes of earlier tasks in that cell. An easy bug fix tells it something about easy bug fixes, and little about hard design work.

There is one useful shortcut. When a cell has too few results, spatz also looks at harder tasks of the same type. If a model did well on hard bug fixes, that is evidence it can handle easy ones. The reverse is not true, so easy results never count for hard tasks.

Rank and hand it over

Now spatz picks, and how it picks depends on how much history it has.

When the cell has enough results, spatz takes the cheapest pair that reaches its bar for success. If no pair reaches the bar, it takes the pair with the best estimate so far.

When the cell has too few results, spatz does the same with the results from harder tasks of the same type, the shortcut from above.

When that is not enough either, Jev's answer to "which model fits best" decides. This is the cold start (the phase when a learning system has too little data of its own). So spatz never needs a full logbook to start. With little history, it leans on Jev.

Without Jev, spatz still checks the history first. Only when that is too thin does a simple rule pick the most expensive pair, the cautious choice.

For tasks that are not critical, spatz also tries a cheaper pair on purpose now and then, so it can learn. Part 4 explains why that is needed.

Critical tasks follow a stricter rule. They start with the most expensive candidate pair. A cheaper pair needs more evidence there than anywhere else before it can win, and spatz does not experiment on critical tasks at all.

The answer is a short ranking: the recommended pair, and up to two next pairs in cost order.

Say you ask spatz for help with "fix the off-by-one error in the pagination helper" (a made-up example). Say Jev calls it an easy bug fix that is not critical. spatz opens the cell for easy bug fixes. If pairs already have enough good results there, it recommends the cheapest of them. If neither this cell nor the harder bug fixes have enough results, Jev's pick decides.

Why it can only recommend

You might expect the tool to just switch the model for you. Through hooks, it can't. spatz connects to Claude Code through hooks (small commands that Claude Code runs on certain events, for example after a tool has run). A hook can watch what happens and add a hint. It cannot set the model or the effort. This was checked in a spike (an experiment meant to be thrown away).

So through hooks, spatz is a recommender, not a router (a service that picks the model and also runs the request for you). The agent reads the recommendation and can start a subagent (a helper that Claude Code starts for one task) with that model and effort. Or the person switches the model by hand. Either way, someone else makes the switch.

A mod is different. It's a plugin that runs inside Claude Code, and it can set the model and effort when a subagent starts or right before a model request. spatz now has such a mod. Part 6 shows it.

This is also where the foreman analogy stops fitting. A foreman can tell a worker to take the job. Through hooks, spatz can only suggest.

Where the results come from

The feedback loop needs to know whether a task went well. spatz gets that from two places.

The first is the hooks. When the agent runs the tests or the build, a hook sees whether they passed or failed. spatz recognizes these runs by known command names. It misses test tools that are not on its list, and it skips commands whose result is unclear, for example when the output goes through a pipe into another command.

The second is an explicit report. At the end of the task, the person or the agent tells spatz whether it worked, partly worked or failed. A report wins over everything the hooks saw.

The hooks never block the session. If something goes wrong inside them, they ignore the error, and the agent keeps working.

Linking a test result to its task

Here is the weak spot. A hook sees a test run, but it does not know which task the test was for. spatz only knows when each recommendation was made.

So spatz uses time windows. A recommendation stays open from the moment it is made until the first of three things: a report for it, the next recommendation in the same window, or a long quiet period with no activity there. The main session has one window. A subagent with its own recommendation gets its own window, so its results don't mix with the main session's. Test and build results that spatz recognizes in a window count for the open recommendation there.

That is a heuristic (a practical rule that can assign a result to the wrong task). Nobody has measured how often it gets it right. Say you ask spatz once and then do two tasks in the same session without asking again. The tests of the second task then count for the first one.

The model that did the work is a guess too. When there is no report, spatz takes the model that wrote the most output tokens (the pieces of text a model writes) in that time window. A report sets the model directly, which is one more reason the report matters.

The foreman sees the work directly. spatz sees only what the hooks catch, and has to guess which job it belongs to.

The learning phase has not started yet. So there is no data yet on how often the recommendations are right, or how much they save.

Next in this series: Is the cheaper model enough?

Update (2026-10-07)

The mod above exists, but I've turned it off for now, so spatz is a recommender again. Its hooks got a new job instead: when a helper or a session ends, they close every suggestion of that run that has no outcome yet, in Claude Code and in Codex. Without a reported result, the outcome is stored as unknown, which counts as neither a pass nor a fail (spatz pull request #101 (opens in a new tab), merged on 2026-10-07). Part 6 explains why I went back to hooks.