Should a decision model pick your model?

Say a tool could pick the model and effort level for every coding task you start. You type the task, and it tells you which model to use. Would you let it?

That's the question of this part of the series. In part 4, "Is the cheaper model enough?", I showed how my tool spatz measures fairly whether a cheaper model is good enough. Here I step back and look at what you gain and what you give up when a decision model makes the choice, and what you could use instead.

What you would hand off

spatz (opens in a new tab) is a proof of concept that recommends a pair of model and effort level for a task and learns which pairs succeed. To sort the task, it asks Jev, a decision model from TypeSafe AI: a model that answers fixed questions instead of writing text. Part 2, "What is a decision model?", explains that idea.

spatz never runs the task, and its command line tool never switches the model. You or your agent make the switch. In Claude Code, hooks (small scripts that Claude Code starts at fixed moments) can't set the model or the effort either. A hook can watch and add a hint, nothing more. So with the command line tool and hooks, handing off the choice means this: you follow a recommendation instead of deciding on your own. Part 6 shows a mod (a plugin that runs inside Claude Code) that can make the switch itself, and why I went back to hooks for now.

Think of a navigation app

The trade-off is easiest to see with a tool you already know. Think of a navigation app. It picks the route, so you don't have to think about it. But it needs to know where you're going. If it picks a bad road, you notice only when you're stuck in it. And if you always follow it, you never really learn the streets yourself.

The route is the pair of model and effort, and where you're going is your task text.

The analogy stops fitting at the start. A navigation app knows the streets on your first trip. spatz has few or no results of its own at first, and until it has enough, it leans on Jev or on caution.

Reasons to hand it off

• Less to weigh. I could run every task on the strongest model at high effort. I don't want to, because that's like shooting sparrows with a cannon, which is where the name spatz (German for sparrow) comes from. My hope is that a recommendation saves me from weighing the choice myself when I'm unsure. That's my view, not a measured result.
• It can save money. The goal of the proof of concept is this: after a learning phase, 30 % of the recommendations that were followed are cheaper than the strongest model, and the success rate stays as high as in a control group that got the strongest pair at random. That's a goal, not a result.
• It learns from your own tasks. spatz keeps track of which pairs succeed on which kind of task. It learns that from the test and build runs it sees, and from a short report at the end of a task.
• It doesn't slow you down. Jev answers the four questions of spatz in one request. Over 5 calls, the median was 235 ms and the slowest call took 255 ms. One full spatz call took 406 ms, including the program start and the first download of the model price list. That was a single run.

Reasons not to

• Your task text goes to a third party. To sort the task, spatz sends its text to TypeSafe. Before that, it checks the text for secrets like API keys, and it doesn't send the text when it finds one. But the check knows only certain patterns, so it can't find every secret. A project can also turn Jev off. Then spatz sorts every task with simple keyword rules instead.
• It needs time to learn. When spatz has too few results for a kind of task, even after it counts harder tasks of the same type, the best guess of Jev decides. Without Jev, spatz picks the most expensive pair. This phase is called the cold start (the time when a system doesn't have enough data of its own yet).
• Learning has a price. For tasks that aren't critical, there's a 10 % chance that spatz tries a cheaper pair on purpose, when a cheaper one is available, and that try can fail. Another 10 % chance puts the task in the control group, which gets the strongest pair even where a cheaper one might have been fine.
• It relies on a heuristic. A heuristic is a practical rule that can be wrong. spatz has to know which task a test run belongs to, and it decides that by time: which recommendation was open when the run happened. Nobody has measured yet how often that guess is right. When it's wrong, a result counts for the wrong task.
• It can misjudge critical tasks. Some tasks are critical: security, money and legal rules, or stored data. For those, spatz starts with the most expensive pair, needs more evidence before it picks a cheaper one, and never experiments. That's a safety floor. But the floor helps only when the task is recognized as critical, and Jev's answer depends on how the options are described. In a spike (an experiment meant to be thrown away), a plain bug fix came back as critical with probability 0.99. With other descriptions, it came back as not critical with 0.97, in a single run. The descriptions differed in wording and in language, so the two runs don't show which of the two changes did it. A task rated too critical only costs money. A critical task rated as not critical loses the floor.
• A switch has a price too. When the model changes, the prompt cache is lost (the part of a request that the API can reuse from the request before). Part 6 looks at that.
• You learn less yourself. This one is my own view. When a tool picks every time, I get fewer chances to build my own sense of which model suits which task.

Three alternatives

spatz isn't the only way to handle the choice. Here are three others, side by side:

• A rules table. A fixed table maps each kind of task to a model. It sends no text anywhere and needs no learning phase. But it doesn't learn either. It stays what you wrote until you change it.
• The agent chooses. If your agent can start work with a model it picks itself, you can let it make the choice. How good those picks are, I haven't measured.
• A router. A router is a service that picks the model and effort and then runs the request itself. Jev Router on OpenRouter works this way. You hand off the choice and the work in one place, and you don't pick at all.

The spatz command line tool only recommends. Its Claude Code mod only shows the recommendation by default, so the last word stays with you or your agent. In its apply mode, the mod sets the model and effort itself. Either way, the run stays with you or your agent.

Where this stands

The learning phase of spatz hasn't started yet. There's no data on savings or success rates, so whether handing off the choice pays off is still pending.

Next in this series: Where should a decision model decide?