Is the cheaper model enough?

Say a tool recommends the strong model for your bug fixes, and every fix works. Was the strong model needed? Or would the cheaper one have done the job just as well?

You can't tell from those results alone. This note shows why, and how my tool spatz (a small command line tool that recommends a model and effort level for a coding task) is built to find out fairly. The short version: it has to try the cheaper model on purpose, and it has to compare against a group that was picked at random.

In part 3, "Building a model recommender", I showed the parts of spatz and the data it learns from. This part is about measuring whether its cheaper picks are good enough.

Learning from past outcomes

spatz sorts each task into a cell, for example "bug fix, medium difficulty". For each model and effort pair it keeps an estimate of how often that pair succeeds in that cell, based on past outcomes. When a cell has enough outcomes, spatz picks the cheapest pair that reaches its success bar. When it has too few, spatz also looks at results from harder tasks of the same type. Only when that is not enough either does it fall back to a first guess, and without that guess it picks the most expensive pair.

That fallback is safe. It is also a trap.

Giving untried models a chance

If spatz always recommended the pair that looks best right now, the cheaper pair would never get a chance. It has no results, so it never looks best. It never gets picked, so it never gets results.

Think of a lunch place you have never tried. You always go to the one you know, because you know it is good. The new place might be just as good and half the price, but you will never find out if you never go there once.

So for tasks that are not critical, each recommendation has a 10 % chance that spatz tries a cheaper pair on purpose. That only happens when a pair cheaper than the normal choice exists. This is called exploration (trying an option you are unsure about, to learn how good it is). The other picks follow what spatz has learned.

Here the analogy stops fitting. A bad lunch costs you one meal. A bad model choice can cost a failed attempt and a round of fixes. That's why spatz never explores on critical tasks, like security, money and legal rules, or stored data. There it starts with the most expensive pair and needs more evidence before it moves to a cheaper one.

The unfair comparison

Now say spatz has run for a while and has results. The cheaper pair succeeded on many tasks. The strong pair succeeded on many tasks too. Can you put those two success rates side by side?

Not fairly. Before spatz, you picked the strong model yourself, and you picked it for the tasks that looked hard. So "the tasks where I used the strongest model anyway" are mostly hard tasks. The cheaper model got the easier ones.

Here is a worked example. The numbers are made up to show the effect, they are not measurements:

The cheaper model looks better. But it had easier work. The table can't tell you whether it would have handled the hard tasks too, or whether the strong model would have scored 100 % on the easy ones. The two groups differ in more than the model.

A random control group

The fix comes from A/B testing (showing two versions to randomly chosen groups and comparing the results). If chance decides who gets which version, the groups are alike on average. With few results, chance can still make one group look better, so you need enough results before you trust a difference.

spatz does the same. For tasks that are not critical, each recommendation has another 10 % chance to get the strongest pair, chosen at random. That's the control group (the group that gets the usual, safe option so you have something to compare against).

Because chance picks the control tasks, they are a mix of easy and hard, like all other tasks. And spatz compares only within the same cell. So it puts the success rate of its learned picks next to the success rate of the control group, for the same kind of task.

If the two rates are about the same, the cheaper picks are good enough. If the learned picks are clearly lower, the cheaper picks lose quality, and spatz should not save money there.

With few results per cell, both rates jump around a lot. A handful of tasks can't show a small difference.

What the extra tries cost

Both groups cost something. Exploration risks a failed attempt on the cheaper pair. The control group spends the expensive pair on tasks where a cheaper one might have been fine.

That's the price of knowing. Without exploration, spatz never learns that a cheaper pair works. Without the control group, it can't tell whether its cheaper picks are as good, or only look good because they got easier tasks.

What counts as success

The proof of concept has one goal. After the learning phase, 30 % of the recommendations that were actually followed should be cheaper than the strongest model, and the success rate should stay as high as the control group's.

This is a goal, not a result. The learning phase has not started yet, so there is no data on savings or success rates. The result is pending.

Next in this series: Should a decision model pick your model?