Testing the food scanner with recorded scans, for free

A wrong scan is the most expensive bug in a food app. Someone who weighed their dinner and watched us miss it stops trusting the numbers. So we wanted a way to know that a change doesn't make an already-fixed case wrong again, without paying for an AI call every time we check.

How does it work?

Each time a scan runs, the AI models answer a few questions: what's on the plate, what the label says, what the barcode reads. We keep those answers, along with what the meal or product really was. To test a change, we feed the stored answers back through the new code and see what it would have picked.

The question is narrow: given what the models already said, does the new code choose better? It costs nothing, takes about a minute, gives the same result every time, and runs on every change before it's merged.

What does it measure?

Everything that happens after the models have answered:

• how we look things up in the food database;
• how the results are ranked;
• how several readings of one item are combined into one answer, and how we judge whether it's plausible;
• when a scan should ask the slower model for a second look;
• how duplicates are merged, and making sure a later reading never replaces a better earlier one.

Today the set holds 640 cases from real scan sessions where we know the truth.

What can't it see?

• A change to the AI model or its instructions. Those change the stored answers themselves, so they need the slower test that calls the models for real.
• Your own history, like products you log often. The replay behaves like a brand new account.
• Second looks recorded under older instructions.
• Steps that failed when the scan originally ran.

We write these limits next to the scores, because a passing replay tells you less than it seems to if you forget them.

Why does free matter?

A check that costs money gets run less often. Because this one is free and fast, it runs on every change to the scanner, and a regression shows up on the pull request with the exact case that changed, not in a user report a week later.