Our AI kept making the same mistake. This trick made it stop for good

Each time you correct a size, era, colour or model the AI got wrong, the fix is saved as a test, and the AI's instructions can't change unless they still pass every saved test. The same mistake stops coming back.

Paul from Beartooth Bargains · Oct 14, 2026

What happened in our store

The same mistake stops coming back; 47 of 49 vintage cases pass against 22 before

Time to set up: A day
Cost: Free beyond model usage when the tests run
Tools: Claude Code
Sells on: eBay

The problem

You correct the AI, it does the same thing next week. Prompt tweaks fix one case and break another.

The workflow

Every edit the seller makes on the review page is logged: field, old value, new value, and the photos.
Each becomes a test case: given these photos and facts, the answer must be the corrected value.
Before any change to the rules or prompt, the agent replays the whole case file and reports how many pass. A change that passes fewer than before does not ship.
Cases only get stronger. A later correction on the same item overrides the earlier one, but a case is never quietly dropped because it is inconvenient.
Test replays cost model usage, so they run in small batches and the seller is told before a big one.

What happened

Vintage calling went from 22 cases passing to 47 of 49 after the rules were rewritten against the case file, and the two failures are known and documented.

For your store

Our honest estimate from running this. Your store and numbers will differ.

What you need: An AI that already drafts your listings, and a spreadsheet to log your fixes. The automatic replay needs someone who can run a script.
First try takes: 20 minutes to start the log. A day or more to build the replay.
First step tonight: Start a sheet: item, field, what the AI said, what you changed it to. Add a row every time you fix a draft.
After a week: A list of the mistakes your AI repeats, in order of how often. That alone tells you which instruction to fix first.

Tools in this one

Claude Code Agent that writes and runs code on your own computer. How most of the Beartooth tools were built.