The gold loop: prove a judgement before you scale it

I was working on a duplicate ad detector problem, where I had to determine if the same ad can be reused across different channels and markets. As typical practice nowadays, I used the grill-me skill from mattpocock, reviewed the problem and solution with Claude Code, wrote spec, concepts and design using my engineering principles skill and let Claude do the implementation. I verified some results in passing, and then ran on the rest of the creatives. Then I created a simple app out of it as a final artifact. While reviewing the app, the results weren’t perfect. While most of the basic duplicates were correctly determined by the AI tagging, something was missing in the overall analysis. There were boundary cases, subtle mistakes which were difficult to write as a rule in the prompt myself, but visually it was apparent. This is a recurring pattern that I have observed, where the quality of the final output is very good, but not enough.

This is the approach that I have settled down on for now:

Step 1: definition on paper before any code

I have gone back to old school for the initial work: I will take 4-5 sample examples and manually write my observations. This includes freestyle writing on what I am seeing, why behind my thought process and first set of rules, that I can count as facts. They become the basis of non-negotiables, for later processing. So the rules can be a list of what is allowed and what is not allowed. For the duplicate detector, a change in resolution or crop was allowed, while a different headline or offer wasn’t allowed.

Step 2: build a small answer key, blind

After that, I would take 30 mixed examples, and let AI create an HTML artifact, with those 30 examples. The mix includes the obvious duplicates, the boundary cases, and the cases where I suspect the current system will get it wrong. If I pick randomly, I mostly get the easy ones, and the easy ones don’t tell me much. I have even created my own design system, so that HTML artifacts look good, and reviewing looks fun and not forced. I write my observations, notes and judgement on the HTML. In parallel, a subagent also makes its own decision on these 30 examples. This is deliberately kept blind, as I have observed across sessions, that I pick the AI recommended response most of the time, and not do the thinking myself. So I deliberately keep AI recommendations blind, until I do the work.

Step 3: test the system on the answer key

I copy these two judgements and comments into the main session, and I review the results and disagreements until we come to a common conclusion. The goal is not to get everything right but agree on a broad set of guidelines and alignment on the scores. The disagreements typically result in 3-4 rules, that I add back to the definition. This dataset is now used for evaluations for all future runs as well.

After we agree, I run the system on these same 30 examples and compare its answers against the answer key. I check the score separately for the obvious cases and the boundary cases. For every miss, I look at what exactly the system read differently and see if there is a rule that can be generalised. If yes, I add it to the definition. Otherwise I don’t try to overfit. So I don’t chase the last 1-2 misses, I just note them as a known failure and move on.

Step 4: scale, then check what actually came out

After that I run AI on the entire dataset, and I have noticed that the results are drastically improved. But still not good enough. What worked for me was iteration in phases. So now, for every 10x examples, I repeat the process of selecting 30 new examples. And never run the AI on all the examples at once.

The new 30 examples come from the output of the run I just finished, and not from the remaining unprocessed examples. A bug between the judgement and the final output, or an answer key that didn’t represent the real data, can still give bad results. That is exactly what happened to me in the app review earlier, where the spot checks looked fine but the app made the misses visible. So these are two different checks: did the logic work, and did the run produce good output. If the second one fails, I don’t use the results. I fix the problem and sample again. This has helped me with getting excellent results, and avoiding misses or mistakes that I was making otherwise, by relying too much on AI.

So that is my current approach for solving most of the problems today, and it has become the gold-loop skill for me. And there are some indirect advantages of the approach:

1. I have a better understanding of the solution.
2. Eval datasets are getting created as part of the process, which seemed to be a separate work earlier.
3. Savings on cost and time, as I don’t have to wait for all the processing to be done.