write the evaluation before you pick the model
The reason most AI features stall is not the model. It is that nobody can say whether a change made things better or worse. Start with the ruler, not the model.
Almost every stalled AI project we have seen shares one symptom: the team can ship a change, but nobody can tell you whether it helped. Prompts get tweaked, models get swapped, and the whole thing moves sideways because there is no ruler.
The fix is unglamorous. Before you choose a model or write a prompt, write down how you will know it works. A set of real inputs, the answers you would accept, and a way to score the gap. It does not have to be fancy - a spreadsheet of cases and a script that grades them beats an opinion every time.
what a usable eval looks like
- real examples pulled from your actual traffic or tickets, not invented ones
- a clear notion of a good answer, including when the right answer is "I do not know"
- a score you can run on every change, so a regression is caught before it ships
Once the ruler exists, everything downstream gets calmer. Model choice becomes a measurement, not a debate. Prompt changes stop being vibes. And when someone asks whether the feature is getting better, you have a number instead of a shrug.
The model is the interesting part to talk about. The evaluation is the part that decides whether any of it survives contact with production.
have a system this describes?
If any of the above sounds like something you run, tell us about it. We answer every message ourselves.