Eval / Benchmark

How do you actually know if one AI is better than another, or if the change you just made to your prompt helped or quietly made things worse? You test it. That test has a name: an eval.

An eval (short for evaluation) is a batch of test questions where you already know what a good answer looks like. You run a model against them and score how well it does. A benchmark is a famous, standardized eval that everyone uses, so different models can be compared on the same yardstick. Eval is the general idea. Benchmark is the industry-wide exam.

It's a report card. Imagine a stack of a thousand questions where you already know the right answer to each. You hand the same stack to every model, let each one answer, and grade the results. Now "this model is smarter" stops being a vibe and becomes a number: this one got 88%, that one got 71%, on the exact same test. That's what the benchmark scores in every AI launch are. Report cards from a shared exam.

Where this gets practical for you isn't the big public benchmarks, it's your own eval. Say you've built an AI that sorts support emails. You write down fifty real emails and the category each should land in. That's your eval. Now every time you tweak the prompt, swap the model, or change the temperature, you rerun those fifty and see if the score went up or down. Without it, you're "improving" things by gut feel and hoping. With it, you actually know.

This is the fix for a trap you'll otherwise fall into: you change a prompt to fix one annoying case, it works, you ship it, and you never notice it quietly broke five other cases. An eval catches that. It's the difference between "seems better to me" and "scored better on the same fifty, every time."

One caution worth keeping. A model can be trained, on purpose or by accident, to ace a famous benchmark while being no better in real life, the same way a student can cram for one specific test and learn nothing lasting. So treat headline benchmark scores as a rough signal, not gospel, and trust your own eval on your own task far more. The test that matters is the one built from your actual work.

An eval is a graded test where you know what a good answer looks like. A benchmark is the standardized one everyone shares. Build your own from real examples and you stop improving your AI by vibes and start improving it by score.