Evaluating model output is the discipline of deciding, repeatably and defensibly, whether a generated answer is good enough for its purpose. The word repeatably is doing the work. Anyone can read an answer and form an opinion. The problem is that opinions do not aggregate, do not survive personnel changes, and cannot tell you whether last week’s prompt change helped or hurt.
What goes wrong without structure
- Vibes-based review. Someone reads ten outputs, declares the system good, and ships. The ten were not representative and nobody can reproduce the judgement.
- Moving goalposts. Without written criteria, the standard shifts with the mood of the reviewer and the deadline.
- Invisible regressions. A prompt change fixes one category and breaks two others. With no fixed test set, nobody notices until a user complains.
- Unresolvable disputes. Two reviewers disagree, there is no rubric to appeal to, and the argument is settled by seniority rather than evidence.
- No improvement signal. You cannot tell whether you are getting better, because there is no baseline number.
The three questions every evaluation must answer
- Is it right? Factual accuracy, correct reasoning, grounding in the provided source material.
- Is it usable? Does it follow the instructions, respect the format, suit the audience, and come in at the right length.
- Is it safe to ship? Does it avoid harm, confident fabrication, leaked sensitive content and out-of-scope advice.
Evaluation is a measurement system, not a judgement
Treat your evaluation like any other instrument. It has a definition of what it measures, a procedure, known failure modes and an error rate of its own. An evaluation that two trained raters apply to the same output and reach different conclusions is a broken instrument, and fixing it is prior to using it. Module 2 covers that calibration work in detail.
Worked example
A team deployed a model to draft replies to customer billing questions. Review was informal: a support lead read a handful each morning. After a prompt revision intended to shorten replies, complaints rose. Investigation found the shorter replies had dropped the sentence confirming the billing period, so customers could not tell which invoice was being discussed. Nobody had caught it because the informal review had been checking tone, not completeness, and the sample had happened to contain only single-invoice accounts.
The team built a fixed evaluation set of 120 real anonymised questions covering nine categories, with a four-dimension rubric: factual accuracy, completeness against a required-elements list, instruction compliance, and tone. The required-elements list for billing questions explicitly included the billing period. Re-scoring the shortened prompt against that set produced 0.71 on completeness against a baseline of 0.94, which made the regression obvious in twenty minutes rather than two weeks. The fix was one line in the prompt. The value was not the fix; it was the instrument that found it.
Common mistakes at this stage
- Starting with automated metrics before anyone has defined what good means.
- Evaluating only on examples the system already handles well.
- Letting the person who wrote the prompt be the sole judge of its output.
- Treating a single aggregate score as the whole picture, hiding category-level collapses.
Action Step
Write a one-page evaluation brief for one system you care about. Name the task in one sentence, the audience, the three to five quality dimensions that matter, the single worst failure you must never ship, and who the accepting decision maker is. Then collect twenty real inputs, including at least five that you expect to be hard. That page and those twenty inputs are the seed of everything built in the rest of this course.