AI Evals

You change an AI prompt. The new version sounds better on three examples. But did invoice extraction improve overall? Did another capability become worse? Traditional deterministic assertions aren't always enough.

Evals measure model behaviour on test cases that represent real tasks. Unit and integration describe test scope; evals assess AI task quality and can use either scope. Answers can vary across runs, and several answers may be valid. Evals use exact checks for facts and defined criteria for open-ended answers. Repeated runs measure consistency, while outcome checks verify that an agent's actions actually happened.

Prompt v1 → test set → results
Prompt v2 → test set → results