Eval Harnesses Before Prompt Tweaking
The fastest way to lose trust in an AI feature is a quiet regression: yesterday the agent booked the meeting; today it invents a calendar that does not exist. Teams usually respond by editing the system prompt. That feels productive. It is rarely enough.
If you cannot measure the task, you are not iterating — you are gambling.
What an eval harness actually is
Skip the research-paper framing. For a product agent, an eval harness is:
- A fixed set of scenarios with known ground truth (or a graded rubric)
- A runner that executes the agent the same way production does
- A score you refuse to ship below
That is it. Fancy dashboards come later.
I treat the harness like a unit test suite for behavior: boring, versioned, and blocking merges when it fails.
Start smaller than you want
Ten scenarios that represent real failures beat a thousand synthetic prompts. Good early coverage:
- Happy path the demo always shows
- Missing tool / API error mid-run
- Ambiguous user intent that should ask a clarifying question
- Out-of-policy request that must refuse
- Long-context case that used to truncate silently
Each scenario records the expected outcome: tool call sequence, final structured fields, or a binary "should escalate to a human."
Score what users feel
Token counts and latency matter for cost. Product quality needs different signals:
| Signal | Example |
|---|---|
| Task success | Did the booking land with the right attendees? |
| Tool correctness | Right tool, right args, no hallucinated IDs |
| Refusal quality | Declined when required, without leaking policy text |
| Regression rate | Pass rate vs. last release on the same set |
A single "LLM-as-judge" score is fine as a secondary metric. Do not let it be the only gate — judges drift, and you will tune prompts to please the judge instead of the user.
Where prompt diffs fit
Prompts still matter. They just move last:
- Fail an eval
- Inspect the trace
- Fix retrieval, tools, or state if that is the real bug
- Change the prompt only when the model choice or instruction was wrong
- Re-run the suite before you merge
This order keeps "prompt engineering" from becoming a dump of tribal knowledge with no test coverage.
Make it cheap to run
If the suite takes an hour or costs a week's budget, people will skip it. Practical defaults:
- Run a smoke subset on every PR (5–15 cases)
- Run the full suite nightly or on release candidates
- Pin model IDs and prompt template versions in the report
- Store failures with the full trace, redacted
At Digitized, the observability work only paid off once we could replay a bad answer. Evals are the proactive version of that replay.
The meta habit
Treat agent behavior like any other reliability surface. You would not ship a payment API on vibes. Do not ship a planner that moves money, email, or calendar state that way either.
Build the harness first. Then the prompt tweaks start meaning something.