In August I almost shipped a bad prompt change to the WhatsApp agent I'm building for a medical practice, the one that handles appointment scheduling. My tests said it worked, and the number was good enough that I stopped asking questions about it. It only fell apart when I made the test play the conversation all the way to the end, instead of grading the first reply.
That wasn't the last time. Over the following weeks, the thing that kept misleading me was the harness I built to catch a broken prompt, more often than the prompt itself. Below are those moments, one by one: the number I trusted, what it was hiding, and what I check now before believing a result.
Where it started
I already had prompt evaluation running inside the workflow, built on n8n's native Evaluations node, testing every change against a small dataset before I trusted it anywhere near production.
As the workflow and use cases grew, I realized it wasn't enough:
- Every run replayed the whole workflow, line by line, against the real calendar and the real database.
- No isolation, no way to check a single case on its own.
- The check itself was just text comparison: did the output string look like the expected string. It couldn't tell me whether the agent called the right tool with the right arguments, only whether two paragraphs read alike.
That gap sent me looking for something else. What I needed was specific: a way to test prompt changes without touching real data, with something stronger than string matching.
It was not yet the time to build, so I looked for open-source options. After my research two options came up: Promptfoo and DeepEval. DeepEval is Python and leans more on tracing across multi-agent pipelines. Promptfoo is Node, matches the rest of my stack, and it's built for side-by-side comparison, the same kind of testing I'd already been doing by hand to decide between two model choices.
I was leaning towards Promptfoo, but one thing nearly changed my mind: an open issue on GitHub describing friction with multi-turn conversations. I checked how conversation history is actually injected in my system prompt, and it doesn't arrive as a real back-and-forth. It's flattened into one block of text before the model sees it. Therefore, each test case becomes a single completion call. Whatever that friction looked like for other setups, it wasn't going to be a blocker for my setup.
So I decided to go ahead with Promptfoo.
What my test was actually scoring
For a while, it worked. Promptfoo caught real regressions: a reply drifting out of format, the wrong tool called, an argument malformed. That's the layer it's built for, and it did the job pretty well.
But the agent I was testing doesn't stop at the first reply. It calls a tool, gets a result back, and decides again from there. That's a different kind of multi-turn from the one in the GitHub issue, and I hadn't checked for it when I picked Promptfoo. My test was scoring what the model said on the first try, not what actually happened by the time the conversation ended.
I found that out right before it got expensive. I rewrote a chunk of the prompt and measured it against my test cases: 93% success, looking only at the first response. Good number, I thought. I almost shipped it. Then to double check, I built a small loop simulator on top of Promptfoo, with three rules:
- It feeds the model the same fake tool result every time.
- It caps how many times the model can go back and forth.
- It only scores what the conversation actually ends on, not what the model says along the way.
The number dropped really hard. More than half of the "near misses" from the first-turn measurement were looping back into the same issue (re-asking questions the patient had already answered), never actually closing the conversation. The 93% was measuring the wrong thing.
Since then, nothing gets scored before the conversation actually ends.
That's the point where Promptfoo stopped being the whole answer and became the foundation for something else I would build on top of it.
Same prompt, same test case, different numbers
Once the loop simulator was in place, a few things surfaced that changed how I trust any of these numbers.
The first was noise. I ran the exact same prompt against the exact same test case twice. One run: 7 out of 20. Another run, no changes in between: 17 out of 20. A single test round only tells you which noisy sample you happened to land on. Since then, a real prompt edit gets at least N=20 to 25 runs of each case before I trust the result, not because that number proves anything statistically, but because it was enough to tell a real change from noise. I tested bigger batches as well, and it did not improve, so I stayed within that range.
The second was worse, because it came from the tool itself. On one test, Promptfoo reported something like 0 to 20% success. A separate test script, hitting the same model via API directly, reported 80% on the same prompt. Nearly opposite numbers, for what was supposed to be the same input. I turned on debug logging and looked at the exact payload the tool claimed to send. It was truncated, cut off mid tool call, some internal tracing artifact rather than the real request. It told me nothing about what actually went out over the wire.
So I built a small local proxy, sitting between the tool and the API, and logged the literal bytes it sent. Then I took that exact captured payload and replayed it 15 times, straight against the API, no framework in between. 1 out of 15 succeeded. That was the real number. The 0 to 20% had been a broken debug log, not a broken prompt. What actually explained the 80% from that separate script, I never fully pinned down, only that whatever it sent wasn't the same request.
That single afternoon reset how I read every result the harness produces now: check the wire before trusting the log.
Testing the test itself
The next two were on me.
The first was a harness easier than production, and it took four steps to see it:
- I measured a chunk of the prompt as safe to remove, 120 out of 120 in the harness, and published the shorter version.
- The next real test failed: right after the patient said yes, the agent asked for a name and appointment type it already had.
- I reverted, captured that real conversation, and replayed it in the harness. 15 out of 15 passed.
- What found it, again, was reading what production actually sent.
That payload showed three places where the harness was kinder than production:
- Conversation history. From the second model call onward, the workflow sent the prompt with every variable empty, history included. The harness never lost it.
- How messages were packaged. Production sent everything as one block of text, the harness sent structured messages.
- Empty tool results. Production returned an empty string where the harness returned an empty list.
The text I'd removed was the only thing covering for the missing history, and the harness could never show me that, because in the harness the history was always there. Since then, before I remove anything from a prompt, I want to know what it's there for. If I don't know, it stays until I find out.
The second one stung the most: a test case I wrote myself, meant to prove a bug was fixed, passed clean every time. It turned out I'd written it too easy, it gave the answer away before the model even had to work for it. The real bug was still there, failing 40% of the time, caught only because another, harder test case happened to exist for an unrelated reason. Four green checks on the screen, and the defect was still live. Since then, I write the test case first and run it against the broken version before I trust it. If it doesn't fail there, it isn't testing anything real.
None of this ended with a tool I'd call finished. It's a handful of scripts sitting on top of Promptfoo: a loop simulator, an ablation runner, a local proxy for the one time I needed to see the real wire. Every one of them exists because a number I trusted turned out to be wrong in a specific, traceable way.
The pattern repeats enough that I stopped treating it as bad luck. The harness that's supposed to catch a broken prompt is code too, with its own bugs. Testing an agent turned out to mean testing the test as often as testing the agent.