My agent testing process used to be world-class: open the test chat, fire off five questions, watch it answer, nod approvingly, ship it. Bulletproof. … Just kidding. That’s not testing. That’s vibes.
And vibes don’t scale. They especially don’t scale when a stakeholder walks over and asks why the agent gave a confidently wrong answer to a customer last Tuesday, and your best evidence is “well, it looked fine when I tried it.”
So here’s the good news. Agent Evaluation in Copilot Studio has been generally available since March 2026, and it’s been quietly getting better ever since. It started as a preview back in October 2025, hit GA together with multi-turn conversation tests in March, and has since picked up CI/CD automation and version comparison. This is the feature that turns “I think it’s fine” into “here are the numbers.” Let me walk you through what it does, how it works, and where it actually earns its keep.
What Agent Evaluation Actually Is
In plain terms: Agent Evaluation lets you build a set of test cases, define what “good” looks like, run them against your agent automatically, and get scored results back. No more clicking through the same ten prompts by hand after every change.
There are two flavors, and the distinction matters:
| Test type | What it checks | Size limits |
|---|---|---|
| Single response | Does the agent give the right answer to one question? | Up to 100 test cases per set |
| Conversation (multi-turn) | Can the agent hold context across a real dialog? | Up to 20 cases, up to 12 messages (6 Q&A pairs) each |
The single-response set is your regression safety net, the “did I break anything?” check. The multi-turn set is the interesting one, because it tests whether your agent can keep context, ask for clarification, and finish a multi-step task instead of falling apart on turn three. That’s where most real-world agents quietly fail.
One practical note before you go all in: test results stay available in Copilot Studio for 89 days, so if you want a long-term quality history, export them to CSV.
Building Your First Test Set
You don’t have to handcraft every test case, which is the part I appreciate. You’ve got three ways to fill a test set:
- Write them manually — full control, but tedious.
- Import from a spreadsheet — my preferred route once you’re past the first round, because you can manage cases in Excel and version them properly.
- Let AI generate them — Copilot Studio reads your agent’s description, instructions, and capabilities and proposes test cases for you.
For multi-turn, the AI generation is genuinely useful. A Quick conversation set spins up 10 short conversations automatically based on what your agent is supposed to do. A Full conversation set goes deeper, using your agent’s knowledge and topics to build longer dialogs. And if you just had a great manual test chat, you can convert it straight into a test case. That last one is gold — you stumble onto a tricky edge case while playing around, and instead of forgetting it, you bank it as a permanent test.
The Part That Matters: How It Grades
Creating test cases is easy. Defining good is where the actual thinking happens, and this is where Agent Evaluation got better.
The default — and the one every test set starts with — is General quality. It uses a large language model as the judge across four criteria:
- Relevance — did it actually answer the question?
- Groundedness — is the answer based on your source data, or did the model freestyle?
- Completeness — did it cover everything it should?
- Abstention — did it even attempt an answer, or did it bail?
Here’s the catch worth knowing: a response has to clear all four to count as high quality. Miss one — incomplete, or not grounded — and the case gets flagged for improvement. No partial credit. And the nice part is you don’t need to define expected answers for this method; the LLM judges against the retrieved context.
For my money, groundedness is the one to watch. An agent that sounds great but invents details is worse than one that says “I don’t know,” because the confident hallucination is the one that costs you trust with a customer. If you only look at one metric, look at that.
General quality is the flexible, “no exact answer expected” option. But it’s not the only method — you can stack several on the same test set, and the rest are more targeted:
| Method | What it checks |
|---|---|
| Compare meaning | Does the answer capture the intent of your expected response, regardless of wording? (default pass score: 50) |
| Text similarity | Does the wording closely match the expected answer? Use it when phrasing matters — legal text, disclaimers. |
| Exact match | Character-for-character match. For codes, numbers, fixed phrases. |
| Keyword match | Does the answer contain specific terms (any or all)? |
| Tool use (a.k.a. capability use) | Did the agent actually call the tools or topics you expected? |
| Custom | Your own rule. Write plain-language instructions and pass/fail labels. |
That last one, the Custom test method (Microsoft’s answer to “custom graders”), is the escape hatch for domain-specific rules: “must always include the order number,” “must never quote a price without a disclaimer,” “must stay HR-compliant.” You describe the behavior in natural language, define your pass/fail labels, and it grades against them.
One thing to keep straight: not every method works on multi-turn sets. Conversation test sets are limited to General quality, Keyword match, Capability use, and Custom. The wording-exact methods (Compare meaning, Text similarity, Exact match) are single-response only.
Where This Pays Off in Practice
Here’s the workflow this unlocks, and why I think it’s a bigger deal than a typical feature drop.
Build your agent. Create a test set — start with AI-generated cases, then add your own painful edge cases over time. Run the evaluation, look at the scores, fix the weak spots, run again. Now you have a baseline. The next time you tweak instructions, swap a model, or add a knowledge source, you re-run the same set and see immediately whether you improved things or quietly broke them.
That’s regression testing for agents. It’s the difference between shipping an agent and maintaining one. For anyone deploying agents to actual users — customers, colleagues, whoever — this is the maturity step the platform needed. Up to now, “testing” an agent meant a maker eyeballing a few responses. Now you can hand a stakeholder an actual number and a trend over time.
And you don’t have to run it by hand. Since April 2026 you can trigger evaluations from the Copilot Studio connector in Power Automate, and there’s a REST API (in preview) to wire it straight into a CI/CD pipeline. Run your test set automatically on every change, fail the build if the score drops — that’s where this stops being a manual chore and becomes part of your delivery process. You can also compare two agent versions side by side to confirm an improvement actually improved things.
The honest pitch: this won’t make a bad agent good. But it will tell you, with receipts, exactly how good your agent is — and whether your last change helped or hurt.
My Take
I’ll be direct: this isn’t a flashy feature. There’s no shiny new chat experience, no demo that makes a keynote audience gasp. But it’s exactly the kind of unglamorous capability that separates a proof-of-concept from something you’d actually put in front of a customer.
If you’ve got an agent running in production, or worse, one you plan to put in production without a real test process, go set up a test set this week. Start small: ten single-response cases for your most important scenarios, one multi-turn set to check that it doesn’t lose the plot mid-conversation. Run it, look at groundedness, and you’ll probably learn something uncomfortable but useful within the first ten minutes.
That’s the best kind of feature: the one that quietly makes you better at your job. Give it a try.
Sources: Microsoft Copilot Studio — What’s new, About agent evaluation, Choose evaluation methods, Create a conversational test set, Build smarter, test smarter (Copilot Blog).
Discover more from Lubenheimer
Subscribe to get the latest posts sent to your email.