How to test an Agentforce agent: from utterance batches to full-conversation simulation

There’s a moment in every Agentforce project when someone asks the only question that matters: how do we know it won’t say something wrong to a customer? For deterministic code, sixty years of testing practice answers that. For an agent — probabilistic, context-dependent, capable of phrasing the same answer thirty ways — the honest answer in 2024 was “we typed things at it and watched.” That answer is now obsolete, and if your test plan still looks like a spreadsheet of sample utterances, you’re testing an agent the way people did two release cycles ago.

The tooling grew up fast. Salesforce announced Agentforce Testing Center on November 20, 2024 as “first-of-its-kind” agent lifecycle tooling, with general availability in sandboxes that December. The first version was useful but narrow — batch-tested single turns, results shipped in and out via CSV. Then in April 2026, the product team rebuilt it inside Agentforce Studio: full-conversation simulation with synthetic personas, custom natural-language evaluations, inline editing, and a CLI that lets a CI/CD pipeline block a deployment until the agent passes its suite. The same post opens with a pointed question — “It’s 2026 — are you still editing CSV files to manage your testing suite?” — which tells you how quickly the floor has moved.

This post is the practitioner’s map: what each testing layer catches, what it misses, and the order we run them in before an agent goes live.

Why agent testing is not test coverage

Start with the mindset shift, because it decides everything downstream. Apex has assertions: given input X, the method returns Y, forever. An agent gives you a distribution of behaviours. The same refund question, asked twice, can produce two differently-worded answers — both acceptable — or one acceptable answer and one hallucinated policy. As Gearset’s engineering-focused guide to Testing Center puts it, agents “operate in a world of probabilities and interpretation,” and the flexibility that makes them useful “also introduces a higher risk of inconsistency.”

So agent testing has to answer four questions that traditional QA never asked:

1. Routing — did the agent pick the right topic and action for the request? This is the closest thing to a classic assertion, and it’s what the original Testing Center shipped to validate at scale: topic and action selection against batches of synthetic utterances.
2. Grounding — did the answer come from your data, or from the model’s imagination? No test suite fixes retrieval built on bad data; we’ve written about why data quality decides agent quality separately.
3. Policy — did it stay inside the rules: identity verification before account details, refund limits, disclosure language? The strongest version of this isn’t a test at all — it’s moving the rule out of the prompt and into deterministic logic with Agent Script, then testing that the gate holds.
4. Conversation quality — across a whole multi-turn exchange, did the customer actually get what they came for?

That fourth question is the one nobody could test in an automated way until this year.

The 2026 Testing Center: conversations, personas, and your own definition of “good”

The April 2026 release moved Testing Center out of Setup and into Agentforce Studio as a dedicated tab beside Agent Builder and Observability — a small change that fixes a real workflow problem: you now build, test, and monitor in one surface, with run history kept so you can compare suites across agent versions instead of eyeballing screenshots.

The headline capability is conversation-level testing. Where the old flow evaluated one utterance and one response at a time, the new one simulates an entire dialogue: you pick a predefined persona — the examples Salesforce ships include a “frustrated customer,” a “non-native English speaker,” and a “distracted user” — and the system generates the multi-turn conversation automatically, then scores it on metrics like task resolution: did the agent complete what the user originally asked? Voice agents are covered too, down to playing back the AI-generated voice conversations from the run. If you’ve ever watched a well-behaved text agent fall apart the third time a tester rephrased the same question, you know exactly which failure mode this exists to catch.

The second addition matters just as much for regulated teams: custom evaluations. The built-in metrics judge completeness and resolution, but you can now define your own scorer in natural language — the documentation’s example is “Rate the politeness of the agent response on a scale of 0 to 5” — with descriptions per score level, example responses, and a pass/fail threshold. Your metric then appears as a column in the testing grid alongside the standard ones. This is how a bank encodes “never states an interest rate without the disclaimer” and a healthcare org encodes “never speculates about symptoms” as scored, repeatable checks rather than tribal review knowledge.

Two quality-of-life changes round it out. Inline editing finally kills the download-CSV, edit, re-upload loop for fixing test cases. And a rich JSON viewer exposes the full execution trace per test — which subagent was selected, which action fired, inputs, outputs, the LLM judge’s reasoning, and latency at each step — any of which you can pull into the grid as a column. When a case fails, the “why” is in the trace, not in a support ticket.

One honest caveat belongs here: several of these evaluations are themselves LLM-judged. An LLM judge is consistent enough to catch regressions at batch scale, but it isn’t a compliance officer. Keep a human sampling the transcripts — especially the passes near your threshold, which is where judge optimism hides.

Wiring tests into the pipeline: the CLI and deployment gates

The change that makes agent testing feel like engineering rather than QA theatre is the command line. Salesforce’s ADLC (Agent Development Lifecycle) Skills expose Testing Center through the Salesforce CLI — sf agent run test to trigger a suite, sf agent get test status to poll results — without opening a browser, and the same surface works from AI-powered IDEs.

For teams with a delivery pipeline, this closes the loop that Apex closed years ago: the CLI integrates with DevOps Testing Center so a failing agent suite can block the deployment, exactly like a failing unit-test run. An agent change — a reworded instruction, a new action, a model swap — goes through a pull request, the suite runs, and “the agent got worse at refund policy questions” becomes a red build instead of a production incident. If you’re already running source-driven Salesforce delivery, agent metadata slots into the same flow; the original launch made DevOps Center, the CLI, and even Change Sets work with Agentforce and Data Cloud metadata precisely so agents wouldn’t live outside your release discipline.

Sandboxes are the other half of that discipline. Testing Center went GA in sandboxes first, deliberately — batch runs consume model capacity and can fire actions, so they belong in an environment that mirrors production data and configuration, not in production itself.

What the tooling still won’t catch

Testing Center validates the agent you meant to build. Three failure classes live outside it.

Adversarial input. Batch tests are generated from your agent’s own structure — they probe competence, not malice. A customer pasting “ignore previous instructions” into chat, or a poisoned knowledge article steering answers, is a different threat model with its own testing discipline; we’ve covered prompt injection and agent security testing in depth. Before go-live, someone on the team should spend a day genuinely trying to break the agent — jailbreaks, data exfiltration attempts, requests just outside policy phrased just inside it. Budget for it the way you’d budget for a penetration test, because that’s what it is.

Humans behaving like humans. Structured QA still finds what synthetic personas don’t. The best field guide we’ve seen is Salesforce Ben’s QA strategy write-up, from a team that ran a large human testing program: give testers scenarios and sample utterances but room to improvise, deliberately include misspellings and missing punctuation, add a “Partial Pass” status because agent responses often land between right and wrong, and document exact inputs and outputs because you will not reproduce a failure from memory. One operational detail from that piece is worth planning around: updating an agent’s instructions or actions takes it offline mid-update, which halts testing — so schedule change windows, or your QA team spends afternoons staring at an unavailable agent.

Production drift. A suite that passed in June says nothing about September, after two knowledge-base rewrites and a model version bump. This is where the Observability tab and utterance analysis take over from testing — watching containment, escalation quality, and topic distribution on real traffic. That’s a monitoring discipline of its own; our production lessons post covers what to watch after launch, and the launch-day metrics that matter are worth instrumenting before you need them.

The one-page test plan

Here’s the whole stack on one page, in the order each layer earns its keep:

Layer	What it catches	Tool	When it runs
Deterministic gates	Policy violations that must never happen	Agent Script conditions	Enforced at runtime; tested in every suite
Utterance batch tests	Wrong topic/action routing at scale	Testing Center batch runs (AI-generated + hand-written cases)	Every agent change, via CI
Conversation simulation	Multi-turn breakdowns, lost context, unresolved tasks	Testing Center personas + task-resolution scoring	Before each release cut
Custom evaluations	Your org’s definition of “good” — tone, disclaimers, policy language	Natural-language scorers with thresholds	In the same suites, as grid columns
Adversarial pass	Prompt injection, jailbreaks, boundary-pushing	Human red team + security review	Before first launch; after major scope changes
Human exploratory QA	Realistic mess: typos, ambiguity, partial passes	Structured QA sessions	Before launch; sampled quarterly
Production observability	Drift, degradation, new failure patterns	Studio Observability + utterance analysis	Continuously after go-live

And the sequence we run it in, for a first agent going live:

1. Move policy out of prompts first. Every rule that can be an Agent Script condition becomes one. Tests then verify a gate, not a hope.
2. Build the batch suite while you build the agent. Generate cases with AI, then hand-write the ones that keep you up at night — the AI generator probes your agent’s structure, not your industry’s edge cases. Mix both, as Gearset’s walkthrough recommends.
3. Add custom evaluations for your three non-negotiables. Not twenty. Three scored metrics a stakeholder actually signed off on.
4. Run conversation simulations with the awkward personas. The frustrated customer and the non-native speaker will find more than a hundred polite single turns ever will.
5. Red-team it, then let humans loose on it. Adversarial pass first, exploratory QA second — security findings change scope; QA findings change wording.
6. Gate the pipeline before launch, not after. sf agent run test in CI, deployment blocked on failure, from the very first release. Retrofitting a quality gate onto a live agent is how quality gates get postponed forever.

The uncomfortable truth about “done”

Traditional software has a finish line: tests green, ship it, move on. An agent never quite reaches one — the model underneath it updates, your data changes, customers invent phrasings your personas didn’t. The teams doing this well have stopped treating testing as a phase and started treating it as infrastructure: a suite that grows with every production surprise, evaluations that encode every stakeholder argument as a scored metric, and a pipeline that refuses to ship an agent that got worse.

That’s also the quiet good news. Eighteen months ago, proving an agent was safe to ship meant a spreadsheet, a prayer, and a pilot group. In July 2026 it means simulated conversations, scored policy checks, execution traces, and a red build when the numbers drop. The tooling has caught up to the ambition. The only question left is whether your release process has — and if you’d like a second pair of eyes on that, adversarial testing before go-live is literally step three of how we deliver.

Understanding the basics

What is Agentforce Testing Center?

Agentforce Testing Center is Salesforce’s built-in tool for testing AI agents at scale. Announced in November 2024 and now a tab inside Agentforce Studio, it runs batches of test cases against an agent — including AI-generated ones — evaluates topic and action selection, simulates full multi-turn conversations with synthetic personas, and scores results against built-in and custom metrics, with a detailed execution trace for every run.

Can Agentforce tests run in a CI/CD pipeline?

Yes. Salesforce’s ADLC Skills expose Testing Center through the Salesforce CLI — sf agent run test triggers a suite and sf agent get test status retrieves results — and the integration with DevOps Testing Center lets a pipeline block an agent deployment until its test suite passes, the same way failing Apex tests block a deploy.

What is conversation-level testing?

Conversation-level testing, added to Testing Center in April 2026, simulates an entire multi-turn dialogue instead of evaluating one utterance at a time. You choose a synthetic persona — such as a frustrated customer or a non-native English speaker — and the system generates the conversation, runs it against your agent (including voice agents), and scores outcomes like task resolution: whether the user’s original request was actually completed.