Testing Agentforce agents in CI/CD: AiEvaluationDefinition, the CLI, and the pipeline the Testing Center can't give you

Here is the failure mode nobody demos. You ship an agent, it works, everyone’s happy. Six weeks later someone adds a topic, tightens an action description, and re-grounds a subagent on a fresh knowledge set. All reasonable changes. None of them touched the topic that quietly stops matching an utterance it used to handle perfectly — and you find out because a customer’s “where’s my order” now routes into the returns flow. The agent didn’t break because the model got worse. It broke because a non-deterministic system changed underneath a test suite that only ever ran by hand, in a browser, when someone remembered to open it.

The Testing Center — the clicks-based evaluation UI in Setup — is genuinely useful, and we’ve written about using it and full-conversation simulation to prove an agent works before launch. But it answers a one-time question: does this agent work right now? The question that actually keeps an agent in production is a different one: did the change I’m about to deploy break anything? That question has to be answerable by a machine, on every pull request, with no human clicking anything. This post is that path — Agentforce agent tests as versioned metadata, run from the command line and the Connect API, wired into the same pipeline that deploys the rest of your org.

The Testing Center is the author; the pipeline is the runner

Start with the mental model, because conflating the two surfaces is where teams get stuck. Agentforce agent tests are a metadata type: AiEvaluationDefinition. Each definition holds a set of test cases; each test case takes an input — an utterance — and a set of expectations about how the agent should respond. That definition is the same object whether you author it by clicking through the Testing Center or by writing it as source. The Testing Center is one editor for it. The CLI and the Metadata API are another. They all produce and consume the identical underlying artifact, which is the fact that makes CI possible: a test you built in the UI can be pulled into source control and run headless, and a test you wrote as source shows up in the UI.

That means the discipline is the same one we argue for everywhere else in release management — source-controlled metadata over point-and-click drift. An agent is metadata: topics, actions, instructions, and now its tests. If the agent is deployed from source, its tests belong in the same repository, moving through the same sandboxes, gating the same promotion to production.

What an AiEvaluationDefinition actually asserts

The tests aren’t assertions about text. An agent’s answer is generated, so you can’t diff it against a golden string and expect green. What you can pin down is the part of the agent’s behavior that has to be stable: which topic did it choose, and which actions did it run. Those are the two load-bearing expectations, and they have exact names in the metadata:

• topic_sequence_match — a topic test. Given the utterance, did the agent classify into the topic you expected? This is the single most valuable regression guard, because topic misclassification is the most common way an agent silently goes wrong: it picks a plausible-but-wrong topic and confidently does the wrong job.
• action_sequence_match — an action test. Did the agent invoke the expected action, or sequence of actions? This catches the case where the topic is right but the agent skipped the lookup, or called the refund action when it should only have called the status action.

Each expectation carries an expectedValue, and when a case fails the result tells you the expected topic or action versus the one the agent actually used — which is exactly the diff you need to debug it. The result field to read is metricScore, and for topic and action tests it returns PASS or FAILED.

Two nuances matter before you build a suite on this. First, not every metric is binary. Agentforce also supports an instruction-adherence check whose score isn’t PASS/FAILED but one of HIGH, LOW, or UNCERTAIN — a graded signal about whether the response followed the topic’s instructions, which you have to threshold deliberately rather than treat as a hard gate. Second, beyond the built-in expectations you can add custom evaluation criteria to a test case, so you’re not limited to topic-and-action matching when a use case needs a sharper assertion. The built-ins cover the regressions that bite most; the custom criteria cover the ones specific to your agent.

Authoring the tests as source: the testspec workflow

You don’t hand-write the XML. The Salesforce CLI’s agent plugin generates it from a readable spec. The workflow has three commands, and they map cleanly onto author → build → run.

1. Generate a spec. sf agent generate testspec walks you through creating a YAML test specification — the human-editable list of cases. The generated file looks roughly like this:

# order-status-agent.testspec.yaml
subjectType: AGENT
subjectName: Order_Status_Agent
testCases:
  - utterance: "Where is my order 10432?"
    expectedTopic: "Order_Status"
    expectedActions:
      - "Get_Order_Status"
    expectedOutcome: "Returns the current shipment status for the order"
  - utterance: "I want to send this back"
    expectedTopic: "Returns"
    expectedActions:
      - "Start_Return"
    expectedOutcome: "Initiates a return, does not expose refund amount"
  - utterance: "Cancel my account"
    expectedTopic: "Escalate_To_Human"
    expectedActions: []
    expectedOutcome: "Hands off to a human with context, takes no destructive action"

Keep the spec in the repo next to the agent metadata. It’s the artifact your team reviews in a pull request — far more legible than the compiled definition, and the place where a reviewer can catch “wait, that utterance should never trigger the refund action.”

2. Build the metadata. sf agent test create converts the spec into an AiEvaluationDefinition and deploys it to your org. The compiled definition is what actually runs; conceptually it’s a set of test cases each carrying named expectations:

<AiEvaluationDefinition xmlns="http://soap.sforce.com/2006/04/metadata">
    <subjectType>AGENT</subjectType>
    <subjectName>Order_Status_Agent</subjectName>
    <testCase>
        <number>1</number>
        <inputs>
            <utterance>Where is my order 10432?</utterance>
        </inputs>
        <expectation>
            <name>topic_sequence_match</name>
            <expectedValue>Order_Status</expectedValue>
        </expectation>
        <expectation>
            <name>action_sequence_match</name>
            <expectedValue>["Get_Order_Status"]</expectedValue>
        </expectation>
    </testCase>
</AiEvaluationDefinition>

Treat the exact element names as version-sensitive and let sf agent generate testspec produce the current shape for your API version — the point to internalize is the structure: a case, an utterance, and named expectations with expected values.

3. Run it. sf agent test run executes the definition against the agent in a target org and reports each case’s pass/fail, the expected versus actual values, the score, and how long it took. Critically for automation, it supports machine-readable output:

# Run and block until complete, emitting JUnit for the CI system to parse
sf agent test run \
  --api-name Order_Status_Agent_Tests \
  --target-org ci-scratch \
  --wait 10 \
  --result-format junit \
  --output-dir ./agent-test-results

# Long-running suite: kick off async, capture the job id, resume later
sf agent test run --api-name Order_Status_Agent_Tests --target-org ci-scratch --json
sf agent test resume --job-id <returned-id> --result-format junit --output-dir ./agent-test-results

The --result-format flag takes human (the default table), json, tap, or junit, and --output-dir writes the results to files instead of the terminal. JUnit is the one that matters for CI: every mainstream pipeline — GitHub Actions, GitLab, Jenkins — knows how to render a JUnit report and fail the job on it. That single flag is what turns an agent test from a thing a person reads into a gate a machine enforces.

Running tests from the Connect API when the CLI isn’t in the loop

The CLI is the right tool inside a build agent. But some pipelines drive Salesforce through their own service, or you want to trigger an evaluation from an orchestration layer that isn’t a shell. For that, the same tests run through the ai-evaluations Connect API. You start a run with a POST and then poll:

# Start an asynchronous evaluation run
curl -X POST \
  "$INSTANCE_URL/services/data/v63.0/einstein/ai-evaluations/runs" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "aiEvaluationDefinitionName": "Order_Status_Agent_Tests" }'
# → returns a run id

# Poll for status, then pull the detailed results
curl "$INSTANCE_URL/services/data/v63.0/einstein/ai-evaluations/runs/<runId>" \
  -H "Authorization: Bearer $ACCESS_TOKEN"

The run is asynchronous — you start it, get an id, poll status until it finishes, then fetch results that include each test case and the outcome of every expectation, with the same metricScore values you’d see from the CLI. It’s the same evaluation engine; the API is just the headless door into it for systems that live outside the CLI. Verify the exact request-body field names and endpoints against the current Testing API reference for your API version before you wire it up, because the resource has evolved across releases.

Wiring it into a pipeline that actually gates

Put the pieces together and the shape is unremarkable, which is the point — it should look like every other test stage you run. On each pull request that touches agent metadata:

1. Spin up (or reuse) a dedicated scratch or CI sandbox, and deploy the agent source and its AiEvaluationDefinition tests into it. Do not run against production — agent tests invoke the live agent, and you don’t want evaluation traffic or any side-effecting action landing on real records.
2. sf agent test run --result-format junit --output-dir ./results for each suite.
3. Fail the build on any FAILED topic or action case, and on instruction-adherence scores below your chosen threshold.
4. Only promote the change through your normal sandbox-to-production deployment when the suite is green.

The subtle discipline is what you assert, not how you run it. A good agent suite is mostly topic tests, because classification is where non-determinism does the most damage, plus action tests on every path that touches data or money, plus a handful of “must escalate, must not act” cases for the utterances where the safe behavior is to do nothing and hand off to a human. That last category is the one teams forget and the one that saves you: a test that asserts the agent takes no destructive action on an ambiguous or hostile utterance is a regression guard against exactly the kind of over-eager behavior that turns a helpful agent into an incident.

The gotchas that bite in practice

Three things are worth knowing before you scale this up.

Every test case is a real agent invocation, and it costs. Running a suite isn’t free assertion-checking against cached output — each case actually invokes the agent, which means model calls and the same Flex Credit consumption a live conversation incurs. A 200-case suite fired on every commit by a busy team adds up. Run the full suite on merges and nightly; run a focused subset on each push. Budget the credit cost of testing the same way you budget the credit cost of running the agent, because it draws from the same meter.

Non-determinism means a single run isn’t proof. The agent can classify correctly nine times and wrong the tenth on the same utterance. A test that passes once hasn’t proven the topic is stable — it’s proven the topic can match. For the utterances you truly depend on, expect to run cases enough times to see the failure rate, and treat a flaky topic as a signal to sharpen the topic and classification descriptions, not as noise to re-run away. This is the honest difference between testing deterministic code and testing an agent: green once is necessary, not sufficient.

Tests are metadata, so they drift too. When you rename a topic or an action, the expectedValue in your tests still points at the old name and the suite goes red for the wrong reason. Regenerate the spec, or update it, as part of the same change that renames the thing — the test suite is only a safety net if it’s maintained alongside the agent it guards, not a museum of what the agent used to be called.

The takeaway

The Testing Center proves an agent works; a pipeline proves your next change didn’t break it, and only the second one keeps an agent alive in production. The mechanism is already there: agent tests are AiEvaluationDefinition metadata, authored from a readable testspec, compiled with sf agent test create, and run headless with sf agent test run --result-format junit or the ai-evaluations Connect API — all consuming the same evaluation engine the UI does. Assert on topics and actions, not on generated text; put a hard gate on “must escalate, must not act” cases; run the full suite where the credits are worth it and a subset everywhere else; and keep the tests in the same repo, moving through the same sandboxes, as the agent itself. Do that and a misrouted “where’s my order” fails your build on a Tuesday afternoon instead of a customer’s conversation on a Saturday night.

Understanding the basics

What is AiEvaluationDefinition in Agentforce?

AiEvaluationDefinition is the Salesforce metadata type that represents an Agentforce agent test. Each definition contains a set of test cases; each case supplies an input utterance and one or more expectations about the agent’s response — most importantly topic_sequence_match (did the agent choose the expected topic) and action_sequence_match (did it invoke the expected actions). Because tests are metadata, they can live in source control, deploy through sandboxes, and run in a CI/CD pipeline. The Testing Center in Setup is a UI for authoring the same definitions, so a test built by clicking and a test written as source are the same underlying object.

How do you run Agentforce agent tests from the command line?

Use the Salesforce CLI agent plugin. sf agent generate testspec creates a readable YAML spec of your test cases; sf agent test create compiles that spec into an AiEvaluationDefinition and deploys it; and sf agent test run --api-name <name> --target-org <org> executes it. For automation, add --result-format junit and --output-dir so the pipeline can parse the results and fail the build on any failed case. Long-running suites can be started asynchronously and picked back up with sf agent test resume --job-id. Run tests against a scratch org or CI sandbox, never production, because each case invokes the live agent.

How do you gate a deployment on agent test results?

In the pull-request stage of your pipeline, deploy the agent and its AiEvaluationDefinition tests into a CI sandbox or scratch org, run sf agent test run with --result-format junit, and configure the job to fail on any FAILED topic or action case (and on instruction-adherence scores below your threshold). Only promote the change to production when the suite is green. You can trigger the same runs from outside the CLI through the ai-evaluations Connect API — POST to /services/data/vXX.0/einstein/ai-evaluations/runs, then poll the run for status and results.

Does running agent tests cost Flex Credits?

Yes. An agent test isn’t a cheap string comparison — each test case actually invokes the agent, so it consumes model calls and draws on the same Flex Credit budget a real conversation would. A large suite run on every commit adds up quickly, so most teams run the full suite on merges and nightly and a focused subset on each push. Treat the credit cost of testing as part of the same budget as the credit cost of running the agent in production.

---

Trying to get an Agentforce agent out of “works in the demo” and into a governed release process with real regression gates? Talk to us. Building the tests, the pipeline, and the topic discipline that keeps an agent trustworthy after the launch is exactly the work we do.