Agentforce
Agentforce A/B experimentation: choosing a winner without rewarding bad answers
Salesforce announced Agentforce A/B Experimentation for an October beta, with variant setup and an analytics scorecard in Builder. The hard part is deciding what counts as a better agent. Plan an experiment that measures task completion, keeps safety failures visible, and distinguishes a useful change from a different mix of customer requests.
An agent can lower your handoff rate by making it harder to reach a person.
That looks like an improvement on a dashboard until you read the conversations. The unresolved customer gives up, the session ends, and a metric designed to measure containment rewards the wrong behavior. An experiment makes that mistake faster if you never define a successful outcome.
Salesforce’s September 28 Dreamforce roundup announced A/B Experimentation for Agentforce, with an October beta. It describes variant setup and an analytics scorecard in Agentforce Builder, with traffic split across cohorts. Prepare your experiment now, and verify availability in your org before planning a live launch. The winner should complete the customer’s task while preserving the rules your current agent must follow.
The announced beta answers a production question
An Agentforce test suite tells you how an agent behaves on selected scenarios. A production experiment asks whether a specific change improves outcomes for the traffic your customers send. You need both answers before treating a promising revision as the new default.
The announced capability supports comparing variants and reviewing their results inside Builder. Salesforce’s roundup labels it “Beta October,” rather than generally available. That month-level announcement doesn’t establish the exact day a given org receives access, its edition requirements, or the controls your administrator will see.
Avoid building a rollout plan around an invented Setup path. Confirm the current documentation and your org’s feature access. Until then, you can still define the hypothesis, prepare the variants, assemble evaluation cases, and establish the outcome data you’ll need.
Use that preparation to separate an experiment from a migration. Moving every conversation to a newly edited agent gives you a before-and-after chart. It also mixes the change with seasonality, campaigns, staffing, and whatever else happened that week. Comparable cohorts give you a better basis for attribution.
The same distinction matters when Agent Optimizer proposes a revision. A proposed fix is a candidate to evaluate. It hasn’t earned promotion simply because the tool found a failure pattern and generated a plausible correction.
Write a hypothesis about one customer task
Start with a recurring failure that has a traceable outcome. For example, your order-status agent retrieves the right order but gives a vague answer when shipping information is missing. You want the candidate to explain the missing information and choose the approved next step.
The hypothesis could be that a revised instruction increases successful handling of missing shipping information without increasing unsupported delivery promises. That names the task, the intended improvement, and the failure that must remain visible. “Make the agent more helpful” gives you none of those.
Change one thing first. Revise the relevant instruction while keeping the model, action set, retrieval configuration, and escalation policy stable. If you change all of them, a better result leaves you guessing which change mattered and which one you can safely reuse elsewhere.
Keep a record of the candidate’s configuration. Link the instruction change to its agent version and its test results. Your deployment process should let you identify the version behind a conversation and restore the previous behavior when needed.
An experiment also needs a defined scope. Decide whether it concerns a single task, an agent’s whole service channel, or a particular language. A result from straightforward order-status questions doesn’t establish an improvement for refunds, billing disputes, or another language.
Measure completion against an authoritative outcome
Define success before assigning traffic. For an order-status question, success might mean retrieving the correct order, reporting the recorded status, and supplying a permitted next step when tracking is unavailable. The answer should be grounded in the order system, not judged solely by how confidently it reads.
For a refund request, the outcome is different. A refund must meet policy, use the authorized action, and have a recorded result. A friendly response that says the refund is complete while the transaction failed is an unsuccessful session, even if the customer initially accepts the message.
The business system should provide the evidence where possible. Use a recorded update, a verified booking, or another observable completion event. Where success requires human interpretation, write a rubric and review representative conversations. Record uncertain outcomes separately so they don’t silently become successes.
The custom scorers guide is useful for designing additional evaluations. Keep a machine-generated score distinct from the underlying business outcome. A scorer can assess whether an explanation mentions the right condition; it cannot replace a payment system’s record of whether a payment occurred.
Track cost and latency alongside completion. A candidate that finishes more tasks may still be unsuitable if it adds enough waiting time to lose customers or calls expensive actions repeatedly. Use the task’s ROI assumptions to decide whether the improvement pays for its operating cost.
Safety failures belong beside the success metric
Some failures cannot be traded for a higher average completion rate. An agent that exposes another customer’s record or changes a protected value without authorization needs investigation regardless of how many ordinary requests it handles well.
Define those stop conditions explicitly. For your use case, they may include cross-customer data exposure, unsupported financial promises, unauthorized writes, or a refusal to provide the required escalation path. Decide who receives the incident, how traffic returns to the approved version, and what evidence must be retained.
Review the candidate’s guardrails before production traffic reaches it. Validate record scoping and the action’s authorization. An experiment compares permitted variants; it doesn’t give you permission to relax the policy because the relaxed candidate might resolve more conversations.
Handoff deserves its own assessment. A lower handoff rate is useful when more customers complete the task. It is harmful when the agent blocks an appropriate escalation. Measure whether the human handoff occurs when policy or the customer’s request requires it.
Sample apparently successful sessions as well as failures. An unsupported promise can look fluent and produce no technical error. If you review only error traces, you can miss the behavior most likely to mislead a customer.
Comparable cohorts matter more than an attractive chart
Your allocation should follow the feature’s documented controls once available. In the experiment design, aim for cohorts that are comparable on the factors that affect the task. Language, channel, account segment, and request complexity can change the outcome independently of the revision.
Also decide how repeat visits are handled. A customer who receives one policy explanation today and another tomorrow can create inconsistent service and muddy the measurement. Check what assignment behavior the beta supports rather than assuming it offers a particular form of persistent allocation.
Read the results by relevant segment before promoting. A candidate might improve common English questions while degrading the smaller language cohort. An overall average can hide that tradeoff because the largest segment supplies most of the observations.
Set the review period and stopping rule in advance. Don’t end the experiment the moment the chart favors the candidate, and don’t continue it indefinitely because an early result disappointed you. Ask for statistical review when the size or risk of the decision warrants it.
There is no universal number of conversations that proves an agent is better. Required evidence depends on the baseline rate, the improvement you need, traffic variability, and the consequences of getting the decision wrong. Describe those assumptions in the review instead of adding a made-up minimum to your rollout checklist.
Understanding the basics
What is Agentforce A/B Experimentation?
Salesforce announced a capability for comparing agent variants with cohort traffic splits, setup, and an analytics scorecard in Agentforce Builder. Its September 28 announcement lists an October beta. Confirm the feature’s current availability and controls in your org before planning production use.
Does A/B testing replace Agentforce Testing Center?
No. Offline testing checks known scenarios before a candidate meets customers, while a live experiment compares outcomes under production traffic. Run your regression and access checks before assigning customers to either variant.
Which metric should decide the winning agent?
Use a task-completion measure tied to an authoritative outcome, with cost, latency, and safety failures assessed alongside it. A lower handoff rate or a higher generated score can hide unresolved requests. Decide what counts as completion before starting the experiment.
How many conversations does an Agentforce experiment need?
There isn’t one valid sample size for every agent. It depends on the baseline outcome rate, the change you need to detect, and the decision’s risk. Set an evidence threshold and review plan before reading the results.
Promote the change you can explain
At the promotion review, bring the configuration change, the task outcome, the safety findings, and the segment results. A reviewer should be able to understand what improved and why the evidence applies to the traffic that will receive the new version.
Keep the previous version available, assign an owner to watch production behavior, and preserve the experiment’s findings for the next revision. An experiment is useful when it changes a release decision. A scorecard without that decision process is another dashboard to maintain.
Preparing an Agentforce experiment or reviewing an optimizer’s proposed fix? Talk to us. Our Agentforce implementation work connects evaluation to production outcomes; the Agentforce ROI calculator helps you set the cost assumptions behind the experiment.
