Agentforce Custom Scorers: grading an agent session against your own definition of resolved, not Salesforce's

Here’s a report you can pull today from Agentforce Observability: your service agent’s average quality score is high, escalations are down, deflection is up and to the right. Here’s the report you can’t pull: of the four thousand sessions it handled last week, how many resolved the customer’s problem, not “responded coherently,” not “avoided escalation,” but closed the loop the agent exists to close. Those are not the same question, and the gap between them is exactly where an agent that looks healthy on the dashboard disappoints the people it talks to.

That gap is what Custom Scorers, new in the Winter ‘27 release (Beta), are built to close. They let you define your own evaluation logic for an agent session, a pass/fail question tied to what your specific use case needs, and, crucially, run it against real production sessions, not just synthetic test cases. This is a different tool from the CI/CD regression tests we’ve written about, and a different tool from the standard quality metrics that ship in the box. Getting the three straight is the whole point of this post, because teams reach for the wrong one constantly and then wonder why their “green” agent is generating complaints.

Three questions, three tools: stop conflating them

Agentforce now gives you three distinct evaluation surfaces, and they answer three different questions. Conflate them and you’ll either over-test the cheap thing or under-measure the expensive one.

Standard quality metrics (Observability). Was the response relevant and helpful? These ship out of the box: per-interaction quality scores, plus session traces showing which topics and actions fired, latency, and proxy outcome signals like escalation, deflection, and abandonment. Salesforce’s own framing is honest about the limit here. Escalation and deflection are proxy signals, not ground truth. A session can deflect (no human touched it) and still have failed the customer completely.
AiEvaluationDefinition tests (Testing Center + CI/CD). Did my change break the routing? These are deterministic, pre-deployment tests: an utterance goes in, and you assert on the expected topic and action sequence. This is the regression gate that belongs in your pipeline, and it’s the single best guard against the topic-misclassification failure that silently sends “where’s my order” into the returns flow. But it runs against synthetic inputs before you ship, and it asserts on mechanics (topic, action) not on whether the whole conversation achieved anything.
Custom Scorers. Did this real session accomplish the outcome I care about? This is the missing middle: an LLM-as-judge that reads an actual session and returns a pass/fail against a criterion you wrote: “did the customer drop off before resolving,” “did the agent quote an entitlement the customer was eligible for,” “did it hand off cleanly when it should have.” It runs on production sessions, and its verdict is domain-specific in a way the built-in metrics structurally can’t be.

The built-in metrics answer “was the response good?” Custom Scorers answer “was the outcome good?” Those diverge exactly where money and trust live.

The subtle thing worth internalizing before any XML: a scorer is a reusable evaluation primitive, not a one-off report. The same scorer object can grade synthetic test cases inside the Testing Center and real sessions inside Observability. What’s new in Winter ‘27 is pointing that judgment at live traffic.

What a scorer is: an engine and an output mapping

A Custom Scorer is a metadata component: the type is AiAgentScorerDefinition, and it lives in the aiAgentScorerDefinitions folder with an .aiAgentScorerDefinition suffix, deployed like any other metadata through the Metadata API. Structurally it has two parts, and keeping them separate in your head makes the whole feature legible:

The engine, the logic that evaluates the session. The documented engine is PromptTemplate: an LLM, driven by a prompt template you author, reads the session and produces a raw result. This is the “judge”. You are literally writing the rubric a model applies to your agent’s own conversation.
The output mapping: the rules that translate the engine’s raw result into an outcome: pass, fail, or not applicable. The engine says “0” or “3 out of 5” or “yes”; the mapping is what decides that “0” means pass and “1” means fail, or that a politeness score below your threshold is a fail.

That two-part shape is why the same primitive is flexible enough for very different questions. The engine can be a blunt binary judge or a graded rubric; the mapping is where you set the bar. Salesforce’s own worked example is a session drop-off scorer: a prompt template inspects the session, decides whether the customer abandoned the conversation before it resolved, and outputs 0 for no drop-off (pass) or 1 for drop-off detected (fail). Simple, but it measures something none of the standard metrics do, a customer who gives up looks identical to a happy customer in a deflection count.

Because the engine is a prompt template, the design discipline is the same one we harp on for grounded prompt templates that don’t drift into fiction: a vague rubric produces a vague, non-reproducible verdict. “Did the agent do a good job” is not a scorer. “Did the agent confirm the order number before initiating a return, and refuse to state a refund amount” is a scorer, because a model can adjudicate it against the transcript.

Writing the scorer

The engine points at a GenAiPromptTemplate, a Prompt Builder template, that you author to ask exactly your question of the session. You can express the criterion in plain language: the documented pattern is something like “Rate the politeness of the agent’s responses on a scale of 0 to 5,” accompanied by descriptions of what each level means, a few example responses, and a pass/fail threshold. The clearer the level definitions and examples, the more stable the verdict, because you’re constraining the judge instead of hoping it shares your standards.

The scorer’s reach over the conversation is governed by its inputScope. At run time inside Observability, the supported scope is Session, the scorer evaluates a whole agent session, which is the right granularity for outcome questions (“did this conversation resolve”) as opposed to per-utterance quality. To reference the most recent turn from inside a session-scoped template, you use the getLastInteraction invocable action, the signal that the judging model is being handed real session context, not evaluating in the abstract.

Conceptually, the definition wires those pieces together: an engine of type PromptTemplate, a reference to the prompt template, an input scope, and the output-mapping rules:

<!-- Illustrative shape of an AiAgentScorerDefinition, verify exact element
     names against the current Metadata API reference for your API version;
     this feature is Beta and the schema is still moving. -->
<AiAgentScorerDefinition xmlns="http://soap.sforce.com/2006/04/metadata">
    <masterLabel>Session Resolved Without Drop-off</masterLabel>
    <engineType>PromptTemplate</engineType>
    <promptTemplate>Session_Dropoff_Judge</promptTemplate>
    <inputScope>Session</inputScope>
    <!-- map the template's raw 0/1 output to a pass/fail outcome -->
    <outputMapping>
        <passValue>0</passValue>
        <failValue>1</failValue>
    </outputMapping>
</AiAgentScorerDefinition>

Treat that block as a mental model, not a copy-paste artifact. Search-level documentation confirms the pieces (the AiAgentScorerDefinition type, the PromptTemplate engine, inputScope, and an output mapping to pass/fail/not-applicable) but the exact element names and the shape of outputMapping are the kind of thing that shifts across a Beta’s release cycle. Author it, then diff it against what the Metadata API returns for your org’s API version, the same way you would with any evolving metadata surface.

Two deployment mechanics are worth knowing before your first push:

Sequential versioning. Scorer versions must be numbered sequentially starting from 1, and each scorer supports a maximum of 100 versions. You iterate on the rubric over time, tightening the criterion as you learn what “resolved” really means for this agent, and each iteration is a version, not a new object.
Deploy the template first. If the engine is a PromptTemplate, the referenced GenAiPromptTemplate must already exist in the org or deploy alongside it, and in your package.xml, list GenAiPromptTemplate before AiAgentScorerDefinition, because the Metadata API deploys in the order you list and the scorer will fail to resolve a template that isn’t there yet.

Running it against real sessions

Once deployed, a scorer can run in two places, and this is where the “reusable primitive” idea pays off. Inside the Testing Center, you can attach custom scorers to test cases; the scorer’s verdict shows up as a new column in the testing grid alongside the built-in evaluations, so a synthetic conversation gets graded by your rubric before you ship. Inside Observability, the same scorer runs against real sessions, and its outcomes surface next to the standard quality metrics, which is the new capability: your own definition of “resolved,” applied to live production traffic at session scale.

That production side has a hard dependency worth stating plainly: Agentforce Observability runs on Data 360. The session traces, the metrics, and the scorer outcomes are collected and stored on the Data Cloud foundation, so if you’re weighing whether you need Data 360 for Agentforce, production-grade agent measurement is another line in the “yes” column. You cannot meaningfully score real sessions without the data layer that captures them.

The honest limitations

Custom Scorers are powerful precisely because the judge is a model, and that’s also the whole caveat, because you are now measuring a non-deterministic system with another non-deterministic system.

A prompt-template scorer is an LLM call per session, and LLM calls aren’t free. No public source breaks out scorer-specific metering, but the mechanism is unambiguous: every scored session invokes a model to render the verdict, on top of the Data 360 consumption Observability already incurs. If you scored every one of a high-volume agent’s sessions, you’d be running a second inference workload the size of the agent’s own. The sane pattern mirrors what we argued for testing in CI, where every test case is also a real, metered invocation: score a representative sample continuously, score everything only for a window when you’re investigating a regression, and treat the credit cost of measurement as a real line item, not an afterthought.

The judge can be wrong, so validate the validator. Before you trust a scorer’s aggregate number, hand-label a few dozen sessions yourself and check that the scorer agrees with you. A rubric that sounds precise to a human can still produce a verdict the model applies inconsistently, the same failure mode that makes agent testing “green once” insufficient applies to the thing doing the grading. A scorer you haven’t calibrated against ground truth is a confident number, not a trustworthy one.

It measures; it doesn’t fix. A drop-off scorer that reads 18% tells you where to look, not what to do. The value is in closing the loop (feeding a bad score back into sharper topic and action design, better grounding, or a cleaner escalation path) and then watching the score move. A scorer nobody acts on is a more expensive dashboard.

It’s Beta. In Winter ‘27 (release notes published August 19, 2026) this is a Beta feature, which in Salesforce terms means the schema, the limits, and the surfaces can shift release to release. Build with it, put real weight on the idea (outcome-level, use-case-specific session scoring) but keep the exact metadata shapes loosely held and re-verify them against the current reference when you go to production.

The takeaway

The reason so many agents look fine on the dashboard and still underdeliver is that the default metrics measure the response, not the result, and “helpful” and “resolved” diverge exactly where it matters. Custom Scorers give you a way to write your own definition of done as a prompt-template AiAgentScorerDefinition, map its verdict to pass/fail/not-applicable, and run it against real sessions on the Data 360 foundation Observability already uses. Keep it distinct in your mind from the two tools it sits between: the deterministic AiEvaluationDefinition tests that gate your deploys, and the standard quality metrics that measure per-response helpfulness. Write scorers that adjudicate something a model can judge from a transcript, calibrate them against hand-labeled sessions before you trust the aggregate, sample rather than score-everything to keep the credit bill sane, and, the part that matters, act on what they tell you. Measured honestly, an agent’s real resolution rate is the number that predicts whether anyone keeps talking to it.

Understanding the basics

What is a Custom Scorer in Agentforce?

A Custom Scorer is a Winter ‘27 (Beta) metadata component, type AiAgentScorerDefinition, that evaluates an Agentforce agent session against evaluation logic you define, and returns an outcome of pass, fail, or not applicable. It has two parts: an engine (the documented one is PromptTemplate, an LLM-as-judge driven by a Prompt Builder template you author) and an output mapping that translates the engine’s raw result into the pass/fail outcome. Unlike the standard quality metrics that ship with Observability, a Custom Scorer answers a domain-specific, use-case-defined question, for example, whether a customer dropped off before the conversation resolved.

How are Custom Scorers different from AiEvaluationDefinition tests?

They answer different questions and run at different times. AiEvaluationDefinition tests are deterministic, pre-deployment regression tests: you supply a synthetic utterance and assert on the expected topic and action sequence, then run them in CI to catch a change that breaks routing. Custom Scorers are LLM-as-judge evaluations that grade real production sessions against an outcome criterion you write, and surface the verdict in Observability alongside the standard metrics. The same scorer object can also grade synthetic test cases inside the Testing Center, but the new Winter ‘27 capability is scoring live sessions. Use test definitions to protect the deploy; use scorers to measure the outcome.

Do Custom Scorers require Data Cloud, and do they cost credits?

Yes to Data Cloud: Agentforce Observability, where scorers run against production sessions, is built on Data 360, so the session data and scorer outcomes are collected and stored there. On cost, there’s no published scorer-specific credit figure, but a PromptTemplate scorer makes an LLM call to render each verdict, so scoring high volumes of sessions is a real, metered inference workload on top of the Data Cloud consumption Observability already incurs. The practical pattern is to score a representative sample continuously rather than every session, and reserve full-coverage scoring for periods when you’re actively investigating.

Standing up Agentforce Observability and trying to measure whether your agent resolves what it’s supposed to, not just whether it sounds helpful? Talk to us. Building the scorers, the CI regression gates, and the tuning loop that keeps an agent trustworthy after launch is exactly the work we do.