Measuring Data 360 retriever quality: how to know your grounding works

The agent is grounded on Data 360 and it still gives a thin, slightly-wrong answer.

The usual next move is to re-tune: smaller chunks, a different fusion ranker, a higher result count. Sometimes that helps. More often it is guessing, because nobody has established whether the retriever fetched the right paragraph in the first place or the model fumbled a paragraph it did have. Those are two different failures with two different fixes, and you can’t tell them apart by reading answers.

So measure them apart. Score the retriever on whether it pulled the relevant chunks, score the model on whether it stuck to the chunks it got, and read those two numbers before you touch a setting.

A wrong grounded answer is usually a retrieval miss wearing a generation costume. Data 360 and Agentforce now ship the surfaces to prove which one you have, and this is how to use them.

The two failures, and the scores that catch each

A retrieval-augmented answer passes through two stages. The retriever searches your index and returns a set of chunks. The model reads those chunks and writes an answer. Each stage fails in its own way, and the evaluation literature has clean names for both.

For the retriever, two scores matter:

• Context precision: of the chunks the retriever returned, what proportion were relevant. Low precision means the model is reading noise and being asked to find the signal in it.
• Context recall: of all the chunks that were relevant in your corpus, what proportion the retriever found. Low recall means the answer was doomed before the model saw a token, because the evidence never arrived.

For the model, two more:

• Faithfulness (groundedness): is every claim in the answer supported by the retrieved chunks, or did the model add something from its own weights.
• Answer relevancy: does the answer address the question that was asked, which a factually correct but off-target answer can still fail.

The split is the whole point. Context precision and recall grade the retriever; faithfulness and answer relevancy grade the generation on top of it.

When answers are bad and context recall is low, you have a retrieval problem, and re-tuning the prompt is wasted effort. When recall is high but faithfulness is low, the retriever did its job and the model is going off-script. One table of four numbers ends most of the re-tuning arguments.

What Data 360 reports natively

You no longer have to compute all of this by hand. Two native surfaces now carry retrieval quality.

The first is a data model object. Data 360 added AI Retriever Quality Metric (AiRetrieverQualityMetricDmo), available from version 254, which stores per-request scores for grounded calls.

It carries a Context Precision Score and an Answer Relevancy Score, plus reference IDs that tie each row back to the underlying gateway request, retriever request, and response generation. Because it is a DMO, you can query it like any other Data 360 object, trend precision over time, or filter to the retriever that is dragging an agent down.

The second is Agentforce analytics quality scores, the dashboard view. It reports an overall Quality Score on a 1-to-5 scale, and three dimension scores on a 0-to-1 scale: answer faithfulness (adherence to the grounded source), answer relevance (fit to the question), and context relevance (whether the agent retrieved the correct source at all).

An external LLM acts as the judge that assigns them. To get these, an admin turns on Audit and Feedback and the knowledge-and-RAG quality metrics in Setup first, so budget that step before you expect the dashboard to populate.

A caution worth stating, since the two surfaces use overlapping but non-identical names. The DMO’s “context precision” and the dashboard’s “context relevance” are not guaranteed to be the same calculation, and I wouldn’t treat a number from one as interchangeable with the other.

Use the dashboard for a fast read on live agents, use the DMO when you want the raw per-request data to analyze yourself, and keep track of which score you are quoting.

Build a golden set, or you are still guessing

The native scores tell you that quality dropped. A golden set tells you why, and it is the one piece you have to build yourself.

A golden set is a fixed list of test cases. Each case is a real user question, the ideal answer written by someone who knows the domain, and, the part people skip, the IDs of the specific chunks that contain the evidence. That chunk-ID field is what separates a retrieval miss from a generation miss: you can check whether the retriever returned the right chunks independently of whether the model used them well.

The workflow is unglamorous and it works:

1. Write 50 to 100 cases that mirror what users really ask, including the awkward ones.
2. Run each question through the live retriever and capture the chunks it returned and the answer the model produced.
3. Compare returned chunk IDs against the expected ones for a retrieval score, and the answer against the ideal for a generation score.
4. Read the worst cases by hand before you trust any aggregate. A broken prompt template or an empty retriever return shows up in thirty seconds of reading and can hide behind a decent-looking average.

You can pull the pieces from Data 360 through the Query API and, for the generation side, lean on the same testing harness you already run for agents. If you have written custom scorers, a pass/fail scorer that checks “did the expected chunk appear in the retrieved set” is a natural addition and runs against real sessions.

One honest limit of golden sets: they are biased toward questions your corpus can answer. The questions users ask that your data simply doesn’t cover won’t be in the set, because you wrote it from the corpus you have. Keep a separate bucket of real “we had nothing good for this” questions, because that gap is a content problem no retriever setting will fix.

What moved the numbers, from Salesforce’s own test

When Salesforce’s data-science team wrote up a Data 360 RAG pipeline, they reported moving accuracy from 62.5% to 92.5% on knowledge-base question answering, and reaching 86.8% end to end on complex PDFs full of tables and charts, with no custom code and no model training. The gains came from pipeline configuration rather than a bigger model, which matches what the scores above tend to show in practice.

Three changes did most of the work, and all three are things you can measure the effect of with a golden set:

• Intelligent parsing of rich documents, so tables stay intact and section hierarchy survives chunking instead of being shredded into unusable fragments. This is the ingestion half of the Intelligent Context story, and it mostly moves context recall.
• Hybrid search, combining vector similarity with keyword matching so exact terms like SKUs and brand names rank correctly. This mostly moves context precision, and the fusion-ranking knobs are where you tune it.
• Cross-encoder reranking, a second-pass model that re-scores each query-and-chunk pair before the top few reach the LLM. Salesforce describes it as the highest-return upgrade after hybrid search, and it almost entirely targets precision.

The useful discipline is to make one of these changes at a time and watch which score moves. If reranking lifts precision but recall stays flat and answers are still incomplete, your problem was never ranking, it was that the evidence isn’t being retrieved at all, and you look upstream at parsing and chunking.

Measuring is not free

Running an evaluation draws the same consumption as serving real traffic, so size it before you automate it. Hybrid search costs noticeably more per query than plain vector search, with practitioner analyses putting it at roughly double, and firing a 100-case golden set through a hybrid retriever several times a day adds up. The LLM-judged quality scores add generation calls on top of the retrieval cost, because an evaluator model has to read every answer.

None of that is a reason to skip measurement, it is a reason to run the full golden set on a schedule rather than on every change, and to keep an eye on the credit side of Data 360 the same way you would for production load. A nightly evaluation run you can afford beats a real-time one you turn off after the first invoice.

What to do

Turn on the quality metrics in Setup and let the dashboard tell you, over a week of real traffic, whether context relevance or faithfulness is the weaker number. That one reading points you at the retriever or the model and saves a round of blind tuning.

In parallel, write a small golden set with chunk IDs, because the dashboard tells you something is wrong and the golden set tells you where. Then change one thing, parsing or hybrid search or reranking, and confirm the matching score moved before you keep it.

Grounding quality stops being a matter of opinion the moment you score the retriever and the model as two separate things.

Understanding the basics

How do I measure RAG quality in Salesforce Data 360?

Score the retriever and the model separately. For the retriever, use context precision (how much of what it returned was relevant) and context recall (how much of the relevant material it found). For the model, use faithfulness (is the answer supported by the retrieved chunks) and answer relevancy (does it address the question). Data 360 reports some of these natively through the AI Retriever Quality Metric object and the Agentforce analytics quality scores, and you supplement them with a golden set you build yourself.

What is the AI Retriever Quality Metric in Data 360?

It is a data model object (AiRetrieverQualityMetricDmo), available from Data 360 version 254, that stores per-request retrieval quality scores for grounded calls, including a Context Precision Score and an Answer Relevancy Score, with reference IDs linking each row to the gateway request, retriever request, and response generation. Because it is a DMO you can query and trend it like any other Data 360 object.

Why does my grounded agent still give wrong answers?

Usually one of two reasons, and they need different fixes. Either the retriever didn’t fetch the right chunks (low context recall or precision), which you fix with parsing, chunking, and search configuration, or it did fetch them and the model ignored or embellished them (low faithfulness), which you fix with the prompt and model. Measuring both scores tells you which, instead of re-tuning the half that was already working.

Does evaluating retrieval quality cost credits?

Yes. An evaluation query consumes the same way production retrieval does, and hybrid search costs more per query than vector search, with third-party analyses putting it at roughly double. LLM-judged quality scores add generation calls on top. Run a fixed golden set on a schedule rather than on every change, and verify current consumption rates against Salesforce’s billing documentation before you automate it.

---

Grounding an agent and want to know whether it is retrieving the right thing before it meets a customer? Talk to us, or start with the Agentforce readiness assessment. Getting retrieval measured and trustworthy is the work that decides whether an agent helps or hallucinates.