Data streams in Data 360: the category choice you cannot undo, and the refresh mode that silently drops deletes

Every Data 360 project starts the same way: something has to get into it. Before you can resolve an identity, build a segment, or ground an agent, the customer data sitting in your warehouse, your object store, your marketing platform, and your home org has to arrive as something Data 360 can model. The mechanism is the data stream, and the setup looks reassuringly like a wizard: choose a source, pick some fields, map them, click through. Most of it is that easy.

But two of those clicks are load-bearing in a way the interface never flags. One of them, the stream’s category, is effectively permanent; get it wrong and your only fix is to delete the stream and start over. The other, the refresh mode, decides whether records deleted in the source ever disappear from your unified profile, or linger forever as ghosts your segments keep counting. Teams discover both of these the expensive way, in production, months after the wizard closed.

This post is the working map of ingestion: the connector landscape and how to choose among copying data in versus federating it in place, the DLO-to-DMO path every stream produces, and, the part nobody documents as a warning, the two decisions that are hard to walk back. One naming note first: Salesforce rebranded Data Cloud to Data 360 in October 2025; the platform, objects, and APIs are unchanged, and older docs and endpoints still say “Data Cloud.” Read them as synonyms. We wrote the full rebrand explainer if you want the history.

Four ways in, and the first question is whether to copy at all

“Ingestion” is a slightly misleading word, because the highest-use decision is often not to ingest. Before choosing a connector, decide which of these four shapes your source is:

Batch connectors, scheduled pulls from cloud object storage and SaaS sources. Data 360 ships out-of-the-box connectors for Amazon S3, Google Cloud Storage, SFTP, Marketing Cloud Engagement, the Salesforce CRM home-org connector, and a growing catalog of advertising and application sources. This is the workhorse for large, periodic loads: files land on a schedule and become rows.
The Ingestion API, a programmatic push for data that doesn’t sit in a file, covered in depth in the Ingestion API guide. It has a bulk path for large loads and a streaming path for near-real-time micro-batches: the route for event data and app telemetry.
Zero-copy federation, don’t move the data at all. Point Data 360 at tables living in Snowflake, BigQuery, Redshift, or Databricks and query them in place, as zero-copy explains. No stream, no refresh schedule, no second copy to drift.
Connected app streams from other Salesforce clouds. Marketing Cloud, Commerce, and the like, via their managed connectors.

The rule of thumb: if the data already lives in a modern warehouse and you mostly need to read it, federate; if you need it resident for high-frequency use, low-latency identity resolution, or you’re pulling from files and APIs, ingest. Copying has a cost in credits and in freshness lag; federation trades some query performance and adds a dependency on the source being up. The mistake is defaulting to “copy everything in” because the wizard makes it easy, a billion-row event table you query occasionally is a zero-copy candidate, not a data stream.

Everything below is about that ingest path, the streams themselves.

Every stream produces a DLO, then you map it up

A data stream doesn’t land data directly into your unified model. It creates a Data Lake Object (DLO): the raw, ingested table, faithful to the source. You then map that DLO to a Data Model Object (DMO), the governed, semantic layer the rest of the platform uses: segmentation, retrievers, identity resolution, activation. If the difference between those two layers isn’t crisp for you, data lake versus data model objects is the post to read first, because the mapping step is where most modeling mistakes get baked in.

The shape to keep in your head:

Source (S3 / API / warehouse)
   │  data stream
   ▼
Data Lake Object (DLO)      ← raw, source-faithful
   │  mapping (+ optional transforms)
   ▼
Data Model Object (DMO)     ← governed, unified, queryable
   │
   ▼
Identity resolution · Segments · Retrievers · Activation

The map step is also where a batch or streaming transform can reshape, clean, or derive fields between the DLO and the DMO (normalize a phone format, drop a junk column, compute a field) so you’re not resolving identities on top of dirty source data.

Decision one: the category you can’t change

When you create a stream, Data 360 asks for a category, and this is the click that’s permanent. There are three, and they are not interchangeable:

Profile: descriptive data about a person or account: names, contact details, demographics, CRM identifiers. The stuff that describes who someone is. Profile data is what identity resolution matches on.
Engagement: time-series records of things that happened: opens, clicks, purchases, page views, app events, service interactions. Engagement streams require a date-time (event time) field, because the whole point of the category is that each row is anchored to a moment.
Other: mixed, mutable, or reference data that fits neither cleanly: product catalogs, store lists, mutable records that get updated in place, IoT-ish data with shifting timestamps.

Here’s the trap, stated plainly: once a stream’s category is set, you cannot change it. If you ingest an events table as Profile and later realize segmentation and calculated insights want it as Engagement (with all the time-series behavior that unlocks), there is no toggle. You delete the stream and rebuild it, along with anything downstream that depended on the DLO. The category isn’t a label; it changes how Data 360 treats the data. So the five minutes you spend deciding category before you click create is the cheapest five minutes in the whole project.

The practical test: is each row a description (Profile), an event at a time (Engagement), or neither (Other)? A “customers” table is Profile. A “transactions” table is Engagement. A “product master” is Other. When a table has both, a purchase with a mutable “last updated”, that mutability usually pushes it to Other rather than Engagement.

Decision two: the refresh mode that decides whether deletes propagate

The second load-bearing choice is the refresh mode. How each new load reconciles against what’s already there. The two you’ll choose between behave very differently at the one moment that matters: when a record is removed from the source.

Full Refresh replaces the dataset each run. Existing data is cleared and rebuilt from the latest load, so the DLO always mirrors the source exactly, including deletions. If a record vanishes from the source file, it vanishes from Data 360. The cost is that you’re moving the whole dataset every time, which is heavier and more expensive on large tables.
Upsert merges each load by primary key: matching keys update in place, new keys insert. It’s efficient, you ship only what changed, and it’s the sensible default for large, frequently updated sources. But it has a sharp edge: upsert never deletes. A record removed from the source stays in Data 360 forever, because upsert only ever inserts or updates; it has no signal for “this is gone.”

That asymmetry is the bug teams don’t see coming. You choose upsert for efficiency, everything works, and six months later your active-customer segment is inflated with people who churned and were deleted upstream, because those deletes never propagated. Meanwhile a calculated insight counts them, an agent grounds on them, and a report overstates the book. Nothing errored; the data just went stale in a direction you weren’t watching.

The design rule: use upsert for scale, but have an explicit plan for deletions. That plan might be periodic full refreshes to reconcile, a soft-delete flag in the source that upsert can carry (a status column the row updates to “deleted”, which your transforms and segments then respect), or, for anything governed by right-to-be-forgotten obligations, a deliberate deletion process rather than a hope that upsert handles it. It does not. Treat “how do deletes reach Data 360?” as a question you answer at design time, per stream, not one you discover in an audit.

The limits that shape the design

Ingestion is metered and bounded, and a few numbers drive real architecture decisions. These are the documented figures at the time of writing; because platform limits move, verify the current ones against Salesforce documentation before you build to them:

Streaming Ingestion API payloads are capped around 200 KB per call and processed asynchronously (on the order of every few minutes, not instantly) with a short additional lag (roughly 30 seconds) after processing before the data is queryable. “Streaming” here means near-real-time micro-batches, not sub-second. If you need true instant reflection, that expectation needs managing.
Batch file loads are sized for volume, individual files up to very large sizes (documented in the hundreds of GB range) and up to on the order of a thousand files per scheduled run for object-store connectors, which is exactly why batch is the right tool for the big periodic loads and the Ingestion API’s streaming path is the right tool for the small frequent ones.
SFTP ingestion of CSVs is bounded per stream (documented around 4.5 GB), which matters if you’re piping large extracts over it rather than S3.
Every ingested row and every refresh is compute, and compute is metered against the consumption model. A stream set to full-refresh a huge table every hour when the data changes daily is burning credits for nothing. Refresh cadence is a cost lever, not a “set it as fast as possible” setting.

The through-line: match the ingestion method to the data’s size and velocity. Big and periodic → batch connector. Small and frequent → streaming Ingestion API. Enormous and query-occasionally → don’t ingest, federate.

After the stream: monitor it like a pipeline

A data stream is not fire-and-forget. Sources change schemas, files arrive malformed, a refresh fails silently, and the first sign is usually a segment count that looks wrong. Treat every stream as a monitored pipeline: watch run status, row counts, and failures through Data 360 pipeline monitoring, and alert on the stream that stopped rather than finding out when a stakeholder asks why the numbers moved. The unified profile is only as trustworthy as the least-watched stream feeding it, and once identity resolution is matching on that data, a broken stream isn’t one bad table, it’s a distorted view of every customer it touches.

Takeaways

Decide copy-versus-federate first. If the data lives in a warehouse and you mostly read it, zero-copy beats a data stream. Ingest when you need the data resident, low-latency, or it comes from files and APIs.
Category is permanent. Profile (descriptions), Engagement (time-stamped events. Requires an event-time field), or Other (mixed/mutable). You cannot change it later; a wrong choice means delete and rebuild. Spend the five minutes up front.
Refresh mode decides deletions. Full Refresh mirrors the source including deletes but moves everything each run; Upsert is efficient but never deletes. Choose upsert for scale only with an explicit plan for how removed records leave Data 360.
Match method to size and velocity. Batch connectors for big periodic loads; the streaming Ingestion API (≈200 KB payloads, near-real-time micro-batches) for small frequent ones; verify current limits before building to them.
Every stream lands a DLO you map to a DMO, and that mapping, with optional transforms, is where clean, governed data gets made. Monitor every stream like the pipeline it is.

Getting data into Data 360 is mostly a wizard, and mostly easy. The two clicks that aren’t, the category you can’t undo and the refresh mode that silently drops or keeps your deletes, are worth slowing down for, because they’re the ones you’ll otherwise be unwinding in production long after everyone forgot which box got checked.

Understanding the basics

What is a data stream in Data 360?

A data stream is the configured pipeline that brings data from a source (cloud storage, SFTP, an API, another Salesforce cloud, or the CRM home org) into Data 360. Each stream lands its data as a Data Lake Object (DLO), the raw source-faithful table, which you then map to a Data Model Object (DMO), the governed layer used for identity resolution, segmentation, retrievers, and activation. A stream carries a permanent category (Profile, Engagement, or Other) and a refresh mode that controls how each load reconciles with existing data.

What are the three data stream categories, and can I change one later?

Profile is descriptive data about people or accounts (used for identity matching); Engagement is time-series event data and requires a date-time field; Other is mixed, mutable, or reference data that fits neither. The category cannot be changed after the stream is created, if you chose wrong, you delete the stream and rebuild it, along with anything downstream that depended on its DLO. Because the category changes how Data 360 treats the data, it’s worth deciding deliberately before you create the stream.

What’s the difference between Full Refresh and Upsert refresh modes?

Full Refresh replaces the whole dataset each run, so the data always mirrors the source exactly, including deletions, at the cost of moving everything every time. Upsert merges by primary key (update matches, insert new) and only ships what changed, which is efficient for large frequently updated sources, but it never removes records: anything deleted in the source stays in Data 360 indefinitely. Choose upsert for scale only if you have an explicit plan (periodic full refreshes, a soft-delete flag, or a deliberate deletion process) for how removed records leave the system.

When should I use zero-copy instead of a data stream?

When the data already lives in a supported warehouse (Snowflake, BigQuery, Redshift, Databricks), you mostly need to read it, and copying it in would add cost, freshness lag, and a second copy to keep in sync. Zero-copy federation queries those tables in place, no stream, no refresh schedule. Ingest with a data stream instead when the data comes from files or APIs, needs to be resident for low-latency identity resolution or high-frequency use, or isn’t in a federatable source.

Standing up Data 360 and trying to get the ingestion layer right: copy versus federate, the category and refresh decisions, and a data model that won’t need unwinding later? Talk to us. The foundation is the part that decides whether everything above it is trustworthy.