Data 360
Ingesting files into Data 360 from Amazon S3: the connector and the auth choice
Your data lands as CSV or Parquet in an S3 bucket, and you need it in Data 360 to ground an agent or build a segment. The connector is a wizard, but two decisions inside it, which way you authenticate and whether you copy at all, are the ones you live with. Here is the setup, the IAM-role path most teams skip, and the deletes gotcha nobody flags.
Your warehouse, your billing system, or a partner drops files into an S3 bucket every night. Clean CSVs, or Parquet if someone upstream was thoughtful. You need that data in Data 360 so an agent can ground on it or a segment can target from it. The S3 connector is how you do it, and most of the setup is a wizard you finish in twenty minutes.
Two choices inside that wizard are not twenty-minute decisions. The first is how Data 360 authenticates to your bucket, where the quick option and the right option are different. The second is whether you should be copying these files into Data 360 at all, or federating them in place. Get those two right and the rest is clicking through a form.
My default: use the IAM-role path, not an access key, for anything that outlives a prototype, and copy with the connector only when you need the rows materialized in Data 360. If you need the data queryable but not duplicated, this is the wrong tool.
Copy or federate: decide before you open the wizard
The S3 connector copies. It reads your files and materializes the rows as a Data Lake Object inside Data 360, on a batch schedule.
Copying is what you want when the files are the system of record for data that does not already live in a queryable warehouse, or when you need the rows local for low-latency grounding and segmentation.
The connector is the wrong choice when the data already sits in a lakehouse you could query in place. If these files are an export of tables in Snowflake, BigQuery, Databricks, or Redshift, look at zero-copy federation first, which queries the source without duplicating it and without paying to store a second copy. If the files are Parquet managed as an open table, Apache Iceberg file federation reaches them where they are. The Salesforce Connect versus zero-copy decision applies to files too: copy when you must, virtualize when you can.
The S3 connector earns its place when the answer is “these files are the source and I need the rows in Data 360”. The rest of this is for that case.
Step one: the connection, and the permission you need to make it
Connections live in Data Cloud Setup under External Integrations. Open More Connectors, choose Amazon S3 as the source, and give the connection a name and an API name. You supply the bucket name and a parent directory, pick an authentication method, run Test Connection, and save. Once it validates, you can create data streams for any objects under that parent directory.
The gotcha here is not technical, it is access. Creating a connection requires the System Administrator profile or the Data Cloud Architect permission set. If a data engineer without those rights is doing the setup, they will get partway in and stall, so sort the permission before the session, not during it.
One more thing to check on the AWS side: if the bucket restricts access by IP, the test will fail until you add the Data Cloud IP addresses to the bucket’s allowlist. A connection that tests fine from an admin’s laptop but refuses to run on schedule is often this.
Step two: the auth path, where the quick way is not the keeping way
Data 360 gives you two ways to authenticate to the bucket, and the difference matters more than the docs let on.
The quick way is an AWS access key and secret key. You map the AWS Access Key field to the programmatic username and the Secret Key to its password, for an IAM user that has S3 read permissions, where a read-only S3 policy is the usual grant.
It works immediately, which is exactly why it ends up in production and sits there. A long-lived key pair stored in a connection is a standing credential someone has to rotate and nobody will, fine for a proof of concept and a liability as a permanent arrangement.
The way to keep is an IAM role through an identity provider. In AWS you add an OpenID Connect identity provider that uses your Salesforce My Domain URL as the provider URL and audience, then create a role whose trust policy lets Data 360 assume it, keyed on an External ID you enter in the trust policy before testing the connection.
No static secret lives in Salesforce; Data 360 assumes the role to get short-lived credentials. For anything carrying real data, that is the posture you want.
One constraint decides it for some teams: IdP-based authentication is only supported for structured data. If this same connection also has to pull unstructured files, you will be back on the access-key path for that part, which is its own argument for keeping structured and unstructured ingestion on separate connections.
Step three: the data stream, the formats, and the sizes
With the connection saved, create the stream. Under Data Streams choose New, select the S3 connector, and pick a file type of CSV or Parquet and an optional Import from Directory path. Wildcards work, so *orders*.csv matches every file with “orders” in the name, which is how you point one stream at a rolling set of daily drops. To pull everything already sitting in the folder, run a one-time Refresh Now and choose Refresh All the Files; after that the stream picks up new files on its schedule.
The formats have sharp edges. Data 360 accepts delimited text such as CSV, and Parquet from S3, Google Cloud Storage, Azure Storage, and SFTP.
With Parquet there is a silent failure mode: if a column uses a data type Data 360 does not support, that column is ignored and not ingested. The stream succeeds, the row count looks right, and a field you needed is simply absent. Diff the fields you expected against the fields that landed, especially on the first run.
Size guidance is published as optimal, not as hard caps. The limits and guidelines page puts the optimal size per CSV file at 500 MB and per Parquet file at 50 GB, and it warns that the maximum file size and the maximum files per scheduled run are not meant to be pushed to their limits at the same time. A useful trick the docs call out: create the stream against a small sample file, same headers, around a thousand rows, so schema detection is fast, then drop the real, larger files into the path once the stream exists.
Refresh runs as a batch job. Older Salesforce guidance describes scheduling it as often as hourly or as infrequently as monthly. Hourly is the floor, so if your grounding needs fresher than that, the file connector is not the mechanism, and you want streaming ingestion or an External Object instead.
The deletes gotcha the wizard never mentions
The connector inherits the same refresh-mode trap as every other data stream in Data 360, and it is the single most common way a file-fed stream goes wrong months later.
Upsert refresh is efficient and never deletes. A record that disappears from your source files stays in Data 360 forever, because upsert only inserts and updates and has no signal for “this row is gone”.
Choose it for scale, which you should, and your churned customers, cancelled orders, and retired SKUs accumulate, inflating every segment and calculated insight built on top. Nothing errors. The numbers just drift in a direction no one is watching.
So decide up front how deletions reach Data 360 for this stream. The options are a periodic full refresh to reconcile, a soft-delete flag your source carries so upsert can update a row to “deleted”, or a deliberate deletion process for anything under right-to-be-forgotten obligations. Pick one at design time, not during an audit.
What about PDFs and unstructured files
The S3 connector described above is for structured CSV and Parquet. If the files are PDFs or other unstructured content, the path is different and worth not conflating: you create an unstructured data lake object, set up file notifications, and authenticate with OAuth using an RSA key pair and an x509 certificate, then let the blob store feed the UDLO. The full unstructured ingestion pipeline has its own post. The thing to carry over here is that it is a separate connection and a separate auth story, and that IdP role auth does not cover it.
After it runs: watch the stream, not the dashboard
A file stream is not fire-and-forget. Source schemas change, a nightly export arrives malformed, a run fails silently, and the first anyone notices is a segment count that looks wrong. Treat every stream as a monitored pipeline. Watch run status and row counts through pipeline monitoring, and when rows vanish on ingestion, Problem Records is where the rejected rows went and how you get them back.
The connector is easy. The two decisions around it, the IAM role over the standing key, and copy only when federation will not do, are the ones that keep the pipeline cheap, secure, and trustworthy a year from now. Spend your attention there, and the wizard is just a wizard.
Understanding the basics
Should I use the S3 connector or zero-copy federation?
Use the connector when the files are the source of record and you need the rows materialized in Data 360 for low-latency grounding or segmentation. Use zero-copy federation when the data already lives in a warehouse like Snowflake, BigQuery, Databricks, or Redshift and you would rather query it in place than pay to store a second copy. Copying is the exception, not the default.
Access key or IAM role for authenticating to the bucket?
Use an IAM role through an OpenID Connect identity provider for anything beyond a prototype, so no long-lived secret sits in a Salesforce connection. The access-key path is quicker to stand up but leaves a standing credential nobody rotates. IdP role authentication supports structured data only; unstructured files still need the key-based flow.
What file formats and sizes does the S3 connector support?
CSV and Parquet. The published optimal sizes are 500 MB per CSV file and 50 GB per Parquet file, and you are advised not to push maximum file size and maximum files per run to their limits at once. Parquet columns whose data type Data 360 does not support are silently dropped, so verify the landed schema against what you expected.
How often can the S3 connector refresh?
It runs as a scheduled batch job, as frequently as hourly. If you need data fresher than an hour old, the file connector is the wrong mechanism; look at streaming ingestion for pushed data, or an External Object or live action for a single current fact.
Getting files into Data 360 reliably is the unglamorous foundation every agent and segment rests on. If you want the connection, the IAM trust policy, and the refresh strategy set up so they hold under real volume, talk to us.
