Platform
Salesforce Bulk API 2.0: jobs, queries and load failures
Follow the Bulk API 2.0 job lifecycle and compare ingestion and query workflows. Review timeout troubleshooting and when Bulk API 1.0 still matters.
The migration script had run fine against the ten-thousand-row sandbox. Pointed at the eighty-million-row production object, the extract died on the first query with QUERY_TIMEOUT, and the load, written one REST call per record, was on track to exhaust the day’s API allowance somewhere around lunchtime. Neither failure was the data’s fault. Both came from using the wrong channel for the volume.
Bulk API 2.0 is the channel for volume, and the headline is that it does the hard part for you. My rule: anything above roughly two thousand records in a run belongs in a Bulk 2.0 job, and below that you want a synchronous call instead.
Salesforce itself draws the line there. The developer guide recommends Bulk API 2.0 for operations involving more than 2,000 records and synchronous REST or SOAP for smaller sets. Get the volume onto the right meter and the timeouts and the allowance problems both go away.
What 2.0 took off your plate
Bulk API 1.0 made you the batch manager. You created a job, split your data into batches of up to 10,000 records yourself, submitted each, polled each, and stitched the results back together.
Bulk API 2.0 removed that work. You create one job, upload the whole dataset as a single CSV, and Salesforce divides it into batches internally and processes them for you. It also retries failed records automatically and processes in parallel.
Three practical differences follow from that. Bulk 2.0 ingest speaks CSV only, where 1.0 also took XML and JSON. You no longer size or submit batches, so a whole class of tuning simply disappears.
And 2.0 performs PK chunking automatically on large queries, which in 1.0 you had to enable and configure by hand. For the overwhelming majority of loads and extracts, 2.0 is simpler and the right default.
The ingest job lifecycle
An ingest job moves through a small state machine, and knowing it makes the API obvious. You create the job, upload the data, tell Salesforce you are done, and poll for the result.
# 1. Create the job
curl -X POST https://MyDomainName.my.salesforce.com/services/data/v68.0/jobs/ingest \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{ "object": "Contact", "operation": "upsert",
"externalIdFieldName": "External_Id__c", "lineEnding": "LF" }'
# 2. Upload the CSV to the job's contentUrl
curl -X PUT ".../services/data/v68.0/jobs/ingest/$JOB_ID/batches" \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: text/csv" \
--data-binary @contacts.csv
# 3. Close the job so Salesforce starts processing
curl -X PATCH ".../services/data/v68.0/jobs/ingest/$JOB_ID" \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-d '{ "state": "UploadComplete" }'
From UploadComplete the job moves to InProgress and then JobComplete, Failed, or Aborted. When it finishes you pull three result sets from the job: successfulResults, failedResults, and unprocessedRecords.
Those three are the whole story of the run, and a well-built loader reads all three rather than trusting that JobComplete means every row landed. The upsert operation with an externalIdFieldName is the one I reach for most, because it matches on your own key and sidesteps the need to know Salesforce ids in advance.
Note what counts against what. The HTTP calls that create, upload, close, and poll the job are ordinary API requests, but there are only a handful per job. The records themselves draw on a separate allocation. That split is the entire reason bulk relieves pressure on the daily allowance: eighty million rows cost a few API calls plus record-allocation capacity, not eighty million calls.
The record meter, and the line below which bulk is overkill
The volume ceiling is generous. The daily limit is 150 million records across Bulk API and Bulk API 2.0 combined, over a rolling 24 hours. Most orgs never approach it.
The line below which you should not use bulk matters more in daily work. Asynchronous jobs have overhead: you create, upload, close, and poll, and the result comes back when the queue gets to it, not immediately.
For a few hundred records that overhead is pure cost, and a synchronous sObject Collections or Composite call returns the result in one round trip. Two thousand records is the rough crossover Salesforce names. Below it, go synchronous; above it, go bulk.
Query jobs, and why extracts time out
Bulk query jobs, available since v47.0, are the other half of the API and the place the opening story went wrong. A query job runs your SOQL asynchronously and streams the results out in pages. The output is capped at 15 GB, split into 1 GB files you retrieve in sequence by following a locator, so even a very large extract comes back in manageable pieces.
The timeout is the thing to understand. Each attempt to retrieve a chunk of a bulk query has to return within a standard two-minute window, and a query that cannot be processed in time fails with QUERY_TIMEOUT. On a small object you will never see it. On a very large object, a query that has to scan the whole table cannot finish a chunk in two minutes, and that is exactly when you need the next tool.
PK chunking: for the extracts that are huge
PK chunking splits a query on the record’s primary key into chunks that each cover a range of ids, so Salesforce scans a slice of the table at a time instead of all of it. In Bulk 2.0 this happens automatically for large queries. If you are still on a 1.0 query job, you enable it with a header:
Sforce-Enable-PKChunking: chunkSize=250000
Salesforce’s guidance is to use PK chunking on tables with more than 10 million records, or when a bulk query consistently times out. The default chunk size is 100,000 records and the maximum is 250,000.
Two constraints come with it. Each chunk is processed as a separate batch, so a huge object produces a lot of batches against your allocation, and you retrieve each chunk’s results separately. And it only works on straightforward queries: no subqueries, and filtering limited to the kind of WHERE clause that can be chunked on the key.
For the table that reliably times out, PK chunking is the difference between an extract that completes and one that never does.
When Bulk 1.0 still earns its place
For most work, 2.0 wins and it isn’t close. There are two narrow cases where I still reach for 1.0.
The first is lock contention on parallel loads. Bulk 2.0 processes in parallel, which is faster and which also raises the chance of UNABLE_TO_LOCK_ROW when many records in flight touch the same parent, the same owner, or the same shared lookup.
Bulk API 1.0 offers a serial processing mode that runs batches one at a time, trading throughput for the absence of row locks. When a parallel load keeps failing on locking, serial 1.0 is sometimes the pragmatic answer, after you have first tried ordering the data by parent so related records land together.
The second is a query where you want explicit control of the chunk size and batch boundaries rather than the automatic behaviour. That is a specialist need, and if you do not know you have it, you do not have it.
The gotchas that waste an afternoon
A few things bite regardless of version:
- CSV formatting is stricter than a spreadsheet export suggests. Declare the line ending on the job (
LForCRLF) to match your file, or rows fail to parse. - Match on your own key with
upsertand an external id field wherever you can. It makes loads idempotent and reruns safe. - Order your input by the parent record when children share a lookup, to keep related rows in the same internal batch and cut locking.
- Always read
failedResults. A job can reachJobCompletewith a pile of rejected rows sitting in the failure set.
When the answer isn’t bulk at all
Three jobs look like bulk and aren’t. A few hundred related writes belong in the Composite API, synchronous and atomic, not an async job you have to poll. A UI that reads an object and its related records belongs on the GraphQL API, which returns just the fields you ask for and caches them. And a near-real-time feed of changes belongs on the Pub/Sub API with Change Data Capture, which pushes events instead of making you poll and draws on its own event allocation. The budgeting view across all of these, which meter each one spends, is in Salesforce API limits.
What I’d do
Default every load and extract over a couple of thousand records to Bulk API 2.0, let it chunk for you, and build the loader to read all three result sets so a partial failure can’t hide. Reserve PK chunking for the specific extracts that time out on a large object, and reserve Bulk 1.0 for the parallel load that won’t stop locking.
The deeper point is the one the opening story makes. A migration script that works in a small sandbox tells you nothing about how it behaves against a production object a thousand times larger. The channel that was overkill at ten thousand rows is the only one that works at eighty million, and the time to learn which is which is before the cutover, not during it.
Understanding the basics
What is the difference between Bulk API and Bulk API 2.0?
Bulk API 1.0 makes you split data into batches, submit them, and track each one yourself. Bulk API 2.0 takes a single upload and divides it into batches internally, retries failed records automatically, processes in parallel, and performs PK chunking on large queries for you. For most loads and extracts 2.0 is simpler and the better default; 1.0 remains useful mainly for serial processing to avoid row locks.
When should I use Bulk API 2.0 instead of REST?
Use Bulk API 2.0 for operations above roughly 2,000 records, where its record allocation and asynchronous processing avoid both timeouts and burning the daily API request allowance. Below that, a synchronous sObject Collections or Composite call returns results immediately without the overhead of creating, uploading, closing, and polling a job.
What causes QUERY_TIMEOUT in a Bulk API query?
Each attempt to retrieve a chunk of a bulk query must complete within a standard two-minute window. On a very large object, a query that scans the whole table cannot finish in time and fails with QUERY_TIMEOUT. Enabling PK chunking, which splits the query into id ranges, is the standard fix for tables over about 10 million records.
How many records can Bulk API process per day?
The daily limit is 150 million records across Bulk API and Bulk API 2.0 combined, over a rolling 24-hour period. The HTTP calls that manage a job count as a handful of ordinary API requests, but the records draw on this separate allocation, which is why moving volume to bulk relieves pressure on the daily API request limit.
Facing a migration or a nightly sync that keeps timing out or blowing the API allowance? Talk to us. Getting data in and out of Salesforce at volume is what we do.