Just migrated from Segment to a "modern" CDP last quarter. The sales deck promised seamless historical backfill. Now, trying to push three years of user events through their Batch API feels like trying to empty a reservoir through a drinking straw.
Their rate limit is 500 requests per minute, with a max of 10,000 events per request. Sounds reasonable until you do the math.
- 500M historical events / 10k per request = 50,000 API calls needed.
- At 500 calls/minute, that's 100 minutes of ideal, sustained throughput.
- Reality: their API starts throwing 429s after 2 minutes, their retry-after header is 60 seconds, and their SLA for batch jobs is… non-existent.
My current script, which handles batching and naive retries, is getting us nowhere.
```python
# The "this will take a year" implementation
for batch in event_batches:
while True:
try:
client.ingest(batch)
time.sleep(0.12) # Aim for 500/min
break
except RateLimitError as e:
time.sleep(int(e.headers.get('Retry-After', 60)))
```
So, the postmortem question before the incident: has anyone actually survived a backfill of this scale with a tier-1 CDP?
**Specifics I'm auditing:**
* Did you negotiate a temporary limit lift? (Their support just sent me the public docs.)
* Did you shard by API key or use multiple destination IDs?
* Is the real play to load the data into S3 and use their "magic" import, which probably just calls the same throttled API?
I refuse to believe the only answer is "just backfill less data." The compliance team would have a field day with that one.
- Nina
- Nina