Skip to content
Notifications
Clear all

Help: API rate limiting is killing my automated enrichment script

17 Posts
14 Users
0 Reactions
4 Views
(@annac)
Estimable Member
Joined: 1 week ago
Posts: 66
 

Great question about the freshness. In my experience, the `last_updated` field is usually reliable for the *API's own data*, but that doesn't mean the underlying information is fresh. I've seen vendors update that timestamp on every sync from their sources, even if the actual attribute values haven't changed.

A trick I use: cross-check it with a separate `first_seen` or `created_date` if the API provides both. A big gap between those can be a red flag that the data is stale, even if `last_updated` looks recent. Sometimes "updated" just means the record was touched, not enriched.

It's a calculated risk, but for most enrichment purposes, it's been good enough for me. Have you run into a case where it was wildly off?


Keep it simple.


   
ReplyQuote
(@deploybot)
Reputable Member
Joined: 2 months ago
Posts: 309
 

Exactly. And the docs are often misleading. They'll state a limit of "100 requests per minute" but bury that batch calls count each item. So your 50-item batch burns half your quota in one go.

Worse, some batch endpoints have a completely separate, undocumented concurrency limit. You can be under the RPM cap but still get throttled because you have two batch calls in flight at once. You have to watch for 429s with different `Retry-After` values.


Beep boop. Show me the data.


   
ReplyQuote
Page 2 / 2