What's the best practice for retrying failed API calls without double counting?

I work as a tech lead at a 100-person SaaS company; we handle a high volume of B2B payments and lead syncs. We've tested several patterns on our Pipedrive/Stripe/Marketo stack and now run idempotency keys backed by Redis.

**Key Considerations**

* **Implementation Complexity**: The simple database flag takes maybe 2-3 days to add, audit, and test. A dedicated queuing system like RabbitMQ or Temporal adds 2-4 weeks of dev and ops overhead.
* **Runtime Cost & Scale**: Idempotency keys with a fast cache (Redis) cost us about $120/month on AWS, handling about 800 req/s at peak. A database flag for everything can spike your transactional DB costs if you're not careful.
* **Failure Case Handling**: A separate queue is best for "at-least-once" delivery and complex retry delays, but it introduces a new service to monitor. Idempotency keys are simpler but only work if the same caller sends the same key on retries (network timeouts can break this).
* **Integration & Vendor Fit**: For third-party webhooks (like Stripe), you must implement your own idempotent receiver; they send an idempotency key you must store. For internal services, a queue is easier to sell to other teams than enforcing a key protocol.

My pick is idempotency keys with Redis for most external API calls, especially payments and CRM writes. It's the fastest to get right. If you have very complex, multi-step workflows where order matters, then you need a queue. Tell us if you control both sides of the API and the average retry delay you need.

You're absolutely right that the theoretical advice falls apart when a payment webhook fires three times. I've seen teams implement idempotency keys but then store them in their primary database, which just moves the race condition. The key is making the idempotency check a single, atomic operation.

For your example of a payment webhook, the only method I've seen work at scale is an idempotency key sent *by the originating system* (like Stripe) stored in a distributed lock service, not just a database. When you receive webhook attempt #2, the check against Redis or even a dedicated key table needs to happen before *any* business logic runs, and the lock must be held until the entire transaction is committed. A simple flag on the record often fails because the check and the update aren't atomic, letting two threads through.

The separate queuing system is overkill for most, but becomes necessary when the retry itself is a complex, multi-step process that could also fail partway.

Totally agree on the atomic check being the critical bit. I've seen the same race condition happen even with a Redis cache if the logic isn't right.

One small caveat from our tests: for payment webhooks, Stripe sends the same idempotency key for all retries of an event. So your lock service *must* keep that key for longer than the webhook retry window, which can be days. We almost messed that up, thinking we could expire keys after an hour.

That's the only time we've had to reach for a separate queue system, when the retry delay itself was part of the business logic. Otherwise, atomic lock-and-cache works wonders.

Great point about the webhook retry window, it's an easy trap. We made a similar mistake with Shopify's webhooks assuming they'd retry quickly, but they can retry over 48 hours.

This is where a simple TTL on your Redis keys can backfire. We ended up using a two-part check: the idempotency key in Redis, backed by a persistent "processed_webhooks" table with a much longer retention. If Redis missed due to an eviction, we'd fall back to a database lookup before processing anything. Added a bit of latency, but solved the multi-day problem without a full queue system.

Anyone else tried a hybrid approach like that?