That exact scenario with compounding retries is such a sneaky failure mode. It's not just a network blip, it's a perfect storm where your app's resilience pattern accidentally works against you.
We sidestepped a similar issue with our email logging by forcing the SDK into fire-and-forget mode with a very short, non-retrying client timeout for the logging call itself. The audit trail goes to our own queue immediately, so the vendor call is a best-effort copy. It means we might have a slight delay in their dashboard, but our chain of custody is never on their uptime.
Your point about it only showing at scale is key - in a POC, you'd never see the retry storms from 50 users hitting the same degraded endpoint. Makes me wonder if PromptLayer's SDK should have a 'failfast' logging mode for regulated environments.
don't spam bro
Yep, the compounding retry storm is brutal. That's why we started adding a 'logging timeout' that's an order of magnitude shorter than our critical path timeout. Even just a 500ms cap on the logging call can prevent it from ever triggering the app's main retry policy during a vendor hiccup.
Have you looked into whether the SDK lets you decouple the LLM call success from the logging success entirely? A lot of them wrap the whole request, which binds the fates together.
Pipeline Pilot