That critical path vulnerability is so real. We hit something similar in our GitLab CI pipelines when we added a logging service to track build metrics.
The timeout cascade you described is exactly why we had to implement a "fail-open" circuit breaker right in our wrapper. If the logging call takes more than, say, 100ms, we just skip it and let the main LLM call proceed. The audit trail has a gap, but the user-facing workflow doesn't freeze.
Your point about the compounded retries is key though. Did you consider adding a flag to your retry logic that checks if the underlying SDK is already in a retry loop? That was a game-changer for us.
Pipeline Pilot
That fail-open circuit breaker is a solid pattern. We use a similar approach, but we also log the skipped events to a local disk buffer as a fallback. It's not perfect, but it gives us a chance to replay them later if the vendor comes back online quickly.
The flag for the SDK's retry state is clever. We actually added a shared "degraded mode" flag in our app config that both the business logic and the logging wrapper check. If the vendor's error rate spikes, we set the flag and both layers back off in sync. It stops the retry pile-up.
Your GitLab CI example hits home. We saw the same in our ArgoCD rollouts when a metrics push blocked the sync. Now any sidecar logging has a hard kill switch if the main workflow starts to lag.
Keep deploying!
That critical path vulnerability is the classic scaling trap with these logging services. You can get away with the extra hop in a POC because volume is low and you're not hitting regional blips.
A pattern that's worked for us is to treat the SDK like any other external dependency and wrap it with our own circuit breaker. We don't let its retry logic interact with ours. If the call to PromptLayer's logging endpoint fails or hangs, we drop the log event and let the primary LLM call proceed unimpeded. It creates an audit gap, but it prevents the timeout cascade from taking down the user workflow.
The key is deciding what's more important in the moment, a perfect audit trail or a responsive application. For a clinical setting, I'd argue the latter usually wins.
catdad