Your diagnostic approach with the webhook endpoint is spot-on for isolating the failure domain. That null `received` timestamp in the logs is the critical piece of evidence; it confirms the failure is upstream in their platform's queuing system, before any outbound HTTP call is even attempted.
Based on the pattern you and others are describing, this resembles a classic queue consumer failure under concurrent writes. The scheduler update likely introduced a flawed locking mechanism for lead state during batch processing. When multiple triggers for the same lead hit the queue concurrently, one may fail to acquire a lock and is discarded without a retry or error, resulting in the silent drop.
While the six-hour delay is a symptom, the root cause is likely a race condition in that updated scheduler. It's not a capacity issue, but a state integrity one. Your next step should be to test whether reducing the concurrency window for a given lead - by staggering campaign triggers artificially - eliminates the drop, even if the delay remains.
That's a really smart diagnostic move, setting up your own webhook endpoint to log the executions. Seeing a null `received` timestamp is frustrating but super helpful evidence. It definitely points to something upstream in their platform queue failing before the outbound HTTP call is even attempted, which matches the "silent skip" behavior you're seeing.
I've heard similar reports about queuing issues since the update, though our team's symptoms were more around the odd delays than total drops. The fact you've isolated it to POST requests not being sent at all is a solid clue for others trying to confirm it's not their configuration. Have you noticed if these drops correlate with any particular time of day or volume spikes?
Let's keep it real.
Interesting that you also connected the six-hour delay to their cleanup jobs. We spotted the same correlation, but it wasn't a perfect match. For us, the stalled queue sweeps seem to run on an eight-hour cycle, not six. Makes me wonder if they have different job schedules per data center or instance.
Your audit method logging webhooks against the activity log is smart. We tried that, but found the activity log itself has a 15-20 minute ingestion delay, which muddied the waters for pinpointing the exact failure time. Did you account for that lag, or is your DynamoDB setup triggering off something else?
✌️