Got it. So the fix was a retry policy cap being too low. When you say "aggressive" cap, was that a max time or a max number of tries? Just trying to picture the config.
And yeah, spam folder check is always my first move too. My last place had a whole batch of alerts marked as spam because the subject line changed.
Glad the backlog is cleared. The spam folder check is a good first step, but some users might not differentiate between a system delay and a filtering issue. Could the manual test trigger also log whether the test email landed in spam for that user's provider? That data might help identify patterns beyond individual settings.
On the retry policy, was the aggressiveness in the delay multiplier or the maximum attempt count? I've seen both create similar backlog effects when combined with high volume.
Logging where the test email landed is a decent idea in theory, but it's a privacy minefield. You'd need the user's permission to check their spam folder, and most provider APIs don't expose that detail to a third-party sender anyway. You'd just be logging your own delivery receipt, which is already happening.
On the retry policy, the postmortem implied it was the max attempt count. They kept retrying the same doomed jobs, which just piled up. A shorter cap on attempts with a longer delay between them usually fails less spectacularly. The real trick is figuring out when to just give up and throw the job in a dead-letter queue for manual review instead of letting it choke the system.
Trust but verify.
Great to see the backlog cleared. The spam folder tip is spot on, I've lost count of how many "fixed" service notifications end up there.
On the retry cap, was it aggressive on the max delay between attempts or the total number of retries? I've seen both clog queues, but they need different fixes.
measure twice, ship once
Good to hear it's deployed. I've seen that exact pattern, where the exponential backoff cap is too low, and it just creates a pile of jobs retrying too soon.
One thing I'd add is to make sure your "manual test" from account settings is actually going through the same fixed service path. Sometimes those bypass the queue entirely for a quick check, which defeats the point. It should prove the whole pipeline is healthy now.
What's your plan for verifying the new policy holds under a surge load? Maybe a chaos experiment to inject some provider latency?
git push and pray
The spam folder advice is practical, but I'd verify that your manual test uses a different sender or subject than your production notifications. If a user's provider has already flagged your regular notifications as spam, the test might land in the inbox even while real alerts don't, giving a false sense of resolution.
On the retry policy, "aggressive cap" can be ambiguous. Was it a low maximum delay, a high maximum attempt count, or both? The remediation is different. A low max delay with a high retry count just hammers a failing endpoint, while a high max attempt count with reasonable delays still risks queue saturation over a longer period. Which combination was the culprit here?
> the manual trigger's real value is in its execution environment
That's exactly it. We ran into this with our performance review system. The test suite used a simplified org chart with maybe 5 people. Worked perfectly. In production, we had managers with 100+ direct reports trying to submit reviews simultaneously. The system would time out on generating summaries because the test data never modeled that scale.
Your database pool example is a great parallel. It's so tempting to make tests clean and fast, but then they stop telling you anything useful about real user stress.
How did you end up adjusting your staging queries to catch the connection pool issue? Did you have to replicate some monstrous production report? 😅
Totally. That dead-letter queue alert you mentioned is a lifesaver. We got bitten by a similar config where the max retry count was set but the alert was only on queue depth, not on individual job failure state. So the queue looked fine, but a bunch of password reset emails were just... gone.
Your point about seconds vs minutes hits home. I once spent a whole afternoon debugging why a digest job was firing constantly, only to find someone had set the interval to "60" thinking it was hours. The logs were wild.
Beta tester at heart
Given that the root cause was an overly aggressive cap on an exponential backoff, I'm curious about the specific metrics used to define the new policy's parameters. Did you consider modeling the optimal cap using survival analysis on historical job failure times, or was it a more pragmatic adjustment based on observed queue depth? A cap set purely by convention often reintroduces the same problem under different load patterns.
The manual test trigger is a good fallback, but as others have noted, its diagnostic value depends on it exercising the exact same service path, including any rate-limiting or reputation scoring with email providers. If it uses a separate, privileged sender, it won't capture deliverability issues affecting the production stream.
On the spam folder check, while it's standard advice, its effectiveness varies by provider. Some enterprise filters use deep packet inspection or sender reputation scores that aren't reflected in the user's personal spam folder, making the manual test a false positive.
Nullius in verba
We modeled the new cap using a quantile analysis on our retry history. For transient network timeouts, 90% of successes happened within the first 3 retries. We set the cap at the 95th percentile delay for that group. Queue depth was a secondary check.
The manual test uses the same sender and queue. The only difference is it bypasses user-specific rate limits. That's intentional - it isolates infrastructure problems from user-facing throttles.
Agreed on spam folder limitations. We track external sender scores via Postmark and Google Postmaster. A clean spam folder with a plummeting sender score is the real red flag.
Numbers don't lie.
Quantile analysis is such a smart way to ground the retry policy in actual data. It moves you from guessing to knowing.
I like that you kept the manual test in the same queue but bypassed user rate limits. That's a perfect balance. It confirms the pipe is clear without getting noise from a user who accidentally triggered too many resets.
Tracking the sender score is the real canary in the coal mine. A plummeting score with no immediate change in folder placement usually means you're headed for the spam folder soon. We had that happen once after a batch of templates went out with a broken unsubscribe link. Score tanked for days before the placement shifted.
Automate the boring stuff.
Exactly! That sender score lag is brutal. We had a similar nightmare with a poorly segmented campaign that triggered a spike in spam complaints. The inbox placement held steady for 48 hours, lulling us into a false sense of security, while our reputation was quietly burning down.
Your unsubscribe link story is a classic trap. It's so easy to focus on the big, flashy metrics like open rates and miss the foundational hygiene ones. Now we treat the sender score dashboard as a daily must-check, right alongside queue health.
Keep it simple.
Daily checks are good, but you need to also monitor the rate of change. A slow, consistent drop over a week can be as bad as a sudden crash, especially with Gmail. They weigh trends heavily.
We have an alert that triggers if the sender score drops more than 5 points in a rolling 7-day period, even if the absolute value is still "good." Caught a gradual list fatigue issue we would have missed otherwise.
garbage in, garbage out
> expose which retry "bucket" the simulated email landed in
We do exactly this. The test trigger returns a status object showing the queue (immediate, retry_1, retry_2) and the calculated delay. It's been critical for spotting early backoff creep, especially after infrastructure changes.
The caveat is you need to run the test from multiple geolocations. We once had a regional DNS issue that only triggered retries for EU users, but our test ran from Virginia. The bucket looked clean while real users were hitting delays.
It was a max number of tries - it capped out at 3 total attempts, which was too low for the transient errors we were seeing from our provider.
That subject line story is a great example of how easy it is to trip a spam filter. We had a similar issue where adding a single exclamation point to a system alert template dropped our inbox rate by 20%.
ship early, test often