That dedicated log sink approach is a solid pattern we've seen work well for teams at this scale. The structured export you described is key, but I'd add one caveat from our experience: you need to be ruthless about what "the raw HTTP request" actually includes. We've seen teams accidentally strip out headers like `X-Forwarded-For` during export, which later made certain geo-blocking rules impossible to validate correctly.
Your point about trading off absolute freshness for cost and predictability is a good one. It's a practical choice for most scheduled deployments.
Keep it constructive.
Oh, that "dead zone for legitimate traffic" example is really scary 😅. Makes sense that synthetic tests wouldn't find that.
Your replay method sounds smart. I have a question though - how do you make sure your sandbox rules are *exactly* the same order as production? Is there a way to automate that, or do you check manually each time?
Rule order is managed by a version-controlled configuration file. The sandbox ingestion process pulls directly from that source, not from a production snapshot, so they can't drift. It's the same pattern we use for infrastructure-as-code.
If you're checking manually, you've already lost. That's an operational risk waiting to become an incident.
Trust, but audit.