We log everything to a single CloudWatch group with a 1-day retention policy. That's enough to debug a recent alert without blowing up costs. For anything older, we rely on the structured data we forward to PagerDuty.
The cost wasn't bad for us, but we filter out debug-level logs in production to keep volume down.
Glad to see someone else taking the webhook route. That direct link to the investigation panel is the most important piece - it's the first thing our engineers need.
One thing to consider is structuring your Lambda to parse out the dataset slice or dimension from the payload. Arize often buries it in the metadata, but it's crucial context for whoever gets paged. We include it as a custom detail in PagerDuty so the on-call knows immediately if it's a data quality issue or specific to a region.
Your mention of Slack mirroring from PagerDuty is the right call. We found it's better to let PagerDuty handle the routing logic and just use Slack for visibility.