We built a post-mortem workflow around Sumo Logic saved searches that cut our mean-time-to-resolution (MTTR) by about 40% last quarter. The real trick wasn't just finding the logs faster—it was automating the collection of the *right* logs so the on-call engineer isn't scrolling through a million irrelevant entries at 3 a.m.
Here's the core of it. We treat saved searches as reusable, version-controlled building blocks instead of one-off queries typed into the search bar. For a given service, we have a set of them that map to common failure modes:
* **Search A: Latency Spike Detector** – Tracks P95/P99 across key endpoints, triggers if thresholds breach for 5+ minutes.
* **Search B: Error Budget Burn** – Calculates error rate against our SLO, highlights the specific failing dependency (database call, external API, internal service).
* **Search C: Deployment Correlation** – Automatically scopes logs to the 10-minute window following any deployment to the affected service.
When an alert fires, the incident commander runs a "Post-Mortem Initiation" dashboard. It's just a panel that executes these three saved searches in parallel for the alert time window. Within 60 seconds, we have:
- The symptom (Search A)
- The probable cause, often pinpointed (Search B)
- Whether a recent change is implicated (Search C)
The cost angle? We were bleeding credits on ad-hoc, poorly constructed broad searches during incidents. This workflow reduced our average search scan volume by about 65% for incident response, because the searches are tightly scoped with good `where` clauses and use fields we actually index. That's a tangible SaaS cost saving, not just an efficiency win.
Hidden fee to watch: If you go this route, be ruthless about what you extract as a field. Every extracted field in your parsing rules is a line item. Our saved searches only rely on fields we already need for billing and core monitoring—no "nice-to-have" extractions that double your bill.
Anyone else using saved searches as a formal part of their incident process? Curious if you've hit any Sumo limitations with sharing or permissioning them across teams.
Cloud costs are not destiny.