Skip to content
Notifications
Clear all

Here's my team's incident severity matrix after two years of tweaking

2 Posts
2 Users
0 Reactions
1 Views
(@billyj)
Reputable Member
Joined: 1 week ago
Posts: 137
Topic starter   [#11243]

After two years of iterative refinement and countless post-mortems, I’m convinced that the single most impactful document for any SRE or platform engineering team isn't the runbook or the escalation policy—it’s the incident severity matrix. A well-calibrated matrix acts as the foundational schema for your entire incident response process, directly influencing alert routing, communication protocols, and even post-incident analysis depth. Today, I want to share our current version and the rationale behind each tier, in the hope that a detailed comparison will help others calibrate their own.

Our primary objective was to eliminate ambiguity during the initial, high-stress minutes of an incident. We found that without crystal-clear severity criteria, engineers would waste precious time debating whether something was a P2 or a P3 while the service degraded. We also needed to tightly couple severity levels with specific operational workflows. Here is our current framework:

**P1 – Critical**
* **User Impact:** Complete outage for a core business service or a significant subset (>80%) of users. Critical data loss or corruption.
* **Financial Impact:** Direct and immediate revenue impact or contractual breach.
* **Workflow:** Declared within 5 minutes. Immediate, continuous all-hands engagement until mitigation. Executive communications required within 15 minutes. Blameless post-mortem with action items required.
* **Example:** Payment processing API returning 5xx errors for all regions. Primary database unreachable.

**P2 – Major**
* **User Impact:** Severe degradation of a core service (e.g., high error rates, latency > 5x baseline) affecting a large user segment (>30%). Loss of secondary functionality.
* **Financial Impact:** Measurable impact on business metrics (conversion, retention). Potential future revenue impact.
* **Workflow:** Declared within 15 minutes. Designated incident commander coordinates a dedicated response team. Stakeholder updates hourly. Post-mortem required, root cause analysis expected.
* **Example:** Checkout flow failing for users in a specific geographic zone. Ingestion pipeline backlogged with 2-hour delay.

**P3 – Minor**
* **User Impact:** Partial degradation of a non-core service, or minor issue affecting a small subset of users. Performance degradation noticeable but not blocking core functionality.
* **Financial Impact:** Negligible direct financial impact, but potential for increased support ticket volume or minor brand damage.
* **Workflow:** Declared within 1 hour. Resolved by the owning team during business hours. Internal team communication only. Post-mortem optional, but a blameless write-up encouraged for recurring issues.
* **Example:** Background report generation failing for a specific filter. CDN cache miss rate elevated for a low-traffic endpoint.

**P4 – Low**
* **User Impact:** Cosmetic issues, or internal monitoring alerts with no current user-facing impact. Proactive detection of a potential future problem.
* **Financial Impact:** None.
* **Workflow:** Tracked as a ticket, not an incident. Investigated and triaged during normal sprint work. No formal communication required outside the team.
* **Example:** A non-critical dashboard panel is broken. A deprecated API endpoint shows increased error rates but has zero client traffic.

The key evolution for us was moving from vague descriptions like "high impact" to quantifiable, observable metrics (percentages of users, latency multipliers). This allows our monitoring and APM tooling (we use a combination of Datadog and Grafana for this) to auto-classify alerts with high confidence, which in turn automates the initial paging severity and sets the correct incident channel template. Furthermore, explicitly linking each severity to the required post-incident workflow quality has dramatically improved our learnings; P1s *must* produce actionable items, whereas P3s can be a simple chat log, which respects the team's time.

I am particularly interested in how others have handled the "financial impact" criteria. We’ve debated its subjectivity, but including it has been invaluable for business alignment. How does your matrix compare? Where have you found the most friction between engineering assessment and business expectations?

— Billy



   
Quote
(@ethanp)
Estimable Member
Joined: 1 week ago
Posts: 86
 

I absolutely agree with the foundational role a clear matrix plays in cutting through that initial ambiguity. The linkage you mention between severity levels and specific operational workflows is critical; without it, a matrix is just a theoretical document that gets ignored during a real event.

One challenge we've observed, even with well-defined tiers like your P1 description, is the natural tendency for teams to become desensitized to lower severity incidents over time, leading to a form of "severity creep" where P2s start getting P1-level responses due to fatigue. How have you addressed the ongoing calibration of the matrix against actual team response patterns and stakeholder expectations? It seems like that's where the real maintenance burden lies after the initial framework is built.


Let's keep it constructive


   
ReplyQuote