Hey folks! 👋 Been a minute since I've shared something here. I was neck-deep in a gnarly incident last week—a cascading failure across our payment service—and the *worst* part, as always, was the evidence scramble. Logs here, metrics there, config diffs somewhere else. By the time we mitigated, my browser had 47 tabs open and my notes were a mess.
It hit me: why do we still do this manually every time? We have runbooks for *fixing* things, but not for *capturing* the state of the world during the fire. So I spent some downtime building a script I'm calling `incident-snapshot`. The goal is simple: run one command at the start of a major incident to automatically gather a standardized evidence pack. This has been a game-changer for our postmortem accuracy and speed.
Here's the core idea. It's a bash script that orchestrates a bunch of queries, saving everything with timestamps to a dedicated directory. My version pulls from our specific stack, but the pattern is universal. Here's a snippet of the main logic:
```bash
#!/bin/bash
INCIDENT_ID="${1:-incident_$(date +%Y%m%d_%H%M%S)}"
EVIDENCE_DIR="./evidence/${INCIDENT_ID}"
mkdir -p "$EVIDENCE_DIR"
# 1. System & Service Topology
echo "Capturing Kubernetes pod state for affected services..."
kubectl get pods -n production -l app=paymentservice -o wide > "${EVIDENCE_DIR}/k8s_pods_$(date +%s).txt"
# 2. Critical Metrics Snapshot
echo "Pulling last 15min of error rate and latency from Prometheus..."
curl -s -G 'http://prometheus:9090/api/v1/query_range'
--data-urlencode 'query=rate(http_requests_total{job="paymentservice", status=~"5.."}[5m])'
--data-urlencode 'start='$(date -d '15 minutes ago' +%s)
--data-urlencode 'end='$(date +%s)
--data-urlencode 'step=30' > "${EVIDENCE_DIR}/metrics_errors_$(date +%s).json"
# 3. Recent Deployment & Config Diff
echo "Checking last config change..."
git -C /config/repo diff HEAD~1 HEAD -- paymentservice.yaml > "${EVIDENCE_DIR}/config_diff_$(date +%s).diff"
# 4. Alert State
echo "Grabbing firing alerts from Alertmanager..."
curl -s http://alertmanager:9093/api/v2/alerts | jq '.[] | select(.status.state=="firing")' > "${EVIDENCE_DIR}/alerts_firing_$(date +%s).json"
```
I've also added sections for grabbing recent relevant logs (with `tail` and `jq`), a quick network connectivity check, and a dump of relevant SLO burn rates. The output is a tidy folder I can zip and attach directly to the incident ticket. No more "what was the error count at 14:23?" debates.
Some lessons from using it live:
* **Pre-approve your queries.** Make sure the script uses read-only credentials and APIs that are okay to hit during high load. Test it in a drill!
* **Keep it modular.** We have different service types (containers, VMs, serverless), so I built it with pluggable modules. The bash script sources a `collectors` directory.
* **Timestamp everything.** The `date +%s` in filenames is crucial for ordering later.
* **This isn't for debugging.** It's for evidence. It won't replace your observability tools, but it creates a coherent snapshot for the *after-action* review.
The biggest win? It reduces cognitive load during the storm. I run it, forget about data collection, and focus on mitigation. Our postmortems now start with a complete data set, which makes finding root cause so much faster.
Would love to hear how others are solving this. Do you have similar scripts? What else do you automatically capture? Any pitfalls I should watch for as I expand this?
@sre_journey
This is such a smart idea! I've never even thought about automating the evidence part, only the fixes. A script like this would've saved me so much time during our last email platform outage, just trying to grab screenshots of send queues and error rates before they changed.
Do you have it tag things with the incident start time automatically? That'd be perfect for our postmortem timelines.
Oh absolutely, it timestamps everything! The script's first step is to generate a unique incident ID (like `inc-20250314-1430`) that becomes the root folder name. Every log dump, metrics screenshot, and config export gets saved inside that, so the whole evidence pack is naturally organized by the moment you triggered it.
That said, a caveat from our first real use: if your incident spans multiple timezones (team members, cloud regions), you'll want to standardize on UTC in the filename metadata. We had a confusing moment where a log timestamp didn't match the folder name because of a local time assumption.
Have you run into other timestamp quirks with your postmortems?
Totally feel the 47 tabs thing, been there. This is a great idea.
Our team had a similar "aha" moment, but for a different reason. We found that if someone *else* declared an incident, the evidence I captured manually was often missing key context they saw at the start. A standardized script run by the first responder would solve that bias.
One question: does your script handle partial failures gracefully? Like, if one log source is down, does it skip and continue, or fail the whole collection?
PipelinePadawan
Oh wow, that's brilliant! I'm just starting to set up processes for my small team, and honestly the idea of trying to document an incident while it's happening makes me break out in a cold sweat. I'd be so worried about missing something crucial for later.
The runbook for capturing evidence instead of just fixing... that's a total lightbulb moment for me. So many tools focus on the response, but then you're left trying to piece together what actually *happened* after the fact. Having a standardized pack sounds like it would make postmortems way less scary.
How did you decide what goes into the script versus what you leave out? I feel like I'd go overboard trying to capture *everything*.
Small team, big decisions