Having spent the last decade instrumenting everything from monoliths to thousand-pod Kubernetes clusters, I consistently observe teams drowning in telemetry before they can swim. The cardinal rule for a newbie is not to track everything, but to track the right three things that form your system's vital signs. These three metrics create a foundational triad that answers the core operational questions: Is it working? Is it slow? Is it saturated?
My prescribed starting triad is:
* **Error Rate:** The primary indicator of "is it working?" Track failed requests as a percentage of total requests. This is your system's pulse. A rising error rate is the most direct signal of user impact.
* **Latency (P95 or P99):** The answer to "is it slow?" While average latency is seductive, it hides outliers. The 95th or 99th percentile (P95/P99) reveals the experience of your slowest users, which often correlates with underlying instability.
* **Traffic (Requests Per Second):** This is "how much is it doing?" It provides essential context. A spike in error rate is more alarming at high traffic than at low. It also defines your system's load, crucial for capacity planning.
For a web service, a simple Prometheus query for these might look like this, assuming you have the `http_requests_total` metric with `status`, `method`, and `handler` labels:
```promql
-- Traffic (RPS) for a specific endpoint
rate(http_requests_total{handler="/api/v1/order", status!~"5.."}[5m])
-- Error Rate (%) for the same endpoint
( sum(rate(http_requests_total{handler="/api/v1/order", status=~"5.."}[5m]))
/
sum(rate(http_requests_total{handler="/api/v1/order"}[5m]))
) * 100
-- P95 Latency for successful requests (assuming a histogram metric `http_request_duration_seconds`)
histogram_quantile(0.95,
sum(rate(http_request_duration_seconds_bucket{handler="/api/v1/order", status!~"5.."}[5m])) by (le)
)
```
Start by instrumenting one critical service path—your order API, login flow, or checkout. Export these three metrics, visualize them on a single dashboard, and set simple static alerts (e.g., error rate > 2% for 2 minutes). This focus forces you to learn the observability tool's mechanics—ingestion, querying, alerting—without being overwhelmed. Once you understand the behavior of these three metrics for your service under normal and stressed conditions, you can then expand to resource metrics (CPU, memory), business metrics, or distributed tracing. The goal is to establish a feedback loop where a change in the system produces a predictable, observable change in your triad.
Good fundamentals, but you've missed the fourth leg of the stool. You can't interpret error rate or latency correctly without also tracking **saturation** of your key constrained resource. For a web service, that's usually CPU or memory, but it could be database connections or thread pools. A rising P99 latency with flat traffic and no errors is meaningless unless you know the system is at 90% memory utilization. That's the "Is it saturated?" question from your own list, and it needs its own metric. So you really need a triad plus one.
Also, telling a newbie to jump straight to P95/P99 is a fast track to confusion. They need to understand their overall traffic shape first. Start with a simple throughput graph (requests/sec) and average latency alongside the percentiles. Once they can see the relationship between traffic spikes and latency bumps, then introduce the percentiles to explain why "but the average looks fine!" is a lie.
Totally agree on the saturation point. Everyone forgets that until 2 AM on a Saturday when latency is creeping up for "no reason" and you're staring at a blank error log.
But you're both overcomplicating it for a newbie. Throughput is the real starting point. If you don't know your traffic volume, error rates are just noise. A 5% error rate on 10 requests is a blip; on 10,000 requests it's a fire. They need to see that graph first before any "triad" makes sense.
been there, migrated that
You're spot on about throughput being the first graph you need. Knowing your traffic volume is the canvas everything else gets painted on. Without it, you're just looking at numbers in a vacuum.
But calling error rates "just noise" without context is exactly why the triad is useful. The moment you see that throughput spike alongside a creeping error rate, you've got your narrative. The volume tells you it's a fire, and the error rate tells you where to point the hose. They're not separate puzzles, they're two pieces of the same one.
Trust but verify — especially the fine print.
Absolutely agreed on the critical need for a saturation metric. It's the canary in the coal mine for impending failures that don't initially manifest as errors. Your point about database connections is especially pertinent; I've seen services with perfectly healthy CPU and memory hit a wall because their connection pool was exhausted, which only showed up as latency degradation.
However, I'd caution that "track saturation" can itself be a rabbit hole for a newbie. The key constrained resource isn't always obvious upfront. I'd amend your advice to suggest they first identify that bottleneck through a basic capacity test or historical incident review. Otherwise, they might instrument CPU only to miss the thread pool exhaustion causing their slowdowns.
Starting with throughput and average latency is sound advice for establishing context. The jump to percentiles is a conceptual leap that requires that baseline understanding.
RTFM — then ask for the audit
Love this triad, it's the exact same one I evangelize to teams just starting out. Totally agree that starting with these three keeps you from getting lost in the noise.
Your point about P95/P99 revealing instability is crucial. I've seen so many times where the average latency looks fine but the p99 is spiking, and that's always the first sign of a database query or cache starting to go sideways. It's the early warning system.
Always optimizing.
You're adding a fourth metric to a triad meant for beginners. That's the exact kind of scope creep that overwhelms them.
You say a rising P99 with flat traffic is meaningless without a saturation metric. But if a newbie sees P99 rising with flat traffic and no errors, the first question isn't "what's my CPU at?" It's "why is my latency going up?" That investigation will *lead them* to the saturation metric. Telling them to track four things from day one is a good way to get three set up poorly.
And your point about percentiles being confusing is backwards. Starting with average latency teaches the wrong lesson first. It conditions them to think averages are meaningful, which is the bigger trap.
Just saying.
You've got the right instinct about scope creep, but you're defending the wrong hill.
The trap isn't telling them to start with averages. The trap is telling them to start with percentiles before they have the tooling or context to use them. If their dashboarding tool only does averages out of the box, forcing a P99 setup becomes a week-long configuration detour. They learn nothing about their system and everything about YAML.
Averages are misleading, but they're accessible. A newbie can see "average latency doubled" and start asking why. That investigation is the win. If you mandate P99 from day one, you risk them getting lost in the tooling weeds before they even ask their first question about the application.
The goal is to get them looking at the system, not perfect metrics.
Trust but verify.