Hey folks! Been deep in the trenches with our LLM-powered features and noticed our P99 latency graphs were looking... spooky. The averages looked fine, but those nasty tail-end spikes were slipping through our standard monitoring. I wanted a way to get a real-time poke the *moment* something felt slow to a user.
So, I built a simple Slack bot that watches our LLM call traces and fires an alert when the P99 latency jumps beyond a threshold. It's been a game-changer for catching issues before they affect too many users. The core idea is straightforward: aggregate recent spans, calculate the P99, and compare it to a rolling baseline.
Here's the heart of it, using OpenTelemetry data and a bit of Python. I'm using Langfuse for tracing, but this would work with any OTEL-compatible backend.
```python
import requests
from datetime import datetime, timedelta
from statistics import quantiles
SLACK_WEBHOOK = "your_webhook_url"
P99_THRESHOLD_MULTIPLIER = 1.5 # Alert if P99 is 1.5x the baseline
def fetch_recent_traces(minutes=10):
# Fetches trace data from your observability backend
# This is a simplified example using a hypothetical client
end_time = datetime.utcnow()
start_time = end_time - timedelta(minutes=minutes)
traces = langfuse_client.get_traces(start_time=start_time, end_time=end_time)
return [trace for trace in traces if trace.name == "llm_call"]
def calculate_p99(latencies):
if len(latencies) (baseline_p99 * P99_THRESHOLD_MULTIPLIER)):
message = {
"text": f"🚨 LLM P99 Latency Spike Detected!n*Current P99:* {current_p99:.2f}msn*Baseline P99:* {baseline_p99:.2f}msn*Ratio:* {current_p99/baseline_p99:.1f}x"
}
requests.post(SLACK_WEBHOOK, json=message)
# Run this on a cron every 5 minutes or via a streaming pipeline
check_and_alert()
```
The key pieces that made this work well for us:
* **Focusing on P99, not average** – This catches the "pain points" our users actually feel.
* **Dynamic baseline** – Comparing to a recent rolling window accounts for normal time-of-day variations.
* **Including trace IDs in the alert** – We added deep links to the specific slow traces in our tracing UI, so we can jump straight to debugging.
It's already helped us catch a few sneaky issues: a degrading external API dependency and a prompt change that inadvertently increased response variability.
Has anyone else built similar real-time alerting for LLM performance? I'm curious about how you're setting thresholds and avoiding alert fatigue.
-- Weave
Prompt engineering is the new debugging
That's a solid approach. Moving from average-based monitoring to P99 alerting is a critical shift for LLM workloads, where the worst-case experience often defines user perception.
One nuance to consider is the baseline calculation itself. Using a fixed multiplier on a rolling baseline can trigger false positives during predictable low-traffic periods, like overnight. The variance in your dataset is much higher when you have only a handful of requests per minute. You might want to incorporate a minimum request count threshold before evaluating the P99, or switch to a more sophisticated baseline like a 7-day same-hour-of-day median.
Also, if you're using OpenTelemetry traces directly, remember that span duration includes queueing or client-side serialization time, not just the LLM provider's processing. Correlating these P99 spikes with a metric like your concurrent request count can help isolate whether it's a backend issue or a self-inflicted saturation problem.
The code snippet you started shows you're on the right track, but don't forget to add some exponential backoff or deduplication logic for the Slack alerts. You don't want a single prolonged issue to flood the channel with repeated pings every time your script runs.
You're absolutely right about the baseline calculation being a trap, but a 7-day median can also bite you during weekly release cycles or marketing blitzes where your traffic pattern shifts permanently. I've seen teams miss real degradations because their "normal" baseline slowly climbed with them.
The minimum request count is non-negotiable. I usually set it to something like 50 in the trailing five-minute window. Alerting on P99 with three requests is a recipe for your team to mute the channel.
Your point about span duration is crucial. That's why my alerts tag on the concurrent request count and the queue depth metric from our load balancer. If P99 is up and concurrency is flat, it's probably the provider. If they both spike, we're the problem.
Speed up your build
Exactly. The baseline creep during release cycles is a real issue that's bitten us before. We ended up implementing a dual-baseline approach: a short-term 1-hour rolling baseline for quick detection, and a longer-term 24-hour one that's compared only if the alert fires, as a sanity check against permanent shifts. It adds a bit of logic but cuts down on the noise.
> Alerting on P99 with three requests is a recipe for your team to mute the channel.
Couldn't agree more. We also added a cooldown period for the alert after a deployment, because the first few minutes of P99 data are always chaos while things stabilize. It's not perfect, but it prevents the immediate post-deploy pager storm.
Tagging on concurrency and queue depth is smart. Do you find you need to adjust your P99 threshold based on the concurrent request count, or does the simple correlation check usually point you in the right direction?
yaml is my native language