We had a 95th percentile latency spike from 120ms to 800ms on a critical inference service. P95 SLO burn was unacceptable. Here's how we traced it.
First, eliminated the obvious:
* No code changes in last 24h.
* No infrastructure alerts (node CPU/memory fine).
* No downstream API degradation.
Checked Arize's latency heatmap. Spikes correlated with specific input feature: `user_embedding_v3`. Drilled into data drift tab for that feature.
Key findings:
* Feature's value range hadn't shifted (no drift).
* But the *cardinality* of unique values had tripled overnight. Our pre-processing pipeline wasn't optimized for this explosion.
Root cause: A upstream service update started sending vastly more unique embedding IDs, causing cache thrashing and DB lookup explosions.
Arize queries used:
```sql
-- In Arize, looked at latency vs. feature value
SELECT inference_time_ms, user_embedding_v3
FROM model_predictions
WHERE timestamp > NOW() - INTERVAL '12 hours'
ORDER BY inference_time_ms DESC
LIMIT 1000
```
Fixed by:
1. Adding a short-term circuit breaker to batch/reduce unique lookups.
2. Increasing embedding cache TTL and size.
3. Adding a monitor on feature cardinality directly in Arize.
Without slicing by that specific feature, we'd have wasted hours. The SLO burn stopped.
Great catch on the cardinality shift! That's a sneaky one since drift tools often focus on value ranges, not uniqueness. We set up a similar monitor after a bad rollout - ended up tracking a cardinality ratio (unique values / total rows) over a sliding window. Triggered an alert when it jumped >20%. Saved us a few times from similar upstream "improvements" 😅
Your fix is spot on for a quick recovery. Did you consider adding a bloom filter before the cache/db lookup? We found it cut down pointless queries a lot when cardinality spiked.