I've been using HuggingChat's API for some light synthetic checks over the last three weeks, routing the results into Prometheus. The uptime itself still looks good on paper (99.9%+ if you just count HTTP 200s), but the latency histograms and the rise of "model loading" messages tell a different story.
During my local peak hours (roughly 10:00-12:00 UTC), I'm seeing a significant increase in request duration, specifically in the 95th and 99th percentiles. The successful requests are fast, but the long tail is getting longer. More telling are the occurrences of non-standard responses. My scraper is now catching more of these:
```
{
"error": "Model is currently loading",
"estimated_time": 45
}
```
From an SLO perspective, if you define latency based on *usable responses* and not just any 200, the service is degrading. For my use case, a response that isn't immediately actionable is a failure.
I've had to adjust my alerting rules to account for this. A simple `up{job="huggingchat-api"}` is useless. I'm now tracking a custom metric that flags any response containing "loading" or "estimated_time" as a partial outage.
**Current alert rule snippet:**
```yaml
- alert: HuggingChatHighDegradedRequests
expr: |
rate(huggingchat_degraded_responses_total[5m]) / rate(huggingchat_requests_total[5m]) > 0.05
for: 10m
annotations:
description: "More than 5% of requests are returning 'model loading' delays for 10 minutes."
```
Has anyone else quantified this? I'm curious if others are just accepting this as the cost of a free tier, or if you've found workarounds—like specific endpoints that are more stable. For serious workflows, this pattern would force implementing circuit breakers and fallback models, which adds complexity.
That's a clever way to track it with a custom metric. I'm new to Prometheus and seeing "model loading" a lot too, just from the web UI. I thought it was just me being impatient 😅
When you see that "estimated_time": 45, do you have your scraper wait and retry, or do you just count it as a failure right away? Trying to figure out a better way to handle it for my own zaps.