As many of you in this community are aware, I've been systematically testing various LLM providers, with a particular focus on coding assistants and their performance in RAG pipelines. A recurring challenge in these evaluations is the lack of persistent, visually accessible performance data. While Hailuo provides API endpoints with latency and token usage in the response headers, correlating this data across multiple models, prompts, and timeframes manually is impractical.
To address this, I have developed a custom Grafana dashboard that ingests, aggregates, and visualizes key performance metrics from Hailuo's API. The system is designed for researchers and developers who require more than anecdotal evidence when comparing models like `hailuo-70b` against `hailuo-8b` or when fine-tuning prompt strategies.
The core architecture involves a lightweight middleware service that intercepts API calls, extracts metrics, and forwards them to a time-series database (I'm using Prometheus, but InfluxDB is equally viable). The extracted metrics are far more comprehensive than what's typically logged in a simple console output. They include:
* **Request Duration:** Broken down into total duration, time-to-first-token (TTFT), and time-per-output-token. This is crucial for distinguishing between initial latency and stream speed.
* **Token Efficiency:** Prompt tokens, completion tokens, and total tokens per request, aggregated by model and endpoint.
* **Rate Limit & Financial Metrics:** Tracked requests against the daily limit and an estimated cost per request based on current pricing.
* **Status Code Distribution:** To immediately spot periods of elevated errors or service degradation.
Here is a simplified example of the metric collection function written in Python, which decorates the standard API call:
```python
import time
import prometheus_client as pc
from functools import wraps
REQUEST_DURATION = pc.Histogram('hailuo_request_duration_seconds', 'Total request duration', ['model'])
TIME_TO_FIRST_TOKEN = pc.Histogram('hailuo_ttft_seconds', 'Time to first token', ['model'])
TOKENS_USED = pc.Counter('hailuo_tokens_total', 'Total tokens used', ['model', 'type'])
def monitor_hailuo_call(model_name):
def decorator(func):
@wraps(func)
def wrapper(*args, **kwargs):
start_time = time.time()
result = func(*args, **kwargs)
duration = time.time() - start_time
REQUEST_DURATION.labels(model=model_name).observe(duration)
# Assuming result contains metrics from response headers
ttft = result.get('ttft', 0)
TIME_TO_FIRST_TOKEN.labels(model=model_name).observe(ttft)
prompt_tokens = result.get('prompt_tokens', 0)
completion_tokens = result.get('completion_tokens', 0)
TOKENS_USED.labels(model=model_name, type='prompt').inc(prompt_tokens)
TOKENS_USED.labels(model=model_name, type='completion').inc(completion_tokens)
return result
return wrapper
return decorator
```
The dashboard itself is organized into several key panels:
1. **Latency Overview:** A time-series graph showing request duration and TTFT, with filters for model and context length.
2. **Token Usage Heatmap:** A bar chart comparing average prompt/completion tokens across different models, which is invaluable for cost forecasting.
3. **Error Rate & Status Codes:** A stat panel and pie chart to quickly assess service health during a testing window.
4. **Performance vs. Context Length:** A scatter plot correlating input token count with response latency, revealing the scaling characteristics of each model.
Initial observations from running this dashboard over a week of varied workloads have been revealing. For instance, the `hailuo-70b` model shows a predictably higher TTFT than the 8B variant, but its time-per-output-token does not scale linearly, making it more efficient for longer, complex generations. Furthermore, intermittent latency spikes are clearly correlated with specific status codes, allowing for more precise troubleshooting.
I am interested in collaborating with others who are conducting rigorous LLM evaluations. Would there be interest in a shared, anonymized instance of this dashboard for the community, or perhaps a repository for the collector configuration and Grafana JSON? Additionally, what other metrics would you consider critical to add? I am considering integrating qualitative evaluation scores (e.g., from LLM-as-a-judge on code correctness) as a separate metric series, though that presents its own data modeling challenges.
Prompt engineering is engineering