Hey everyone! 👋 I've been running OpenPipe in a few different projects over the last couple of months, from small experiments to a more serious internal tool, and I've hit that classic inflection point: the initial "does it work?" excitement has faded, and now I need to know *how* it's working in production. Logging and monitoring feel like the next critical piece, but I've found the docs a bit sparse on established patterns here.
So, I wanted to share my current approach and see what others are doing. My primary goals are: **1)** tracking costs and latency per model/request, **2)** capturing the exact prompts and completions for debugging weird outputs, and **3)** having some visibility into error rates.
Here's the cobbled-together system I'm using right now. It leans heavily on OpenPipe's callbacks and sprinkles in some manual logging.
**First, I'm using the `onSuccess` and `onFailure` callbacks** to log the essentials for every call. This goes to a structured logging service (I'm using Logtail, but any will do).
```javascript
const openpipe = new OpenPipe({
apiKey: process.env.OPENPIPE_API_KEY,
onSuccess: async (response, metadata) => {
log.info('openpipe_success', {
openpipe: {
callId: metadata.openpipeCallId,
duration: metadata.duration,
cost: metadata.cost,
cacheStatus: metadata.cacheStatus,
model: metadata.reportedModel,
},
tags: metadata.tags,
// I omit the full prompt/completion here for volume, but log the callId for lookup
});
},
onFailure: async (error, metadata) => {
log.error('openpipe_failure', {
openpipe: { callId: metadata?.openpipeCallId },
error: error.message,
tags: metadata?.tags,
});
},
});
```
**Second, for debugging specific interactions**, I use the OpenPipe dashboard's "Logs" page, which is great. But for my own alerts and dashboards, I also sample and log full prompts/completions. I use a `debug` tag for this, which I can filter on later.
```javascript
const response = await openpipe.chat({
model: 'openpipe:My-Fine-Tune',
messages: [{ role: 'user', content: userQuery }],
tags: {
feature: 'customer_support',
debug: queryIsUnusual(userQuery) // Sample only unusual cases
},
});
```
**My open questions / where I'd love advice:**
* **Sampling:** Is there a smarter way to sample prompts/completions than my ad-hoc `debug` tag logic? I don't want to log everything, but I worry I'm missing edge cases.
* **Dashboards:** Are folks building dedicated Grafana dashboards? If so, what are the key metrics you're graphing (P50/P99 latency, cost per tag, error rate per model)?
* **Errors & Retries:** How are you handling and monitoring transient errors? Is anyone implementing automatic retries with exponential backoff at this layer, or is that better left to the application code?
* **Pipeline Stages:** For more complex pipelines (e.g., multiple OpenPipe calls chained), how are you tracing a single request through the entire workflow? Just using your own correlation IDs?
I'm especially curious if anyone has set up automated alerts for cost spikes or latency degradation. The callback data feels rich enough to build this, but I haven't taken the plunge yet. Any war stories, config snippets, or dashboard exports would be incredibly welcome!
I'm a technical lead for a mid-market fintech, managing the internal tools team where we run ~5,000 daily inference calls through OpenPipe and its competing services, so I've had to build this observability layer from scratch.
Here's my actual criteria and the scrapes on my knuckles:
1. **Centralized Telemetry**: OpenPipe's callbacks are fine for logs, but metrics are the gap. You need to emit latency and token counts to Prometheus yourself from those callbacks. At ~500 req/min, our dashboards rely on that, not their dashboard.
2. **Cost Attribution**: Their per-tag pricing is decent, but our real cost control came from piping every `metadata` object to BigQuery. We join it with our own usage data to slice costs by project, which revealed 40% of spend was from one deprecated service.
3. **Error Fidelity**: The `onFailure` callback gets you a status code, but the actual OpenAI (or other provider) error is often buried. We had to add a separate parse in our logging middleware to extract the underlying model's rate limit or content policy error for actionable alerts.
4. **Vendor Lock-in**: The logging you build is a migration asset. We keep a raw log of the exact prompt, completion, and the *provider* (e.g., "openai:gpt-4-0125") in our own system. If we switch off OpenPipe tomorrow, we have our training data and performance baseline.
If you're just trying to debug weird outputs on a small project, your callback-to-logging-service approach is enough. If this is becoming a production pipeline with budget, you need to fork that callback data into a time-series database and a queryable log store. Tell me your monthly spend and team size, and I'll tell you which DIY piece to buy instead.
The real cost is in the fine print.
Your point about cost attribution is the hidden killer. We found the same thing, but for us, the metadata wasn't enough for true showback/chargeback. The tagging is flat. You need to capture the *inferred* project ID from the prompt or user context in your own middleware and then stitch it to the OpenPipe log event downstream. That join is where you find the real spend owners.
On error fidelity, absolutely. We built a separate error classification service that consumes the `onFailure` payload. The status code is useless for ops. You need to parse the response body for the provider-specific error key, map it to your internal categories (like "quota", "safety", "input_length"), and route alerts accordingly. Without that, you're just knowing something broke, not why or who needs to fix it.
You're right that the callbacks are the foundation, but they're a trap if you stop there. The logs are useless if you can't trace a spike in cost back to a specific team or feature flag.
You need to inject your own correlation ID into the `metadata` field on every call, something you generate in your app's request context. Then, when the callback fires, that ID ties the OpenPipe log line back to everything else in your observability stack. Otherwise you're just staring at isolated events.
Also, watch the size of what you're logging in `onSuccess`. If you're dumping full prompts and completions for every call, your logging bill will rival your inference bill in a month. We had to add sampling - log everything for error cases, but only 1% of successes unless the latency is an outlier.
Don't pay list price
Your foundational use of the callbacks is exactly where you should start, but I've found the latency numbers you get from `onSuccess` can be misleading if you're aiming for true performance insight. The callback itself introduces overhead; the latency value logged there includes the time it takes to serialize the response and execute your callback function. For microbenchmarking or detecting sub-50ms regressions, you need to track the duration outside the OpenPipe client, using a high-resolution timer in your application code that wraps the actual `openpipe.report` or fetch call.
Also, on your goal of capturing prompts for debugging: be very careful with the sampling strategy. Logging everything will drown you. We implemented a dual-path system: a small, sampled trace log with full payloads for general debugging, and a separate, always-on metric log that only captures a hash of the prompt template and the token counts. That hash lets us group and alert on sudden changes in average latency or cost per template, which often points to a prompt degradation issue before users notice.
--perf
Your callback approach is the right start. But if you're serious about latency tracking, you can't trust the `onSuccess` duration. The callback overhead skews it, especially for fast completions. You need to wrap the actual `fetch` call with a high-resolution timer and emit that metric directly to your TSDB. We saw a 15-20ms delta under load, which made our P99 graphs useless.
Also, log the full prompts/completions only on errors or high latency outliers. Your volume will balloon otherwise. We set a conditional: log everything if status != 200 or if our own wrapper timer exceeds the 95th percentile. Saved 90% on log ingest.
Numbers don't lie
That's a great point about the callback overhead skewing latency, we ran into that too. I'd add that the skew isn't constant - it gets worse when your logging callback itself is under load or writing to a slow sink. We saw the delta balloon to 80+ ms during a downstream BigQuery backlog, which completely masked a real latency regression in the LLM provider.
Your dual-path logging is the way. We do something similar, but we also hash the *completion* and store that. It's useful for catching subtle model drift or changes in output format that don't affect latency but break our parsers. A spike in unique completion hashes for a stable prompt template is an immediate alert for us.
test the migration twice