I've been using Langfuse for a few months now to trace our LLM calls, and the trace comparison feature is invaluable for debugging. However, our team frequently iterates on prompt templates outside the deployed application—in notebooks, one-off scripts, or simply in draft documentation. Comparing a new prompt version against historical traces required a manual, copy-paste process into the Langfuse playground, which was becoming a bottleneck.
To streamline this, I built a simple internal CLI tool. It fetches a specific trace from our Langfuse project (identified by trace ID) and allows us to "replay" it by replacing the original prompt with a new version while keeping all other parameters (model, temperature, etc.) constant. The tool then calls the LLM provider's API directly, logs the new trace to Langfuse under a separate `session_id`, and outputs a side-by-side comparison of the outputs.
The core workflow is straightforward:
1. Fetch the original trace data via Langfuse SDK (`langfuse.get_trace`).
2. Extract the prompt, model configuration, and any system instructions.
3. Accept a new prompt markdown file as input from the user.
4. Construct a new generation call, inheriting all non-prompt parameters.
5. Send the request and log the new trace, linking it to the original via tags.
This has been particularly useful for:
- **A/B Testing at Scale:** We can now systematically replay a batch of representative error traces with a corrected prompt to estimate fix impact.
- **Cost Forecasting:** By comparing token usage of the new prompt against the old across hundreds of traces, we can model the cost implication of a prompt change before rollout.
- **Onboarding:** New engineers can test their prompt modifications against real, historical user scenarios without affecting production.
Has anyone else built similar tooling around Langfuse? I'm curious about alternative approaches, especially concerning how you handle stateful conversations (multi-turn traces). I'm also considering whether to open-source this if there's interest, though it's quite tailored to our internal stack (AWS Bedrock, mostly).
—A
Every dollar counts.
Interesting approach. I've built something similar but focused on batch evaluation across multiple traces. One challenge I ran into was handling parameter inheritance when the original trace contains nested tool calls or parallel generations. Did you consider extracting the entire invocation chain, or are you limiting replay to the root prompt?
Also, for the side-by-side comparison, are you calculating any metrics (e.g., token usage delta, latency, semantic similarity scores) automatically, or is it purely qualitative?
Prompt engineering is engineering