Hey folks, been using LangChain to instrument some LLM workflows for our internal observability docs. The initial setup with GPT-4 was straightforward, but now I'm looking at other models for cost, latency, or specific task reasons. The list is getting long—Claude, Llama, Cohere, etc.
My instinct is to treat the LLM choice like selecting a database or a metrics backend: it's about the SLOs for your specific chain. Are you building a high-throughput logging parser, or a complex, multi-step APM analysis agent? The requirements differ.
For example, I needed a model to consistently output structured JSON for dashboard config generation. GPT-4 was reliable but expensive for the volume. I switched to using Anthropic's Claude for that specific chain because its structured output is solid and it cut my token cost significantly. Here's a simplified version of how I set that up:
```python
from langchain_anthropic import ChatAnthropic
from langchain_core.pydantic_v1 import BaseModel, Field
class DashboardConfig(BaseModel):
title: str = Field(description="Dashboard title")
metrics: list[str] = Field(description="List of metric names")
structured_llm = ChatAnthropic(model="claude-3-sonnet-20240229").with_structured_output(DashboardConfig)
```
So my question is: what's your framework for deciding? I'm thinking along these axes:
- **Accuracy for task**: Does it follow complex instructions? Test with a representative sample of your prompts.
- **Latency & Throughput**: What's the P95 latency for your chain's context length? Can it handle your load?
- **Cost per trace**: Model cost + token usage. Instrument your chains and see the spend!
- **Context window**: Needed for your RAG chunks or long prompts?
- **Tool calling/Function calling**: If your agent relies on it, not all models support it equally.
Would love to hear real use cases. What model are you using for what, and what metrics are you tracking to know it's the right fit? I've been logging model choice, token counts, and response quality scores as spans in my traces—super helpful for comparison.
Observe all things.
DataDogDodger