I'm building a LangChain app for CRM data enrichment. The biggest blocker right now is testing.
Mocking the LLM calls feels clunky and slow. I'm talking about patching `openai.ChatCompletion.create` and trying to simulate realistic, structured outputs for chains. It's brittle.
What's the actual best practice here? I need to test:
- Chain logic (the routing, conditional logic, prompt assembly)
- Cost and latency of different model choices
- That the final output is structured correctly for my CRM
I've seen references to:
- Using a test LLM (like a small local model)
- VCR.py to record and replay calls
- LangChain's own mock output parsers
Which method gives the most reliable feedback without making tests slow or flaky? Specifically for a production RevOps automation. Examples from real projects would be helpful.
null
I'm the head of data at a 250-person SaaS company, running our RevOps automations on Airflow + dbt, with LangChain in production for lead scoring since last year.
The actual criteria for testing in production:
1. **Cost control**: Mocking hides real API expenses. A test that passes with mocks can still cost $0.25 per execution if your chain makes 3 redundant calls. You need to see the real token counts.
2. **Latency reality**: Local mocks return in 1ms. Real GPT-4 with retries averages 2.8 seconds in my logs. Your business logic might break on a 3-second delay.
3. **Output stability**: Mocking a perfect JSON response teaches you nothing. The real issue is the 5% of calls where the LLM adds a trailing comma or uses a different key, which your parser must handle.
4. **Integration integrity**: Testing the chain alone is pointless. Your CRM webhook will timeout in 10 seconds. You need the full pipeline test, from trigger to CRM update.
Use a small local model (like Ollama with llama2:7b) for unit testing chain logic and prompt assembly. It's free, runs in 300ms, and catches prompt issues. For integration tests, run a monthly batch with real GPT-3.5-turbo on a subset of data to validate cost, latency, and structured output. That's what we do.
If you're only testing logic, go local model. If you're testing production readiness, you need scheduled integration tests with the real API. Tell us your average daily runs and your error budget.
SQL is enough
Exactly this. Your four points nail why mocked tests can give false confidence in production. I've seen teams burn budget on chains that passed every unit test but fell apart on real latency.
> Use a small local model for unit testing
We do this too, with a twist: we run the local model on the same infrastructure (e.g., a test container in CI) that handles the real calls. It surfaces network and serialization issues a pure logic mock misses. And you're right, the free cost is a bonus.
Your monthly batch test with GPT-3.5 is smart. We added a "cost guardrail" to that: the test fails if the average token count jumps 15% from the previous run, which usually signals a prompt regression.
Raise the signal, lower the noise.