Hey everyone! 👋 Our legal ops team was drowning in old contracts, so we benchmarked all the major API models on a real-world task: extracting key clauses (like termination dates, liability caps, and governing law) from a messy dataset of 500 PDF contracts.
We focused on three metrics: **extraction accuracy** (manual check), **average latency per document**, and **cost per 100 documents**. We used a consistent, structured JSON output prompt across all providers.
Here’s the high-level rundown:
**Top Performers for Accuracy (F1 Score)**
* **Claude 3 Opus** (~95%): By far the most accurate, especially on nuanced language. Slow and expensive, but worth it for critical docs.
* **GPT-4 Turbo** (~92%): Very strong, with great consistency. Latency was good.
* **Claude 3 Sonnet** (~90%): Excellent balance. Became our default for bulk processing.
**Biggest Surprise**
* **Gemini 1.5 Pro** (~89%): Did extremely well on complex tables within contracts, but occasionally hallucinated on dates.
* **GPT-3.5 Turbo** (~82%): Fast and cheap, but struggled with longer, convoluted clauses. Fine for simpler extractions.
**Cost & Speed Trade-offs**
* For a high-accuracy, low-volume task, Opus is unbeatable.
* For processing thousands of documents, **Sonnet** offered the best blend of speed, cost, and accuracy.
* **Llama 3 70B** (via Groq) was blazing fast and very cheap, but accuracy (~80%) dropped on legal jargon.
**Key Takeaway:** No single "best" model. It totally depends on your budget, speed needs, and accuracy tolerance. We now use a tiered approach: Sonnet for bulk, with Opus spot-checking high-risk sections.
Happy to share more details on our prompt structure or the exact evaluation dataset if anyone's interested. What's everyone else using for document extraction?
Happy benchmarking!
Always testing.