I've seen a lot of hand-wavy claims about AI code assistants being "better" or "faster" without any real methodology. So I built a controlled benchmark to compare Continue's performance using GPT-4o versus Claude 3.5 Sonnet, since both are now supported. The goal was to measure actual code generation quality and reasoning on realistic data engineering tasks, not just toy examples.
I constructed a suite of 15 tasks across four categories relevant to our work:
1. **SQL Optimization:** Transforming a poorly written multi-join query into a performant version, adding appropriate indexes.
2. **Stream Processing:** Writing a PySpark structured streaming job to consume from Kafka, apply a stateful transformation, and write to Delta Lake.
3. **Orchestration Logic:** Creating a robust Airflow task with error handling, retries, and idempotency for a data validation step.
4. **Schema Design:** Drafting a protobuf schema for a change data capture event stream from a relational source.
Each task was run three times with each model via Continue, using the same system prompt template for consistency. I evaluated outputs on three axes:
* **Correctness & Completeness:** Does the code run and solve the full problem? (0-2 points)
* **Production Readiness:** Includes error handling, logging, and avoids common pitfalls. (0-2 points)
* **Explanation Quality:** Does the assistant explain its choices and trade-offs clearly? (0-1 points)
**Here are the aggregate results (average score per category, max 5):**
| Task Category | GPT-4o (via Continue) | Claude 3.5 Sonnet (via Continue) |
| :--------------------- | :-------------------- | :-------------------------------- |
| SQL Optimization | 4.2 | **4.7** |
| Stream Processing | **3.8** | 3.5 |
| Orchestration Logic | 3.5 | **4.1** |
| Schema Design | 4.0 | **4.5** |
| **Overall Average** | **3.88** | **4.20** |
**Key Observations:**
* Claude Sonnet consistently provided more thorough explanations and considered edge cases, particularly in schema design and orchestration. Its SQL suggestions were more nuanced regarding index selection and join ordering.
* GPT-4o was slightly faster in generating initial code and was better at the boilerplate structure for the PySpark streaming job. However, it more often omitted critical checkpointing configurations.
* Both models failed spectacularly on the same specific task: implementing a correct session window in the streaming job with late data handling. Neither's code worked without significant correction.
* The context window advantage of Claude Sonnet was evident. When I provided a lengthy existing pipeline code file and asked for an extension, GPT-4o sometimes ignored earlier parts of the context, while Sonnet integrated the whole context more reliably.
**The Takeaway:**
For data engineering work, Claude 3.5 Sonnet currently provides a higher floor of quality and reasoning in my tests. GPT-4o is quicker for more straightforward, boilerplate generation. The "best" choice in Continue is now heavily task-dependent. I strongly recommend you run your own benchmark on your specific codebase patterns before making a wholesale switch. Blindly assuming one model is superior is a good way to introduce subtle bugs.
I've published the exact prompt sequences and evaluation rubric on my GitHub. Would be interested to see if others in the community are running similar tests on different problem domains, like frontend or systems programming.
—davidr
—davidr
I'm a principal engineer at a mid-sized logistics company, where we run a ~70% microservice API fleet integrating our homegrown TMS, a couple of legacy ERPs, and Snowflake. I've personally written and maintained the middleware that pipes data between all of them, and we use Continue daily to fight the boilerplate.
Your benchmark is exactly the kind of rigor I wish I saw more of. Based on my own wrangling with both models through Continue for actual integration code:
* **Cost vs. Payoff Structure:** GPT-4o is cheaper on the API ($5/1M input, $15/1M output), but Claude Sonnet ($3/1M input, $15/1M output) often reaches a correct solution in fewer, more deliberate tokens for complex logic. If you're refactoring 500 lines of gnarly legacy API glue, Sonnet's slightly higher reasoning budget can be cheaper overall because it needs fewer revision cycles.
* **Local Development Velocity:** For the rapid, iterative "write a FastAPI endpoint that does X with Y validation" tasks, GPT-4o is consistently faster. Its latency from Continue's IDE plugin is noticeably lower, sub-second for most completions vs. Sonnet's 1-3 second think time. This matters when you're spiking out three connector options in an afternoon.
* **Reasoning Depth on Messy Systems:** Sonnet wins when the prompt includes 200 lines of existing, poorly-documented interface code and asks for an extension. It's better at holding the entire context and producing something that doesn't break an implicit pattern. GPT-4o will get it right 70% of the time, but that 30% failure is a subtle serialization bug that costs you an hour.
* **Incidental Complexity Handling:** When I need to generate code that involves a specific, less-popular library (think `aio-pika` vs. `pika` or `tortoise-orm`), Sonnet's output requires fewer corrections for imports and method signatures. GPT-4o hallucinates more aggressively with niche dependencies.
I'd recommend sticking with GPT-4o for general day-to-day API and boilerplate generation where speed and cost are primary, but switch the Continue project config to Claude Sonnet for any task involving refactoring, detailed system explanations, or working with obscure libraries. If you can share your team's monthly token spend and whether most of your work is greenfield or brownfield, the choice gets much clearer.
APIs are not magic.