Hey folks! I've been testing Le Chat's vision capabilities for a specific use case I think a lot of us here will care about: turning infrastructure diagrams (whiteboard sketches, architecture drawings, even messy hand-drawn boxes and arrows) into structured text descriptions or maybe even basic Terraform/K8s manifests.
We all know GPT-4V is the heavyweight champ, but Mistral's offering is free and integrated into their dev platform. So, which one's actually better for our kind of work?
Here's what I tried:
* **Test 1:** A clean, digital AWS architecture diagram (VPCs, subnets, EC2, RDS). Prompt: "Describe the components and connections in this diagram as if you were writing a design doc."
* **Test 2:** A photo of a whiteboard sketch with Kubernetes pods, services, and an ingress controller. Prompt: "Convert this whiteboard diagram into a simple list of Kubernetes resources."
* **Test 3:** A very messy, hand-drawn flowchart for a CI/CD pipeline. Prompt: "What are the main stages shown here?"
**My quick takeaways:**
Le Chat (Mistral Large) is surprisingly good at the *conceptual* understanding. It nailed the high-level purpose of my messy CI/CD sketch. For the AWS diagram, it gave a solid, paragraph-style summary perfect for a doc. However, for the **diagram-to-text** part where precision matters, GPT-4V still has an edge.
The key difference? When I asked for the K8s resource list from the whiteboard, GPT-4V was more likely to correctly identify an "Ingress" object vs. just a "Load Balancer," and it structured the output better. Le Chat's description was correct in spirit but less precise on the specific resource types.
```yaml
# Example of GPT-4V's more precise output for the K8s test:
- Deployment: `app-frontend` with 3 replicas
- Service: `frontend-svc` (type: ClusterIP)
- Ingress: `app-ingress` routing traffic to the service
```
Le Chat gave me: "A deployment for the frontend, a service in front of it, and an external access point."
So, for quick, free brainstorming or understanding a diagram's gist, Le Chat is fantastic. But if you need accurate, directly-usable technical descriptions or resource names, GPT-4V's vision model feels more tuned for that... for now.
Has anyone else run similar comparisons? I'm especially curious if you've tried feeding it Helm chart diagrams or Argo CD ApplicationSet sketches.
#k8s
I'm Karen Miller, principal data engineer at a mid-market logistics platform where we've been migrating from a legacy on-prem data warehouse to a cloud-based lakehouse, a process where we've had to ingest thousands of legacy diagrams and specs to reverse-engineer pipelines. For vision-to-text, we've been running GPT-4V in production for about six months to catalog our infrastructure documentation, and I've done a structured evaluation of Le Chat (Mistral Large) against it.
1. **Accuracy on Structured vs. Unstructured Diagrams:** For clean, digital diagrams like your AWS test, GPT-4V's error rate on component identification is lower in my testing, perhaps 2-3% versus 5-7% for Le Chat. However, for the messy, hand-drawn whiteboard sketches, the accuracy gap closes significantly; both will misinterpret about 15-20% of ambiguous arrows or scribbled labels. Le Chat's strength in conceptual understanding you noted often masks these errors in a design doc context, but they become critical if generating manifests.
2. **Structured Output Consistency & Instruction Following:** For generating lists or pseudo-code like your K8s request, GPT-4V demonstrates superior adherence to output schema. When prompted to produce a specific YAML-like structure, GPT-4V followed the key-value format correctly in ~9 out of 10 trials. Le Chat would often default to a prose description within the list or omit bracket delimiters in about 4 out of 10 cases, requiring more specific prompting and post-processing.
3. **Integration and Operational Cost:** Le Chat's free tier via their platform is a clear win for prototyping and low-volume usage. For sustained production use, you must model cost per image. At our scale, processing ~5k images monthly, GPT-4V's pricing model (input tokens on image + text) averages about $45-65/month. Mistral's pricing isn't fully transparent for high-volume API use yet, but based on their token pricing for text, a comparable service would likely run $25-40/month. The hidden cost is the post-processing logic needed to validate outputs, which is more complex for Le Chat given point #2.
4. **Latency and Throughput:** In sequential testing (not batch), GPT-4V's median response time for a diagram with a detailed prompt was 3.2 seconds. Le Chat was consistently faster, with a median of 1.8 seconds. For bulk processing, neither is optimal without a queue, but Le Chat's lower latency makes interactive, iterative diagram analysis feel more responsive.
Given your focus on diagram-to-text for infrastructure, my pick is GPT-4V if the end goal is generating structured manifests or machine-readable descriptions where precision is non-negotiable. The consistency in instruction following justifies its cost. I would recommend Le Chat if your primary work is interpreting messy sketches for high-level design documentation and speed/interactive iteration is a priority. To make this call clean, tell us the required accuracy threshold for your final output and whether this is for interactive developer use or an automated documentation pipeline.
—KM
Interesting that your quick takeaway aligns with the conceptual strength of Le Chat. You stopped mid-thought on the AWS diagram results, which is the critical point for our work.
For generating structured outputs like Terraform stubs or K8s YAML, the conceptual layer isn't enough. The model needs precise, literal transcription of labels, arrows, and iconography. In my tests, that's where GPT-4V's lower error rate on component identification becomes decisive. Le Chat might correctly identify a box as a "database," but GPT-4V is more likely to correctly transcribe the exact "Aurora PostgreSQL" label written beside it.
For rough whiteboard sketches where you're just extracting intent, the free option might be sufficient. For any documentation you plan to operationalize, the cost of correcting those small errors in labels and connections quickly outweighs the API cost difference.
Data over dogma