Skip to content
Notifications
Clear all

Check out my comparison matrix: Kling, Claude, and local LLMs for our use cases.

3 Posts
3 Users
0 Reactions
4 Views
(@davidl)
Eminent Member
Joined: 3 days ago
Posts: 20
Topic starter   [#19203]

We've been evaluating AI coding assistants across our engineering teams for the past six months. The goal wasn't to find a "best" one, but to map each tool to specific, high-cost engineering activities where they either save significant time or introduce unacceptable risk. The hype around "AI coding" is deafening, so we built a concrete decision matrix based on our actual workflows.

The core criteria we measured were:
* **Latency & Throughput:** Time to first token and tokens/sec for code generation tasks. This directly impacts flow state.
* **Context Handling:** Accuracy and reliability with large codebases (50k+ lines), specifically for refactoring and understanding distributed system flows.
* **Infrastructure Cost:** Direct cost (API) vs. indirect cost (self-hosted GPU infra, engineering time to maintain).
* **Determinism & Control:** Ability to get the same output for the same input, crucial for generating configs (Terraform, Ansible, Prometheus rules).
* **Local Data Security:** For dealing with production incident data, architecture diagrams, and performance benchmarks.

Here's the distilled matrix for our three primary use cases:

| Use Case | Primary Tool | Why | Key Benchmark / Trade-off |
| :--- | :--- | :--- | :--- |
| **Interactive Day-to-Day Coding (IDE)** | Claude (via API) | Consistently superior at understanding nuanced requests and complex existing context. Its longer context window is reliable for cross-file refactors. | Latency is higher than Kling (~1200ms vs ~400ms avg TTFT), but the reduced iteration cycles due to better comprehension net a 15-20% time save per task. |
| **Generating Boilerplate & Configs** | **Kling** | Unbeatable for speed and deterministic output on well-defined tasks. Generating repetitive but intricate Prometheus alerting rules or Kubernetes manifests is where it shines. | We scripted it using the API. Example benchmark for generating 10 varied Prometheus rules: Kling completed in 4.2s avg, Claude 9.8s, local LLM 22.1s. Cost per 1000 tasks is negligible. |
| **Internal Code & Log Analysis** | Local LLM (Llama 3 70B on our own infra) | Non-negotiable for security. We feed it production error logs (sanitized), performance profiles, and full service code to ask "why is this slow?" No data leaves the perimeter. | Throughput is poor (18 t/s), but data sovereignty is absolute. Initial GPU cluster cost is high, but marginal cost per query is low at our scale. We would **never** send this data to a hosted service. |

**The Pitfall Most Teams Will Hit:** They'll pick one tool for everything. That's a mistake. Kling is mediocre at deep code analysis but stellar for scripted generation. Claude is slower but understands subtle requirements. Local LLMs are expensive to run but essential for sensitive data.

**Our Integrated Workflow:**
1. **Scripted Generation (Kling):** We use the API to generate Terraform modules, monitoring configs, and load test plans from templates.
```bash
# Example: Generating a standardized set of dashboards
KLING_PROMPT=$(cat ./templates/grafana-dashboard-prompts.txt)
curl -X POST https://api.kling.ai/v1/completions
-H "Authorization: Bearer $KLING_KEY"
-d "{
"prompt": "$KLING_PROMPT",
"model": "kling-code",
"max_tokens": 4000,
"temperature": 0.1
}" > ./generated-dashboards/json
```
2. **Deep Refactoring (Claude):** Manual use in the IDE for large-scale changes, like splitting a monolithic service.
3. **Security-Sensitive Analysis (Local LLM):** Internal tool that bundles code and logs, sends to the local inference endpoint, and returns analysis.

**Verdict:** Kling has earned a permanent place in our toolkit, but **only as a specialized high-velocity code generator**. It's not our generalist. The ROI was clear once we stopped trying to make it do everything and instead automated the tasks where its speed and determinism excel. The moment you need nuanced understanding or have sensitive data, you must switch tools.

—DL


Benchmarks or bust


   
Quote
(@jacksonj)
Estimable Member
Joined: 6 days ago
Posts: 64
 

Love this approach. "Hype is deafening" is so true. I keep seeing "best AI for coding" lists that feel totally detached from real work.

Did you consider "determinism & control" for any non-config tasks, like generating test data? I've had issues with that.

Also, curious - did any tool surprise you by being bad at a specific, common thing you assumed it'd handle?


Thanks!


   
ReplyQuote
(@devops_contrarian_42)
Estimable Member
Joined: 4 months ago
Posts: 117
 

Good question on determinism. That's the hidden tax with these tools. You can't version-control a non-deterministic output. For test data specifically, I gave up and went back to old-school libraries like Faker. Less magic, but at least the build doesn't break randomly.

The biggest surprise for me was how bad they all are at straightforward YAML/configuration. They'll hallucinate non-existent properties for common tools like Ansible or Terraform. You'd think that's the easiest part.


Keep it simple


   
ReplyQuote