Skip to content
Notifications
Clear all

Rolled out Cursor to 20 engineers on a K8s platform - what broke

2 Posts
2 Users
0 Reactions
63 Views
(@cloud_cost_owen)
Reputable Member
Joined: 7 months ago
Posts: 181
Topic starter   [#26946]

Rolled out Cursor to our platform team last week. Big win for dev velocity... until our staging cluster started throwing weird errors. Turns out, a few engineers used its "Agent" mode to generate some K8s manifests.

The main culprit? It kept injecting `imagePullPolicy: Always` into every Pod spec, even for our internal ECR images tagged with specific SHAs. This bypassed our local image cache and hammered our container registry, hitting throttling limits.

```yaml
# Example of the generated patch
spec:
containers:
- name: app
image: 123456789.dkr.ecr.us-east-1.amazonaws.com/my-app@sha256:abc123
imagePullPolicy: Always # <-- This was the problem
```

**Quick fixes we had to implement:**
* Added a pre-commit hook to validate manifests for the rogue `imagePullPolicy`.
* Updated our internal docs with a Cursor context file to guide proper K8s spec generation.
* Brief team sync on using "Agent" for exploration, but final configs should be from standard Composer mode.

Overall, still a net positive! But a good reminder that AI-generated infra code needs the same guardrails as human code.

#savings



   
Quote
(@grafana_knight_shift_2)
Honorable Member
Joined: 6 months ago
Posts: 472
 

Good catch on the registry throttling. That's the kind of subtle, cascading failure that's easy to miss until your alerts fire.

We saw something similar when an AI coding assistant kept setting Prometheus scrape intervals to odd values like `17s`. It broke a bunch of our latency calculations and SLO definitions because they were all built on the assumption of `15s` or `30s` intervals.

Your pre-commit hook is smart. I'd also suggest adding a metric to track image pull failures from the registry. It can give you an early signal before you hit a hard throttle limit. Something like:

```yaml
- record: cluster:ecr_pull_errors:rate5m
expr: rate(kube_pod_container_status_terminated_reason{reason="ImagePullBackOff"}[5m])
```

This kind of thing is exactly why we treat any generated config as untrusted code that needs full validation. The speed gain is real, but so is the blast radius.


Sleep is for the weak


   
ReplyQuote