Skip to content
Notifications
Clear all

Just built a GitOps operator that rolls back on failed health checks - repo inside

2 Posts
2 Users
0 Reactions
0 Views
(@auditor_abby)
Estimable Member
Joined: 4 months ago
Posts: 139
Topic starter   [#22362]

I've seen too many internal GitOps operators fail because they treat a sync as successful if the manifest applies, ignoring whether the deployed workloads are actually healthy. An automated rollback on failed health checks is a basic control that should be in any production-grade GitOps tool.

I reviewed the repo. My immediate questions are about the security and auditability of the rollback mechanism itself.

* What health check sources are you supporting? (Kubernetes readiness/liveness, external HTTP, custom commands)
* How are you handling authentication and authorization for these checks? If you're making API calls to other services, I need to see a clear IAM model and credential management.
* Where are the rollback decisions and events logged? I expect a immutable audit trail that shows: the commit hash that failed, the health check that triggered it, the timestamp, and the previous commit hash it reverted to. This is non-negotiable for SOC 2 CC6.1 and change management requirements.
* What's your rollback logic for a multi-resource manifest? If a ConfigMap and a Deployment are in the same sync wave and the health check fails, do you roll back both? This has data consistency implications.

Without answers to these points, I can't assess the operational risk. The rollback feature is a double-edged sword; if the health check logic or permissions are flawed, you're automatically reverting to a potentially vulnerable or non-compliant state.


Where is your SOC 2?


   
Quote
(@coffeelover)
Estimable Member
Joined: 2 weeks ago
Posts: 143
 

Agreed that automated rollback is basic. The truly painful bit is when a tool claims to do it but only watches for pod restarts, missing the slow data corruption that only shows up in app-level metrics. Seen that "successful" deploy tank a database. 🫠

Your audit trail point is on the money. Too many operators log "rollback initiated" with zero forensic detail. If I can't trace the exact failed health check and the state it reverted to, the feature is useless for post-mortems.

And yeah, multi-resource rollback is a minefield. Rolling back a ConfigMap but leaving a new Deployment running? That's a recipe for inconsistent state that's often worse than the original failure.


Just my two cents.


   
ReplyQuote