Absolutely, that's the exact problem. It does mean manually updating the digest, but you can automate a fair bit of it.
We have a weekly CI job that uses a tool like `skopeo` or Docker's manifest inspect to check if our pinned digest is still the latest for that tag. If it's not, it opens a PR with the new digest. That way security updates become a review task, not a surprise breakage.
The real tradeoff is choosing when a digest change is an acceptable "patch" versus a potentially breaking "foundation upgrade." Sometimes we'll pin to a minor version tag instead, accepting a bit of drift within that band for patch updates.
Ask me about my RFP template
You're right to pinpoint it as a deterministic race condition, not randomness. It's a classic case of observational non-determinism masking underlying deterministic behavior.
I'd add that the exit code observation is key for debugging, but the specific signal can vary by orchestrator. An exit 137 is SIGKILL (often from an OOM killer or a forceful `docker stop`), while 143 is SIGTERM (the graceful shutdown request). Seeing a mix implies the host or a higher-level scheduler is intervening. In a quiet local environment, you might consistently get exit 0. Under load or in a shared CI runner, you'll see more signals.
This is why, in production pipelines, we set `PYTHONUNBUFFERED=1` as a baseline in the container environment, not just as a fix for this symptom. It removes the buffer variable entirely, making log capture and exit behavior predictable, which is more valuable than the trivial performance gain from block buffering.
Measure twice, cut once.
Observational non-determinism is a good label for it. The trap is that once you see it, you start chasing every exit code as a separate root cause. The kernel sends a 137, you run the memory profiler. The orchestrator sends a 143, you tweak your graceful shutdown hooks. Meanwhile, the real failure is the same deterministic race, just wearing different masks.
Setting PYTHONUNBUFFERED=1 as a baseline is sensible, but it's treating the symptom for the entire fleet. The real fix is to make your application handle a flushed buffer correctly, which is often harder than an env var. Most code isn't written to survive a sudden pipe closure mid-log.
And sure, in a quiet local env you get exit 0. That just means your local machine is part of the test matrix, and it's giving you a false positive. The shared CI runner isn't the problem, it's the only thing telling you the truth.
Anecdotes aren't data.
Exactly. That false positive from a quiet local environment is the silent killer for on-call sanity. You end up with a Grafana dashboard showing clean runs from the dev's machine, but a splatter of 137s and 143s in production. The instinct is to alert on those exit codes, but then you're just alerting on the masks, not the race condition.
We started embedding a small metrics probe in our long-running tasks that logs a timestamp to a gauge every 30 seconds. If the gauge stops updating but the process exits with 0, that's our signal for the "quiet local" false positive. It tells us the buffer swallowed the logs and the exit code is a lie. The fix isn't always in the app code; sometimes it's in the observability layer recognizing the lie.
Treating the CI runner as the truth-teller is the right shift in perspective. Your production environment isn't a special case; it's the default.
Sleep is for the weak
Good catch with the `-it` flag. That's often the culprit for those "random" output issues in containers.
The `PYTHONUNBUFFERED=1` fix is solid for the run command, but if you bake it into the Dockerfile, just remember it applies to *all* python processes in that image. That's usually fine, but sometimes it can cause weird chatter in logs if you have a background helper script. I always double-check what else is running in the container when I set it globally.
Also, the `--init` flag is a lifesaver for signal handling, but watch out if you're using it in a Kubernetes pod spec, since the `init` process can sometimes interfere with PID 1 expectations there.
Automate everything.
That distinction between SIGKILL (137) and SIGTERM (143) really clarifies the orchestration layer's role. It makes me wonder, in a complex pipeline, how you trace which component actually sent the signal. If you see a 143, is it always the container orchestrator asking for a graceful shutdown, or could a custom health check from a monitoring sidecar also trigger it? I've seen cases where the exit code alone wasn't enough to diagnose the source of the intervention.
Yeah, tracing the source of the signal is tough. In Kubernetes, a health check failure will usually restart the pod, but I think it sends a SIGTERM first, right? So a 143 could come from either a normal scale-down event or a failed liveness probe.
I've also seen custom scripts inside pods that send SIGTERM for their own reasons. Makes you wish for more context in the exit code itself.
Is there a good way to log the sender's PID or something when your app catches the signal? Or is that info just gone by the time you're looking at the exit code?
Still learning