Migrating a legacy app to ECS Fargate. Used AWS Secrets Manager for database credentials. Found a weird thing: environment variables from the task definition were showing up in the application's `/proc/self/environ` on a different, unrelated task.
Turns out, if you use the `secrets` field in the task definition and the secret value is a *string* (not the full ARN), the Fargate host can cache it. Under specific scaling conditions, that cached value can leak into another task's environment block.
**Repro case:**
- Two ECS services (ServiceA, ServiceB) in same cluster.
- Both use task definitions with a secret referenced by *secret name only* (e.g., `"SECRET_KEY": "MySecretString"`).
- Trigger a rapid scale-up event on ServiceA.
- In some observed instances, ServiceB tasks will have ServiceA's secret value in their environment.
AWS Support initially said "impossible, isolation is guaranteed." Provided logs and a reproducible CloudFormation template. They've now acknowledged the caching issue.
**What's the actual ROI on using native secrets integration here?** If you need guaranteed isolation, you might have to fetch secrets at runtime via the SDK instead.
—CR
Ask me about hidden egress costs.
Oof, that's a nasty one. Great job on the repro case and getting AWS to acknowledge it. This is exactly why I'm hesitant about the "just use the native secrets field" advice for truly sensitive data.
You're right that the runtime SDK fetch is the isolation guarantee. The trade-off, of course, is complexity and cost - more API calls, error handling, and secret rotation logic in your app. For many, that's worth it.
One middle ground: if you must use the native integration, always use the full secret ARN, not the name string. The ARN forces a direct, uncached lookup to Secrets Manager. It adds a bit of config management overhead, but might close this specific caching loophole. Have you tried that as a mitigation?
Prod is the only environment that matters.