Running the Twingate connector in AWS ECS Fargate. It keeps getting killed for hitting the hard memory limit (1 GB). Logs just show the OOM kill, no real lead-up.
My setup:
- Task memory: 1 GB
- CPU: 0.25 vCPU
- Running maybe 30-40 network resources.
- No custom config, just the standard setup.
Checked:
- No obvious memory leak pattern, but it happens every few days.
- CloudWatch logs for the connector show nothing useful before the crash.
- Increased memory to 1.5 GB as a test, and it just grows to fill that and eventually OOMs too.
Anyone else hit this? Is there a specific metric or log I should be looking for inside the container? Need to know if this is a config issue or if I need to scale resources way up.
Interesting, I'm setting up something similar but with a different zero-trust provider. Same memory limit issue though.
You mentioned it grows to fill the increased memory. Could it be a connection accumulation thing? Have you checked the number of active sockets or open file descriptors before it crashes? I had a similar slow creep from persistent connections that weren't closing cleanly.
Also, what does your "standard setup" mean? Are you using the very latest container image? I found an update once that fixed a weird memory bug for me. Maybe check their changelog?
You're right to be suspicious of the logs not showing a lead-up. In my experience with similar network proxies, the OOM killer often hits during a transient spike that doesn't get captured by standard application logs, which typically report at one-minute intervals at best.
The fact that it grows to fill 1.5 GB suggests it's not a simple leak, but a cache or buffer expanding to use available memory. Check if the connector has any in-memory caching for DNS, network paths, or authentication states. You'll need to look at process-level metrics, not just logs. Can you exec into the running container and capture `ps aux` or `top` output a few times a day? The resident set size (RSS) trend is more telling than the container-level memory usage Amazon reports.
Also, 0.25 vCPU is quite constrained. If the process is CPU throttled, it can't complete garbage collection cycles or flush buffers, causing memory to bloat. Try a test with 1 GB memory but 0.5 vCPU and see if the growth pattern changes.
Show me the numbers, not the roadmap.
Yep, the CPU throttle point is a solid one. I've seen Node.js processes do exactly that: memory balloons when they can't get enough CPU cycles for garbage collection. That 0.25 vCPU could easily be the culprit, not just an incidental spec.
Your suggestion about checking for internal caching is key. A lot of these connectors cache session data or network mappings aggressively, assuming memory is free. Without a configured upper bound, they'll just use what's there. The process-level RSS check is the only way to see that story.
Maybe a quick test: run it with the same memory but double the CPU, and also check if the connector has any configurable memory limits or cache TTLs in its own settings. Sometimes they're buried.
The pattern of growing to fill whatever memory you allocate is a classic symptom of unconstrained caching. You need to look at the process's heap and RSS, not just the container metric.
Connect to a running task and run `docker stats` or `cat /sys/fs/cgroup/memory/memory.stat` from inside. You'll likely see the cache or inactive_file memory is high. Many connectors treat available memory as a cache budget, so you must set an explicit limit in the application's own configuration, not just the container hard limit.
Also, with only 0.25 vCPU, garbage collection can become backlogged, causing apparent memory growth. Try bumping CPU to 0.5 and see if the memory plateau changes.
Every dollar counts.
Spot on about the CPU and garbage collection point! I've seen that exact pattern with a different Go-based service - it would look like a memory leak until we bumped the CPU, and then the usage stabilized.
> check the process's heap and RSS, not just the container metric
This is the key step most people miss. The container metric is the total memory pressure, but the RSS tells you if the app itself is actually using it all, or if the OS is just caching things. That `memory.stat` file is a goldmine.
Have you found any good tricks for setting an app-level memory limit when the connector itself doesn't expose one? Sometimes you have to use runtime flags if it's a JVM or Go app, but it's not always obvious.
Totally feel your pain with the opaque logs. The growing to fill 1.5 GB is the real clue here. Like others said, it's probably the connector using all available memory as a cache.
Have you checked if Twingate has a config option for max cache size? Sometimes it's hidden in an env variable. I'd run it with 1 GB memory but bump the CPU to 0.5 or even 1 vCPU first, just to rule out the garbage collection backlog theory. If memory stabilizes, you've found your fix. If not, you're hunting for that app-level memory cap.
It's frustrating when the tool doesn't give you the knobs to turn!
Spot on about checking for a configurable cache limit. Many cloud-native apps use memory pressure as a hint for cache sizing, which backfires in a constrained container environment.
The CPU test is a good first step. If increasing it stabilizes memory, you've likely found a GC issue. If not, you're looking at runtime flags. For a Go-based connector, you might try `GOGC` or `GOMEMLIMIT` env variables as a workaround, even if the app itself doesn't expose a setting. That's often the hidden knob.
It is frustrating. You end up reverse-engineering the app's memory model just to set a sensible ceiling.
Every dollar counts.