I've been conducting a long-term cost and reliability analysis of our Twingate deployment, which utilizes auto-scaling groups for Connectors in AWS, and I've identified a persistent operational issue that appears to contradict the intended stateless architecture. The core problem is that our Connector instances within the ASG are recycling far more frequently than anticipated—often multiple times per day—despite stable load. This is triggering a cascade of inefficiencies, not just in user connection churn, but also in cloud cost implications due to the lifecycle hooks and increased data transfer during constant re-initialization.
My hypothesis is that the Connector is retaining or requiring some form of state that is not being adequately managed by the standard launch template, causing health checks to fail and triggering termination. I have methodically ruled out the typical AWS culprits: the ASG scaling policies are sound, target tracking is based on stable CPU metrics, and the launch template uses the latest Twingate-provided AMI. The instances successfully join the network and pass initial checks, only to fail later.
I need to understand the precise conditions for a Twingate Connector to be considered truly stateless. My investigation thus far points to several potential areas of statefulness:
* **Local ephemeral storage:** Is there any logging or caching mechanism writing to the instance's root volume that could fill up or become corrupted? We've configured CloudWatch Agent for logs, but perhaps there are undisclosed local artifacts.
* **In-memory state:** Could there be a memory leak or a stateful table related to active connections or network topology that accumulates and eventually causes a health check failure?
* **Dependency health:** The Connector's internal health check might be probing a downstream dependency (like a Relay or the Controller) and marking itself unhealthy based on a transient fault elsewhere in the system.
* **Configuration fetch at boot:** If the instance retrieves its full configuration from the Twingate Controller at every launch, what could cause that process to hang or fail intermittently after the instance is already marked healthy?
From a FinOps perspective, this instability negates the cost benefits of an ASG. We're not seeing the steady-state utilization that reserved instance planning requires, and the constant churn generates a non-trivial amount of additional data transfer costs (both to the Twingate service and between AZs) that are difficult to forecast.
Has anyone performed a deep-dive analysis on their Connector ASG lifecycle? Specifically, what are the exact health check parameters, and are there any recommended modifications to the UserData script or instance IAM policy to enhance resilience? A truly stateless Connector should be able to be terminated and replaced with zero impact on ongoing connections and with minimal overhead—our data suggests we haven't achieved that.
-- Liam
Always check the data transfer costs.