After deploying Twingate across multiple production environments to secure backend services, I identified a critical visibility gap: operational health metrics for the connectors themselves were primarily accessible through the Twingate Admin Console or scattered CLI commands. For a platform built on zero-trust principles, the inability to centrally monitor the vital components facilitating that trust—especially across large fleets—was a notable operational hurdle.
To address this, I developed a monitoring dashboard using the Twingate Connector's local metrics endpoint and Grafana. The connector exposes a `/metrics` endpoint (on port 8081 by default) providing Prometheus-formatted data on resource usage, network activity, and connection states. This post details the configuration to scrape, store, and visualize these metrics effectively.
**Core Components of the Setup:**
1. **Prometheus Scrape Configuration:** A dedicated job in `prometheus.yml` targeting the connector pods/instances.
```yaml
- job_name: 'twingate-connectors'
scrape_interval: 30s
static_configs:
- targets: ['connector-host-ip:8081']
labels:
component: 'twingate_connector'
environment: 'production'
metrics_path: '/metrics'
```
2. **Key Grafana Panels & Queries:** The dashboard focuses on actionable, stateful metrics.
* **Connector Status & Uptime:** A simple `up{job="twingate-connectors"}` query for instant state.
* **Active Tunnel Count:** The most direct measure of connector load.
```
sum by (instance) (twingate_connector_tunnels_active)
```
* **Network I/O for Relay/Resource Traffic:** Separates traffic directed to Twingate relays vs. your protected resources.
```
rate(twingate_connector_network_bytes_received{type="relay"}[5m])
rate(twingate_connector_network_bytes_sent{type="resource"}[5m])
```
* **Memory & CPU Utilization:** Standard process resources from the `twingate_connector_process_*` metrics.
3. **Alerting Rules Example:** Configured in Prometheus to trigger on conditions indicative of degradation.
```yaml
- alert: TwingateConnectorDown
expr: up{job="twingate-connectors"} == 0
for: 2m
labels:
severity: critical
annotations:
summary: "Twingate connector {{ $labels.instance }} is down."
- alert: HighConnectorTunnelCount
expr: twingate_connector_tunnels_active > 200
for: 5m
labels:
severity: warning
annotations:
summary: "Connector {{ $labels.instance }} has a high number of active tunnels ({{ $value }})."
```
**Pitfalls & Considerations:**
* Ensure firewall rules allow Prometheus to reach port 8081 on all connector instances.
* The `twingate_connector_tunnels_active` metric is a gauge, not a counter; use `sum()` for aggregation across instances, not `rate()`.
* Correlate spikes in `twingate_connector_network_bytes_received{type="relay"}` with increased user activity or potential DDoS patterns against your network entry points.
This dashboard has been instrumental in capacity planning and incident response, moving connector health from a "black box" to a first-class citizen in our observability stack. I'm interested if others have explored similar integrations or are tracking different key metrics from their connectors.
-ck