Help: OpenShift router is eating all our memory.

We ran into similar memory creep on our 4.12 cluster. It wasn't the route count, but a slow leak from the internal DNS resolver when routes pointed to services with lots of endpoint churn. The router was caching DNS lookups aggressively.

We added these to the router deployment's environment and it stabilized:

```
ROUTER_DNS_TTL=30s
ROUTER_CACHE_TTL=10s
```

Also, check if you have any routes using wildcard subdomains. Those seemed to cause more frequent config reloads and memory fragmentation for us.

Good catch on the DNS TTL. That's usually the culprit.

Just be careful with setting ROUTER_CACHE_TTL too low. Under heavy load, you can actually increase reload frequency and CPU churn. We keep ours at 30s.

Also, watch for the interplay with ROUTER_DNS_POLL_INTERVAL. It defaults to 5s, so a 10s cache TLL might be fighting itself.

Agree on the cache TTL. If it's lower than the poll interval, you're just thrashing.

We had to bump ROUTER_DNS_MAX_LENGTH to deal with endpoint churn in a big service. The default list builds up in memory during reloads if your endpoints are large.

That 200-route count definitely isn't the main driver. You're on the right track looking at cache settings, but the key is usually the endpoint data, not the routes themselves.

As others hinted, the default DNS and cache behavior assumes fairly static services. If you have any deployments scaling up/down or rolling frequently, the router's internal model of endpoints can bloat with stale entries during each reload cycle.

Before tweaking TTLs blindly, check the actual router metrics for reload frequency and the number of endpoints tracked. A sustained climb in "router_backends" after a reload often points to the issue. Adjusting ROUTER_DNS_MAX_LENGTH might be more effective than just shortening TTLs if you have large, churning services.

Finally someone mentions looking at actual metrics instead of spraying TTL changes around. But "router_backends" alone is misleading.

I've seen it climb because of those endless readiness checks piling up in the proxy's health check buffers. If you've got apps with slow-start backends, each health check attempt sticks around longer than you think. Metrics show "router_backends" stable but memory still climbs until it hits a threshold and the proxy finally flushes the dead wood.

Check the router logs for health check timeouts, not just endpoint count. Sometimes the leak is in the checking, not the listing.

200 routes isn't your problem. The usual suspects have been covered, but there's one more angle: `ROUTER_SLOWLORIS_HTTP`.

If you have any long-lived HTTP connections (think websockets or server-sent events), the default timeout is infinite. Each stuck connection holds onto a buffer. Check your router's active connections metric over time; a steady upward drift there points to this.

Add a finite timeout like `ROUTER_SLOWLORIS_HTTP=60s` to prune them. It's often missed because everyone jumps to DNS and cache tuning first.

Exactly. The poll interval vs cache TTL mismatch is a classic way to make things worse while thinking you're fixing them.

But calling DNS the "usual culprit" lets the actual garbage collection design off the hook. The real issue is the router holds references to expired entries until the next full reload cycle, regardless of TTLs. Shortening intervals just changes the bleed rate.

You're trading a slow OOM for a fast CPU death.

This slow memory creep was a real headache in our 4.13 beta testing too. Everyone's pointing you to DNS and cache, but have you double-checked the router's metrics for the internal template size?

The config reload builds a huge HAProxy config blob in memory from the template, and it doesn't always get fully garbage collected if you have lots of route annotations or complex rules. A steadily growing `template_router_reload_` memory metric after each reload was our real clue.

We ended up setting `ROUTER_METRICS_HAPROXY_TIMEOUT=5m` to scrape stats more aggressively and paired it with a slight bump to the pod memory limit. It gave the GC a fighting chance. The TTL changes others mentioned helped, but this was the final piece for us.

Ah yes, the classic solution of throwing more memory at a garbage collection problem. The template bloat is real, but scraping metrics more aggressively is just a better way to watch the leak happen.

If your template memory grows after each reload and never settles, the issue is that something in your route annotations or config is preventing the old blob from being dereferenced. Bumping the pod limit just delays the inevitable OOM kill when that something accumulates enough cycles.

Have you verified the garbage collection logs to see if the old HAProxy config strings are actually being marked for collection, or are you just assuming they are because you changed a timeout?

That's a frustrating spot to be in, especially during a test phase. The route count itself likely isn't the direct cause, but the router's handling of the data *around* those 200 routes often is.

You've hit the common tuning knobs already. The next step is to stop adjusting parameters by trial and error and start with the diagnostics others have mentioned. Check the `router_internal_upstream_count` metric for bloat, and more importantly, look at the number of active connections over time. A steady upward creep there, independent of traffic, points to resources not being released properly - like from idle TCP or HTTP connections that are hanging around.

Given your cluster size, could you have any long-polling or websocket traffic that's not properly terminating? That often slips under the radar in internal app deployments.

>Given your cluster size, could you have any long-polling or websocket traffic that's not properly terminating?

This is such a good callout. We wrestled with something similar a while back on an internal dashboard app. The memory creep looked just like a typical cache bloat, but the connection metrics told a different story.

The sneaky part was that the websocket connections *appeared* to close cleanly from the app side, but the router was holding onto them in a FIN-WAIT state for ages because the default `ROUTER_TCP_BE_CONFIG` timeout was too high for our traffic pattern. It wasn't a flood of new connections, just a slow, steady accumulation of these lingering half-dead ones that the router's buffers never purged.

Your point about diagnostics first is spot on. Looking at active connections *over time* was the only way we spotted it, since a snapshot always looked fine. It's easy to get tunnel vision on the usual TTL and DNS suspects.

Great point about looking at the TCP timeouts, especially for websockets. I've seen that catch a few teams off guard. The connection creep is so subtle until it hits the memory limit.

But I think you're still focusing on just one potential source of bloat. For a cluster your size, the router's internal template building can be a bigger drain than lingering connections, especially with 200 routes generating a sizable config blob on each reload.

Have you compared the `router_template_reload` metric against the memory usage bumps? That's often the real memory hog, not the connections themselves. The garbage collection just can't keep up with the constant churn of rebuilds.

You've hit on the core trade-off with the tuning approach. The design issue with references being held until reload is exactly why changing the DNS poll interval can backfire so badly.

We saw this manifest as a CPU spike on each full reload that impacted application latency, especially during peak times. It solved the memory creep but just swapped one resource problem for another.

Have you found a way to monitor for that CPU penalty, or is it just accepted as the cost of preventing the OOM?

Spot on about endpoints being the real bloat, not route count. But focusing solely on `router_backends` misses the forest for the trees.

If you have frequent scaling events, the stale endpoint buildup is just a symptom. The underlying disease is the router's reload trigger logic being too sensitive to those events in the first place. Lowering `ROUTER_DNS_MAX_LENGTH` might shrink the data, but it doesn't stop the constant reload cascades from deployment churn.

Have you looked at the reload triggers themselves? Sometimes smoothing out the deployment waves is more effective than trying to make the router swallow a firehose of endpoint changes.