I am currently analyzing a significant and unexpected cost driver in our OpenShift 4.12 deployment that has manifested as a severe memory consumption issue, and I require the community's analytical input. The primary suspect is the OpenShift router component, which is built on HAProxy. Our operational metrics indicate that router pods are experiencing sustained, linear memory growth, leading to periodic Out Of Memory (OOM) kills and subsequent pod restarts. This is not a transient spike but a consistent pattern observed across multiple production clusters hosting approximately 150-200 services each.
Our initial investigation points to the dynamic nature of our environment. We frequently create and destroy `Route` objects for development and testing purposes. It appears the router's memory is not being efficiently garbage-collected after these objects are deleted. The memory footprint of a single router pod often grows from an initial ~500MiB to over 2.5GiB within 48-72 hours, at which point the pod is recycled. This constant churn and the resource overhead required for restart cycles are creating both reliability concerns and tangible cost implications.
We have reviewed the standard configuration parameters. The current router deployment configuration is as follows:
```yaml
spec:
template:
spec:
containers:
- args:
- --publish-metrics
- --metrics-type=haproxy
- --expose-metrics
- --stats-port=1936
- --name=$(NAME)
env:
- name: ROUTER_THREADS
value: "4"
- name: ROUTER_SERVICE_HTTPS_PORT
value: "443"
- name: ROUTER_SERVICE_HTTP_PORT
value: "80"
- name: ROUTER_TCP_BALANCE_SCHEME
value: source
- name: ROUTER_METRICS_TYPE
value: haproxy
resources:
requests:
memory: 256Mi
cpu: 100m
limits:
memory: 2Gi
cpu: 500m
```
Key questions for which I need quantitative or deeply technical perspectives:
* What is the precise relationship between the number of `Route` objects and HAProxy memory allocation within the OpenShift router? Is there a known memory leak in specific versions of the router image, or is this expected behavior for a high-churn environment?
* Have you implemented successful garbage collection tuning or operational patterns to mitigate this? For instance:
* Does adjusting `ROUTER_THREADS` or implementing a `ROUTER_MAX_CONNECTIONS` have a measurable impact on memory retention?
* Is there a viable strategy for scheduled router pod rotation (e.g., via a `CronJob`) as a cost-effective stopgap versus increasing memory limits?
* What is the cost-benefit analysis of sharding routers by namespace or application type versus scaling vertically?
* Are there specific `HAProxy` global or defaults section tunables that can be injected via the `ROUTER_HAPROXY_CONFIG` environment variable to control memory allocation more aggressively?
My immediate priority is to stabilize memory usage to prevent OOM kills, but the underlying financial objective is to right-size the resource requests and limits to avoid over-provisioning. Simply increasing the memory limit to 4Gi feels like an inefficient and costly workaround. I need to understand the root cause to apply a precise and cost-optimized fix. Please provide any relevant metrics, configuration snippets, or operational experiences from your own cost analysis.
Show me the bill.
CostCutter