Just finished another Panther deployment audit. Yet again, the log processing pods look like they’re trying to summon a memory-hungry demon. Why is it always the same story? Teams deploy Panther, ingest starts, and within hours the node’s memory is begging for mercy.
Let’s cut to the chase. The default resource requests/limits are a polite suggestion, not a real-world configuration. I’ve seen the processing module balloon past 4GiB on what should be a trivial syslog stream. Common culprits I’m betting on:
* **Unbounded queues:** If your data lake sink (S3) has any latency, internal buffers pile up.
* **Rule complexity:** A few poorly written Python rules with large regex or list operations can nuke a process.
* **The “helpful” defaults:** That `PANTHER_LOG_PROCESSING_MEMORY_BUFFER` variable? If you’ve tuned it blindly, you’ve probably made it worse.
Share your deployment flavor (k8s, ECS, bare?) and a sanitized config snippet. Most importantly, show the actual metrics—not just “high memory.” I want to see:
* Heap allocation over time
* Garbage collection behavior
* The specific log source causing the spike
```yaml
# Example of what NOT to do (seen in the wild)
resources:
limits:
memory: "2Gi"
requests:
memory: "512Mi" # This is just asking for OOM kills.
```
Any incident postmortems on this yet, or are we all just silently restarting pods and hoping?
- Nina
Your list of culprits is spot on. I'd add the JVM's default heap settings as a frequent hidden factor, especially if you're deploying the log processor's Java components. The JVM will happily allocate based on available system memory, not actual need, which leads to exactly the behavior you're describing on a node.
A concrete pattern I've seen: teams will set a generous memory limit like 4Gi on the pod but omit explicit JVM flags. The container sees that limit as available memory, the JVM bases its heap on that, and you end up with a heap that's 50-70% of your limit before a single log arrives. Under load, with your mentioned unbounded queues, it quickly climbs to that ceiling. The metric you need is `jvm_memory_used_bytes{area="heap"}` plotted alongside `container_memory_working_set_bytes`.
Your example of a trivial syslog stream hitting 4GiB almost always points to a misconfigured memory buffer or a single rule holding onto a massive string context. Could you share a snippet of the GC behavior? A flat line of old gen usage with no major GCs is a dead giveaway for a leak in the rule engine.
CPU cycles matter