Let's get this out of the way: you don't deploy a suite that ingests, correlates, and analyzes telemetry without expecting someone to point that same lens back at it. So after our third "unexplained" latency spike in threat intelligence feeds, I did the obvious thing and wired up its own metrics to Grafana. Turns out, watching the watcher is a depressing hobby.
Anomali's docs talk a big game about scalability and performance, but they're curiously quiet about what normal operational telemetry looks like. You're left guessing. So I hooked into the API for job status, tapped the Kafka consumer lag for the internal streams (yes, they use Kafka under the hood, surprise), and pointed Prometheus at the JVM metrics from the various Spring Boot services. The dashboard isn't complicated. It shouldn't need to be.
The highlights, or rather, the lowlights:
* The "Match" service heap usage looks like a sawtooth wave from a broken signal generator. It climbs steadily to 95%, triggers a GC, drops to 40%, and repeats every 90 minutes. This isn't a leak, per se, but it's the kind of lazy memory management that tells you no one ever tuned the JVM args for a sustained production load.
* Internal Kafka topic lag spikes to the thousands during what should be routine IOC updates. The consumer groups are clearly not keeping up, and the partitioning strategy seems naive. We're not even at high volume.
* The PostgreSQL connection pool for the main platform database sits at 90% utilization during idle periods. God help us during a full corpus update. I found the connection string config; it's using the default Hikari settings. Amateur hour.
Here's the basic Prometheus query that exposed the connection pool issue. It's not rocket science.
```sql
avg by (application) (hikari_connections_active{application="anomali-platform"}) / avg by (application) (hikari_connections_max{application="anomali-platform"}) * 100
```
The real kicker? All of this is observable with the tools they bundle and the APIs they expose. They just don't do it themselves, or if they do, they ignore it. It feels like the engineering focus stops at "feature works in demo." Operational maturity is an afterthought.
I'm now spending more cycles tuning and monitoring Anomali than I am on the actual pipeline it's supposed to support. The cost of this "insight" is needing another layer of insight to keep it alive. The irony is thicker than a stale log file.
Has anyone else gone down this rabbit hole? I'm particularly interested in whether you've found sane baseline thresholds for their JVM GC pressure or if you've re-architected their Kafka stream topologies. I'm about to start tweaking broker settings and I'd rather not learn all their internal topics the hard way.
-- old salt