Has anyone benchmarked ThreatConnect's correlation engine speed vs. others?

I'm a security architect at a 500-person MSP. We've run ThreatConnect (TC) in production for 4 years, correlating feeds from over 50 client networks, and I've stress-tested its API against our custom-built ELK stack and a MISP instance.

* **Correlation API Latency:** The `/api/v2/indicators/search` endpoint gets sluggish past 200k indicators. A direct ID lookup on a warm cache is sub-200ms. A broad correlation search on a text value (like `descriptionContains`) against that same 200k database often takes 3-5 seconds. It's not linear, it degrades.
* **Webhook-to-Action Loop:** Using TC as the correlator, our median time from receiving a webhook to firing a script via Playbook is 1.2 seconds. The 95th percentile spikes to 8 seconds, which we traced to JVM garbage collection pauses during bulk indicator insertions happening concurrently. You need to isolate your correlation node.
* **Bulk Enrichment Throughput:** Parsing and correlating a batch of 1000 observables in a JSON payload takes about 45 seconds via their batch API. For pure parsing, it's snappy (under 2s), but the correlation add-on is heavy. We routinely hit timeout issues until we split batches into chunks of 200.
* **Hard Limitation on Scale:** Their document size limit for indicator custom objects (where you stash context) is 4MB. If you're pulling in rich intelligence reports, you will hit this and have to fragment data, which kills correlation speed because now you're making multiple DB calls.

For your scenario of 50 observables via webhook, expect a median response of 1.5 seconds, but plan for the 5+ second outliers that will bottleneck your automated response. If sub-second, consistent correlation on >500k indicators is your requirement, TC won't hit it without significant hardware and careful tuning. For that scale, I'd look at a purpose-built streaming pipeline.

My pick: I recommend ThreatConnect only if you operate in the 50k-300k indicator range and need the integrated Playbooks for automation. If your primary need is raw, high-speed IOC correlation on a massive scale, a custom ELK/OpenSearch build is cheaper and faster, but you lose the built-in workflow. Tell us your exact indicator volume and whether you need the built-in SOAR more than raw speed.

Your specific scenario with 50 observables per webhook is where we saw unpredictable bottlenecks. The correlation isn't a single check, it's often multiple API calls per observable for different IOC types. That 3-5 second latency per call user413 mentioned means your total response time balloons fast.

We got around it by shifting the correlation logic. Instead of querying ThreatConnect's API directly in the hot path, we now pull a daily snapshot of high-fidelity indicators into a local Redis cache. The webhook handler checks there first, which is sub-millisecond. It only calls out to TC's API for slower, complex correlations or if the local check flags something. It's an extra layer but keeps the automation loop under a second.

For 10k indicators, the API is generally fine. The real performance cliff comes from the complexity of your search queries and the total database size, not the batch size of a single call. Avoid using broad text-matching operators in your automation if you need consistent speed.