We've been running Splunk ES on a mid-sized deployment (~400 GB/day ingestion) for several months. Our search head performance has degraded significantly, specifically during correlation search execution. After isolating the issue, the primary bottleneck appears to be our Asset and Identity lookup tables.
Our lookup configuration is relatively standard: we are using the `identity_lookup_expanded` and `asset_lookup_expanded` macros with CSV-based lookups that have grown to approximately 35,000 and 18,000 entries respectively. The searches invoking these lookups, particularly those with high concurrency like `Notable Event - Account Lockout`, are now taking 4-5 times longer than before. CPU usage on the search head spikes during these periods.
I have reviewed the Splunk documentation on lookup optimization. We've already implemented:
- Enforced field name matching to reduce unnecessary field extraction.
- Verified the CSV files are stored on the search head's local filesystem with appropriate permissions.
- Attempted to prune stale entries from the lookup tables.
My specific questions for the community are:
- At what scale have others moved from CSV lookups to a dedicated database (like a KV store or external SQL database)? Is 35k identities the tipping point?
- For those who transitioned to a KV store lookup, what was the observed performance delta for ES correlation searches? Were there any unexpected complications with ES's built-in macros or data model acceleration?
- Has anyone implemented a hybrid approach—for example, using a scheduled lookup cache to a summary index for high-volume identity fields—and seen measurable gains?
I am particularly interested in concrete metrics: reduction in search execution time, changes in search head resource utilization, and any operational overhead introduced by the new method. A side-by-side comparison of the different lookup backend options would be extremely valuable.
Your bill is too high.