We are approaching our annual Splunk license renewal and facing a significant storage cost increase due to the unchecked growth of our `notableevents` index. Our current retention policy is set to the default, and the volume of closed events is becoming unsustainable.
Our primary concern is maintaining a complete audit trail for compliance (SOX, PCI-DSS) while reducing active storage. We need to archive older, closed notable events, but we must ensure they remain accessible for historical review and audit purposes. Simply deleting them or adjusting the retention policy to delete them is not an option.
Key requirements for the solution:
* Archived events must remain searchable, ideally with their original context and investigative actions intact.
* The process should not break any correlation searches or risk-based alerting that might reference historical data patterns.
* The operational overhead of the archival process itself should be minimal.
We are evaluating a few approaches and would like practical feedback on their long-term viability and total cost of ownership:
* **Cold Storage / Frozen Archiving:** Using Splunk's built-in cold-to-frozen archiving to move data to a cheaper object store. What are the practical restore times and operational complexities you've encountered during audits?
* **Summary Indexing:** Creating a pruned summary index of closed events after a certain age. What is the best way to maintain the necessary field structure, and how do you handle the loss of raw event data?
* **External Data Export:** Exporting events to an external system (e.g., a relational database or dedicated log archive). What tools have you used successfully for this export, and how do you handle searchability?
What has been your experience regarding the ROI of each method? Specifically, we are looking for details on:
* Implementation and ongoing maintenance costs.
* Retrieval efficiency during an actual audit.
* Any hidden pitfalls that affected your security workflow.
Buy once, cry once.