I've been evaluating Google Chronicle for centralized log analysis, specifically for its performance in high-throughput, low-latency query scenarios common in security investigations. While its underlying infrastructure is impressive, I'm compiling a list of alternatives that prioritize similar architectural goals—columnar storage for fast analytics, petabyte-scale ingestion, and sub-second query latency on recent data—but are not the usual enterprise suspects (Splunk, Microsoft Sentinel).
My primary metrics for comparison are:
* **Query Latency:** P95 and P99 times for complex JOINs across 30 days of telemetry.
* **Ingestion Performance:** Sustained throughput in MB/sec/event-processing-core without pipeline backpressure.
* **Hot Data Access:** Query performance on data from the last 48 hours, which is critical for active investigations.
From my profiling, Chronicle's BigQuery backend provides strong analytical depth but can introduce variable latency on complex historical searches. I am seeking alternatives that might offer more predictable performance profiles.
Candidates on my current evaluation shortlist include:
* **Elastic Stack (Elasticsearch):** With proper tuning (e.g., time-series indices, warm/cold node architecture). The query latency for term aggregations on hot data is excellent, but managing the cluster for petabyte-scale has operational overhead.
* **Graylog:** Particularly with its OpenSearch integration. Focused on real-time processing, its pipeline engine can be less latent than Chronicle's rule matching for certain streaming use cases.
* **Humio (now part of CrowdStrike):** Its claim to fame is real-time, unbounded streaming query performance. I've seen benchmarks showing sub-second full-scans on ingested data, which is intriguing.
* **Datadog Security:** While an APM-first platform, its log management performance for correlated queries between traces, metrics, and logs is noteworthy from a latency perspective.
Has anyone conducted structured performance benchmarks between these and Chronicle for specific workloads, such as:
* Concurrent query throughput under load?
* The cost-performance ratio for storing >1PB of telemetry with fast retrieval?
* The impact of schema-on-read versus schema-on-write on query execution time?
I am particularly interested in any profiling results or flame graphs that reveal bottlenecks in the query execution path for these platforms.
sub-10ms or bust