Hey folks, I've been evaluating centralized logging platforms for a new data platform we're building, and Sumo Logic keeps coming up. The marketing materials talk a good game about petabyte scale, but I'm always skeptical until I hear from teams actually running it at serious volume.
We're projecting about 12-15 TB of log/event data daily, mostly from our ETL pipelines (Airflow, dbt), application servers, and cloud service audit trails. My main questions for anyone running at this scale:
* **Ingestion performance:** Any major throttling or delays during peak hours? How's the agent/collector overhead?
* **Query patterns:** Are complex, multi-line parse queries still responsive on, say, a week's worth of data? What's your typical time-to-first-result?
* **Cost reality:** Does the pricing model hold up, or do you get hit with unexpected charges from certain query types or data scans?
* **Pipeline integration:** Do you use their hosted collectors or push directly to an HTTP source? Any issues with data durability or schema management?
I've been prototyping with a smaller dataset and the _parse regex_ and _timeslice_ operators are powerful, but I'm worried about that scaling. For example, a query like this on a few GB is fast, but what about 10TB?
```sql
_sourceCategory=production/kubernetes
| parse regex "(?d+.d+.d+.d+).*?(?d+)ms"
| timeslice 5m
| count by _timeslice, ip
| transpose row _timeslice column ip
```
Would love any war stories, config tips, or even "we switched to XYZ because..." insights. Especially interested if you're coupling it with a data warehouse (BigQuery, Snowflake) for longer-term trends.
--diver
Data is the new oil - but it's usually crude.
We've run Sumo Logic at a similar scale, around 10TB/day from a mix of Kubernetes, AWS native services, and application logs. On your specific points:
Ingestion is generally stable, but you'll hit delays if your data isn't partitioned correctly. We use hosted collectors with HTTP Sources, and the main overhead is network egress from our cloud provider, not the collector itself. The durability is fine, but you must design for their partition keys - if you don't, you'll see ingestion lag during peak hours as you overwhelm a single partition.
The query cost is the real catch. Complex parse operations, especially `_parse regex` across a week of data, will scan every byte and burn through your contracted data scan allowance extremely fast. Your time-to-first-result might still be under ten seconds, but your finance team will get a nasty surprise. We had to implement very strict query guardrails and train engineers to use pre-computed fields and scheduled views.
I'd advise you to model your expected monthly query volume just as carefully as your ingestion volume. The pricing model holds only if you treat ad-hoc, full-scan queries as exceptional events, not daily operations.