Hey everyone! 👋 I've been lurking in this subforum for a while, learning from all your insights, and now I find myself at a crossroads with my current team. We're a mid-market SaaS in the healthcare tech space, and our recent growth has... well, let's just say our current logging setup (a patchwork of self-hosted Elasticsearch, some CloudWatch, and a lot of hope) is creaking at the seams. With our HIPAA compliance requirements, the stakes are incredibly high for audit trails, security monitoring, and data integrity.
We've been evaluating Sumo Logic pretty seriously for the last quarter. The built-in compliance certifications (HIPAA, SOC 2, etc.) were a massive initial draw, and the way it handles log ingestion with field-level encryption is reassuring. But I wanted to reach out to this community for real-world, gritty details before we commit. My background is in databases and ETL pipelines, so I tend to look at these tools through that lens—how is the data *really* structured once it's in there? Can I build efficient dashboards without pulling my hair out?
Here’s a snippet of the kind of parsing logic we had to write for our old system, which I'm desperately hoping to avoid:
```sql
-- A simplified example of trying to correlate auth logs with DB audit logs
-- This is the 'before' dream we want to leave behind.
SELECT
l.user_id,
l.ip_address,
MAX(l.timestamp) as last_auth,
(SELECT COUNT(*) FROM db_audit_log d WHERE d.user_id = l.user_id AND d.timestamp > DATE_SUB(NOW(), INTERVAL 1 HOUR)) as db_actions_last_hour
FROM
auth_log l
WHERE
l.timestamp > DATE_SUB(NOW(), INTERVAL 24 HOUR)
GROUP BY
l.user_id, l.ip_address
HAVING
db_actions_last_hour > 100;
```
My specific questions for those of you using Sumo Logic in similar environments:
* **Search Performance:** For those dealing with terabyte-scale daily ingestion, how responsive are the searches when you need to trace a single user's journey across microservices? Do complex joins (or the Sumo equivalent) feel sluggish?
* **Cost Predictability:** The pricing model based on ingestion is clear, but have you been bitten by unexpected spikes (e.g., verbose debug logs during an incident causing a bill shock)? What's your strategy for keeping it predictable?
* **HIPAA-specific Workflows:** How seamless is it to manage and prove compliance for audits? Are there particular features like live-tail for real-time security monitoring that you rely on?
* **The Ecosystem:** We're increasingly cloud-native (K8s, AWS). How well does the collector/agent play with containerized environments? Any pitfalls during deployment?
I'm eager to hear both the glowing reviews and the painful pitfalls. Our migration from MongoDB to Postgres last year taught me that the devil is always in the details, especially around data schema and indexing. I'm applying that same scrutiny here!
Thanks in advance for sharing your wisdom.
—B
Backup first.
I'm a platform lead for a 125-person fintech. Our stack is Go/Kotlin on k8s, and we ship with GitHub Actions. We run DataDog for logging, metrics, and APM in production, and I've previously managed Splunk and ELK stacks at other shops.
* **Mid-market fit:** Sumo is built for your exact scenario - compliance-first mid-market. DataDog also targets this space but is more feature-broad. Splunk is the enterprise heavyweight; their sales cycle and minimums are often too much for companies under 500 people.
* **Real pricing and hidden costs:** Sumo's ingest-based pricing is predictable but can lock you in. At my last shop, we hit ~800 GB/day and the bill was steady at ~$12k/month. DataDog's log pricing is similar per GB ($0.10-$0.12), but their strength (and cost) is correlating logs with traces and metrics, which drives usage. The hidden cost with both is the query/analytics layer - complex queries on large time ranges chew through your committed volume fast.
* **Deployment/integration effort:** Sumo's collectors are simple to deploy. The main effort is normalizing your log schemas before ingestion. If you don't, you'll pay for that processing later and dashboard building becomes a pain. DataDog requires the agent on your nodes, which is more ops overhead but gives tighter integration with infrastructure metrics.
* **Where it breaks:** Sumo's query language is fine for basic filtering and aggregation, but for complex joins or programmatic analysis, it feels restrictive. You'll still need an external data lake for deep forensic work. DataDog's logging shines as part of their whole observability platform, but if you ONLY need logs, you're paying for a lot of unused capability.
I'd recommend Sumo Logic for your specific HIPAA-focused, logs-first use case. It's the straight tool for the job. If you see yourselves needing to tightly correlate log events with application performance traces and infra metrics in the next 12 months, then look at DataDog despite the complexity. Tell us your projected daily ingest volume and whether your security team requires on-premise storage.
Your focus on data structure and dashboard efficiency is the right lens for this. Sumo's data model is essentially a flattened JSON document store with metadata enrichment at ingest. The underlying query engine is a proprietary columnar format they call LogReduce. While it's optimized for text search and basic aggregations, complex joins or multi-stage transformations aren't its strength.
If you're coming from a database/ETL background, you'll hit a ceiling when you need to correlate data across disparate log sources with high cardinality. Building a dashboard that, for example, ties a specific user action (from app logs) to downstream API latencies (from proxy logs) and a specific database transaction (from audit logs) often requires cumbersome subqueries or exporting to another system. Their query language lacks the expressive power of proper SQL or even Splunk's SPL.
For your HIPAA audit trails, this becomes critical. You need to reconstruct exact sequences of events, which is a join-heavy operation. Test this during your proof-of-concept: take a real audit scenario and try to build that timeline query. Monitor the query performance as the date range expands. You might find Sumo adequate now, but it will degrade as your retained data grows.
Test it yourself.
You're absolutely right about the correlation challenge. We hit a similar wall trying to tie a patient record access (from our app audit) to the exact PDF generation request in our render service logs and then to the delivery status from our email logs. The subqueries in Sumo Logic became a real performance hog for that 72-hour audit window.
One workaround we had to adopt was pre-joining some of this data at the stream level using a middleware service before ingestion. It adds complexity, but for our core HIPAA-mandated audit trails, it was the only way to guarantee the query performance needed for an auditor's live demo. Feels a bit like cheating the system, though.
Integration Ian