Yes, we've done it, but "without it being a mess" entirely depends on your pre-ingest strategy. The unified platform part works great. The mess happens (or is prevented) long before data hits Devo.
For schemas, we used a hybrid approach: separate parsing pipelines for Infra and Sec teams right up until the final enrichment step. That let each team use field names and structures that made sense for their context. Then, everything fed through that mandatory core schema service others mentioned, which standardized the four critical correlation fields and *renamed* any truly conflicting fields before writing to a common domain. For example, Infra's `host` and Sec's `device_name` both got mapped to a unified `asset_identifier`. It wasn't automatic, but the pre-ingest service made it enforceable.
On performance and cost, the high-volume infra logs were the biggest threat. We tackled it by implementing a two-tier ingest pipeline right at the source. Tier 1 (security-relevant and high-value operational logs) went straight to the real-time Devo pipeline. Tier 2 (verbose debug, raw application trace) gets routed to a cheap object storage bucket first. It's still *queryable* from within Devo for forensics via a linked integration, but it doesn't clog the primary ingestion queue or inflate our hot storage costs. The key was getting both teams to agree on what "security-relevant" actually meant - which, honestly, took longer than building the pipeline itself.
Pipeline is king.
That's a pleasant fiction, but it ignores the political reality of who controls the "lanes on the same highway." The moment you have separate ownership of schemas and dashboards, you get a turf war over the query budget and the dashboard real estate. Sure, Devo can technically handle divergence, but can your organization handle the inevitable blame game when the security team's expensive, complex query blows the monthly spend because the infra team's schema change created a cartesian product nobody anticipated?
Your approach works only if there's an absolute, ironclad governance layer with veto power over any schema change that impacts cross-team joins. In my experience, that layer either doesn't exist or becomes the very bottleneck you were trying to avoid.
You've hit on the real problem - the hard part isn't the technology, it's cost attribution and political accountability. Your point about expensive, complex queries blowing the monthly spend is exactly right. We solved the turf war by implementing a strict chargeback model where the budget for the Devo platform was decentralized.
Each team owned their own query budget and was billed for their dashboard compute. That "blame game" became a straightforward financial report. If the infra team's schema change caused a spike in the security team's costs, the security team had to either adjust their queries or take it up with infra as an inter-departmental budget issue. It forced collaboration because the cost was visible and traceable, eliminating the need for a bottleneck governance layer with veto power. The governance became about cost transparency, not schema approval.
Data > opinions
Great question, and I'm watching this thread closely as we're considering something similar. You mentioned "high-volume, low-value infra logs" impacting security event performance - how do you even define what's low-value upfront?
I like the cost attribution idea from later posts, but I'm curious about the initial setup. Did you guys just start filtering on day one, or was there a period where you ingested everything and then worked backwards? Feels risky to guess what you might need for an incident later.
You've hit on the two biggest practical challenges right out of the gate. Starting with your second point about performance and cost, we absolutely did run into that issue. Our "day one" rule was that only security events and infra logs tied to defined alerting use-cases went into the high-performance, real-time pipeline. Everything else, like verbose debug logs, landed in a separate, cheaper domain with slower query performance. It forced teams to be intentional from the start.
For schemas, we blended into a common structure but *only* for a small set of core correlation fields, like asset identifiers and user IDs. Everything else lived in separate, team-specific tables. This meant security could parse a firewall log their way, and infra could parse a Kubernetes event theirs, without fighting over field names. The key was a mandatory pre-ingest service that mapped those team-specific fields to the common core. It added a step, but prevented the mess later.
The real tension, honestly, came from dashboard and query ownership. Even with separate tables, a poorly written cross-team query could tank performance for everyone. We didn't find a technical fix for that, only a social one: a clear, agreed-upon escalation path when one team's work impacted another's.
Keep it constructive.