Help: Cloud firewall logs are impossible to search during an incident. Alternatives?

We've all been there: the alert fires, something is clearly wrong, and you need to triage by diving into the firewall logs. In our cloud-native environment, we rely heavily on the native cloud firewall services (think AWS Network ACLs, Security Groups with VPC Flow Logs, GCP Firewall Rules with packet logging, Azure NSG Flow Logs). During a recent incident involving lateral movement attempts, the experience was... abysmal. The native tools are fine for ad-hoc, single-resource checks, but aggregating and searching across accounts, regions, and services under time pressure is a nightmare. The latency on VPC Flow Logs to S3/CloudWatch is a particular pain point.

My core issue is the **separation of data generation from analysis**. The cloud providers give you the raw log streams (which is good), but force you into their proprietary, often sluggish, analytics tools or generic object storage. To get real-time search, you're looking at a complex pipeline: log ingestion → transformation → indexing → dashboard. The mental load of managing this *and* ensuring it's capturing all relevant flow data feels like rebuilding a SIEM.

I'm considering a few architectural paths and would appreciate critiques or war stories:

1. **Standardized SIEM Ingestion:** Pipe all VPC Flow Logs/NSG logs directly to a dedicated SIEM (e.g., Splunk, Datadog, Sumo Logic). This seems obvious but the cost at cloud scale can be staggering, and you now have to manage parsers and field extraction.
2. **Open Telemetry Collector Pattern:** Deploy collectors (e.g., FluentBit, Vector) as DaemonSets on Kubernetes or on each subnet/host to scrape cloud-specific log sources, transform them into a unified schema (e.g., ECS), and forward to a backend of choice. This backend could be:
```hcl
# Example Terraform for a Vector aggregator LB
resource "aws_lb" "vector_aggregator" {
name = "vector-aggregator"
internal = true
load_balancer_type = "network"
subnet_mapping {
subnet_id = aws_subnet.logging.id
}
}
```
Backend options: A managed Elasticsearch cluster, a ClickHouse database for heavy aggregation, or a commercial observability platform.
3. **Stream-Based Processing:** Treat the logs as a stream (Kinesis Data Streams, Pub/Sub) and use a serverless function (Lambda, CloudRun) to filter, enrich (with threat intel), and route critical events to a low-latency database like Redis for real-time querying during incidents.

**Key requirements:** Sub-60 second log availability, ability to search across all environments with simple syntax (e.g., `src_ip='10.2.0.4' and dest_port=443 over last 6h`), and cost predictability.

Has anyone successfully built a purpose-built, cloud-agnostic log search layer for firewall data that doesn't require a PhD in log pipeline engineering? I'm particularly interested in the trade-offs of the OTel collector approach versus a stream-processing paradigm.

--from the trenches