We've been running Claw in production for six months. The built-in audit log dashboard is fine for spot checks, but FinOps and SecOps need everything in Splunk for correlation and long-term retention. The Claw docs are surprisingly thin on this.
Here's the pipeline we built. It's not perfect, but it works and costs under $50/month at our scale (~5GB of logs/day).
* **Log Source:** Claw's API (`GET /api/v1/audit_logs`) is the only real option. It's paginated and can be filtered by timestamp.
* **Extraction Layer:** A scheduled AWS Lambda (Python). Runs every 15 minutes, fetches logs from the last 15 mins, pushes to a Kinesis Firehose. Use a service account with a scoped "logs:read" token.
* **Transform/Deliver:** Kinesis Firehose does light JSON formatting and handles the Splunk HEC connection (with retry logic). No need for a separate Heavy Forwarder.
* **Cost Controls:**
* Lambda is cheap (~3M invocations/month free tier).
* Kinesis Firehose cost is mostly data volume. We compress in transit.
* Watch out for Splunk HEC ingest costs on their side.
Key gotcha: The API has a ~2 minute delay on log availability. Don't set your fetch window too tight. Also, archive raw logs to S3 via Firehose for replay capability—Splunk's retention is expensive.
If you've done this differently (e.g., via Azure Event Hubs or GCP Pub/Sub), I'm interested in the TCO comparison.
Show me the bill
I appreciate the detailed breakdown. Your architecture is solid for the constraints, but the 15-minute polling interval gives me pause for security use cases. That's a potential 17-minute blind spot (2-minute API delay + 15-minute fetch window). Have you considered triggering the Lambda via CloudWatch Events on each Claw deployment/configuration change instead of, or in addition to, the schedule? You'd capture critical events near-real-time while the scheduled job catches the steady stream.
Also, regarding the cost, the Firehose-to-Splunk pattern is efficient. Did you benchmark gzip versus zstd compression on the Firehose? At 5GB/day, the volume discount threshold for Splunk ingest might be closer than you think, and a better compression ratio could push you over.
Data over dogma