ELI5: How does Claw's 'data processing' cost actually work?

Hey folks, I've been neck-deep in our observability bills lately, and I keep circling back to the same question that tripped me up initially: **Claw's "Data Processing" line item.**

We all know the basics: ingest, query, retention. But "Data Processing" feels like the mysterious middle child. I finally spent some time with their docs and our own Terraform configs to piece it together. Here's my attempt to break it down like I would for a junior engineer on my team.

Think of it as the "assembly line" cost *after* your data is accepted (ingest) but *before* it's stored (retention). It's the work Claw does to make your data usable. The main operations are:

* **Parsing & Structuring:** Taking your raw log line or span and extracting fields (like turning `status=404` into a field `http.status_code: 404`).
* **Enrichment:** Adding context, like looking up an IP address to add a `country` field.
* **Indexing:** Building the data structures that allow fast querying. This is a big one – high cardinality (lots of unique values) means more indexing work.

So, if you have a log line with 10 parsed fields, that's less processing than one with 50. If you add a custom unique request ID to every span, you're increasing cardinality and, therefore, processing effort.

Here's a real-world Terraform snippet from our config that directly impacts this cost. See the `processing_services` block? This is where you can control it.

```hcl
resource "claw_pipeline" "app_logs" {
name = "my-app-logs"
filter = "service:my-app"

parsing {
type = "json"
}

processing_services {
# Enrichment adds processing cost
geoip {
field = "client_ip"
}
# Each lookup adds a bit more
user_agent {
field = "user_agent"
}
# This is a custom processing step - it runs your own code on the data.
custom_processor {
name = "add_important_field"
source_code = file("${path.module}/processors/enrich.js")
}
}
}
```

The key takeaway? **Processing cost is driven by the *complexity* of your data pipeline, not just volume.** Turning off unnecessary enrichment, simplifying overly verbose logs, and being mindful of high-cardinality fields you parse can directly reduce this part of the bill.

Has anyone else done a deep dive here? I'm curious about the cost trade-off between, say, doing enrichment in Claw versus in our own application code before sending the data.

-- Amy