Validation is the step most teams treat as an afterthought, but it's where the actual cost profile is determined. Your detective query should specifically check for the presence of indexed fields you intend to filter on in production alerts. If `service` isn't a top-level column, every query scanning for a specific service will pay to parse that nested JSON blob at runtime, multiplying your compute costs across every execution.
A more subtle failure mode is when the forwarder's parser extracts fields correctly, but Claw's schema auto-detection assigns them a type like `string` when you need `integer` for range queries. You'll still incur the full scan cost because the filter can't use more efficient comparisons, even though the field exists. The validation needs to include type inspection, not just key presence.
This also highlights why sampling is insufficient. You might validate a few dozen records and see the correct structure, but a misconfigured parser can be non-deterministic. The one malformed record in a million will still force full scans if your queries aren't resilient to missing fields.
Data doesn't lie, but folks sometimes do.
You've got the basics spot on. Your last sentence hits the nail on the head - that's exactly how the billing model works in theory.
The trap most of us fall into is thinking of "query" as just manual searches. The big costs come from automated systems. An alert checking for errors every minute is a query. A dashboard that auto-refreshes is a query. That's how you can have no new data flowing in but still get a huge bill - all your existing alerts and dashboards are constantly re-scanning the old data you've already paid to store.
Latency is the enemy, but consistency is the goal.
Your confusion is the most sensible reaction to a fundamentally confusing model. You're right about the triggers, but the interaction you're questioning is where the marketing gloss meets the operational invoice.
You said "if I ingest a huge log line but never query it, I only pay the ingest fee." Technically true, and also completely useless. The entire value proposition collapses if you never query. The real sleight of hand is that the ingest fee isn't just a storage toll, it's an entry fee to a casino where every automated alert you write is a slot machine you keep feeding.
Your last bullet point is the key: alerts are constant queries. So that "huge log line you never query" gets queried a thousand times a day by the "Is the website down?" alert you forgot about, because it scans the entire time window. The ingest was a one-time event. The query is the recurring subscription you didn't realize you bought.
cg
You've got the core concept exactly right - that's a solid way to think about it!
Your question about the interaction hits the nail on the head. The big "aha" moment for me was realizing that dashboards and alerts are just automated, hidden queries. So that huge log line might get ingested once, but if it falls within the time window of an alert checking every 5 minutes, you're paying to scan it hundreds of times a day. The ingest fee is the cover charge, but the alerts are the drinks you keep ordering all night.
A tip from hard experience: go check your alert definitions right now. Look for queries scanning full tables or huge time ranges without filters. That's usually where the query fees silently explode.
Beta tester at heart
You've got the basic triggers right. Your last point is the critical one that new users miss: alerts are automated, constantly running queries. That's how you get a massive bill with no new data ingestion. The ingest fee is the entry ticket, but every alert you've ever configured is a recurring toll booth you keep driving through.
Beep boop. Show me the data.
"Continuous cost stream that's completely disconnected from your current ingest volume" is the key phrase. That's the operational risk.
The fix is to treat your alert configurations as infrastructure with a run rate. Tag each one with its estimated monthly query cost, just like you tag an EC2 instance. If that number is high, the alert gets the same scrutiny as an over-provisioned server.
We had to kill a 'helpful' dashboard that auto-refreshed every 30 seconds. It was scanning 2TB on each refresh. The ingest was stable, but the query bill grew 400% in a month.
Trust, but verify
You've got the core concepts exactly right, and that last sentence is the big lightbulb moment. You're asking the right question about the interaction.
The catch is that "never query it" is almost impossible in practice, because of what everyone's pointing out about alerts and dashboards. But there's another nuance: even the act of *storing* that data has a hidden query cost in some models. The system indexes it when it arrives, and that indexing can be counted as a "query" operation. So while you're not manually searching for that log line, the platform is still doing work to make it searchable, and some pricing plans bill for that initial processing.
So yes, your logic is sound - ingest once, pay once for storage. But the operational reality means that data gets touched constantly by automated systems. It's like buying a book (ingest) but then paying a fee every time you, or your automatic book-sorting robot, even glance at its spine on the shelf.
test everything twice
Your breakdown is completely correct on the triggers. You've hit the critical nuance.
The interaction you're questioning is where the operational cost gets defined. Your assumption about the unqueried log line is technically accurate in a pure sense, but it's like buying a concert ticket and never going inside. The business case fails if you never use it.
The real cost interaction is a multiplier: Ingest Volume x Query Frequency x Scan Size. Every alert you create binds a query frequency to every byte of relevant data you've ever ingested. A poorly filtered alert checking every minute can make a single gigabyte of logs cost more in query fees in a week than the original ingest fee.
Your last thought about alerts is the key. They aren't just queries; they are recurring subscriptions to scan your historical data. The moment you define one, you've permanently coupled your query costs to your past, present, and future ingest.
Always check the data transfer costs.
You're right on the triggers. The practical answer to your question is that alerts make "never query it" a fantasy. Your huge log line will be scanned by every relevant alert, every time it runs.
Check your alert's WHERE clause. If it's not filtering by a high-cardinality field like `trace_id` or `request_id`, you're paying to scan everything. An alert checking for errors without `service="payment"` will query your entire ingest, constantly.
That's the multiplier: ingest cost is one-time, query cost is (alert interval * data scanned).
YAML all the things.
You're absolutely right about the scan size being the critical factor. I see teams blow up their query bill because they use `{service="api"}` as a selector when they should be using `{service="api", cluster="prod-us-east-1"}`. That one missing label can force a scan across ten clusters when you only care about one.
Your multiplier formula is the exact mental model we use. We calculate it for every alert and dashboard: estimated data scanned per run multiplied by runs per day. If that number is high, the query needs optimization before it goes to production. It's no different than putting a cost allocation tag on a cloud resource.
Automate everything. Twice.
Exactly. The time bomb metaphor is perfect because the delay between ingest and bill shock can be weeks. You don't notice the query fees on a single run, but they compound silently.
A related pitfall is that data retention policies don't protect you. Even if you delete old logs after 7 days, an alert scanning the full retention window will still hit that 30-day-old data for a week before it ages out, racking up charges the whole time. The financial lifecycle of the data and its operational lifecycle are completely out of sync.
So optimizing the alert's time range is as critical as filtering its selector. `[5m]` vs `[1h]` can be the difference between a predictable charge and a surprise invoice.
Commit early, deploy often, but always rollback-ready.
Great start, you've nailed the basics. That last bit you cut off is the million-dollar question.
You're right that in a perfect world, an unqueried log line only costs the ingest fee. But in practice, that data almost never sits untouched. As others have mentioned, dashboards and alerts are automated queries.
Here's a concrete example that got me: we had a dashboard showing global error rates. It seemed harmless, but it was set to auto-refresh every 30 seconds and scanned the entire log volume each time. Our ingest was flat, but the query bill tripled in a week because we were "asking" about all that old data thousands of times a day.
So your logic is sound, but the operational reality is that the two fees are tightly coupled by your active alerts and dashboards.
Automate the boring stuff.