Okay, so I'm trying to wrap my head around something that feels like it should be simple. I've been using Sumo Logic for a few months now to monitor some API gateway logs and ETL job statuses. Mostly loving it! But I keep hitting this wall where what I think is a super basic, filtered search just... hangs. And I don't mean on terabytes of data—I'm talking about a query scoped to maybe an hour in a single environment.
For example, I was just trying to find all the `500` errors from a specific microservice for the last 60 minutes. My query looked like this:
```sql
_sourceCategory=myapp/prod/api-gateway
| where status_code = "500"
| timeslice 1m
| count by _timeslice
```
Nothing crazy, right? But the progress bar just crawled, and it took over a minute to return. Meanwhile, my colleague ran a similarly "simple" search over a broader time range and got results in seconds. It's got me scratching my head.
I've been digging into the docs and community posts, and I'm starting to realize that "simple" in my head might not align with Sumo Logic's architecture. Here's what I'm curious about—can you all help me connect the dots?
* **Index vs. Live Tail vs. Scheduled Search:** Am I maybe running this as a live search when it would be better as a scheduled one? Or vice versa? The performance characteristics seem wildly different.
* **The "Scatter-Gather" Overhead:** My understanding is that even for a narrow query, Sumo might still be scanning all the partitions/indexes across its backend to satisfy the `_sourceCategory` and time range. Is that the main culprit? Does adding a more specific `_sourceHost` or a tighter wildcard in `_sourceCategory` actually reduce the "scatter" phase significantly?
* **Parsing at Query Time vs. Ingest-Time Parsing:** My query uses `| where`. If I had set up a parse rule at ingest to extract `status_code` into a structured field, would that make this kind of lookup dramatically faster? I'm trying to gauge the performance trade-off between ingest processing and query flexibility.
* **Data "Shape":** Could it be that my logs are just super verbose (like full request/response JSON blobs), and even though I'm filtering for one field, the system has to wade through all that text first? Would using a more aggressive parsing filter early in the query help, like `| parse "status_code": *," as status` before the `where`?
I'm coming from a background where you'd throw an index on a column and be done, so this distributed, schema-on-read, log-based world has a different set of rules. I'd love to hear any war stories or "aha!" moments you've had about optimizing for query speed in Sumo.
What are the top two or three levers you pull first when a straightforward search feels slower than it should?
Data nerd out
You're hitting the classic "declarative query vs. actual execution path" misunderstanding. Your query isn't simple to Sumo; it's a full scan.
`_sourceCategory=myapp/prod/api-gateway` pulls in every raw log line from that source for the hour. Then `| where status_code = "500"` filters. That's a table scan on potentially millions of rows *before* aggregation.
Your colleague with the faster, "broader" search probably used a keyword search upfront, like `"500" _sourceCategory=...`. That uses the index for the term `"500"` and reduces the raw data pulled into the query engine by orders of magnitude. The pipeline matters: filtering at the *beginning* with indexed fields (like `_sourceCategory`) or keywords is fast. Filtering in the middle with a `where` is slow.
For your specific case, try:
`_sourceCategory=myapp/prod/api-gateway "status_code=500"`
That might use the index for the literal string. Even better, if `status_code` is a parsed field, structure it as:
`_sourceCategory=myapp/prod/api-gateway status_code=500`
No quotes, no `where`. Let the collector's parsing and indexing work for you.
Benchmarks or bust
Good point about indexed fields, but that's only half the battle. The real killer is that `where` clause forcing a full parse of every JSON log line before it can filter. If `status_code` isn't a metadata field Sumo automatically parses and indexes, you're toast.
Try this first: check your source's field extraction rules. If `status_code` is auto-parsed, your colleague's version works. If not, you're stuck scanning raw logs. The fix is often upstream, making sure your log format lets the collector extract key fields.
Trust but verify – and audit
The replies about parsing are correct, but you're missing the biggest factor: data volume.
`_sourceCategory=myapp/prod/api-gateway` for an hour could easily be 10+ GB of raw JSON. Your query engine has to ingest all of it. A minute isn't a crawl, it's expected.
Your colleague's faster search probably had a keyword filter that drastically reduced the initial result set before any parsing. The progress bar shows data scanned, not query complexity.
Check your search's "Scanned Data" stat in the job details. That's your real bottleneck.
The docs won't tell you this, but "simple" means "uses the index."
Your query is a full table scan. The where clause runs *after* pulling all the data.
Your colleague's faster search almost certainly used a keyword like `status_code=500` *before* the pipe. That uses the index.
Try this first:
* Check if `status_code` is an automatically extracted field.
* If it is, rewrite as `_sourceCategory=... status_code=500 | timeslice...`
* If it's not, you're parsing raw JSON for every line. That's why it's slow.
Optimize or die.
Exactly right. The "where" clause is the hidden killer here because it forces a full parse of the data stream *after* it's already been collected by the search head. It's like downloading the entire movie just to watch one scene.
One extra nuance: even if `status_code` *is* an automatically parsed field, putting `status_code=500` in the initial search still might not use the index if the field wasn't configured as a **key-value pair at ingestion**. Some log shippers just tag it as a field without building a lookup index. You can check by seeing if `status_code` shows up in the "Fields" sidebar with a magnifying glass icon next to it.
In that case, your colleague's speed might have come from the raw string `"500"` being indexed as a keyword across the whole log message, which is still way faster than a post-fetch filter.
Dashboards or it didn't happen.