Persistent error: 'Invalid schema' when onboarding CloudWatch logs.

Ah, the dreaded 'Invalid schema' error - that one can be a bit of a head-scratcher. Welcome to the club.

You're right to double-check the role permissions, that's usually step one. But since that's in order, let's look at the log format itself. Panther is very particular about the JSON structure it expects from CloudWatch. The most common trip-up I've seen is when the raw log event contains a JSON string within a JSON object, which creates a nested mess for the schema detector.

Could you share a redacted example of a log event from the group/stream you're trying to onboard? Often, it's about the `message` field being a string instead of a parsed JSON object. If your application logs are JSON strings, you might need to enable the 'Parse JSON' option on the source, or adjust your log producers to output plain JSON objects.

The connection test pulls a sample to infer the schema, so if that sample is malformed or inconsistent, the inference fails.

You've pinpointed the most likely culprit with the nested JSON string. I'd add that the schema inference can also fail if your sampled events have inconsistent fields, which is common during application startup or error conditions. A log stream mixing structured JSON objects with plain-text error messages will confuse the detector.

If enabling 'Parse JSON' doesn't resolve it, try creating a test source using a log group with only a few, consistent, well-formed events to verify the pipeline works. Once confirmed, you can tackle the inconsistency in your main log group.

Ah, the 'inconsistent fields' problem - that's a classic. The schema detector takes a small sample, and if you're unlucky, it hits a startup banner or a stack trace instead of your normal log shape. This can fail open or, more frustratingly, fail closed with a cryptic error.

I'd suggest a more aggressive sampling strategy. If possible, pull a few hundred events from the target log group via the CLI first. Manually inspect them for the actual variance. You often find it's not just startup logs, but health checks, load balancer pings, or cron jobs polluting the stream. Sometimes the fix is adding a filter pattern at the source, not just hoping for consistent sampling.

The filter pattern idea is interesting, but I'm a little worried about accidentally filtering out something important. If I set a pattern to exclude health check pings, could that also mask a real issue that looks like a health check? Or does Panther let you test the filter without committing to it?

Also, when you pull those few hundred events via CLI - what's your go-to command for that? I'd like to take a look before I mess with the source config.

Yeah, it's usually the JSON structure. The docs for the CloudWatch setup gloss over how rigid the schema detection is. Did you enable the 'Parse JSON' option on the source itself? If your log message field is a stringified JSON object and that's off, it'll choke every time.

Permissions are the easy part. The messy part is what your app is actually writing.

The initial connection test failing indicates the schema is being inferred before data even starts flowing, which narrows this down. The most common trigger for that specific test failure is that the IAM role, while having the necessary `logs:DescribeLogGroups` and `logs:FilterLogEvents` permissions, is being denied on a `logs:GetQueryResults` or `logs:StartQuery` action for the CloudWatch Logs Insights API.

Check if your role's policy includes the `logs:StartQuery` and `logs:GetQueryResults` permissions. Panther's initial test often uses a Logs Insights query to sample events for schema detection, and missing those permissions results in a generic 'Invalid schema' error because it can't retrieve a sample to analyze. It's a misleading error message that frequently sends people down the log format rabbit hole first.