Tutorial: Creating a custom log parser for our in-house app format.

Exactly. Mapping the fields is the critical step they don't emphasize enough. Your `lvl` needs to map to `p_log_level` for proper classification, and `ctx` to something like `p_any_context`. Also, don't forget to flatten that `extra` object or its fields will be ignored in queries.

You'll want to add something like this after your parse:
event['p_log_level'] = parsed.get('lvl')
event['p_any_context'] = parsed.get('ctx')
# For the nested object
if 'extra' in parsed:
event['retry_count'] = parsed['extra'].get('retry_count')

Otherwise, your detection might fire but the data won't be usable in the data lake.

While that json_parse approach is the standard path, I've found you can sometimes get away without a custom rule by modifying the schema itself in Panther's UI. You can define a custom schema that accepts `lvl` and `ctx` directly, then use field mappings within the schema configuration to point `lvl` to the standard `p_log_level` internally.

But your point about flattening the nested object is critical, and that's where the schema method falls short. It often can't handle dynamic nesting gracefully. For that, your programmatic rule is definitely the more robust solution. The overhead of maintaining a custom Python rule versus a schema definition is a real trade-off.

You'll also need to handle the type conversion for `p_log_level`. Panther's classification expects specific strings like `INFO`,`ERROR`. If your source uses `ERR`, you need to standardize it.

event['p_log_level'] = parsed.get('lvl', '').upper().replace('ERR', 'ERROR')

Otherwise your dashboard filters won't group correctly.

You're right that the docs don't prepare you for the first real encounter with an internal log format. The initial excitement of "it's just JSON" quickly fades when you realize how much implicit structure Panther expects.

One thing I'd add to your starting point is to validate the parse immediately. That `json_parse` call on `raw_log` will fail silently if the field is missing or empty, returning an empty dict. I usually wrap it in a try/except and return False early, otherwise you'll spend hours debugging why a rule isn't firing when it's actually a parsing problem.

Also, consider the `ts` field. If it's not in a Panther-recognized format, you'll need to map it to `p_event_time` manually, or your timelines will be off.

Your example's too clean. Real internal logs are broken. Missing fields, malformed JSON, mixed string/integer types in the same key.

json_parse will fail on those. And you haven't said how you'd test this rule against actual log volume. Are you just hoping it works?

The bigger fight isn't mapping fields. It's dealing with the garbage data that will inevitably flow through.

You're right that the docs don't prep you for the real fight - they assume clean JSON. The initial example is just the starting line. The real marathon begins when you deploy that parser and get flooded with ten variations of that same log from different service versions.

A tip I haven't seen mentioned yet: before you even start writing field mappings, run a sample of your raw logs through a simple rule that just does `json_parse` and returns the parsed structure as a string field. Let it run for a day. You'll discover three more nested fields, four different timestamp formats, and that `cust_id` is sometimes a string, sometimes a number, sometimes missing entirely. That's your real schema.

The mapping part is easy once you know what you're actually dealing with. The discovery phase is what they skip in the tutorial.

Oh, the bliss of that first "just use json_parse" step before the abyss opens up. You're absolutely right that the docs make this part sound like a one-liner, but they completely skip the crucial next sentence: "and now spend the next two weeks becoming an archaeologist for your own application's data stratum."

Your example is the museum-ready artifact. The logs you'll actually get will look like it was dug up, chewed on, and dropped by a badger. I'd add one immediate caveat to your stripped-down start: never trust `event.get('raw_log', '{}')`. What if the field is there but it's `None`? Or an empty string? That default empty dict will parse just fine and your rule will happily process nothing. I've seen it happen.

So yeah, define the detection. But maybe define a beer budget for the debugging session, too.

Exactly. The empty string case you mentioned is a real parser killer. If raw_log is an empty string, json_parse returns an empty dict, and your rule runs on nothing, creating a false sense of security.

I'd push that validation a step further. Don't just check for None or empty string. Check the actual result of the parse.

parsed = json_parse(event.get('raw_log', ''))
if not isinstance(parsed, dict) or not parsed:
return False

Because if your raw_log contains the literal string 'null', you'll get a None object back from json_parse, and trying to call .get on that blows up the rule entirely. Found that one in production after a "successful" test phase.

That's a really good catch about the string 'null'. It's easy to miss in testing because you're feeding it valid JSON, but in real traffic you'll get every edge case. Your validation check is solid.

I'd also suggest logging the actual raw_log string when that validation fails, maybe with a debug flag. Otherwise, when the rule stops firing, you're left guessing whether it's because there's no data or because the data is broken.

Good point on the standardization, but that `.replace('ERR', 'ERROR')` is too simplistic. It will also incorrectly convert strings like `TERRITORY`. You need a more precise mapping, ideally with a dictionary.

```python
level_map = {'ERR': 'ERROR', 'WARN': 'WARNING', 'DBG': 'DEBUG'}
standard_level = level_map.get(parsed.get('lvl', '').upper(), parsed.get('lvl', '').upper())
if standard_level in ['INFO', 'ERROR', 'WARNING', 'DEBUG']:
event['p_log_level'] = standard_level
```

Also, remember to handle the case where the mapped value still isn't a valid Panther level; you might want to default to `INFO` or drop the field to avoid classification errors.

Your mapping is better but still incomplete. Your `if` check at the end is key, but you're silently discarding invalid mapped levels. That's dangerous for monitoring.

If `standard_level` isn't in your valid list, you should explicitly set a default or, better, tag it as `UNKNOWN`. Silently skipping means you lose visibility into bad data, and your log-level dashboards will be wrong.

Also, that `level_map` will need constant updates. You'll find `CRIT`, `FATAL`, `SEVERE` all in the wild. Put it in a shared location.

Absolutely! That starting example is the perfect illustration of the gap between theory and reality. You've got the right instinct to skip the built-in schemas and go straight for a custom parser.

But I'd add one thing before even writing that first `json_parse`: talk to the dev team. That `extra` object is a trap. It's a dumping ground, and you need to know what *could* be in there. Ask them for the three most recent "we added something to the logs" commits. You'll often find the real schema hiding in git history, not in any documentation.

And once you start mapping, flatten that `extra` field right away. Trying to handle nested objects later in your rules is a headache.

Sure, the official docs make it sound like `json_parse` is the whole solution. But that's just the hook to get you started. The real work begins when you realize you have to map `lvl` to `level`. Now you're writing a translation layer for your own team's bad naming decisions. And once you start mapping one field, you'll end up writing a custom schema for the entire object, which defeats the whole point of using a tool that supposedly handles this automatically.

You also assume the `extra` field is a neat nested object. In my experience, it's often a string that someone dumped `json.dumps` on, or a dict that occasionally contains another dict under a key like `metadata`. Good luck flattening that reliably.

You're hitting the real pain point. Once you start mapping `lvl` to `level`, you're signing up to maintain that translation layer forever. Next week they'll add `sev` or `priority_code`.

And you're dead on about the `extra` field being a string. I've had to write pre-parsers that do `if isinstance(extra, str): extra = json.loads(extra)`, only to find the string is sometimes invalid JSON with a trailing comma. At that point, you're not writing a parser, you're writing a full-time data janitor script.