Your 5-sample validation rule is the bare minimum. I've seen parsers pass 20 test logs and still fail silently in production because they hit a log line with a null value in a nested field that wasn't in any sample. The parser doesn't just skip that field, it discards the entire event.
You can mitigate it by setting every XPath mapping to optional in Dev Studio, but that's a manual checkbox for each field and it's easy to miss one. Even then, a null in a critical field like timestamp will kill the parse. The silent failure is the worst part; you only find out when your dashboards go empty.
Automate everything. Twice.
Spot on about the silent failure. That's the real danger.
You can have "optional" set on every field and still lose events. It's not just nulls. A missing closing brace, a single quote where it expects a double quote, any malformed JSON that's still technically delivered by the syslog daemon will just vanish.
The TCO isn't just the build time, it's the ongoing maintenance and hunting for missing data.
Exporting samples is a good start, but that "live parse rate" metric you mentioned is a lagging indicator. By the time you see it drop, you've already lost data. You need a proactive validation step before deployment.
Instead of just mapping fields, build a small Python script that uses LogRhythm's API to test parse your 20-30 samples against your parser *before* you push it to Dev Studio. Use their test endpoint to see the raw parsed output. You'll catch those optional field misses immediately, not hours later. It adds maybe 30 minutes to your process and saves you from the silent failure hell everyone's describing.
Speed up your build
Welcome to the community, and great question. You've already put your finger on the core issue - the built-in UI config and XML rules are really meant for flat structures, so they'll butcher nested JSON.
Given your product analytics mindset, I'd say dive straight into the FlexParser. That schema-mapping step everyone's mentioned will feel very natural to you, and it's the only way to properly define your hierarchy. But before you even open Dev Studio, go double-check your log source is set to 'Verbose' in the Console. If it's not, the parser gets stripped data and you'll be building a solution for a problem that doesn't exist anymore.
The silent failure on nulls or malformed JSON is real, though. Once you build that mapping document, consider using the API test endpoint to validate your parser against a wide sample set, including edge cases, before you deploy. It catches things the UI won't.
Keep it constructive.