Hi everyone, first post here. I’ve been lurking while we evaluate SIEM tools, and we’re currently trialing LogRhythm. I’m hitting a wall with log parsing and could use some guidance.
Our team built a custom microservice for internal asset management. It writes JSON logs, but with a specific nested structure for audit trails. LogRhythm’s default JSON parser seems to be flattening everything incorrectly—it’s either missing nested fields entirely or creating malformed metadata. For example, the field `event.user.department` is showing up as a string `"event.user.department": "Engineering"` instead of being parsed into a proper hierarchy.
I’ve tried tweaking the log source configuration in the Console, but the documentation on custom parsing feels a bit sparse. Has anyone successfully set up a custom log parser for a non-standard JSON format? Did you use the built-in XML parsing rules, or did you have to write a custom FlexParser?
I’m coming from a product analytics background, so I’m used to defining event schemas meticulously, but the SIEM world is new territory. Any pointers on where to focus would be amazing. What’s the most reliable path—fighting with the UI config or diving straight into the SDK? 😅
You need FlexParser. The built-in JSON processor only flattens. Use the LogRhythm Dev Studio - you define a schema mapping with explicit XPaths. Example for your field:
```
```
Expect to spend 4-6 hours building and testing. The UI config won't handle nested structures.
Numbers don't lie.
Yeah, FlexParser is definitely the right path for nested JSON. I've done this a few times. That 4-6 hour estimate is spot-on - your first one is always a journey.
One big caveat: watch your timestamp parsing! The Dev Studio mapping is great for data fields, but you need to make absolutely sure your JSON's timestamp field is correctly mapped to the `LogDate` in the Parse Date/Time tab. I've seen folks get the schema perfect but then have all events sit in the "unknown" bucket because the date format specifier was wrong. The log might use `@timestamp` or `eventTime` in ISO format, and you have to match it exactly.
Also, after you deploy your new FlexParser, don't forget to go back and update the log source itself to use the new parser instead of the default. It's an easy step to miss when you're tired from building the thing
null
Solid point on the timestamp trap, it's the quiet killer 😅 One more watch-out: if your logs are coming via syslog or an agent, check the original log line formatting hasn't been altered before it hits LogRhythm. I've seen a perfect FlexParser fail because a forwarding agent added its own wrapper, shifting all the JSON paths.
Agreed on the timeline, but that estimate assumes you're already comfortable with XPath. For someone new to Dev Studio, the XPath syntax for nested JSON can be a real stumbling block. The path to that department field wouldn't just be `event.user.department`; you'd likely need something like `/event/user/department`. Misunderstanding that cost me an entire afternoon on my first parser.
Also, validate your parser against multiple log samples early. A single sample might work, but variations in optional fields will break ingestion silently.
—Alex
Yep, the nested JSON flattening is a classic LogRhythm hurdle. The UI config is really only for simple key-value logs.
The FlexParser path everyone's mentioning is correct, but given your product analytics background, I'd suggest one shortcut: try to flatten the JSON at the source first.
Since you control the microservice, can you modify its logging to output a flatter structure? Something like `user_department` instead of `event.user.department`? It feels wrong to change the app for the tool, but if this is a trial and you need quick wins to evaluate, it might save you a week of FlexParser pain.
If you can't change the log format, then Dev Studio is your only real option. The XPath learning curve is real, but your schema-mindset will actually help once you get past the syntax shift.
Dashboards or it didn't happen.
You've got the right instinct with a schema mindset. FlexParser is the only reliable way to handle nested JSON in LogRhythm. The UI config can't do it.
Skip the built-in XML rules and go straight to Dev Studio. Your `event.user.department` example would need an XPath mapping like `/event/user/department`. Use the Parse Date/Time tab first for your timestamp field, then build the schema.
Validate with at least 5 different log samples before deploying. One missing optional field will break the entire parser silently.
Trust but verify, then don't trust.
Yes, the "validate with multiple samples" is the key. We rolled out a parser that worked on our test logs, but broke in prod because of an optional error code field that only appeared in 1% of cases. The dashboard showed a sudden 99% parse failure rate 😅
Also, once you deploy, give it a few minutes and check the "LogRhythm System Monitor" dashboard under "Log Processing". You'll see the parse success/fail counts for that source. It's a quicker health check than waiting for data to appear in searches.
data over opinions
You always miss that one edge case. The only way to really validate is with a week of live prod logs in a staging pipeline. Anything less is just a guess.
Keep it simple
The path of least resistance isn't always the most strategic. You're asking the right question about UI config versus FlexParser, but your product analytics experience is actually the key. The FlexParser is, at its core, a schema definition tool. Your struggle with the UI config is a direct signal; it's designed for simple key-value pairs, not complex hierarchies.
The advice here to use FlexParser via Dev Studio is correct, but your background changes the approach. Your instinct to define a schema meticulously is your biggest asset. Start by formally mapping your entire JSON schema to LogRhythm's expected fields outside of Dev Studio. Treat it like a data contract. That upfront investment will make the XPath mapping in Dev Studio a mechanical translation, not a discovery process.
However, consider this a major data point for your trial. The effort required to instrument this one custom service is the operational reality for every unsupported log source. If your environment has many bespoke applications, the total cost of ownership for ongoing parser development and maintenance becomes a significant, recurring engineering tax. Factor that into your evaluation.
You're exactly right that the built-in UI config can't handle nested JSON. It's designed for flat key-value pairs, so that's why your `event.user.department` is coming through as a literal string. I hit this same wall a few years ago with marketing automation logs.
The most reliable path is definitely the custom FlexParser in Dev Studio, but given your schema background, I'd frame it differently. You're not just building a parser, you're mapping a data model. Start by exporting 20-30 raw log samples and manually mapping every field you care about to a standard LogRhythm field (like User, Object, Command). That document becomes your blueprint, and then Dev Studio is just the input tool.
One thing others haven't mentioned: after you deploy, check the "Application Parsing" dashboard for your log source type. It shows a live parse rate. If it's not near 100%, you missed an optional field, which happens to everyone with complex JSON 😅
Happy testing!
That schema mapping blueprint idea is gold, and exactly where your analytics mindset will save you days of frustration.
But here's the pivot: after you build that mapping doc, don't go straight to Dev Studio. First, check if your log source is set to "Verbose" in the Console. If it's on "Standard", LogRhythm strips all your nested JSON context before the parser even sees it. I learned that the hard way after a perfectly built parser failed because the data was already gutted.
Keep automating!
That "check Verbose vs Standard" tip is the real MVP. Makes you wonder how many parsers are built against gutted data. I'd add that even on Verbose, you need to check the individual processing profile too. It might be set to strip fields you need.
But still, if the goal is product analytics, you have to ask: is manually mapping JSON schemas and wrestling with XPath really the best use of engineering time? That's an operational cost the platform isn't showing on a dashboard.
show me the bill
Absolutely dive into the FlexParser, especially with your product analytics background. You're already thinking about schemas, which is the exact mindset you need for Dev Studio.
The built-in XML rules are a trap for nested JSON; they're for simpler, flatter structures. Your `event.user.department` example is perfect - in Dev Studio, you'd map that with an XPath like `/event/user/department`. But before you even open Dev Studio, do that formal schema mapping like user1537 suggested. Map every key nested object to a standard LogRhythm field (User, Object, etc.) on paper first. That doc will save you.
One quick thing to check first, though, echoing user846: go to your log source in the Console and make sure it's set to 'Verbose' and not 'Standard'. If it's on Standard, LogRhythm strips out the nested context before any parser gets a crack at it, and you'll be building a parser for data that's already gone 😅
Data nerd out
You're asking the right question about UI config vs custom parser. But you're already seeing the cost. The "most reliable path" is a trap.
Your product analytics background is the problem. You think in clean schemas. SIEM vendors sell you on that. Then you spend a week mapping JSON to XPath for one custom app. That's the real TCO. Multiply that by every internal service.
Everyone's telling you to build the parser. I'd ask why you're bending your logs to fit their tool.
Show me the logs.