You've already identified the core difficulty, which is mapping formats. The initial advice here to validate the raw output is correct, but I can add a methodological point.
Instead of aiming for a single props.conf, start by classifying the event types you actually care about monitoring. Run your netcat capture and isolate, for example, the payload for a "session started" event and a "privilege elevation failed" event. You'll often find they have different structures, which explains the format mismatch frustrations others mentioned. Build one extraction stanza per critical event type, not per source. This approach saves you from creating a monolithic, brittle parsing rule that fails on new event subtypes.
Also, check the transport mechanism. Are you using syslog TCP? The appliance sometimes batches events, and Splunk's default line breaking can mangle the JSON. You might need to configure a `TRANSFORMS` rule in props.conf to handle multiline events before any field extraction occurs.
Yeah, those format mapping issues are the worst. The docs make it seem like a clean 1:1 mapping, but real session data is never that neat.
I'd echo the advice to start by seeing what's actually on the wire with netcat, but add one thing: capture a few *different* event types. A normal login, a privilege failure, and a session start often have slightly different structures even within the same chosen format. That's likely the root of your mapping headache.
Build your props.conf around those captured samples, one stanza per event type you care about, not one giant rule for the whole source. Start with the 2-3 fields you need for your first critical alert and expand from there. Trying to map the entire spec at once is a trap.
Totally agree on the incremental approach. We did the same thing, starting with a simple "high risk session started" alert. That quick win got the team on board before we even thought about the messy data.
One caveat we ran into: even with a minimal props.conf, you've gotta watch out for future app updates. They can change a field name slightly, and your one critical alert can silently break. We added a simple dashboard that just charts the count of those parsed events daily - it's an easy sanity check that the pipeline is still alive.
Did you set up any similar monitoring for your parsing logic, or just rely on alert triggers?
Ship fast. Learn faster.