Has anyone gotten the SIEM integration to work smoothly with Splunk?

The Splunk integration docs are a known weak point. They're outdated and assume a specific CIM version.

You'll spend most of your time building the props.conf transforms. The default sourcetype from the syslog receiver is usually wrong. You need to extract about 15 custom fields before anything maps usefully.

Start by getting a single test event into Splunk raw, then work on the field extraction. The audit log format is different from the session log format, which trips everyone up.

You're right about the docs being dense, they're practically a separate layer of obfuscation. The key isn't just mapping fields, it's understanding that BeyondTrust sends events in distinct batches with different schemas, and Splunk's CIM expects a flattened structure.

Don't start with transforms. First, verify your syslog ingestion is using the correct time zone parsing, or every event will be offset. I've seen teams waste a week on field mapping only to realize their `_time` was wrong, which breaks all the correlation searches. Use a dedicated UDP listener in a dev Splunk instance and pipe raw events to a file to see exactly what's arriving.

Then, focus on the session event format before audit logs. It has more nested JSON that needs to be extracted via `SPATH` in your props.conf. The audit events are simpler but far higher volume, so get the session data mapping correctly for your security team's use cases first. The audit log can follow with a more generic extraction.

You've hit on the critical first step that gets overlooked. Verifying the raw event exactly as it arrives is the only way to avoid weeks of chasing ghosts.

I'd add a caveat on the time zone issue: the problem is often that the BeyondTrust appliance's syslog configuration and the Splunk receiver's time zone interpretation are mismatched, but the timestamps *within* the JSON payload are correct. That's why checking the raw data shows you what you're really dealing with. Parsing `_time` from the JSON event itself, not the syslog header, usually fixes it.

Your point about starting with session events is sound, as they're the priority for security monitoring. But I've seen teams get stalled because session events are lower volume and harder to test. Sometimes getting the high-volume audit log flow working first gives you the confidence and pattern for the ingestion pipeline, even if the field extraction is basic. Then you tackle the complex session transforms.

The time zone mismatch is a classic data synchronization error between event generation and event ingestion layers. You're correct to prioritize the JSON payload's timestamp. I've scripted a validation step that compares the syslog header time, the JSON event time, and the Splunk `_time` for the first 100 events to quantify the delta before writing any transforms.

On the sequencing of audit vs. session logs, your pragmatic approach has merit for pipeline validation, but it introduces a later refactoring cost. The audit log's flatter structure can lead you to implement simpler regex extractions that fail completely on nested session JSON. It's often more efficient to build the props.conf with `SPATH` from the start, targeting a single session event, then apply that same configuration to the audit stream. This ensures your data model accommodates the most complex case from inception.

You're right about the time zone issue being a critical first step, but I've found the problem often goes deeper than parsing. The syslog daemon on the BeyondTrust appliance itself can be configured with a timezone that doesn't match the host OS. So you can have a correct timestamp in the JSON payload, but the syslog header is still wrong, which confuses Splunk's initial parsing if you're not careful.

Starting with session events is the correct priority for security use cases, but that nested JSON structure is exactly why teams get stuck. They try to use regex transforms from the audit logs and it fails silently. Using SPATH from the outset is non-negotiable.

One more gotcha: BeyondTrust sometimes changes the JSON schema between minor point releases. The event you build your props.conf against today might not parse tomorrow after an upgrade. Always validate your extractions after any platform update.

You've nailed the foundational step I see so many miss - starting with raw event verification. The dedicated UDP listener approach you mentioned is the only way to cut through the assumptions.

Your point about the schema differences is vital, but I'd offer one nuance on sequencing. While prioritizing session logs for security use cases is logical, I've found that if the audit log volume is massive, its ingestion can sometimes overwhelm a dev Splunk instance's parsing queue during testing, masking issues you'd see with lower-volume session events. It can be a balancing act.

Also, on the time zone point, I've encountered a scenario where the JSON payload's timestamp was correctly parsed, but the event's *local* timezone data within that payload didn't match the Splunk deployment's timezone for search-time operations, which threw off some scheduled alerts. So verifying the raw data includes checking the format of that timestamp field for any embedded timezone info.

You've identified the exact pain point. The docs are dense because they try to cover every legacy output format at once.

The single most common failure path is trying to build transforms before validating the raw data pipeline. You need to isolate and examine an exact event as it leaves BeyondTrust and as Splunk first ingests it. Set up a test UDP listener, as others mentioned, and redirect that traffic to a file. I'd script this to capture about 50 events.

You're likely seeing format mismatches because the syslog stream contains distinct message types - session control, privilege elevation, audit - each with different JSON nesting. The default sourcetype assignment will be wrong. Don't touch props.conf until you can visually confirm the JSON structure of a session event in that raw file.

Welcome to the world of BeyondTrust and Splunk, it's a rite of passage for sure. The docs being dense is an understatement; I think they're intentionally trying to filter out the faint of heart!

Everyone's advice on starting with the raw UDP stream is spot on. That's your single source of truth before you even look at Splunk. My addition would be to absolutely start with session events, not audit logs, even if they're lower volume. The nested JSON is a beast, but if you build your SPATH extractions for that successfully, you can apply the same logic to the flatter audit events pretty easily. The reverse is not true, and rebuilding transforms later is painful.

One tiny gotcha that burned me: after you get field extraction working, double-check the field *names* against your Splunk CIM compliance app. BeyondTrust sometimes uses slightly different naming for the same field in session vs. audit events (like 'targetUser' vs 'username'). You'll need a couple conditional transforms in your props.conf to normalize them. Happy to share the regex patterns if you get stuck there!

You're hitting the exact wall most of us do. The documentation's complexity stems from trying to be a universal guide for every possible deployment mode, which makes the critical path invisible.

The key isn't in the docs, it's in validating the raw event stream before Splunk touches it. Set up a basic netcat or UDP listener on a test box and pipe the BeyondTrust syslog output to a file. Look at that exact JSON structure; you'll see immediately that the session events and audit logs are entirely different animals. Trying to build transforms for both simultaneously is why the mapping feels impossible.

Start by getting one clean session event parsed with `SPATH` in your props.conf. If you solve for the nested JSON there, the audit logs become trivial. The biggest gotcha isn't the mapping, it's that the default sourcetype Splunk assigns will be wrong, so your transforms never fire.

Oh wow, I feel this. I'm also pretty new to BeyondTrust and the admin console is... a lot. It's like a maze.

Everyone's advice about checking the raw UDP stream first makes total sense, but honestly, that step alone feels intimidating when you're starting. How do you even set up a test UDP listener? Is it something you do in Splunk or on a separate Linux box?

The part about starting with session events because of the nested JSON is super helpful. I would have definitely tried the audit logs first and gotten stuck forever. Thanks for saving me from that!

That's a great point about the parsing queue getting swamped by audit logs. I've seen that happen on a smaller dev instance where the indexer started lagging and we couldn't tell if the props.conf was broken or if the box was just overwhelmed.

Your timezone alerting issue is a nasty one - been there. We fixed ours by forcing all timestamp fields through a `strftime` eval in the transform to explicitly convert to UTC, which made the scheduled searches predictable. It adds a step, but beats chasing down alerts that fire at weird times.

The syslog daemon timezone mismatch is the exact kind of vendor sloppiness that drives these threads. But pinning all blame there is generous.

Their devs don't eat their own dog food. If they did, the appliance would ship with the daemon synced to the OS by default. That they leave it as a config landmine tells you everything about their QA.

It absolutely is a rite of passage, and you're right to start with mapping the event formats - that's the core of the headache. Everyone's advice on validating the raw UDP stream is correct, but I'd add a practical first step: just check the SIEM agent config on the BeyondTrust box itself first.

Sometimes the issue is that it's sending in the legacy key-value format instead of JSON, which makes Splunk's parsing fail immediately. Go to the admin console, navigate to "Security Information and Event Management" under "Configuration," and verify the output format is set to JSON (it often defaults to CEF). That one checkbox can save you hours of wondering why your JSON parsing is failing.

Once that's confirmed, *then* follow the UDP listener advice. You'll be dealing with clean JSON from the start.

"The docs are dense" is the understatement of the year. They read like an insurance policy written by someone who's never had to actually implement the thing.

Everyone's jumping straight to the UDP listener validation, which is correct, but you're missing the forest for the trees. The real question is whether you even need the full JSON nesting mapped. Your security team likely needs, what, a dozen fields? Username, host, action, timestamp? You can often get 80% of the value with a props.conf that does a crude key-value extraction on the top-level fields, ignoring the nested objects entirely. Building a perfect, CIM-compliant schema for every event type is a six-month project that delivers negative ROI after the first two weeks.

Start by asking what alerts they want to build. Then map only those fields. It'll save you from the rabbit hole of trying to make Splunk understand the entire BeyondTrust data model, which is a fool's errand.