Hey everyone. I've been using Splunk ES for a few months now, mostly relying on the built-in correlation searches and security content. It's great, but I'm hitting a wall where I need to start building custom searches for some of our specific internal workflows and oddball SaaS apps we use.
Coming from a CRM/RevOps background, I'm used to building alerts in Salesforce or HubSpot for weird data patterns, so the logic makes sense, but the SPL and the ES framework feel like a different beast. I get a bit lost on the "right" way to structure things within the ES app itself.
So, for those of you who've built your own, how did you start? Are there any good templates or a basic structure you follow? I'm curious about things like:
- How you set the data model alignment (do you always use datamodel=...?)
- Best practices for naming and severity assignment.
- Any good examples of a simple, yet effective, custom correlation search you built from scratch.
I'm not looking for anything super complex to start—maybe something like detecting a user's successful login from two geographically impossible locations within a short timeframe, but for a non-AD application. Just trying to understand the scaffolding. 😅
Any migration stories from generic Splunk alerts to proper ES correlation searches would be super helpful too.
Start with the built-in ones. Go to Configure, then Correlation Searches in ES and open a few in edit mode. That's your template. Copy one that's close to what you need and modify the SPL.
For your questions: you don't always need `datamodel=`. If your SaaS data is already CIM-compliant, use the tstats command on the model. If it's weird log data, just search the index/sourcetype directly and map fields later. Naming should follow the ES convention: "Application - Descriptive Name - Rule". Use the built-in severity matrix as a guide, don't just pick high because it feels right.
Your geographic impossible login example is a good start. The structure is already in the ES content. Look at "Access - Multiple Successful Logins From Geographically Distant Locations - Rule". Swap the identity and authentication data model references for your specific index and user/location fields. Test it against a small timeframe first with a `| stats count` to verify logic before you schedule it.
Where is your SOC 2?
Building from user323's point about CIM compliance, I'd add a practical distinction. If your SaaS app logs are already normalized via Technology Add-ons or Common Information Model (CIM) fields, absolutely use `datamodel=` and `tstats`. The performance benefit is massive for scheduled searches. If they're not, starting with a raw search on the index is fine, but you should immediately map the key fields (`user`, `src_ip`, `action`) to their CIM equivalents using `eval` or field aliasing in your `props.conf`. This future-proofs the search and lets you integrate with other ES features later.
For your impossible travel example on a non-AD app, the core SPL structure is straightforward. You'd start with the raw logs, filter for successful logins, then use `transaction` or `stats` to group by user with a time window. The geographic lookup would typically be done with a `lookup` to a geolocation table based on `src_ip`. The main deviation from the built-in search is just your source data.
One template I frequently reuse is this pattern for anomaly detection on low-volume events:
```
| tstats summariesonly=true count from datamodel=Authentication where Authentication.action="success" by Authentication.user,_time span=1h
| streamstats window=5 current=true avg(count) as avg_count stdev(count) as stdev_count by Authentication.user
| eval threshold = avg_count + (3 * stdev_count)
| where count > threshold AND count > 5
```
This flags users whose login count in the last hour is more than three standard deviations above their rolling average. You swap out the data model, the action field, and the grouping key. It's simple, statistically grounded, and avoids static thresholds.
data is the product
Great point about mapping fields early. I've seen too many teams skip that step and then struggle to adopt new ES use cases later. That props.conf work is a bit of upfront pain, but it pays off.
Your anomaly pattern is solid for low-volume stuff. One thing I'd add - when working with raw SaaS logs that aren't CIM-ready, test the search with `eventstats` instead of `transaction` first. It can be way more forgiving with weird timestamp formats, which these apps love to throw at you.
Trust the trial period.
Good call on mapping fields early. The props.conf step is tedious but it's a one-time cost. If you don't do it, you'll be rewriting every search later when you want to use Risk-Based Alerting or asset/identity correlation.
One caveat to the tstats performance advice: it only works if your data model accelerations are actually built. For a brand new, low-volume data source, a raw search might run faster initially until the summaries populate.