Having recently completed the integration of NordLayer's network security logs into our Splunk Enterprise instance, I found the process to be remarkably streamlined—provided you understand the data schema and the requisite Splunk configuration. The official documentation covers the basics, but a practical, step-by-step workflow focusing on data mapping and transport reliability is often what's missing. This guide details the exact procedure I used, which successfully created a parsed and searchable data source within the promised timeframe.
**Prerequisites & Initial Configuration**
First, ensure you have the necessary permissions in both NordLayer (Organization Admin) and Splunk (to create inputs and indexes). The core mechanism is NordLayer's **Activity Logs API**, which provides JSON-formatted events for connection attempts, data transfer, and administrative actions.
1. **Create a dedicated Splunk index and HEC token:** This isolates the data and provides a secure ingestion endpoint.
```bash
# Example of creating the index via Splunk CLI (or use the Web UI)
splunk add index nordlayer_logs
splunk http-event-collector create token nordlayer_token -index nordlayer_logs -description "NordLayer API Logs"
```
2. **Generate your NordLayer Service Token:** Navigate to *Organization settings > API tokens* in the NordLayer Control Panel. Store this token securely.
**The Integration Workflow: Data Fetch to Parsing**
The integration uses a simple Python script as a data fetcher and forwarder, scheduled via cron or a scheduler. The script performs three key functions: API call, minimal transformation, and submission via Splunk HEC.
```python
import requests
import json
import time
from datetime import datetime, timedelta
# Configuration
NORDLAYER_API_URL = "https://api.nordlayer.com/v1/logs/activity"
NORDLAYER_TOKEN = "your_nordlayer_service_token"
SPLUNK_HEC_URL = "https://your-splunk-server:8088/services/collector"
SPLUNK_HEC_TOKEN = "your_splunk_hec_token"
headers_nord = {'Authorization': f'Bearer {NORDLAYER_TOKEN}'}
headers_splunk = {'Authorization': f'Splunk {SPLUNK_HEC_TOKEN}'}
# Fetch logs for the last 15 minutes (adjustable window)
params = {
'start': int((datetime.now() - timedelta(minutes=15)).timestamp()),
'end': int(datetime.now().timestamp()),
'limit': 1000
}
response = requests.get(NORDLAYER_API_URL, headers=headers_nord, params=params)
if response.status_code == 200:
events = response.json().get('data', [])
for event in events:
# Prepare event for Splunk HEC
payload = {
"host": "nordlayer-api",
"source": "nordlayer:activity",
"sourcetype": "_json",
"event": event
}
# Send to Splunk
requests.post(SPLUNK_HEC_URL, headers=headers_splunk, json=payload, verify=True)
```
**Critical Data Mapping & Field Extraction**
The raw JSON payload is nested. For effective Splunk searches, you must configure `props.conf` on your indexers or heavy forwarders to extract key fields. This is the most crucial step for usability.
```
# In $SPLUNK_HOME/etc/system/local/props.conf for index=nordlayer_logs
[nordlayer:activity]
SHOULD_LINEMERGE = false
DATETIME_CONFIG = CURRENT
KV_MODE = json
TRANSFORMS-nordlayer_fields = extract_nordlayer_json
# In transforms.conf
[extract_nordlayer_json]
REGEX = "event_type":"(?[^"]+)".*"account_id":"(?[^"]+)".*"result":"(?[^"]+)"
FORMAT = event_type::$1 account_id::$2 result::$3
```
**Key Considerations & Potential Pitfalls**
* **Rate Limiting:** The NordLayer API has undisclosed rate limits. Implement exponential backoff in your script if you plan to poll frequently.
* **Idempotency:** The script should handle partial failures to avoid duplicate data ingestion. Consider logging the last successfully ingested timestamp to a file.
* **Time Zones:** NordLayer API timestamps are in UTC. Ensure your Splunk instance is configured to interpret them correctly to avoid time-shift discrepancies in your dashboards.
* **Field Consistency:** Review the API sample response thoroughly; while generally stable, new features may introduce additional fields. Your field extractions (`transforms.conf`) may require updates.
This setup now provides our SOC team with real-time visibility into NordLayer connection events, fully integrated into our existing security incident workflows. The total active configuration time was approximately 25 minutes, with the majority spent on validating the field extractions and writing the initial documentation for our internal runbook.
Thanks for kicking this off with such clear, actionable steps. The point about data mapping is crucial. I've seen a few teams get tripped up because the API's nested JSON doesn't always align cleanly with their existing Splunk common information models. Did you have to build any custom calculated fields, or did the default sourcetype parsing handle it for you?
Keep it civil, keep it real
Thirty minutes is optimistic unless you're working with a pristine lab environment. In the real world, getting those permissions approved from two separate teams alone can take days. The 30 minute clock doesn't start until you're actually touching a console.
And let's be honest about "parsed and searchable." Sure, the data lands in an index. That's the easy part. Making it actually useful for correlation usually means wrestling with field extractions and CIM compliance long after the initial victory lap. Did you factor that into your timeline?
Show me the TCO.
You've hit on the critical post-ingestion phase. The default sourcetype parsing for JSON handled the initial field extraction, but alignment with CIM, specifically the `network_traffic` model, required calculated fields.
I had to create three to normalize the data:
* One to concatenate source and destination into a `connection_id` for session analysis.
* Another to translate NordLayer's activity `type` field into a more generic `action` field with values like "allowed" or "denied".
* A third to convert their timestamp into both `_time` and a separate `date_hour` field for faster time-based aggregations.
Without those, the raw fields are searchable, but you can't effectively use out-of-the-box correlation searches or dashboards built for the CIM. That extra work took me another hour, but it's a one-time setup.
Your calculated field for the `action` normalization is a smart move. That translation from vendor-specific terminology is often the hidden step that makes the data truly portable across your security tools. It also future-proofs your searches if NordLayer ever decides to rename those `type` values.
One caveat on the `date_hour` field for aggregations: just be mindful of how you handle time zones if your team is distributed. The API timestamp's timezone might not match your Splunk server's, so I usually bake that conversion right into the calculated field logic to avoid midnight confusion.
Review first, buy later.
Good call isolating the logs to a dedicated index from the start. That's a huge time saver for cost allocation later, especially if you're tagging your indexes by cost center.
One minor adjustment: in your CLI example, the token creation command references an index named `nordlayer`, but you created `nordlayer_logs`. That mismatch will cause the HEC to fail or route data to a default index.
Also, have you considered setting a retention period on that new index during creation? It's easy to forget, and a few months of verbose network logs can get surprisingly expensive.
Oh, good catch on the index name mismatch in the CLI step - that's exactly the kind of typo that wastes an afternoon. Thanks for pointing it out.
You're absolutely right about setting retention at creation. It's too easy to kick that can down the road until the bill arrives. I usually set mine to 90 days, which feels like a good balance for network logs like these.
The dedicated index also saved us when we later needed to apply a different data retention policy for compliance reasons. Much easier to adjust one index than hunt through a general security log.
Automate the boring stuff.
Nice kickoff. That emphasis on the **practical workflow** over just the official docs resonates so much. I've found that the gap between "it's possible" and "it's working reliably in production" is all about those specific transport and mapping details you're hinting at.
Looking forward to the rest of the steps, especially how you handle the API pagination and any error handling you baked into the data collection. I've seen that trip people up when the log volume grows.
Happy testing!
Exactly, pagination and error handling are what separate a quick test from a reliable feed. For the API calls, I wrapped the collection in a script that checks for the `next_cursor` in the response and loops until it's empty. Also added basic retry logic with exponential backoff for any 5xx errors from their end. It's simple but keeps it running without manual babysitting.
The real trick was logging the script's own health - like fetch timestamps and record counts - back into a separate Splunk monitor. That way, if the feed stalls, we get an alert from the gap in its own heartbeat logs.
Automate everything.
Love that you're starting with the dedicated index and HEC. So many folks dump everything into "main" and regret it later when they need to manage volume or access.
> provided you understand the data schema
This is the real 30-minute gatekeeper, isn't it? I'd add one super-quick prerequisite: pull a sample JSON log from the NordLayer API *before* touching Splunk. Having that actual output open in a text editor while you set up the props transforms saves so much back-and-forth. You can spot nested fields that might need extra extraction right away.
It's the only sane way to handle third party logs. Getting a sample JSON first is mandatory, not a suggestion. You can't write a props.conf transform if you don't know the exact field structure you're dealing with.
That pre work is what keeps it to 30 minutes. Otherwise you spend 20 just troubleshooting why Splunk isn't picking up a nested `object.sub_field`.
Ah, the "reliably in production" gap. That's where the 30-minute guide meets reality. The pagination and error handling aren't just implementation details - they're the entire production workload. A script with a simple retry loop sounds good until you realize you've now built a stateful service that needs its own monitoring, deployment, and failure recovery. Suddenly you're not just configuring Splunk, you're responsible for a custom data pipeline. That's the hidden cost they never put in the quick-start guide. What's your plan for when that script crashes at 2am?
Your k8s cluster is 40% idle.
Spot on about needing that practical workflow. The official docs get you to the starting line, but the real race is making the data usable and reliable in your specific Splunk setup.
Pulling a sample JSON first is such a smart time saver. I'd add that you should also grab a few different event types (logins, data transfers, etc.) to see the full schema variation. Nothing worse than building your transforms around a "connection" event only to find the "admin_action" event has a completely different nested structure 😅
Excited to see your steps on the data mapping. That's usually where the clock starts ticking for me.
~Harry
Absolutely. Grabbing multiple event types before you start is the real pro move. I learned that the hard way on a different project - built everything around user login events, then the first 'policy_change' log came in and my extractions were all broken because the field structure was totally different.
It added like an extra hour of rework to untangle. So yeah, your point about "the full schema variation" is clutch.
Looking forward to the mapping steps too. That's where I always get stuck - knowing what to even look for in the logs.
Still learning.
Ninety days is a solid starting point for network logs. We settled on the same, but ended up creating a separate, longer-retention index just for admin and policy change events after our first audit. It let us keep the high-volume tunnel logs on a tighter cycle without losing the crucial security audit trail.
That separation you mentioned, having a dedicated index, also made it trivial to apply different role-based access controls later. The networking team doesn't need to see the admin audit logs, and now they don't.