Hey everyone. I've been running a self-managed ELK stack for log analytics for a few years, and our team recently made the jump to the full Elastic Security solution (with Fleet, Agent, all the new bells and whistles). The migration was mostly smooth, but the mapping phase held a few surprises that cost us some time. I wanted to share what we learned, hoping it might save someone else a headache.
The biggest shift in mindset is that Elastic Security is much more prescriptive about its field mappings than a vanilla ELK stack. In our old setup, we had a lot of custom, dynamic mappings for our application logs. That flexibility disappears when you start using the Elastic Common Schema (ECS) and the pre-built security detection rules. We discovered that many of our existing fields were either incorrectly typed or mapped to non-ECS field names, which meant the security rules just wouldn't fire.
For example, we had a custom field `src.ip` in some logs. In ECS, the canonical field is `source.ip`. Sounds simple, but we had to decide: do we re-ingest all the old data, or use ingest pipelines to rename and re-type fields on the fly? We went with ingest pipelines for speed, but it added complexity. Also, paying attention to field types like `keyword` vs `ip` became critical—some detection rules are very specific.
My advice? Before you flip the switch, spend time auditing your current indices against the ECS reference. Use the Field Mapping Assistant in Kibana Stack Management if you can. And don't underestimate the value of a test environment with a subset of real data to validate that your critical security detections are actually picking up events correctly. The tooling is good, but it requires a bit of a different approach than the old "log it and figure it out later" method.
Anyone else gone through this transition? What was your biggest mapping hurdle?
Keep it civil, keep it real.
I'm a senior data engineer at a mid-market fintech, running a hybrid SIEM setup where we've kept ELK for app logs but shifted our security telemetry to Elastic Security across about 3,000 agents.
- **Mapping and Schema Rigidity:** The biggest operational cost is ECS compliance. If your existing logs aren't already close to ECS, the mapping and ingest pipeline work is non-trivial. We spent roughly 80 engineer-hours retrofitting pipelines for fields like `client_ip` to `source.ip` and enforcing correct types (like `keyword` not `text` for IPs) so detection rules would trigger.
- **Agent and Fleet Overhead:** The Elastic Agent/Fleet model is a win for centralized policy but adds ~15% more resource consumption per host compared to standalone Beats in our environment. The benefit is unified management; the cost is increased memory footprint, which matters at scale.
- **Rule Tuning and False Positives:** The prebuilt security rules are a starting point, not a set-and-forget. Expect to spend the first 30-60 days in a cycle of tuning exceptions and adjusting thresholds. We saw a 40% false positive rate initially on network-based alerts until we tailored them to our specific subnet layouts and service ports.
- **Licensing and Cost Surprise:** List price is per agent per year. The real cost for us was the compute tier required for performant alerting. Running the security features on the same "hot-warm" hardware we used for pure logging led to rule execution latency. We had to bump to dedicated, higher-spec "hot" nodes for the security indices, which increased our Elastic Cloud commit by about $3k/month.
My pick is Elastic Security, but only if your primary need is a tightly integrated SIEM and your logs are already mostly structured or you can commit the pipeline work. If you're still predominantly a log analytics shop with occasional security use cases, the added complexity and cost of the full security suite isn't justified. To make a clean call, tell us your average daily ingest volume and whether your team has dedicated security analysts or if it's the same engineers managing the logs.
—davidr
The 15% overhead is a really useful data point, thanks. We're planning a similar rollout for a few hundred endpoints and that helps with capacity planning.
You mentioned spending 80 hours on mapping pipelines. Was most of that effort figuring out the new ECS field names, or was it more about fixing data quality issues in your source logs? I'm worried our log formats are messy and that's where we'll lose time.