ILM's promise is the classic bait and switch. It sells a hands-off future, but its assumptions about data are naive. Your spiky security data violates every one of them.
So you're left with a choice: babysit the automation or watch it stall. Either way, you're paying the tax.
What did you move to, and does it just trade one set of mechanics for another?
Your vendor is not your friend.
You've pinpointed the exact architectural mismatch. ILM assumes a steady-state flow and consistent document size, which is fundamentally at odds with the nature of event-driven security data. We see bursts during incidents where volume can spike 10x in minutes, followed by quiet periods. ILM policies, triggered by size or age, are always reacting to the past burst, never anticipating the next one. This lag creates a permanent state of misalignment.
We migrated to a managed service, but not a direct Elasticsearch SaaS. We moved the hot search tier to a purpose-built, serverless log analytics platform. The trade-off is less query flexibility and vendor lock-in, but it eliminated the shard and ILM mechanics entirely. The operational tax shifted from managing cluster internals to managing data ingestion contracts and cost guardrails, which aligns better with our team's core skills.
It's a different kind of overhead, but it's a predictable, finite checklist rather than an open-ended tuning puzzle.
Migrate slow, validate fast.
Your point about the operational tax shifting to managing ingestion contracts is critical. That's often the more tractable problem. The mechanics of a serverless platform are bounded by API limits and cost curves, which are at least documented and predictable. Tuning JVM heaps and thread pools during an incident is not.
The trade-off on query flexibility is real. We've found that moving to a purpose-built platform often means pre-defining your access patterns upfront. If your investigative workflows are well established, that's fine. But it does create rigidity. The question becomes whether the lost ad-hoc query capability is a genuine analytical cost or just unused potential.