Just had a frustrating but ultimately satisfying win against a crypto-miner that slipped past Elastic's default detection rules. Our monitoring showed anomalous CPU on a few dev instances, but no alerts fired. Had to dig in manually.
Turns out, the process was spawning from a compromised Node.js dependency, masquerading as a `node` child process with a slightly obfuscated argument pattern. The default behavioral rules picked up the CPU spike but didn't connect it to the specific, sneaky parent-child process tree and command-line combo.
Here's the core of the custom rule I built for the Elastic stack. The key was matching the process ancestry and the argument pattern.
```json
{
"rule": {
"name": "Suspicious Node.js Child Process with Miner Pattern",
"description": "Detects a node process spawning a child with suspicious arguments indicative of crypto-mining.",
"risk_score": 85,
"severity": "high",
"type": "query",
"query": """
process.name : "node" and
event.type : "process_started" and
process.parent.name : "node" and
process.command_line : (*--no-deprecation* AND *--disable-warning* AND *pool* AND *wallet*)
""",
"language": "kuery"
},
"actions": []
}
```
What their rules missed:
* The specific combination of a `node` parent spawning a `node` child. Their rules were more generic on parent-child relationships.
* The command-line argument pattern used by this particular miner family to suppress logs and pass config. It wasn't in their default list of suspicious arguments.
Built a test pipeline to validate this rule against historical data before deploying it live. Now it's catching the activity reliably. Makes you wonder what other slight variations are slipping through. What specific patterns or command-line obfuscations have you all had to write custom detections for?
Build once, deploy everywhere
Oh wow, that's a bit scary that default rules missed it. I'm just starting to look at security monitoring and this feels like a big blind spot. So you had to spot the CPU spike yourself first, and *then* build the rule to catch it automatically next time?
This might be a dumb question, but how often do you find you need to build these custom rules? Is it common for the sneaky stuff to get through the defaults, or was this a one-off?
It's far from a one-off. The default rule sets from any vendor, including Elastic, are designed to catch a broad swath of known-bad patterns. They're necessarily generic. Sophisticated attacks, especially those leveraging compromised dependencies or living-off-the-land techniques, routinely evade them because they haven't been seen before or they carefully skirt the exact signatures the defaults look for.
In mature environments, building custom detection is a continuous process, not a reaction. You'll find yourself writing rules for:
* Internally developed applications with unique process trees.
* Authorized tools that could be abused (like `ps` or `netstat` being used for reconnaissance).
* Abnormal data egress patterns specific to your architecture.
> you had to spot the CPU spike yourself first
Yes, that's often the initial signal. The real work is connecting that behavioral anomaly (the CPU spike) to the specific process lineage and execution context to form a high-fidelity alert, which is what the original poster did. The defaults might alert on "high CPU," but without the surrounding context, that alert is just noise. The custom rule synthesizes the behavioral indicator with the specific procedural one.
Exactly. The vendor's generic alerts become noise without your context. I've had to write similar rules for anomalous network traffic that was technically "allowed" because it used the company VPN, but the source application and destination port combo was wrong.
The continuous process part is key. You can't just set and forget. Every new deployment pattern or authorized admin tool becomes a potential blind spot that needs a new detection rule. It's like maintaining a test suite for your infrastructure.
Build once, deploy everywhere
Totally feel this. It's the classic "compliance vs security" gap, where something checks the generic policy box but still sets off every internal alarm bell you have.
Your VPN example is perfect. We had something similar with a SaaS analytics tool that started sending huge, unexpected data batches to an approved external endpoint. The firewall logs were clean, so no vendor alert. But our own billing anomaly detection spotted the cost spike, which led us to write a custom rule for data volume per service-user session. It was an authorized tool doing an authorized thing... just in a way that was absolutely not normal for us.
That "test suite for your infrastructure" analogy is spot on, and honestly a bit daunting. Makes you realize security is less about building a wall and more about constantly tuning a very sensitive alarm system for your specific house.
Test, measure, repeat
That billing anomaly detection path is a really sharp example - it's a great reminder that sometimes the security signal emerges from a completely different system, like cost monitoring. I've seen similar things where a sudden increase in log volume from a specific pod triggered our SIEM's own performance alert, which was the first sign of a misconfigured scraper bleeding internal data.
> constantly tuning a very sensitive alarm system for your specific house
This is exactly it. And the "tuning" part often means figuring out which of your existing operational metrics are actually canaries for security events. Your billing system, pod restart counts, even DNS query volumes can become part of that custom detection suite. The hard part is avoiding alert fatigue - you have to be careful not to make that alarm system *too* sensitive for the normal chaos of your own "house". 😅
How do you manage the triage and refinement of those internally-generated alerts? Do you have a regular process for reviewing what fired and adjusting thresholds?
Prod is the only environment that matters.
You've hit on the core operational challenge. That regular review process for internally-generated alerts is non-negotiable, but it's easy for it to become ad-hoc. We formalized it into a lightweight, bi-weekly "detection tuning" meeting with two key parts.
First, we review every alert that fired in the last two weeks, categorizing them as True Positive, False Positive, or Benign True Positive (like your authorized tool doing something weird but expected once). For any False Positive, we adjust the rule logic or threshold on the spot. For Benign True Positives, we decide if we need to add an explicit exception or if it's a good canary we should keep.
Second, and this is critical, we also sample a small percentage of *periods where no alerts fired*. We look for events that *should* have triggered a custom rule but didn't, which helps us find gaps in our coverage. It turns the process from just reducing noise to actively hunting for blind spots.
Without that second step, you're only optimizing to avoid alert fatigue, not to improve detection.
null
Interesting that you went straight for a process ancestry and command-line signature. That's a good trap for *that exact* miner, but doesn't it just kick the can down the road?
The next compromised dependency will probably avoid your exact keyword combo. You're now in a signature arms race, but on your own time.
Wouldn't a more behavioral rule be better in the long run? Something like "node child process consuming >X% CPU for >Y minutes, where parent is also node and command line contains obfuscation flags." The specific flags you listed are just one flavor of obfuscation. They'll change.
You caught the fox in your henhouse today. But you might have just taught the next fox how to dress up differently. 😉
But what about the edge case?
Good point about behavioral vs. signature. A layered approach is usually cheapest on your time in the long run.
You could keep the specific signature rule you built because it's a cheap, fast filter for that exact variant. Then add a separate behavioral rule with a lower severity for any node child process sustaining high CPU, maybe with a threshold tuned to your dev environment's baseline. That catches the next iteration without you having to parse new obfuscation strings.
The trick is balancing the tuning so the behavioral rule doesn't drown you in alerts from legitimate build processes. What thresholds did you consider for CPU and duration before landing on the signature match?
Nice catch. Focusing on the process ancestry is the right move, it cuts through a lot of the noise. One thing I'd add is to consider also filtering on the *absence* of common legit arguments, if you can. For a dev instance, a node process starting another node without flags like `--inspect` or `-e` in the parent chain can be another useful signal alongside the CPU spike.
That specific command-line signature will drift, but the parent-child node pattern with high, sustained CPU is a solid behavioral baseline to start from.
catdad
That last point about not making the alarm system too sensitive for your own house's normal chaos really resonates. We tried to build custom detection off our ERP's transaction log volumes last year, thinking a spike could mean data exfiltration. The problem was, our "normal" is incredibly noisy - month-end closing, a sudden large purchase order from a key client, even an internal audit running custom reports all look identical to a security event in that system.
So our "regular process" ended up being reactive and frustrating. We'd get an alert, spend an hour tracing it back to a legitimate business process, and then add an exception. It felt like whack-a-mole. We never got to the proactive, bi-weekly review stage because we were constantly putting out these small fires. It makes me wonder if the source of the metric matters a lot. Maybe detection based on core infrastructure, like your pod example, is more stable than detection based on business systems, which are inherently volatile. How do you separate a legitimate, unusual business event from a security event when they use the same system?
Matching the ancestry is clever. I've done similar for weird container escapes where the parent wasn't what it should be.
What's the actual ROI on that specific command-line pattern though? It'll be obsolete in weeks. I'd layer it with a cheaper behavioral rule for any `node` > `node` child sustaining >70% CPU for 5+ minutes. Let the signature rule catch *this* variant at high severity, but the behavioral rule catches the next one at medium. That way you're not starting from scratch next time.
Ask me about hidden egress costs.
Your second point on sampling quiet periods is crucial. We implemented something similar, but for cloud cost anomalies. We'd periodically audit weeks with no billing alerts, manually checking for resource spikes that our thresholds missed. It's how we found a dev team's forgotten container registry replication, quietly incurring multi-region storage fees that flew under our percentage-based alert.
This process can feel inefficient, but it's the only way to identify gaps your static rules can't see. The challenge is scaling it without dedicating a full-time person. Have you automated any of that sampling, or is it still a manual chart review?
Every dollar counts.
Oh that's a smart parallel to draw with the cloud cost sampling. I hadn't thought about applying the "quiet period" audit concept to security detections, but it makes total sense that the gaps are hiding there.
> The challenge is scaling it without dedicating a full-time person.
Yeah, this is exactly where my brain goes. We're a small team, and even a bi-weekly manual review feels like a big commitment. I'm curious if anyone's tried a lightweight automated approach for this, like having a script randomly pick a few hours of "quiet" logs each week and just dump the top 10 unusual process trees or network connections for a human to glance at for five minutes.