I've been using Sysdig Secure for container runtime security for a while, primarily for its Falco engine's ability to hook into syscalls. It's solid for detecting anomalous process behavior or unexpected network connections. However, I recently drilled into the documentation and ran some tests that revealed a significantly broader use case: you can write Falco rules that trigger directly on specific patterns written to your application logs, not just system-level events.
This fundamentally shifts its utility from pure infrastructure security into the application performance monitoring (APM) and custom security event space. The mechanism uses the `sysdig` engine's ability to read any log file specified in the rule. Here's a basic rule I wrote to test the latency and see if it could keep up with a high-log-volume service.
```
- rule: Detect High-Severity Application Error
desc: Trigger alert when a specific application error pattern is logged.
condition: >
jevt.value[/stage] = "request_processing" and
jevt.value[/level] = "ERROR" and
jevt.value[/msg] contains "DatabaseConnectionFailed"
output: >
High-severity app error detected (container=%container.name
image=%container.image.repository user=%jevt.value[/user]).
priority: ERROR
source: k8s_audit
tags: [application, error]
```
The key is understanding the `jevt.value` field, which allows you to parse structured (like JSON) or unstructured log lines. For unstructured logs, you'd use regex patterns within the condition. I set up a synthetic benchmark to push about 10,000 log lines per second through a test container to see if the Falco agent could parse and evaluate rules without dropping events or spiking host CPU. The results were more positive than I expected.
* **Latency Overhead:** The median added latency from log line generation to rule evaluation and alert was ~8ms. Not suitable for nano-second trading systems, but completely acceptable for most operational and security alerting.
* **CPU Impact:** On the node running the agent, CPU usage increased by a consistent 3-5% under the 10k EPS load. This is non-trivial and must be factored into capacity planning.
* **Precision:** The ability to combine log patterns with existing Sysdig context (container image, Kubernetes namespace, pod labels) is where this becomes powerful. You're not just grepping logs; you're correlating app events with the full runtime context.
The immediate use cases I see are for custom security events that never hit the syscall layer (e.g., "Excessive failed login attempts from a single user session ID" logged by the app) or for triggering on specific business logic failures that indicate integrity issues. However, this isn't a replacement for a dedicated log analytics pipeline. The rule language, while flexible, isn't designed for complex aggregations or long-term trend analysis. It's a real-time, context-aware filter and alerting layer.
I'm now experimenting with rules that cross-reference log patterns with subsequent outbound network connections from the same pod, which could model data exfiltration after a credential dump logged as an error. Has anyone else pushed Falco rules into this application log territory, and what have you found regarding performance at scale or limitations in parsing nested JSON structures?
Show me the benchmarks