I'm in the early stages of evaluating CrowdStrike for our CI/CD and production environments. The sales demo is booked, but I don't want a flashy features tour. I need to know if it will actually work in our pipelines without causing flaky builds or deployment blockers.
What specific, operational metrics should I demand to see in the demo? I'm thinking less about marketing slides and more about data that proves it won't break our automation.
My list so far:
* **False Positive Rate:** The percentage of benign activities flagged as malicious during normal dev/build operations (e.g., Docker pulls, npm installs, terraform apply).
* **Performance Impact:** Measurable increase in pipeline stage duration with the agent loaded. I want to see this for a container build, a unit test suite, and a deployment step.
* **Detection/Prevention Lag:** Time from a known IOCs release (e.g., new CVE) to it being actionable in my policy. Not "we update hourly," but a demonstrated timeline.
* **API & Automation Coverage:** Can I manage policies, fetch alerts, and initiate scans entirely via API/CLI? I need to see actual curl examples or a Terraform provider for the Falcon platform.
* **Container Image Scan Integration:** If it scans images in the registry, what's the failure workflow? Does it just alert, or can it integrate with a pipeline to **break the build** with a clear output?
What else am I missing? What metrics did you wish you'd asked for before rolling it out?
Build once, deploy everywhere
Your list is a solid foundation. I'd push for more granularity on performance impact. Asking for an "increase in pipeline stage duration" is good, but you need to isolate the agent's overhead from network and storage I/O. Demand to see a breakdown: user CPU, system CPU, and memory delta during the build and test stages in a controlled environment. A vendor can hide a 15% overall time increase in noise, but a consistent 8% increase in system CPU across every pipeline run is a real cost.
On false positives, broaden the scope beyond dev/build. Ask for the mean time to acknowledge (MTTA) and mean time to resolve (MTTR) for alerts in a CI/CD context from their existing customers. A low false positive rate is meaningless if the triage process for the positives it does generate requires a security engineer to manually validate every pipeline run.
You mentioned API coverage. Go further: ask for the API's idempotency guarantees and rate limiting tiers. If you're managing policies as code, a non-idempotent `PUT` to a policy endpoint can cause configuration drift. Also, check if their detection lag metrics include the time from their cloud receiving a telemetry event to a rule actually evaluating it. The "IOC release" timeline is external; the internal processing latency is what you'll actually experience.
No free lunch in cloud.