Hey everyone, I've been lurking here for a while and learning a ton from the shared templates. I'm currently in a process at work where we're evaluating a few CI/CD platform vendors (think along the lines of GitLab vs. GitHub Actions vs. a hosted Jenkins service). One thing that wasn't super clear in the initial RFP was how to handle security and audit trails for the build agents themselves.
After reading some horror stories about compromised pipelines, I realized we needed to specifically ask about agent audit logs. I mean, if a job runs on an agent, who triggered it, what code ran, what network calls were made—that's crucial for us junior folks trying to debug and also for security. So I modified the "Security & Compliance" section of my rubric. Here's what I added as a specific line item:
**Agent-Level Audit Logging**
* **Requirement:** The platform must provide immutable audit logs for all actions performed on each CI/CD execution agent (container or VM), including user/process identity, commands executed, network egress attempts, and artifact access.
* **Evaluation Question:** "Walk us through your agent audit logging capability. Show us a sample log entry from a failed build, highlighting security-relevant events."
* **Scoring (0-5 pts):**
* 0: No agent-specific logging.
* 2: Basic job start/stop logs only.
* 4: Detailed command/network logging, but requires manual export.
* 5: Real-time streaming of agent audit logs to a SIEM, with granular filtering.
I also asked vendors to provide a sample log snippet during the demo. It was super revealing! One vendor just showed a generic job log, while another could actually pull up a JSON object showing a `docker pull` command from a specific IP inside the agent. 😮
For anyone else building their rubric, I'd suggest adding this. It pushes the conversation beyond just "yes we have logging" to "show us *exactly* what you log." It really helped us compare apples to apples. Has anyone else had to dig into this level of detail for their procurement? I'd love to hear if there are other agent-specific gotchas I should be looking for.
Learning by breaking
That's a solid addition to your rubric. Immutability is key; without it, the logs aren't trustworthy for a security review.
You'll want to test the granularity in your evaluation. Some platforms log only that an agent executed "a job," while others capture the actual shell commands or container pulls. The latter is much more useful for debugging a flaky step but can generate significant volume.
Consider adding a follow-up question about log retention and export formats. Can you pipe these logs directly to your SIEM or data warehouse for correlation? That's often where the real investigative work happens.
Immutable logs are a good start, but that's where the vendor reality distortion field kicks in. Everyone claims immutability until you ask who holds the encryption keys and whether their support team has a backdoor 'for your convenience.' The real test is asking them to generate a log entry from a *deliberately* tampered-with agent during a proof of concept. Then see if the log magically omits the `rm -rf` or the suspicious curl to an external IP.
Also, be prepared for them to balk at "network egress attempts." Most of these platforms have no visibility into the socket calls from the user's code running inside the container. They'll give you pretty timestamps for job start and stop, and maybe the container image hash. If you want actual network telemetry, you're probably layering on another tool, which they'll happily sell you as an 'ecosystem solution.' So your rubric line might just reveal the feature doesn't exist as you imagine it.
cg
Immutable logs are useless if you can't afford to keep them. Retention beyond 30 days is usually a paid add-on, and they know you'll need it for audits. Ask for the price per gigabyte per month before you get excited about SIEM integration.