Just upgraded our main cluster to v2.1 last week and dove straight into the new audit log feature. I was initially a bit overwhelmed by the verbosity—it logs *everything* from admission controller decisions to the full request/response bodies for some resources. But wow, it just paid for itself.
We have a policy blocking NodePort services in production via a Kyverno policy. The audit logs captured a developer's `kubectl apply` attempt in stunning detail. It didn't just say "denied"; it showed the exact spec of the service they tried to create, the user's identity, and even the specific policy rule that fired. We could trace the whole event end-to-end without digging through three different tools.
Here's a snippet of the log structure that was so useful:
```json
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v2",
"level": "RequestResponse",
"auditID": "abc123...",
"stage": "ResponseComplete",
"verb": "create",
"user": {
"username": "developer@company.com",
"groups": ["system:authenticated"]
},
"sourceIPs": ["10.0.1.5"],
"objectRef": {
"resource": "services",
"namespace": "prod-apps",
"name": "cache-bypass"
},
"requestObject": {
"spec": {
"type": "NodePort",
"ports": [...]
}
},
"responseStatus": {
"code": 403,
"message": "admission webhook 'validate.kyverno.svc' denied the request..."
},
"annotations": {
"policies.kyverno.io": "[{"name": "deny-nodeport-services", "rule": "validate-nodeport"}]"
}
}
```
We've now routed these JSON logs to a dedicated S3 bucket for analysis and set up a simple Athena query to flag policy violations. It's become a fantastic source of truth for our FinOps reviews too—seeing exactly who tried to provision what.
Has anyone else rolled this out? I'm curious how you're managing the volume without breaking the bank on log storage. I'm considering some filtering at the source before it hits S3, but don't want to lose the golden details.
cost first, then scale