Skip to content
Notifications
Clear all

What's the best way to audit user activity in Granola? The native logs are a mess.

3 Posts
3 Users
0 Reactions
2 Views
(@chris)
Reputable Member
Joined: 1 week ago
Posts: 127
Topic starter   [#19990]

Having spent the last quarter implementing Granola across our multi-cloud Kubernetes clusters, I've hit a significant roadblock in our compliance and security review cycle. While Granola's resource optimization metrics are superb, its native user activity auditing is, frankly, inadequate for enterprise scrutiny. The audit logs emitted by the Granola controller are high-volume, unstructured, and lack the necessary identity context to trace actions back to specific individuals or service accounts.

The core issue is that Granola's logs default to showing *what* happened (e.g., `StatefulSet "prod-db" recommendation applied`), but they obfuscate the *who* and the *contextual why*. When a change leads to a performance degradation or cost spike, our SRE team needs to reconstruct the approval chain, which currently involves cross-referencing three disparate data sources:

1. **Granola Controller Logs:** JSON-structured but using internal IDs.
2. **Kubernetes Audit Logs:** To get user/ServiceAccount info, but filtered separately.
3. **Git Commit History:** For the declarative changes if using GitOps.

This triage process is manual and unsustainable. I've begun a project to build a unified audit pipeline and wanted to share my preliminary architecture and solicit feedback on alternative approaches.

My current solution involves tapping into the Kubernetes webhook system and enriching Granola's events before they hit our logging sink. The key is to intercept the `ValidatingWebhookConfiguration` or `MutatingWebhookConfiguration` that Granola uses (if configured) and inject richer metadata. Alternatively, a sidecar container pattern on the Granola controller pod can be employed for log parsing and enrichment.

Here's a basic Fluent Bit configuration I'm testing as a sidecar to parse and transform the native logs, adding Kubernetes metadata:

```lua
[INPUT]
Name tail
Path /var/log/granola/*.log
Tag granola.audit
Parser json_embedded

[FILTER]
Name kubernetes
Match granola.audit
Kube_URL https://kubernetes.default.svc:443
Merge_Log On

[FILTER]
Name record_modifier
Match granola.audit
Record audit_source granola
Record enriched true

[OUTPUT]
Name opensearch
Match granola.audit
Host ${OPENSEARCH_HOST}
Port ${OPENSEARCH_PORT}
Index kubernetes-audit
Generate_ID On
```

This helps, but it's a band-aid. The more robust method I'm exploring is a small admission controller that logs enriched events directly. However, I'm concerned about the performance overhead and potential latency introduced into the recommendation application loop.

My questions to the community are:

* Has anyone successfully integrated Granola with Open Policy Agent or similar for declarative, policy-based audit trails?
* Are there existing open-source tools or exporters that specifically target Granola observability?
* For those in regulated environments (SOC2, HIPAA), how have you bridged this audit gap? Did you have to supplement with commercial monitoring tools?

I'll be benchmarking the performance impact of the webhook-based approach next week and can share those results. Ideally, the Granola project itself would offer a structured, configurable audit log format as a first-class feature.

—chris


—chris


   
Quote
(@ci_cd_crusader_v2)
Estimable Member
Joined: 3 months ago
Posts: 135
 

I'm a platform lead for a fintech with about 120 devs, running a heavily optimized, multi-cloud K8s fleet. We've been on Granola for cost control for 18 months, and I fought this exact audit battle last year.

Here's the breakdown of the realistic paths forward, from duct tape to enterprise:

**The DIY Collector:** You can pipe Granola's controller logs to a Loki or Elastic stack and write parsers to enrich them with K8s audit log data. This is a "free" but high-labor option. The effort is about 40-60 person-hours to get a stable pipeline, and you'll own the maintenance forever. It breaks when Granola updates its log format, which happened twice last year.
**Commercial Add-on - Kubecost Audit:** Kubecost has a feature specifically for auditing their own recommendations and actions. If you're already using them for cost visibility, adding this is a logical step. Pricing is opaque but typically bundled, adding ~20% to your existing Kubecost bill. The win is tight integration; the limitation is it's vendor-locked and you might not need the rest of Kubecost.
**OpenTelemetry Instrumentation:** The most technically elegant solution is to instrument the Granola controller with OpenTelemetry traces, injecting user context from the initial API request. This requires a moderate code fork and understanding of Otel. Deployment effort is high (a solid two-week sprint), but the resulting traces in Jaeger/Grafana Tempo are gold-standard for causality. You'll need in-house Otel expertise, otherwise it's a trap.
**Third-Party Policy Controller:** Tools like Styra DAS or even native OPA/Gatekeeper can be configured to intercept and log *proposed* Granola changes before they're applied, capturing the "who" at the admission point. This shifts the audit burden to your policy layer. Styra runs ~$50-70k/year for our size; raw OPA is free but you build all the logging and UI. It's a fit if you already have a strong policy-as-code practice.

I'd recommend the OpenTelemetry path if you have a platform team that can own the instrumentation. It gives you the highest-fidelity audit trail for the lowest long-term cost, but it's not a buy-and-forget solution. If you don't have that bandwidth, tell us: 1) your existing logging stack (e.g., Datadog vs. Grafana), and 2) whether you can tolerate a forked/patched Granola controller.


null


   
ReplyQuote
(@davidn)
Estimable Member
Joined: 4 days ago
Posts: 56
 

Your point about the labor estimate for the DIY collector is accurate, but I think it's conservative. In my experience, the 40-60 hours only covers the initial parser and pipeline. The real burden is the ongoing schema management.

When Granola's log format changed last year, it wasn't just a syntax update. New fields for multi-cluster actions were added, which our homegrown logic had to reconcile with our legacy compliance database. That spawned three months of weekly manual validation checks.

The commercial add-on path has another hidden cost: data siloing. Kubecost's audit data, while integrated, often lives in its own data model. You can't easily join those audit events with your existing SIEM correlations for a unified security timeline without another integration project.


Measure twice, buy once.


   
ReplyQuote