Having recently completed a migration to BeyondTrust Password Safe for our PAM requirements, a significant operational gap became immediately apparent to our SRE team. While the platform provides excellent native auditing, those logs were trapped within its ecosystem, creating a blind spot in our centralized observability stack. Our security team operates a Splunk-based SIEM, and our platform engineering group relies on a Grafana/Loki/Prometheus setup for infrastructure and application monitoring. The lack of a real-time, streaming log feed from Password Safe into these systems was a compliance and operational risk.
To address this, I've developed and open-sourced a dedicated log connector. It's a Kubernetes-native service written in Go that consumes the BeyondTrust SIEM API, transforms the log entries, and forwards them to a configurable destination. The primary design goals were reliability, idempotency (to handle potential API throttling), and making no assumptions about the downstream consumer.
The core architecture is straightforward:
* A single, lightweight container that can be deployed as a `Deployment` or `DaemonSet`.
* Configuration via `ConfigMap` and `Secret` for credentials and endpoints.
* It polls the BeyondTrust `Event-Log` API endpoint at a configurable interval, using the `lastLogId` as a checkpoint stored in a persistent volume to avoid duplicate entries.
* Log entries are parsed, enriched with Kubernetes metadata (if running in-cluster), and serialized into JSON.
* A pluggable output system currently supports direct HTTP/S endpoints (for Splunk HEC, Loki, etc.) or stdout for sidecar collection by Fluent Bit or Logstash.
Here is a snippet of the core configuration for the connector, showcasing its flexibility:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: beyondtrust-connector-config
data:
config.yaml: |
beyondtrust:
api_url: "https://.beyondtrustcloud.com"
api_key_secret_name: "beyondtrust-api-credentials"
poll_interval_seconds: 30
batch_size: 1000
outputs:
- type: "http"
url: "https://splunk-hec.yourcompany.com/services/collector"
auth_token_secret_name: "splunk-hec-token"
compression: "gzip"
- type: "stdout"
format: "json"
```
Key considerations and lessons learned from the implementation:
* The BeyondTrust SIEM API is paginated and can be subject to rate limits. The connector implements exponential backoff and jitter in its retry logic.
* Log entry schemas are versioned. The connector includes a simple normalization layer to ensure field consistency, which is critical for downstream dashboards and alerts.
* For cost optimization and resource efficiency, the connector is designed to be stateless and horizontally scalable, though a single instance can typically handle the load for most deployments.
* We've integrated this with our Prometheus stack to expose metrics (e.g., `logs_forwarded_total`, `api_poll_duration_seconds`, `last_log_id`) for monitoring the health of the pipeline itself.
The immediate operational benefits have been substantial. We now have:
* Unified dashboards in Grafana correlating PAM events with infrastructure changes and IAM actions.
* Proactive alerting in Splunk for specific high-risk PAM activities.
* A immutable audit trail stored alongside all other platform logs in our cost-effective, long-term storage bucket.
The code, full documentation, and example Kubernetes manifests are available on GitHub. I'm interested in feedback from others who have tackled similar integration challenges, particularly around handling the API's semantics or ideas for additional output adapters.
—Chris
Data over dogma
Interesting approach focusing on idempotency. Did you consider how this handles schema changes in the BeyondTrust API? I've seen similar connectors break silently when fields get deprecated or added.
The Kubernetes-native design makes sense, but I'm curious about the deployment decision. When would you choose a DaemonSet over a standard Deployment for this type of log forwarder? Is it about node-specific routing or something else?
Excellent question about handling API schema changes, as that's a critical failure mode for any integration. My approach uses a combination of defensive unmarshaling and a strict validation schema. The connector unmarshals the API response into a flexible map structure first, then applies a transformation layer that only extracts fields defined in our internal schema. If the API returns a new, unexpected field, it's logged at a warning level but doesn't break the pipeline. The real risk is field deprecation, which is why the transformation logic includes default values for all required fields; if a required source field is missing, the connector populates it with a null sentinel value and raises an alert. This ensures the log stream continues uninterrupted, though with potentially degraded data fidelity, which is preferable to a silent stop.
Regarding your DaemonSet versus Deployment question, the choice is almost entirely about fault domain isolation and egress routing requirements. A DaemonSet becomes necessary if your network policy requires egress traffic to originate from specific node IPs, or if you're using a node-level logging agent sidecar pattern and need to forward locally. For a pure cloud service API call, a standard Deployment with sufficient replicas for availability is typically more operationally simple. The trade-off is that a Deployment introduces a potential noisy-neighbor problem if a single pod instance gets overwhelmed, whereas a DaemonSet naturally scales with your node count. I'd only default to a DaemonSet if you have a requirement to colocate the forwarder with another node-specific service.
Measure everything, trust only data
Ah, the classic "null sentinel and an alert" strategy for deprecated fields. A time-honored tradition of quietly accepting data decay while a Jira ticket languishes forever in the "Security Tech Debt" column.
While I appreciate the pragmatism of avoiding a pipeline crash, I've seen this pattern become a long-term liability in vendor management. That "degraded data fidelity" you mention isn't just a temporary blip. It becomes the new normal. The alert fatigues, the null values get baked into dashboards, and six months later you're in a renewal negotiation with zero leverage because you can't definitively prove their API change broke your compliance reporting. You've traded a loud, catastrophic failure for a silent, chronic one.
And on the DaemonSet rationale - isolating fault domains is valid, but isn't this just institutionalizing a workaround for a broken network policy? If your egress routing requires specific node IPs, you've already lost. Now you're coupling pod scheduling to network topography, which feels like solving one piece of tech debt by creating another, more subtle kind.
Price ≠ value.
Nice to see a Kubernetes-native approach. I'm curious about the "no assumptions about the downstream consumer" part.
Do you have built-in support for multiple output formats, or is the transformation locked to something like CEF for Splunk? I'm thinking about shipping to both Splunk and Loki; they usually need different log structures.
Automate everything.