We've been evaluating Cribl Stream to replace a legacy log routing system, with a primary requirement being strict isolation between our different client data pipelines. While Cribl's Worker Groups and Routes provide logical separation, we needed enforceable boundaries for configuration, secrets, and data spillage prevention in a shared infrastructure.
Our current approach uses a combination of Cribl features and external orchestration:
* **Dedicated Worker Groups per Tenant:** Each client gets a named Worker Group, pinned to specific nodes via tags.
* **Pipeline Namespacing:** All pipelines, Packs, and sources for a tenant are prefixed (`clientA_httpin`, `clientA_parse_filter`). This simplifies UI/API management.
* **Route-Based Guardrails:** Ingress routes are configured with strict filtering on the `__inputId` or a client-specific header/token, directing data *only* to that tenant's pipeline chain. The critical step is a validation function at the start of each pipeline to reject mismatched data.
```javascript
// Pipeline Function: validate_tenant_source
if ( (__inputId !== 'clientA_secure_ingest') &&
(event?.headers?.x_client_token !== 'clientA_defined_secret') ) {
throw new Error("Unauthorized pipeline access attempt.");
}
```
The unresolved tension is around secret management for destinations (S3 keys, Splunk tokens). We're leaning towards storing tenant-specific secrets in our orchestration layer (e.g., HashiCorp Vault) and injecting them as environment variables at Worker Group startup, rather than using Cribl's built-in secret store. This keeps access control external and auditable.
Has anyone implemented a similar multi-tenant model? I'm particularly interested in how you handle:
* Configuration drift between identical pipelines for different tenants.
* Monitoring and metrics separation to provide per-tenant visibility.
* Safe shared Pack management without accidental cross-tenant data references.
--crusader
Commit early, deploy often, but always rollback-ready.