I've been evaluating Cribl Stream for centralizing log management across several of our cloud environments, with a specific focus on meeting GDPR and CCPA requirements for data subject requests. A recurring challenge has been the ad-hoc, often brittle, redaction of Personally Identifiable Information (PII) from disparate log streams before they hit our long-term retention layer. To address this, I've constructed a reusable, condition-based pipeline that I believe others in similar compliance-heavy environments might find useful.
The core design principle is to separate the *identification* of sensitive data from the *redaction* logic. This is achieved through a two-stage process within a single pipeline. The first stage uses a series of `Parser` functions with regex patterns to detect and tag potential PII fields (e.g., email addresses, social security numbers, credit card numbers). The second stage uses a `Filter` function to inspect these tags and apply a consistent masking or hashing transformation only to the confirmed fields, leaving the rest of the event intact.
Here is the critical section of the pipeline configuration, focusing on the SSN detection and redaction logic. The same pattern is replicated for other PII types.
```javascript
// Stage 1: Detect and Tag
function detectPii(pipeline, context) {
const event = pipeline;
// SSN Pattern (simple US-centric example)
const ssnRegex = /b(d{3})-(d{2})-(d{4})b/g;
let match;
// A temporary field to hold our detected PII markers
event.__piiTags = event.__piiTags || [];
while ((match = ssnRegex.exec(event.raw)) !== null) {
event.__piiTags.push({
type: 'ssn',
startIndex: match.index,
originalValue: match[0],
lastFour: match[3]
});
}
// Repeat for email, credit card, etc.
return event;
}
// Stage 2: Redact based on Tags
function redactPii(pipeline, context) {
const event = pipeline;
if (!event.__piiTags || event.__piiTags.length === 0) {
return event;
}
let modifiedRaw = event.raw;
// Process tags in reverse order to preserve string indices
event.__piiTags.sort((a, b) => b.startIndex - a.startIndex);
event.__piiTags.forEach(tag => {
switch (tag.type) {
case 'ssn':
// Redact to 'XXX-XX-1234' (shows last four)
const redacted = `XXX-XX-${tag.lastFour}`;
modifiedRaw = modifiedRaw.substring(0, tag.startIndex) +
redacted +
modifiedRaw.substring(tag.startIndex + tag.originalValue.length);
break;
case 'email':
// Example: transform 'user@domain.com' to 'u***@domain.com'
// Implementation omitted for brevity
break;
}
});
event.raw = modifiedRaw;
delete event.__piiTags; // Clean up temporary field
return event;
}
```
Key considerations and lessons learned from implementing this:
* **Performance:** The regex operations, particularly on high-volume streams, are CPU-intensive. We mitigated this by applying this pipeline selectively via the `Route` function, only to log groups known to contain unstructured user data (e.g., application error logs, support ticket logs).
* **False Positives:** Initial patterns generated many false matches (e.g., ticket numbers resembling SSNs). We refined the regex patterns and added contextual checks (e.g., a field name hint from a prior key-value extraction) to the tagging logic to improve accuracy.
* **Reversibility:** For certain legal requirements, a one-way hash (using a `Hash` function with a salt stored in Cribl's secret store) is preferable to simple masking. Our pipeline allows swapping the redaction method per PII type based on a configuration variable set at the worker group level.
* **Testing:** We built a dedicated test stream of sanitized log samples with known PII placements to validate redaction accuracy and ensure no accidental data leakage or event corruption occurs.
This approach has proven significantly more maintainable than our previous method of embedding complex, all-in-one regex replacements directly in ingestion sources. By abstracting the logic into a reusable pipeline, we can now uniformly update redaction rules across all relevant data flows from a single definition.