Hi everyone, I'm pretty new to all this observability stuff, so apologies if this is a basic question. I've been reading about how expensive it can get, especially with logs.
Our team is starting to send more data to our observability platform, and the finance person is already asking about the projected bill 😅. We also handle customer data, so compliance (like GDPR) is a big concern.
I saw this thread title about "PII scrubbing at the source" and it sounds like a great idea. It makes sense that if you remove sensitive stuff before it leaves your servers, you wouldn't pay to process or store it. But I'm a bit lost on how to actually start.
Can someone explain the practical first steps? Like:
* Where does this scrubbing usually happen? Is it in the application code itself, or in an agent?
* Do you use specific tools for this, or is it mostly custom rules?
* How do you make sure you're catching all the PII patterns without breaking useful log data?
Any simple examples or pointers to guides would be so helpful. I'm trying to build a case for us to implement this early, before our costs and complexity get out of hand. Thanks in advance for any guidance!
Hey, great questions. For the where, I'd push back a bit on doing it directly in your application code. That's messy and can hurt performance.
A better first step is using your logging agent or library. Most of them have configurable processors or filters. For example, you can set up the OpenTelemetry collector with a 'attributes' processor to redact specific keys before anything gets sent upstream.
On patterns, start simple. Make a list of your own field names that contain PII (email, user_id, address). Scrub those first. Trying to catch every possible pattern with regex is a rabbit hole and you might remove useful info. Focus on what you know you log.
dk
> I'd push back a bit on doing it directly in your application code.
Good. It's a bad pattern. You'll have inconsistencies, tech debt, and a new surface for bugs.
But leaning solely on your collector/agent isn't complete either. You still need a codified source of truth for field names. Your first step should be a shared, versioned list across all teams, maybe in a config repo. If devs and SREs don't agree on what `customerIdentifier` means, you'll leak data.
slow pipelines make me cranky