Skip to content
Notifications
Clear all

How do I handle logs from acquisitions that use different log formats?

1 Posts
1 Users
0 Reactions
1 Views
(@datadog)
Estimable Member
Joined: 1 week ago
Posts: 90
Topic starter   [#18095]

Acquisitions mean inheriting a mess of syslog, CSV, and custom JSON. Standardizing at the source is a political battle you'll lose. You need a normalization layer.

Ingest everything into a common pipeline (e.g., Fluentd, Vector) and transform to a canonical model *before* it hits Chronicle. Use a rules engine. Example mapping different auth log formats to a CEL time, principal, and action:

```yaml
# Vector remap example for "user_login" event
transforms:
normalize_auth:
type: remap
inputs: ["raw_acme_logs", "raw_globex_logs"]
source: |
. = parse_apache_log!(.message) if exists(.message)
.timestamp = to_timestamp!(.timestamp, format: "%d/%b/%Y:%H:%M:%S %z")
.event_type = "user_login"
.principal = .user
.target_resource = .path
del(.user, .path)
```

Key steps:
* Map all timestamps to UTC nanos.
* Extract a common set of core fields (principal, target, outcome).
* Tag the source with `company: acquired_co` and `log_type: normalized`.
* Drop redundant or PII-heavy original fields post-extraction.

Chronicle rules then operate on the normalized `event_type` fields. Alerts on `$principal` work universally. Keep original raw logs in cold storage for 90 days in case mapping fails.

—DD


Metrics don't lie.


   
Quote