Syslog forwarding is the correct starting point for reliability, but the cost of the parsing service is often overlooked. Running it on a "stable, centrally-managed host" means you're committing to a perpetually running compute instance, likely 24/7. That's an ongoing operational expense.
You can mitigate this by making the parser stateless and deploying it as a serverless function, like AWS Lambda or Azure Functions. It only runs when syslog messages arrive. The cost shifts from a fixed monthly EC2 bill to per-invocation pricing, which for sporadic log traffic from legacy gear is often negligible. The failure mode then becomes cloud provider queue limits, not VM disk corruption.
The parsing logic itself is the real maintenance burden. Allocate engineering time quarterly to review its metrics, as log format drift from firmware updates is inevitable.
Less spend, more headroom.
The gradual degradation point is spot on. We lost a whole quarter of compliance logs because the appliance's firmware update changed the delimiter from commas to pipes. The parser kept running, but every field was misaligned.
Is there a good way to detect that kind of drift automatically, aside from just watching throughput? A drop in count might not show up if the device keeps sending the same volume of garbage.
Oh, this exact scenario is a special kind of pain. Watching throughput won't catch it because, like you said, the log lines keep flowing. The parser just happily creates nonsense.
What finally worked for us was adding a small validation rule at the start of the parsing stage. We'd sample, say, 1 in every 1000 lines and run them through a set of expected patterns: "does this field look like a timestamp?", "is this field an integer?", "does this column contain only these known enum values?". If the validation failure rate spiked, we got an alert. It's not perfect, but it caught a delimiter change for us once because a numeric field suddenly contained pipe characters.
The real trick is finding a "canary" field that's almost guaranteed to be stable. We used the total length of the line as a cheap heuristic for a while - a sudden shift in average character count was our first clue something structural had broken.
Syslog's the usual answer, but everyone glosses over the parsing tax. A Python script using the API sounds fine until the third time you're up at 2am because a firmware update changed a field.
The real pitfall is thinking this is a one-off project. You're signing up for permanent, unpaid maintenance on a custom pipeline. Budget the ongoing time, or it will fail silently.
—EB