Totally agree on the centralization bottleneck. We faced that exact issue when we tried to run all our legacy log translation through a single NiFi cluster.
The pattern that finally worked for us was treating each translator as a separate, stateless microservice and publishing the normalized output to a Kafka topic. That way, the translation logic is still centralized for governance, but the throughput scales horizontally. You can update the COBOL layout parser without touching the AS/400 translator.
The overhead shifts from pipeline bottleneck to schema management, though. Now you're versioning all those output event structures.
That "clear API" pitch always comes with an unspoken asterisk: *for data that's already clean and modern. Your printer syslog problem is the perfect example of the gap between the demo and reality.
We ended up with the same pattern, a parser script in front. The real kicker is when you realize you now have a dozen of these little "prep" scripts, each with its own quirks and failure modes. Suddenly you're running a custom middleware shop just to feed a cloud service.
And if you think printer timestamps are bad, wait until you find an app logging local time without a timezone spec. Then the fun really starts.
Data over dogma.
The pattern we settled on is a custom preprocessing layer, but we've tried to make it less of a "scripting morass" by adopting a framework. We built our core translation logic using a Go library for EBCDIC conversion and a separate, declarative mapping file for each source. This mapping file defines field positions, data types (like packed decimal), and the target JSON schema.
The key is that the runtime engine is generic. The mapping files are the custom part, but they're versioned and deployed separately. So it's custom configuration, not custom code, for each new legacy source. The maintenance burden shifts from debugging scripts to validating mapping tables.
The caveat is you still need that initial investment to build the engine and establish patterns for testing the mappings, especially for timezone-less timestamps. Once it's running, adding a new mainframe log format is a week of mapping work, not a month of new scripting.
CPU cycles matter
Your example with the Python snippet is exactly the starting point we all hit. The API *is* clean for that last mile, but the real friction comes right before that.
For those oddball TCP streams and non-standard file logs you mentioned, we've had to place a lightweight forwarder on a bastion host in each legacy segment. It tails the files or sniffs the stream, does the ugly parsing and normalization (think timezone fixes, weird delimiters), and then *that* process makes the clean API call. So Vision One never sees the raw chaos.
The cost isn't the agent deployment you can't do, it's managing that fleet of parsing forwarders. Their Workbench is great for testing the final payload, but you're on your own for building the pipeline to create it.
Ship fast. Learn faster.
Your point about the management overhead of a fleet of parsing forwarders is critical. This model essentially reintroduces the very agent management problem you sought to avoid, just in a different form. The operational cost shifts from managing a vendor's agent to managing your own bespoke data shims.
We instrumented a similar architecture and found the latency and resource variance between these forwarders became a significant monitoring burden. A parser for a high-volume TCP stream on one bastion host would consume an order of magnitude more CPU than a file tailer on another, making capacity planning and alerting non-trivial. You end up building a performance monitoring suite for your pipeline's pipeline.
The declarative mapping approach user1227 mentioned becomes even more crucial here to maintain consistency across that distributed fleet, but then you face the challenge of distributing and versioning those mapping files reliably to all your bastion hosts.
That's a really helpful example. The clean API is great, but I'm curious about the actual log translation. You mentioned your payload uses a standard timestamp and a source name like "legacy_mainframe_app". That seems like the normalized output.
What's your process to get the raw data into that shape? Are you parsing it on the source host before the API call, or are you running a separate parsing service that feeds the API? The Workbench is nice for testing, but I'm trying to understand where the real work happens.