What's the best way to handle logs from unsupported appliances or legacy systems?

I'm a senior SRE at a mid-market fintech running about 200 nodes, where I manage our Elastic Stack (200+ TB) for both security and application observability; we ingest logs from everything from modern K8s pods to 15-year-old AS/400 systems.

1. **Syslog Forwarding (UDP/TCP)**: The most common method. Configure your appliance to forward to a syslog server (like rsyslog) running as a dedicated ingestion node. In my env, a single 4-core node reliably handles ~25k EPS (entries per second) for TCP syslog. The hidden cost is parsing: you'll spend significant time writing Grok patterns or ingest pipelines to structure the raw syslog message. Reliability drops with UDP; we saw ~0.5% packet loss under load.
2. **Custom Python Script + HTTP API**: You write a script to tail a file or poll an API on the legacy system, then ship via the Elasticsearch HTTP API or the `elasticsearch` Python library. For a low-volume source (<100 EPS), this is simple. For higher volume, you must implement batching and retries. Our Python forwarder for a mainframe log file uses a 10-second batch window and holds up to ~5000 events in memory before applying backpressure. The win is total control over parsing before ingestion.
3. **Filebeat as a Generic Forwarder**: Even without a supported module, Filebeat's `log` input can tail files. Deploy a lightweight VM or container alongside the legacy system. It handles backpressure, TLS, and retries out of the box. We run this on Windows Server 2008 R2 systems. The config effort is low, but you still need to manage parsing separately. The limitation is it's file-only; it can't poll HTTP endpoints or proprietary APIs.
4. **Logstash as a 'Bidirectional' Gateway**: Deploy Logstash on a central server. Configure inputs for `tcp`, `udp`, or `http` to receive data from scripts or appliances, and use an Elasticsearch output. This adds a processing layer. Its strength is handling multiple input streams and normalizing data, but it's resource-heavy. A single Logstash node in our setup (4 vCPU, 8GB RAM) saturates at ~7-8k EPS when running complex filters.

My pick is **Filebeat** for any legacy system where logs are written to files, because it's resilient and operationally simple. For appliances that can only send syslog or where logs are only accessible via a proprietary protocol, my pick is a **custom Python script feeding Logstash TCP/HTTP input**, as it centralizes the parsing logic. To make the call clean, tell us the approximate log volume (EPS) from these unsupported sources and whether they can run a lightweight sidecar process.

> I'm most comfortable with Python.

That's a great starting point. While syslog is the universal fallback, a custom Python script gives you much finer control, especially for weird legacy formats. I've used the Elasticsearch Python client to build simple file watchers that parse and push data.

The biggest pitfall I've hit is managing state - making sure your script reliably tracks its position in a log file after a restart. Use something like the `watchdog` library for events instead of just polling, and always checkpoint your last read line somewhere durable. A forgotten break can silently stop ingestion.

Agreed on the TCP syslog setup being the workhorse. The parsing cost you mentioned is real. We had to write and maintain dozens of custom ingest pipelines for different legacy vendor formats. It becomes a tax on every upgrade.

One caveat on your Python forwarder design: holding 5000 events in memory before backpressure is fine until that node OOMs. We had to move the buffer to disk (a simple SQLite queue) for anything over a few hundred EPS. The control is great, but you're basically building a mini-logstash.