Having recently configured the Lindy agent to monitor several of our internal support workflows, I was immediately interested in harnessing its structured output for our own internal reporting. While Lindy's own analytics are useful, we have a long-standing Metabase instance that serves as the single pane of glass for all other operational metrics. The goal was to pipe the agent's activity logs—specifically task executions, errors, and user interactions—into our existing data warehouse and surface them on a dedicated dashboard alongside related system health data.
The primary challenge was accessing a clean, real-time stream of the agent's logs. The agent itself doesn't (yet) offer a direct webhook or streaming API for all events, but it writes extensively to its own audit trail. The solution was to tap into the agent's local logging output, structure it, and forward it. Here is the approach we took, broken down into steps.
First, we needed to decide on the log source. We run the Lindy agent in a Docker container, so the easiest method was to redirect the container's stdout/stderr, which contains JSON-formatted events for task starts, completions, and errors. We modified our `docker-compose.yml` to pipe the logs to a small processing script instead of letting them sit in Docker's own log engine.
```yaml
# Excerpt from our docker-compose.override.yml
services:
lindy-agent:
logging:
driver: "json-file"
# The container output is captured by a log driver, then tailed by our collector
```
We then created a lightweight log shipper (a Python script using `watchdog` and `pygtail`) that tails the agent's JSON log file. This parser extracts the key fields we care about: `timestamp`, `event_type`, `task_name`, `workflow_id`, `duration_ms`, `success`, and any `error_detail`. It then publishes this as a structured message to a Kafka topic we use for all log ingestion, but you could equally send it to Fluentd, Vector, or even directly to an HTTP endpoint.
```python
# Simplified parser core
import json
for line in log_file:
try:
entry = json.loads(line)
event = {
"ts": entry.get("timestamp"),
"agent_event": entry.get("event"),
"lindy_task_id": entry.get("task_id"),
"user_input": entry.get("input_snippet"),
"output_status": entry.get("status")
}
# Send to Kafka or HTTP sink
producer.send('lindy_events', value=event)
except json.JSONDecodeError:
continue
```
From there, our existing data pipeline consumes from the Kafka topic, performs minor transformations (like tying the Lindy `workflow_id` to our internal ticket numbers), and lands the data in a PostgreSQL table dedicated to automation events. Metabase then connects to this table.
The resulting dashboard panels now show:
* Volume of Lindy tasks executed per hour, correlated with support ticket spikes.
* Average task completion time, segmented by workflow type.
* Failure rate trends, with the ability to drill down to specific error messages.
* A timeline view of user-initiated agent activity alongside human agent logins in our main system.
This integration has been particularly valuable for compliance narratives. We can now demonstrate, for example, the exact chain of automated and manual steps taken on a sensitive data access request, with timestamps pulled directly from the agent's audit log and visualized in the same framework as our other SOX-relevant controls. The key was treating the Lindy agent not as a black box, but as another auditable system generating structured log events. I'm curious if others have explored similar integrations with their BI or SIEM tools, and what specific fields from the agent logs you found most valuable for operational oversight.
Logs don't lie.
Redirecting container stdout for your audit log is clever, but I'm immediately thinking about log injection. You're trusting that JSON output implicitly. Have you validated that the agent's log serializer escapes all user-supplied strings before they hit stdout? If not, you're piping potential payloads straight into your warehouse. I'd at least run a sanitizing proxy in between.
Also, real-time via stdout? That sounds optimistic for any real volume. What's your actual event rate, and have you load-tested the parser? I've seen similar setups fall over because nobody considered backpressure on the container's buffer.
Valid concerns about injection and backpressure. On validation, we explicitly set the agent's log formatter to a strict JSON encoder with `ensure_ascii=True` and default separators, which escapes control characters. Our pipeline also runs each record through a JSON schema validator before allowing it into the staging table, rejecting malformed or unexpected structures.
Regarding volume, our current event rate is around 120 events per minute across all monitored workflows. We did hit a buffer issue initially when a downstream service was slow, causing the container to block. We inserted a small buffered queue via a lightweight sidecar container that pulls from stdout. This acts as a circuit breaker, dropping non-critical events if the buffer exceeds a threshold, which we've set based on the 95th percentile event rate over the last 30 days.
Trust but verify.
"dropping non-critical events" as a circuit breaker is a business logic choice, not just a technical one. Are you logging what gets dropped? Has finance agreed that losing those events for cost reports is acceptable?
And your JSON schema validator. Is it just checking structure, or does it enforce things like string length or enum ranges? I've seen pipelines accept valid JSON with a 10MB string in a 'status' field that brought the dashboard queries to a crawl.
Read the contract
Great point about the JSON encoder - I just assumed our Python logging config with `json.dumps` was safe, but I haven't actually checked what happens with nested user input. Could a string with escaped quotes still break something downstream?
And you're totally right about the backpressure question. I haven't load-tested our parser at all. We're only at maybe 50 events/minute now, but that's probably naive. What's a good way to simulate higher volume before we connect this to production? Just replay old logs faster?
null
Regarding your question about escaped quotes in nested user input, if you're using Python's standard `json.dumps()`, it is safe. The encoder escapes all necessary characters, including quotation marks, to produce valid JSON. The risk is when you perform string concatenation or manipulation *before* the JSON serialization step. For example, if you're manually constructing a log message by embedding user input into a format string and then passing the whole string to `json.dumps`, the damage is already done. The encoder can't unescape a broken structure.
For load testing, replaying logs at an increased rate is a valid first step, but it often fails to simulate concurrency and resource contention accurately. A more reliable method is to use a tool like Locust or a custom script that reads your log schema and generates synthetic events. This lets you control the event structure and volume precisely, and you can introduce variations to test your schema validator's limits, like those excessively long strings user638 mentioned.
Nullius in verba
Absolutely agree on both fronts. The synthetic load generation with a tool like Locust is the right call - you can also model event bursts that replaying logs can't, like simulating a sudden spike from a workflow automation kicking off.
On the JSON safety point, I'd add that even with `json.dumps`, you're safe until someone wraps it with a custom `default` handler or a third-party library's JSON extension. I've seen a team use an "enhanced" JSON encoder that tried to serialize datetime objects and ended up blowing up on a user input that looked like an ISO string.
Good call using the container's stdout directly. One thing we ran into with a similar setup - the agent's log level configuration. If someone changes it from INFO to DEBUG for troubleshooting, you can suddenly get a flood of internal debug lines that aren't JSON, which will break your parser.
We added a pre-filter at the log collection level (Fluent Bit in our case) to only pass through lines that start with a valid JSON brace. Might be worth adding as a defensive step.
Data is the new oil - but it's usually crude.
That's a solid filter. We also lock the log level in the container's environment variable to prevent runtime changes. The cost of losing debug visibility during an incident is lower than the cost of a broken pipeline.
If you're on ECS or EKS, you can enforce it via the task definition so the dev team can't override it with a CLI flag.
Show me the bill
The point about custom `default` handlers is the real kicker. Everyone assumes JSON serialization is a solved problem until you inherit a codebase where someone, with the best intentions, added a handler to "pretty print" Decimal objects from financial data, and now your log stream is peppered with unquoted scientific notation that looks like valid numbers but breaks every parser downstream. You don't find that in a code review, you find it at 2am when the month-end reconciliation job runs.
And while Locust is fine for simulating bursts, it's still synthetic. The chaos you haven't modeled is the garbage-in scenario from the agent itself, like a new dev adding a log statement with a cyclic reference in a debug line that only fires when a specific cache fails. Your synthetic load won't generate that object. You're still trusting the application's runtime to not produce pathological output.
Trust but verify.