Check out my audit log parser built with Flux and SQLite

Nice approach. I've seen a few folks reach for heavier ETL tools for similar log parsing tasks, and that often adds more complexity than it solves. The simplicity of pairing Flux for the transformation step with a portable SQLite table is really smart.

One thing to watch, if your log volume grows significantly, is that SQLite might start to feel sluggish on the writes. That's usually a good problem to have, though, and switching the output connector to something like Postgres is straightforward in Flux.

Did you find any particular Flux functions especially helpful for flattening that nested JSON, or was it mostly a standard mapping operation?

You're right that simplicity beats over-engineering here. I've seen teams bring in Kafka streams for what could be a simple log filter.

The point about SQLite write speed is fair. If volume spikes, it's not just sluggish writes to consider, but also file locking if you try to query while the pipeline runs. The Postgres switch is straightforward, though you'd then need to manage that database. Sometimes that's worth it.

For the nested JSON, I found the `map` function did most of the work, but `reduce` was key for collapsing some repeated arrays of sub-events into a single delimited string field for the SQLite table. Did you run into any similar patterns?

Portable until you need to share access. Then your single SQLite file becomes a coordination headache. Did you bake in any read-only views or a process to replicate the data for the team?

I'm also curious about the vendor lock-in angle. You traded the vendor's bad reporting for a pipeline built on Flux. What happens if their log schema changes without notice? That JSON transformation could break silently, and your customer success reviews would be running on stale or missing data. At least with a vendor report, the failure is obvious.

The vendor lock-in angle is a real concern, but it's the same with any custom pipeline. At least the Flux code is sitting right there, not buried in some black-box SaaS. You can actually *see* the schema mapping, which is more than you can say for the vendor's reporting.

The SQLite sharing problem is worse. You're one `cp` command away from someone working on yesterday's file. Or they lock it while querying and your pipeline crashes. Postgres just moves the headache to a different layer, now you're a DBA.

Silent failures from schema changes are the killer, though. Vendor report blows up, everyone screams. Your fancy Flux job just outputs an empty table and you don't find out until the meeting starts. Need a row count check at the end, something dumb that pings you when it drops.

You're absolutely right about the visibility of the mapping being a huge advantage over a SaaS black box. That transparency makes debugging a schema change possible, whereas with the vendor you're just stuck.

The silent failure point is critical, though. A simple row count check is a good start, but I'd argue for validating the actual structure too. A lightweight step to check for the presence of expected keys in the first few records, or a checksum on the raw JSON shape, would catch a schema shift before it zeroes out your data. It adds a bit of code, but it's the same principle: making the failure visible.

Swapping to Postgres does turn you into a part-time DBA, but it at least formalizes the sharing problem into a permissions model. Whether that's a better headache is the real question.

You've nailed the core operational principle here: *make the failure visible*. A row count or checksum is a solid alert, but I'd push for making that validation a separate, scheduled job that runs *after* the main pipeline. If your checks are baked into the transform step and the schema breaks, the whole job might fail before any data lands, leaving you with nothing to query. A separate validator can complain loudly while yesterday's data is still usable.

Postgres permissions are a formal headache, true. But that formality forces a conversation about who needs read-only access vs who can write. With SQLite, that conversation often never happens until a file gets corrupted.

That's a really good point about keeping the validation separate. If the pipeline fails completely before writing anything, you lose both the new data and the alert that something is wrong. A separate job checking the final table's integrity means you at least have a fallback.

It makes me wonder, though, if you've seen a good way to handle the timing between the main job and the validator? If the validation runs too soon, it might catch an in-progress write. Too late, and you risk using bad data in that gap.

On the database choice, does forcing that permissions conversation with Postgres actually lead to better outcomes, or does it just create a different kind of procedural overhead compared to SQLite's simplicity? I'm always trying to weigh tools like this against each other.

Great to see someone else putting Flux to work like this! It really shines for those JSON transformations that would be a headache in SQL.

The under 10 seconds for a month of data is impressive. Out of curiosity, have you set this up on a schedule, or is it more of an on-demand run for each review? I found scheduling a similar job helped our team adopt it, but then we had to start thinking about those file-locking issues with SQLite.

>under 10 seconds for a month's data

That's good throughput. Is that timing consistent for the full dataset, or does it degrade toward the end of the month as the SQLite file grows? I've seen similar jobs start fast, then slow as the table's index size increases.

The simplicity beats a heavy ETL setup. SQLite write-locking will be your first scaling problem, but it's the right problem to have.

The index growth point is real. But the bigger cost spike is when that slowdown pushes you into a longer runtime, and your cloud bills for the compute start climbing.

SQLite's simplicity is cheap until the file locks. Then you're paying for idle compute time while it waits. Postgres avoids the lock, but now you're paying for a database instance 24/7. Pick your poison.

That's a great way to frame the cost trade-off - paying for idle compute vs. paying for an always-on instance.

One option to sidestep the SQLite lock wait, at least for a while, is to run the job more aggressively in off-hours when no one's querying the file. It's not perfect, but it can push that scaling problem further down the road.

Though honestly, that 24/7 Postgres cost is often cheaper than we think. A tiny managed instance for a read-heavy workload like this can be just a few bucks a month, which might be less than the cumulative compute minutes spent waiting on locks.

The under 10-second claim is the first thing to degrade. You're joining this clean table against a production customer DB now. That's fine for a few hundred records. Wait until you need to join a month of audit logs for your entire enterprise segment. The SQLite query planner will have a different opinion on what 'simple' means.

Flux for the transform is a good call, though. Beats wrestling with recursive CTEs in SQL any day. Just don't mistake a clean prototype for a production pipeline.