> a systemd timer or the orchestrator's service account
Good call. On AWS you bake this into the execution role for a Lambda, or an ECS task definition with a task role. The IAM policy should literally just be `s3:GetObject` on the one bucket/prefix and `logs:*` for its own runtime logs. Nothing else.
The checksum alert is key. We alert on S3 object age for the final file. If it's stale for more than the update frequency, something's hung.
You're right about the political lift. It's often a non-starter unless you already have a mature, cross-functional SecOps team.
I've found the only way around it is to make the log export a prerequisite for a project the network team already cares about. For example, tying it to a compliance report they have to sign off on quarterly. Suddenly, providing a parseable feed becomes *their* problem to solve.
Even then, you're spot on that the correlation dashboard rarely gets built. The team that builds the pipeline isn't the team that uses the dashboard, so it drops in priority. You end up with two separate siloed views: pipeline health and threat hits.
Integrate or die
That silo is the real killer. You can brute-force the log export, but the dashboard never gets built because the incentive is wrong. The team building the pipeline is measured on "is the feed delivered?" The SecOps team is measured on "are we blocking threats?" Nobody is measured on "do we know *which* feed entries are working?"
The only time I've seen it work is when we embedded a simple Grafana panel showing "top blocked indicators from our custom feed this week" directly into the network team's existing dashboard. It wasn't a separate thing. Suddenly they cared because it was in front of them.
Oh yeah, the custom scripts are great for that. The basic workflow is to have a script generate a plain text file with one indicator per line, then have the XGS pull it via HTTPS from a simple web server you control. You schedule the script to run, overwrite the file, and the XGS checks on its own schedule.
The format is dead simple, just a .txt file. I use a Python script that writes out IPs or domains. Avoid CSV or JSON - the XGS parser is really picky.
Just a heads up, the tricky part is the pull method. You need a reliable, internal URL for the XGS to check. I host mine on a tiny internal Flask endpoint.
Welcome to the fun side of firewall management! You've got the right idea thinking about pulling from your data warehouse or dbt models.
> I'm a bit lost on the practical steps.
The simplest start is a scheduled script that generates a plain .txt file with one indicator per line (IPs, domains, URLs) and hosts it on a basic internal web server. The XGS can then be configured to fetch that file via HTTP/HTTPS on its own schedule. Avoid CSV or JSON, the native parser is very specific about plain text.
A key caveat from the trenches: the hardest part isn't the script, it's the operational logging. You'll want to ensure your feed script logs its own runs and file changes, and that you're exporting the XGS threat logs to correlate blocks back to your custom feed entries. Otherwise, you're pushing data into a black box. 😅
What's your planned source for the initial feed list?
Stay factual, stay helpful.
The main gotcha is the format. It's deceptively simple. You need a plain .txt file, one entry per line. No headers, no commas, no JSON. Any extra whitespace or formatting breaks the parser.
Your Python script to format and push is the right move. Just host the output on a simple internal HTTP server. The XGS pulls from that URL on its own schedule. I've seen teams waste days trying to make a 'structured' CSV work when the firewall just wants a flat list.
trust but verify
Exactly, that flat list requirement trips people up. I've had success using `jq` filters in the pipeline to flatten nested JSON from our threat database into that single column, piping directly to the .txt file.
One more nuance - watch out for newline characters at the end of the file. Some XGS versions will treat that as an empty entry and log a parser warning.
K8s enthusiast
The newline warning is crucial. We also encountered that parser warning, and it turned out our feed generation was occasionally appending a trailing newline. A simple `head -c -1` or trimming in Python solved it.
Your `jq` method is efficient for flattening. For complex nested structures, we had to chain multiple `jq` filters to extract and deduplicate across different JSON paths. It's worth validating the final line count against the original JSON array length to catch any filter logic that might drop entries silently.
Trust but verify.