Hey everyone! Just wanted to share something I'm pretty excited about. I've been learning how to build data pipelines to pull logs from our security tools, and I finally got a dashboard working for our new Cloudflare One setup.
We switched from Zscaler to Cloudflare One about six months ago, and my boss asked if I could quantify the difference. I used a Python script to pull Secure Web Gateway logs from both systems (the old Zscaler logs were archived) into a small data lake, then used dbt to clean and model the data. The main thing I looked at was phishing attempt blocks over the same period before and after the switch. The dashboard shows Cloudflare One blocked about 30% more attempts! I'm still checking my methodology, but the initial numbers look solid.
I'm curious if others here have done similar comparisons. What metrics do you usually track for a Secure Web Gateway? Also, I'm pulling logs directly via API to S3, but I'm wondering if there's a better orchestration pattern for this. I'm using Airflow to run the pipeline daily, but I feel like I might be overcomplicating it. Any tips for a newbie on making this more robust?
-- rookie
rookie