Having recently completed a project to integrate GitHub Advanced Security (GHAS) alert data into our centralized Splunk instance, I've developed and refined a Python-based orchestration script that others in the community might find useful. The primary motivation was to move beyond the dashboard-centric view within GitHub and enable longitudinal analysis, custom correlation rules with other security telemetry, and more granular ownership metrics across our enterprise repository portfolio. While GitHub provides APIs for Code Scanning, Secret Scanning, and Dependabot alerts, constructing a robust, fault-tolerant pipeline that handles pagination, rate limiting, and schema evolution requires careful consideration.
The core script operates as an Airflow DAG, scheduled for daily execution, and performs the following sequence:
* Iterates through all organizations our service account has access to, fetching the list of repositories with GHAS enabled.
* For each repository, it makes discrete calls to the three alert endpoints (`/code-scanning/alerts`, `/secret-scanning/alerts`, `/dependabot/alerts`), respecting the `per_page` limits and `Link` headers for pagination.
* It performs a lightweight transformation on the JSON payloads to flatten nested structures where beneficial for our SIEM's search performance (e.g., extracting the CVE identifier and severity from the Dependabot `security_advisory` object).
* A state file is maintained to record the last successful run's timestamp, allowing subsequent runs to fetch only alerts `since` that time, which drastically reduces API consumption and processing time.
* Finally, alerts are batched and posted to the Splunk HTTP Event Collector (HEC) endpoint, with each alert type tagged appropriately.
Key design decisions and trade-offs encountered include:
* **API Cost Management:** Fetching all alerts for all repositories on every run is untenable for large organizations. The incremental `since` parameter is critical, but note that its support varies slightly across the three alert types; Dependabot's API uses `created_at` filters, while Code Scanning uses `ref` and tool GUIDs for true incremental diffing in some configurations.
* **Error Handling:** The script implements exponential backoff for rate limit (429) and server error (5xx) responses, with alert state checkpointing to avoid re-processing or data loss on transient failures.
* **Data Enrichment:** We opted to join repository metadata (e.g., team ownership, `CODEOWNERS` file data) within the SIEM via lookups post-ingestion, rather than in the extraction script, to keep the pipeline focused and modular.
* **Schema Drift:** GitHub occasionally adds new fields to the alert payloads. The script is designed to ingest the full JSON object per alert, storing it in a semi-structured field in Splunk, which provides future-proofing at the cost of some immediate query complexity.
The most significant pitfall we overcame was underestimating the volume of Dependabot alerts for repositories with large dependency graphs. This necessitated moving from a sequential repository-by-repository processing model to a parallelized approach using Airflow's dynamic task mapping for repositories, while carefully staying within the secondary rate limits for authenticated requests. The current implementation has been running in production for four months, processing an average of 18,000 alerts daily across 350 repositories, with a 99.8% successful delivery rate to the SIEM. I am happy to discuss specific implementation details, alternative approaches for real-time streaming versus batch, or how we structured our Splunk CIM-compliant data models for these alerts.
Data doesn't lie, but folks sometimes do.
This is a fantastic initiative, moving from the isolated GHAS dashboard to a unified SIEM view is such a critical step for mature security operations. The pain points you mentioned around pagination and rate limiting are real - our team ran into the same issues early on.
I'm curious about your approach to handling schema evolution, especially with the dependabot alerts API. We found its response structure changed twice in a six-month period, which broke our ingest until we added version-aware field mapping. Did you bake that kind of logic directly into your script, or are you relying on something external to catch those drifts?
Also, when iterating through all repos in an org, did you implement any logic to prioritize repositories based on commit activity or criticality? We ended up adding a weighting system because a full sequential scan for a large organization took longer than our ingestion window allowed.
Architect first, buy later
Interesting approach with separate calls per repo per alert type. Did you benchmark that against a batched GraphQL query? The sequential nature could become a bottleneck with many repos, even with careful rate limit handling. The GraphQL API for GHAS (still in preview) lets you fetch multiple alert types for multiple repos in a single request, which can dramatically reduce wall-clock time and API call count. The trade-off is complexity in constructing the query and parsing the nested response.
benchmark or bust
That's super helpful, thanks for sharing the high-level flow. I'm just starting to look at our GHAS data. Do you handle private repos differently? Our list includes a mix, and I'm worried about permissions tripping up the service account as it iterates.