A common challenge in structured CI/CD comparisons is the normalization of performance and cost metrics across disparate platforms. While each tool provides its own dashboard, aggregating data for a true apples-to-apples comparison—particularly for pipeline duration, resource consumption, and associated cloud costs—requires a systematic scraping approach. This guide outlines a method to programmatically extract key pipeline metrics from GitHub Actions, GitLab CI, and Jenkins, storing them in a unified format for analysis.
The core strategy involves leveraging each platform's API with a consistent data model. Below is a Python class structure designed to be extended for each provider, focusing on metrics that directly influence cost and efficiency: compute minutes, concurrency, and queue times.
```python
import pandas as pd
from abc import ABC, abstractmethod
from datetime import datetime
class PipelineMetricsScraper(ABC):
"""Abstract base class for CI/CD platform metric scrapers."""
def __init__(self, base_url, auth_token):
self.base_url = base_url
self.headers = {'Authorization': f'Bearer {auth_token}'}
self.metrics_data = []
@abstractmethod
def fetch_pipeline_runs(self, repository, branch='main', limit=100):
"""Fetch recent pipeline executions for a given repo/branch."""
pass
@abstractmethod
def parse_metrics(self, raw_data):
"""Parse raw API response into a standardized dictionary."""
pass
def get_standardized_metrics(self, repository):
"""Main method to retrieve and standardize metrics."""
raw_runs = self.fetch_pipeline_runs(repository)
for run in raw_runs:
parsed = self.parse_metrics(run)
# Add common cost-driver calculations
parsed['estimated_compute_minutes'] = parsed.get('duration_seconds', 0) / 60
parsed['cost_cent'] = self.estimate_cost(parsed)
self.metrics_data.append(parsed)
return pd.DataFrame(self.metrics_data)
def estimate_cost(self, metrics, price_per_minute=0.005):
"""Basic cost estimation. Override for platform-specific pricing tiers."""
return metrics['estimated_compute_minutes'] * price_per_minute
```
For each platform, you must implement the `fetch_pipeline_runs` and `parse_metrics` methods. The key is to map heterogeneous API responses to a common schema. Below is a critical excerpt for GitHub Actions, focusing on the workflow run API.
```python
import requests
class GitHubActionsScraper(PipelineMetricsScraper):
def fetch_pipeline_runs(self, repository, branch='main', limit=100):
# repository format: 'owner/repo'
url = f"{self.base_url}/repos/{repository}/actions/runs"
params = {'branch': branch, 'per_page': limit, 'status': 'completed'}
response = requests.get(url, headers=self.headers, params=params)
response.raise_for_status()
return response.json().get('workflow_runs', [])
def parse_metrics(self, raw_run):
return {
'platform': 'github_actions',
'run_id': raw_run['id'],
'status': raw_run['conclusion'],
'duration_seconds': raw_run['updated_at'] - raw_run['run_started_at'],
'queue_duration_seconds': (raw_run['run_started_at'] - raw_run['created_at']).total_seconds(),
'machine_type': raw_run.get('runner_name', 'unknown'), # Often includes label like 'ubuntu-latest'
'trigger': raw_run['event'],
'timestamp': raw_run['created_at']
}
```
* **Required Metrics to Scrape:**
* **Pipeline Duration:** Wall-clock time from start to completion. Directly correlates with compute costs.
* **Queue Time:** Time spent waiting for an available runner. Highlights agent pool sizing inefficiencies.
* **Runner Type/Label:** Identifies the underlying hardware (e.g., `macos-latest`, `self-hosted-large`). Essential for accurate unit cost assignment.
* **Status (Success/Failure):** To calculate efficiency rates and quantify wasted compute from failed runs.
* **Concurrency Level:** The number of parallel jobs in a run. Needed to extrapolate total compute minutes.
* **Data Storage & Analysis:**
* Store the aggregated `DataFrame` as a Parquet file or in a SQL database.
* For cost analysis, create a lookup table mapping `machine_type` to your actual cloud/provider cost per minute.
* Generate comparative dashboards focusing on:
* Average cost per successful pipeline.
* Compute time efficiency (duration vs. queue time).
* Failure rate impact on monthly compute spend.
The primary obstacle is normalizing the concept of a "runner" across platforms. Jenkins agents, GitLab shared runners, and GitHub-hosted runners have vastly different pricing models and specifications. Your scraping script must annotate each run with sufficient metadata to apply the correct cost coefficient later. Without this, any benchmark will be fundamentally flawed.
-cc
every dollar counts