Everyone obsesses over pipeline runtime, but the real time-sink is often the hidden queue. You're paying for an idle runner while your team waits. Let's measure what you're actually buying.
First, define your metrics clearly:
* **Startup Time:** From the moment a commit triggers the pipeline to the moment the first line of your `script:` job executes. This includes YAML parsing, environment provisioning, and container pull.
* **Queue Delay:** The time your pipeline sits in a pending state, waiting for an available runner. This is where vendor pricing models and your own concurrency limits bite you.
You need to instrument your pipeline to capture this. Don't rely on the platform's dashboard alone—log it yourself. Here's a practical approach using a pre-job script:
```bash
# Capture the initial pipeline creation timestamp (often available as env var)
PIPELINE_START=$CI_PIPELINE_CREATED_AT
# Capture the job start time immediately in `before_script`
JOB_START=$(date -u +%s%3N)
# Later, in `after_script` or a final step:
CURRENT_TIME=$(date -u +%s%3N)
PROVISIONING_TIME=$((JOB_START - PIPELINE_START))
QUEUE_TIME=$((JOB_START - PIPELINE_START - PROVISIONING_TIME)) # Adjust based on your logic
echo "Provisioning/Startup latency: ${PROVISIONING_TIME}ms"
echo "Queue delay: ${QUEUE_TIME}ms"
```
Run this measurement across different times of day, days of the week, and for different pipeline types (e.g., main branch vs. PR). Aggregate the data. Now you can benchmark.
What to look for:
* **Baseline vs. Peak:** Is your queue delay consistently under 30 seconds, or does it spike to 8+ minutes during business hours? This points to an undersized runner pool or a restrictive concurrency tier.
* **Vendor Comparison:** Use this data to compare. A platform charging 2x per minute might have 60% faster startup times, making it cheaper for short jobs. Another with "unlimited" pipelines might have massive queue delays during peak usage.
Without these numbers, you're negotiating a SaaS contract blind. You're buying promises, not performance.
Read the fine print.
VendorNegotiator
Oh, that's a super helpful distinction. I always just looked at the total runtime in the dashboard and got frustrated. The idea that the actual "startup" is different from "queue" time makes total sense.
But, this might be a dumb question... How do you actually get that initial PIPELINE_START timestamp reliably? In your example you used `CI_PIPELINE_CREATED_AT`. Is that a standard variable in most CI systems, or is it something you have to hunt for? I'm on a couple different platforms and they all seem to have their own weird env vars.
If that part is off, won't the whole calculation get messed up?