TIL: You can use webhooks to trigger CI/CD security gates

While implementing a new security scanning pipeline for our PostgreSQL-backed microservices, I discovered that Braintrust's webhook system is far more extensible than their primary UI suggests. The documentation primarily focuses on webhooks for notifications, but with strategic configuration, they can serve as a powerful trigger mechanism for automated security and compliance gates within a CI/CD pipeline. This moves security left from a manual, post-hoc review to an integrated, automated checkpoint.

The core concept leverages the `check_run` webhook event. When a candidate completes a test, Braintrust can send a detailed JSON payload to a secured endpoint you control. This endpoint can then parse the results, apply policy-based logic, and pass/fail the associated CI job. Below is a simplified architecture and a configuration example.

**Architecture Flow:**
1. CI pipeline (e.g., GitLab CI, GitHub Actions) starts a job for a feature branch.
2. Job runs the Braintrust test suite via CLI, storing the log with a unique `project` and `tags`.
3. Braintrust dispatches a `check_run` webhook to your internal webhook router (e.g., a small Flask/Node.js service).
4. The router service queries the Braintrust API for the full experiment details using the IDs in the webhook payload.
5. Business logic evaluates the metrics (e.g., `scores.accuracy > 0.95`, `scores.latency_p90 = 0.95,
'latency_gate': scores['inference_latency_ms_p90'] = 0.94
}

all_passed = all(gates.values())

# Update CI system status via its API
ci_commit_sha = extract_sha_from_tags(payload['data']['tags'])
update_ci_status(
sha=ci_commit_sha,
state='success' if all_passed else 'failure',
description=f"Security/Perf Gates: {gates}"
)

# Optionally, log the decision to a monitoring table in your analytics DB
log_gate_result_to_postgres(payload['data']['id'], gates, all_passed)
```

**Key Considerations & Pitfalls:**
* **Webhook Security:** You must validate the webhook signature using Braintrust's secret to prevent spoofing.
* **Idempotency:** Webhooks can be retried; your handler must be idempotent to avoid duplicate CI status updates.
* **Data Completeness:** The initial webhook payload is a summary. For complex gates requiring full trace data, your handler must perform a subsequent API call to `braintrust.get_experiment`. This adds latency and requires managing API credentials securely.
* **Timeout Management:** Your webhook endpoint must respond quickly to Braintrust's request. Long-running evaluations should be delegated to an async job queue, with the CI status updated later.
* **State Management:** This pattern works best when the CI pipeline pauses at a manual/approval job waiting for the webhook-driven status. Alternatively, use a polling approach in CI if webhooks are unreliable in your environment.

This approach effectively transforms Braintrust from a siloed experiment tracker into a integrated quality control system. The primary trade-off is the operational overhead of maintaining a secure, available webhook handling service versus the benefit of automated, metric-driven deployment gates. For teams already managing microservices, this overhead is minimal compared to the risk reduction.