We needed to enforce specific compliance checks before any code merge. Out-of-the-box solutions were too generic. Used Braintrust's API to build a custom gatekeeper.
Key components:
* A listener service (deployed on Fly.io) watches for Braintrust project events.
* On `task.created`, it fetches the linked code commit from our GitLab.
* It runs a custom script to validate:
* JIRA ticket format in branch name
* Presence of specific file headers
* Absence of banned dependencies
* Posts results back to Braintrust as an `annotation`, auto-approving or flagging the task.
The workflow is now mandatory. It catches about 30% of compliance slips before human review. Main benefit: it integrates directly into our existing Braintrust review queue, no new tooling for teams to learn.
Biggest hurdle was handling API rate limits during bulk imports. Solved with a simple queue system. Their webhook payloads are straightforward, which made the integration fast.
Ship fast, review slower
Integrating directly into your existing review queue is the key efficiency gain. You avoid the tool sprawl that kills compliance adoption.
On the rate limits, that's a common vendor API pitfall. Many teams miss that during initial testing. A queue is the right fix. Did you consider implementing exponential backoff, or was a simple FIFO sufficient for your event volume?
Catching 30% automatically is a solid return. I'd be curious about the false positive rate on those checks, especially the file header validation. Does it ever flag acceptable variations, or is the rule set strict enough to avoid reviewer annoyance?
independent eye