Switched from OSS-Score to Braintrust, here's my migration pain

Another day, another migration from a functional open-source tool to a shiny "platform" promising to solve all my problems. The team decided to jump from our homegrown OSS-Score setup to Braintrust, lured by the siren song of integrated experiments, lineage, and that ever-so-enticing "single pane of glass." I was, predictably, the designated pessimist. Having now shepherded this migration to its painful, mostly functional conclusion, I feel compelled to document the reality. Not the marketing slides, but the actual grunt work, the hidden costs, and the moments where I genuinely missed my simple, ugly scripts.

Let's start with the data model shift. OSS-Score was essentially metrics and tags in a time-series database. Braintrust wants to own your entire AI lifecycle narrative. This meant a massive, tedious ETL process to not just move data, but to reshape it into Braintrust's schema of projects, experiments, and sessions. The "simple" migration script they provide is a fantasy for any non-trivial setup. We ended up writing a custom orchestrator that had to handle partial failures and idempotency, because of course their batch upload API has limits and occasional timeouts. Here's a taste of the "joy":

```python
# The promised land:
# braintrust.log(experiment="my_exp", inputs={...}, output="...", metrics={...})

# The reality during migration:
for batch in fragile_batcher(legacy_data):
try:
response = requests.post(
f"{BT_URL}/api/v1/events/insert",
json=batch,
headers={"Authorization": f"Bearer {API_KEY}"},
timeout=30
)
if response.status_code == 429:
time.sleep(2 ** retry_count) # Hello, exponential backoff my old friend.
# Also, hope you tracked your last successful batch ID.
except requests.exceptions.Timeout:
# Now you get to decide: replay the whole batch and risk dupes, or skip and lose data?
logger.warning("Timeout on batch. Adding to retry queue.")
```

Then there's the cost conversation, which was hand-waved away initially. OSS-Score ran on a couple of modest EC2 instances and some S3 storage. Braintrust's pricing, while seemingly straightforward per-event, quickly balloons when you realize every prediction, every intermediate step, every "session" is an event. Our test load projected a monthly cost 3x our existing infrastructure spend. We had to immediately implement client-side sampling and aggressive filtering of "low-value" logs before we even turned on the firehose, which of course defeats the purpose of comprehensive tracing. So now we have a partial pane of glass.

The worst part? The lock-in. With OSS-Score, if the tool annoyed me, I could fork it, patch it, or just write a new query. With Braintrust, my observability is now a SaaS endpoint. My dashboards, my experiment definitions, my team's workflow—all dependent on their service being up, their API not changing, and their pricing not becoming even more "enterprise." The failure mode has shifted from "I can fix this" to "I hope their status page is accurate and their support ticket gets answered."

It works. The UI is nice. The experiment comparison features are genuinely useful for the researchers. But was it worth the migration pain, the 300% cost increase, and the newfound vendor dependency? For our use case, barely. For yours? I'd strongly advise you to run the numbers on both cost and operational resilience *before* you commit. Map your actual data flow to their event model. Build a prototype that handles failures. Otherwise, you're just trading known, manageable problems for a new set of opaque, expensive ones.

-- cynical ops