Hello everyone. I've been tracking a similar performance dip across two of my client engagements this past week, both coinciding with the latest platform agent update. Seeing a 20-point drop in your composite score isn't just a blip; it's a significant operational signal that demands a structured diagnostic approach. In procurement and vendor management, we treat these shifts as critical incidents requiring a root-cause analysis before any renegotiation or escalation discussions.
Based on my standard vendor performance evaluation framework, I'd recommend immediately auditing the following four pillars. Your issue likely resides in one or more of these areas:
* **Scoring Metric Re-calibration:** The most common post-update issue. Have the underlying weights or definitions of the scored behaviors changed? A "quality" score of 80 last month might be measuring something materially different this month. You need to request the updated scoring rubric or key performance indicator (KPI) definitions from your account team or support channel.
* **Agent Configuration Drift:** Updates can sometimes reset or alter agent profiles, knowledge base linkages, and operational parameters. Verify that all custom instructions, guardrails, and context settings survived the update intact. A single altered parameter, like response length or temperature, can dramatically alter output scoring.
* **Integration & API Impacts:** If your score is derived from outputs fed into another system (e.g., a CRM, a quality assurance platform), confirm the update didn't change the output schema or data structure. A new line break or JSON formatting change can break downstream scoring logic.
* **Baseline & Benchmark Shifts:** It's possible the "score" is now being measured against a new, broader dataset or a higher benchmark standard. Essentially, the goalposts may have moved. You need to clarify if this is an absolute drop in performance or a relative drop due to a changed benchmark.
My immediate playbook would be to open a formal ticket with Braintrust support, but not with a generic "our score dropped" message. Frame it using the pillars above. For example: "Per our service review, following Agent Update vX.X, we observed a 20-point decline in our composite performance score. To diagnose, we request: 1) The updated scoring methodology document, 2) Confirmation of agent configuration integrity for Profile [Your Profile Name], and 3) Disclosure of any changes to the performance benchmark cohort or calculation engine."
This structured approach moves the conversation from "something's wrong" to a collaborative investigation on specific, contractual performance deliverables. Please let us know what you find; documenting these post-update paths is invaluable for the community's collective negotiation and management playbooks.
null
That's a really smart way to break it down, especially checking for scoring metric changes after an update. I wouldn't have thought to ask for the updated KPI definitions myself.
I'm seeing something similar with agent configuration drift on my test portal. The update seemed to unlink a few key response macros from our standard ticket categories, which tanked our first response time score. Maybe check those automation links first?
I get the procurement framework approach, but I'm skeptical of asking for a "rubric" as step one. By the time you get that document from the account team, they'll have retroactively justified the new scoring.
You need an independent data point. Look at your raw event logs for a specific agent interaction from before and after the update. Compare the actual user actions and timestamps against the score you received. The delta tells you what changed, not their marketing material.
If their scoring is a black box, your root cause analysis is already compromised.
If it's not a retention curve, I don't care.
Yeah, that structured approach makes sense for a big vendor contract, but it feels a bit heavy for the average self-hosted setup. I'm with user55 on being skeptical about asking for the rubric first.
If this is a self-hosted agent, the scoring logic should be in your own config or code. Could the update have pulled in a new default scoring config file that overwrote your custom weights? That's the kind of drift I'd check before anything else.
Self-host or die trying.