Exactly. The "compiled binary" comparison is perfect. You're not deploying a model you can monitor, you're calling an API endpoint that's a wrapper around a live, changing system.
The real risk isn't just the shifting baseline. It's that their internal "engagement" composite can change its definition without notice. What they weighted as a "click" last quarter might now heavily factor "share" or "comment," metrics that don't align with your business goal. Your 85 could mean something entirely different after a silent retrain.
So your benchmark suite needs to track correlation with your actual KPIs, not just score stability. If the ranking stops predicting your outcomes, the tool is just generating expensive random numbers.
garbage in, garbage out
Great question, it's the classic black box API problem. I've poked around their endpoints too, and from the response schema, the `predictive_performance` field is a float, not a percentile. That hints it's likely a raw output from an ensemble model, maybe predicting a composite "success" probability.
You won't find the exact label in the docs, but based on their marketing, it's almost certainly a blend of engagement signals like CTR and conversion, weighted towards whatever their clients value most. The normalization baseline is the real mystery - it's probably against their internal training corpus, which shifts.
For variance, I've seen the same headline score bounce +/- 5 points on identical API calls a week apart. That's the retraining drift everyone's talking about. Treat the number as a ranking index, not an absolute metric.
Webhooks or bust.
You're approaching this correctly by looking for the traceable input, but that's precisely what they obscure. The "85" isn't a direct prediction of a single metric like CTR. It's the output of a composite model trained on a proprietary engagement signal, which is almost certainly a weighted blend of multiple actions (clicks, conversions, maybe even post-click dwell time) from their aggregated client data.
Think of it as a proprietary ranking index, not an absolute measure. The normalization baseline is their internal corpus, which drifts with each retrain, hence the variance. I've logged scores for the same seed copy over six months and seen swings of ±7 points without any model version announcement. Your monitoring analogy breaks because you can't inspect the query plan. You're observing the output of a compiled binary that gets silently updated. The only practical baseline is your own: maintain a fixed set of control text samples and track their scores over time to isolate vendor drift from your content's actual performance.
Trust but verify.