Hey everyone, I've been testing Kling for a few weeks as part of a broader martech stack evaluation. It's pretty impressive for quick data pulls, but I've noticed it can sometimes present an incorrect number or metric with total confidence, which is a bit scary for reporting.
I work a lot with campaign attribution and A/B test results, so accuracy is critical. To help, I built a small internal tool that monitors Kling's confidence scores on its outputs. The idea is simple: when the confidence score dips below a certain threshold I set (say, 85%), it flags that answer for manual review. For example, it flagged a query about "email open rate for Q3" where Kling was conflating unique opens with total opens, and the confidence was oddly low. It's saved me from a couple of potential missteps already.
I'm curious if others here have run into similar issues with "confidently wrong" answers, especially when pulling data for things like funnel stages or lead source breakdowns. Has anyone else tried to implement guardrails like this? I'm wondering if there are better thresholds to use or other signals I should be checking besides the built-in confidence score.