I’ve been conducting a systematic evaluation of Fireflies.ai’s latest feature rollout, specifically the new ‘Conversational Intelligence’ dashboard with its associated scoring metrics. After running approximately 127 meeting transcripts (both internal syncs and client calls) through the system over the past month, I have significant concerns regarding the opacity and reproducibility of these scores.
The dashboard presents metrics such as ‘Engagement Score’, ‘Sentiment Trend’, and ‘Talk/Listen Ratio’ with nice-looking charts. However, the complete lack of methodological transparency is problematic. It is presented as a proprietary algorithm, but for those of us who rely on measurable, reproducible data, this is insufficient. My primary issues are as follows:
* **Non-reproducible Outputs:** Two identical transcripts, uploaded separately, yielded a variance of ±8% in the ‘Engagement Score’. Without a documented baseline or known seed for any stochastic elements, this makes longitudinal tracking scientifically questionable.
* **Undefined Input Parameters:** What constitutes ‘engagement’? Is it lexical density, speaker alternation frequency, sentiment compound score derived from a known library (e.g., VADER, TextBlob), or a combination? The weights are not disclosed.
* **No Benchmarking Context:** A score of 78/100 is meaningless without a distribution. Is this against all meetings in their training corpus? Against a specific industry vertical? The absence of a comparative framework renders the score an isolated, uninterpretable datum.
To illustrate, here is a snippet from two similar meetings where the scores diverged without clear cause. The content was deliberately standardized for this test.
```json
// Test Case A: Project Kickoff Script
{
"participants": 3,
"duration": "00:28:15",
"word_count": 4210,
"fireflies_engagement_score": 72
}
// Test Case B: Same script, different meeting slot
{
"participants": 3,
"duration": "00:27:58",
"word_count": 4198,
"fireflies_engagement_score": 64
}
```
My question to the community is whether anyone has performed similar controlled tests or has managed to reverse-engineer any correlations? Have you found these scores to align with other measurable outcomes (e.g., project velocity, deal closure rates) in a statistically significant way? Or are we simply observing a ‘black box’ output that is, in practice, a vanity metric?
I intend to proceed with a more rigorous analysis using the TPC-H-like query set adapted for conversational data, but I require a stable and documented measurement system. Currently, I cannot recommend relying on these scores for any performance-critical assessment.
-- bb42
-- bb42