We rolled out Freeplay to our team of 60+ analysts. The promise was solid: a central spot for all our LLM testing. Reality was messier.
Big win was finally getting a handle on prompt versions and costs. Before, it was a spreadsheet nightmare. Now we can actually see what's worth it.
The pitfall? The "enterprise" setup. The sales talk was all about seamless integration. The actual config was a time sink. Our security team had a field day with the network requirements. Also, watch the user seats. We got hit with a surprise charge for "inactive" users we thought were de-provisioned.
Is the granular cost tracking worth the setup headache? For us, barely. Might be overkill for smaller teams.
The user seat trap is real. Had a similar shock with another tool last quarter. Did you push back on the billing for the inactive users? Sometimes they'll credit you if it's a first-time config issue.
How long did the security review actually take? We're looking at Freeplay now and that's my biggest worry.
Still learning.
The user seat trap is one of those classic SaaS gotchas. We had something similar with a different vendor - our finance team now makes us run monthly seat audits before billing cycles close.
On the security review, ours took about three weeks. The network requirements weren't too wild, but the SSO setup had a few quirks. Did you run into the same with SAML mapping? We ended up writing a small script to sync groups.
The cost tracking is a lifesaver though, especially when you're comparing GPT-4 to cheaper models. That alone probably saves us more than the license cost.
Dashboards or it didn't happen.