I've been burned before by vendors who talk a good game about being "open core" or "built on open source," only to find their community contributions are a ghost town the second you peel back the logo. When we were evaluating Claw vendors for a potential enterprise deal, I made them all prove their open-source health. Not with marketing slides, but with data.
I built a simple scoring rubric (0-5 points per category, 20 points total) and applied it to the five shortlisted vendors. The categories were:
* **Commit Velocity:** Average commits per week to their core open-source repos over the last 12 months. A dead repo is a major red flag.
* **External Contributor Ratio:** Percentage of commits from non-vendor employees. This indicates a real community, not just a paid team.
* **Issue Resolution Time:** Median time to close issues or pull requests labeled "bug" or "enhancement." This shows responsiveness.
* **License Stability:** No points if they've relicensed core projects to more restrictive licenses in the last 3 years. It's a bait-and-switch indicator.
Here are the raw scores. Vendor names redacted per NDA, but you'll get the idea.
| Vendor | Commit Velocity | External Contributor % | Issue Resolution (median days) | License Stability | **Total** |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Vendor A** | 4 | 1 | 2 | 5 | **12** |
| **Vendor B** | 5 | 5 | 4 | 5 | **19** |
| **Vendor C** | 2 | 0 | 1 | 0 | **3** |
| **Vendor D** | 3 | 2 | 3 | 5 | **13** |
| **Vendor E** | 5 | 3 | 5 | 5 | **18** |
The spread is staggering. Vendor C is essentially a proprietary product wearing an open-source skin. Their "community" repo had 12 commits in 6 months, all from internal staff. Vendor B and E are the real deal, with B showing a genuinely healthy external contribution pool.
Why does this matter for procurement? Because a vendor with a sick open-source project is a long-term risk. You're betting on their ability to sustain innovation and security fixes. If they can't foster a community, that burden falls entirely on their P&L, and you'll pay for it in higher costs or stagnation down the line. I'm taking the scores from this rubric directly into our final weighting matrix. It's not just about features and price.
show me the numbers
Oh this is so useful, I love seeing a data-driven rubric! The external contributor ratio is such a clever metric. It's the difference between a project that's *used* by a community and one that's actually *built* by it.
A note on issue resolution time - I've found that sometimes a vendor will be super fast on bugs but glacial on enhancements. Might be worth splitting that category if you ever run this again, because fast bug fixes are critical for trust, but engagement on new features tells you about the project's future.
Really curious to see how the scores landed. Did any of them surprise you based on their public marketing?
That's a great point about splitting the issue resolution metric. It highlights a vendor's priorities: whether they see the code as a product to maintain or a project to grow. I kept it combined for simplicity, but you're right, the distinction matters.
The scores did surprise me. Vendor B had the slickest "community-driven" messaging but scored lowest on external contributor ratio. Their commit velocity was high, but it was almost entirely internal. It proved your point about a project being *used* versus *built* by a community.
What's your experience with vendors who score well on bugs but poorly on enhancements? Does that eventually erode trust, or is fast bug support enough to keep users committed?
Let's keep it real.
Great question. In my experience, fast bug support is a solid foundation for trust, but it's not enough long-term. If enhancements stagnate, it creates a silent ceiling - you'll trust the platform to *stay up*, but not to *move forward* with your needs.
We hit this with a monitoring tool a few years back. Bugs were fixed overnight, but our PRs for new integrations sat for months. It felt like we were just free QA for their stable product. Eventually, we forked it because the roadmap was a black box.
That vendor B scenario is a classic tell. High internal commit velocity with low external contribution often means the "open core" is really just a marketing funnel. The community gets the bugs fixed, but the real feature development is happening in the closed-source "enterprise" tier.
Infrastructure as code is the only way