Hey folks! 👋 Wanted to share something our team tried in our last tool evaluation (we were looking at a new APM vendor) and get your thoughts.
We added a dedicated "Red Flags" column to our standard vendor scorecard, right next to the weighted scoring categories. The idea was to capture any deal-breakers or major concerns that a simple 1-5 score might not highlight. For example:
- **No clear deprecation policy** for older agent versions
- **Data egress fees** that weren't obvious in the initial quote
- **Key feature** (like tail-based sampling) was actually "on the roadmap" with no committed date
It turned out to be super useful during demos. Instead of just ticking a box for "feature present," we were actively listening for caveats and limitations. We logged each potential red flag with a brief note and severity.
Here's a simplified version of how we structured that part in our spreadsheet:
```markdown
| Red Flag Item | Severity (High/Medium/Low) | Notes | First Noted In |
|---------------|----------------------------|-------|----------------|
| Custom metrics cost extra per host | High | Adds ~20% to estimated bill | Demo Q&A |
| No OpenTelemetry collector support | Medium | Vendor-specific agent only | Security review |
| API rate limits too restrictive for our peak loads | High | Could drop data during incidents | Capacity planning doc |
```
Curious if anyone else has tried something similar? Did it help your evaluation committee make a more confident decision, or did it just add noise? I found it especially helpful when comparing 3-4 tools that all *seemed* similar on the surface—the red flags really differentiated them.
Dashboards or it didn't happen.