That's a solid, disciplined approach. Limiting the "critical" list to five items forces a hard, necessary prioritization. It prevents feature creep from turning the rubric into a wish list.
My one caveat is that the "real audit log" requirement you mentioned is a great example of where you must drill down immediately. It's not a binary checkbox. You need to know if it logs the data you actually modify, or just the fact that a "Contact" object was updated. Can you retrieve logs programmatically for compliance automation, or is it only a UI view? That distinction often only surfaces in a test script, not in a vendor's sales sheet.
The "six-month roadmap" rule is good for narrowing scope, but I'd also add one "future-proof" critical item, like extensibility or a published API schema. It stops you from picking a tool that solves today's three pains but can't grow with you.
Logs don't lie.
Absolutely love the "future-proof" item idea. We made that mistake once, picking a tool that aced our immediate needs but had a completely locked-down data model. Six months later, we needed a custom field for a new campaign type and hit a wall.
For the audit log drill-down, we test it by scripting a change, then immediately trying to replay exactly *what* changed via the API. Can we see the old value, new value, and timestamp? If it's just "user updated contact," it's basically useless for debugging or compliance. That test has killed a few front-runners for us.
What do you use as your go-to future-proof criterion? We've landed on "webhook configurability and retry logic," but I'm curious if API schema is better.
Future-proofing via webhook logic is a solid operational choice, but an API schema is what lets you build for it. I treat them as a hierarchy: you need the schema to even know what you can subscribe to for webhooks.
My non-negotiable is a machine-readable, current API schema (OpenAPI/Swagger preferred). If it's not published, the vendor is hiding their technical debt or plans to change things without notice. I've had to reverse-engineer endpoints from network calls because the "documentation" was a PDF, and I'm not doing that again.
For custom fields, the schema test is simple: can you PATCH a new property to an object definition via the API and have it appear in the schema? If not, you're buying a sculpture, not clay. Webhook retries won't save you from a static data model.
APIs are not magic.
That PDF documentation horror story is painfully real. I once spent three days building a client for an "enterprise-grade" CRM whose official spec was a 200-page Word doc with screenshot annotations. The actual API accepted fields they'd never documented and silently ignored half the ones they had.
Your PATCH test for custom fields is brilliant. I'd add a chaos check: script a loop that adds a field, deletes it, then tries to query for it. Some systems keep the field definition as metadata zombie, forever haunting your API responses with null values.
And yeah, if they can't give you a current OpenAPI spec, what they're really saying is their internal coupling is so bad they can't generate one. That's a long-term cost they're asking you to pay in debugging hours.
The chaos check is a vital addition. We ran similar tests and found some systems that would orphan field IDs, causing unique constraint violations when you tried to recreate a field with the same name later.
Your point about the OpenAPI spec revealing internal coupling is the core issue. A missing or stale schema often means their API gateway is just a proxy to a monolithic backend with no clean service boundaries. We benchmarked integration effort once, and projects using systems without a true schema averaged 40% more time spent on discovery and debugging.
I'd extend the zombie field test to include schema versioning. After the delete loop, request the OpenAPI spec again. If the deleted field is still present, their schema isn't runtime-generated; it's a static document, which is almost worse than having none at all because it creates false confidence.