Unpopular opinion: The documentation is comprehensive but poorly organized.

>the number of dead-end searches

I've been wondering about that metric. Does a single search count if the user has to refine it three times before finding the right page? Or is that three dead-end searches?

The confidence score makes sense though. A frustrated person won't want to use the tool again. But how do you score a vendor when their numbers are okay, but that "gut check" story is really bad? Do you just veto them?

Measuring dead-end searches is a classic case of a metric that's easy to gather but tells you nothing useful. You can refine a search three times because you're bad at keywords, or because their taxonomy is completely disconnected from user intent. The vendor will always claim the former.

A gut-check veto is the only real tool you've got. If your team dreads opening their docs, that operational drag translates directly to slower deployments and more billable hours lost to "figuring it out." No tidy SaaS metrics dashboard will ever capture that.

You're right about the metric being misleading, but there's an internal measure I've found more telling: search-to-ticket conversion. When someone executes a search from the docs portal and immediately opens a support ticket from the same session, it strongly indicates the information architecture failed, regardless of search refinements.

While a gut-check veto is powerful, it's subjective and hard to use in procurement. I've started including a practical test in evaluations: we take a core workflow, like deploying a service with a specific security policy, and time how long it takes to build a working Terraform module using *only* the public documentation. The elapsed time and number of distinct sources required become a concrete, albeit anecdotal, data point.

Exactly. Their search function is garbage because it's indexing a bunch of disconnected, redundant pages. I ran a quick test: searched their portal for "agent install status codes." Got seven results.
* Four were basically the same overview paragraph in different "Getting Started" guides.
* Two were dead links to old PDFs.
* One was the actual API error code list, buried on page 17 of a 40-page REST guide.

The information is technically there, but you have to sift through five pounds of chaff to find the ounce of wheat. It wastes more time than missing documentation would.

Totally feel this, and it's not just you. That search example is spot on - you get the marketing copy instead of the actual fix.

I've seen this pattern before. It happens when the documentation roadmap gets built around features, not user workflows. They'll have a "Container Security" doc, an "API" doc, and a "Deployment" doc, but no single guide for "Set up automated container scanning." So you're forced to be the integrator.

Have you tried checking their community forums or GitHub? Sometimes the real "workflow" guides are hidden in a solved forum thread or a readme from an engineer. It's a workaround, but it can save hours.

That search test you ran is a perfect example of why the "information is there" defense rings hollow. It's like saying all the parts for a car are in the warehouse, but they're scattered across different buildings and some boxes are empty.

The redundancy is the real killer. Those four nearly-identical overviews clog up the search results and create maintenance hell. When the API changes, now someone has to remember to update that same paragraph in four different places. No wonder details slip through the cracks.

Your point about dead links is a huge red flag for process, too. It shows a lack of ownership. Someone published a new version of a PDF and didn't clear the old reference, which means there's probably no automated link checker or clear doc deprecation workflow. That's a basic hygiene issue.

That "known issues" page example is brutal, I've been there. I actually asked a vendor's support rep about this once, and they admitted their internal knowledge base has a totally different tagging system and even some internal-only articles. The public portal is a filtered, sometimes sanitized, version.

So to your question, I think the internal resources are often *different*, not just better. The real trouble is when that internal stuff, like known issues, is critical for problem-solving but gets stuck in a separate system. It creates this weird loop where you need support to access the info that would have prevented you from needing support.

That temporal misalignment is a real automator's nightmare. It turns a versioning mismatch into a silent failure mode that only shows up in production.

You get an automated deployment that passes all pre-checks because it's reading the "latest" IaC examples, but those examples are actually referencing a deprecated spec. The logs won't tell you it's a doc problem, they'll just show an obscure API error.

It makes automation feel fragile, because your success depends on the sync schedule of a team you never interact with.

Exactly. This is why I always ask during sales calls about their versioning policy for public examples. Most of them look confused and mumble something about their docs being "living documents." That's a red flag.

You end up having to treat vendor examples like any other third-party dependency. Pin the example URL to a specific commit hash or date stamp in your own automation. It's ridiculous, but it's the only way to get a stable target. Otherwise you're building on a foundation that can shift without notice, and your only clue is a cryptic error from their API.