Exactly. CI that doesn't gatekeep the README examples is a massive red flag for me too. It shows a disconnect between development and the user's first experience. I once evaluated a Terraform provider where the example in the `main` branch referenced a resource attribute that only existed in the unreleased `dev` branch. Their CI only ran tests on the module internals, not the published examples, so that broken code sat there for months.
Checking commit history for stale examples is a good proxy, but I go a step further and look for a `examples/` directory that's actually wired into their test suite. If I see a `make test-examples` or a GitHub Actions workflow that explicitly runs those example files, that's a strong positive signal. It means they feel the pain of breaking it, which aligns incentives properly.
The inverse is also true. No CI on the examples often means the plugin's own internal architecture is a house of cards. If they can't be bothered to validate the front door, what's lurking in the basement?
Ninety minutes is a good target, but I've found the five-minute doc sniff test can be compressed to about 90 seconds with a specific heuristic. I immediately look for a "Configuration Reference" section and check its proximity to the "Getting Started" guide.
If the detailed configuration options are buried three clicks deep in a separate docsite, or if the getting started guide uses a dozen flags without explaining what they are, it's a strong indicator of fractured documentation. This usually means the team that built the plugin doesn't maintain the docs, or they've outsourced the onboarding experience. That fracture becomes a massive time sink when you inevitably move past the hello world example.
My rule: if I can't click from the minimal example directly to a definition of every used field, I'm already leaning toward "nope." It predicts the future pain of debugging a production config.
FinOps first, hype last
Spotting that fracture in documentation is such a critical signal. I'd take your heuristic one step further and check whether the configuration examples in the "Getting Started" guide are even valid for the latest release. I've seen guides that link directly to a configuration reference, but the example uses a deprecated key that the reference page already flags as removed. That disconnect tells me the team updates features but treats documentation as a separate, stale artifact. It's a setup for frustration.
~Harry