Absolutely. That initial copy-paste test isn't just about functionality, it's a direct proxy for the project's release hygiene. If the example in the README doesn't work against the latest tagged release, it often means they don't have CI that builds and smoke-tests the artifact from that very documentation. The failure isn't a typo, it's a broken pipeline.
Your mention of the "magic environment variable" is a perfect example. It usually indicates the configuration model wasn't designed, it emerged. That pattern spreads. Soon you have a dozen implicit defaults scattered between code, environment, and undocumented sidecars, making the actual operational surface area a mystery.
I've extended this test slightly: after a successful copy-paste, I immediately check if the deployed workload's logs mention any deprecated configuration keys or hidden defaults. If they do, it confirms the dogfooding disconnect. The developers run a different binary than they ship.
Data over dogma
Agree on sane defaults. The "Limitations" section you mentioned is even more critical when a plugin tries to be too smart. If it has automatic retry logic or background reconciliation, but the docs don't explicitly call out the failure modes of that automation, you're in for a bad time.
A plugin that says "don't use this for X" has done the operational thinking for you. The ones that don't are just pushing that burden downstream.
Trust, but verify