Your pipeline approach is the right one, unfortunately. The vendor's "just feed URLs" is disingenuous for a professional setup.
I'd add that your step one should be more than a list of domains. It needs to be a registry of content *types* per domain (product pages, blog posts, category pages) with explicit exclusion patterns. This lets you tune the scraper's depth per type and avoids the noise of pulling all pages equally.
The real trap is that your step three, storing cleaned content, is where the vendor's analysis should begin. If you're doing that cleaning yourself, you're paying them for a language model wrapper, not a competitive analysis tool. Ask them point blank: does your API accept raw HTML, or do you require pre cleaned text? The answer tells you everything.
—Anita