My results after building a competitive analysis scraper - works but is slow and expensive.

cloud_infra_newbie · 2026-07-27T21:34:40Z

Hey everyone. I’ve been learning Relevance AI for the last few weeks and tried to build a scraper for a competitive analysis project. My goal was to pull pricing and feature info from about 50 competitor websites. It works... but it's honestly pretty slow and the costs added up faster than I expected 😅. I’m using the Relevance AI SDK with Puppeteer for the scraping part. Here's a simplified version of my main agent flow in Python: ```python from relevanceai import Client client = Client() agent = client.agents.run( agent_name="scraper_agent", configuration={ "task": "Extract pricing table and core features from the given URL.", "steps": [ "Navigate to URL", "Wait for page load", "Extract specified elements", "Structure data" ] }, inputs={"urls": my_list_of_urls} ) ``` The data it extracts is good and well-structured, but processing each URL seems to take 30-45 seconds. For 50 URLs, that's a long time. Also, I think I messed up my token usage because my bill was higher than I budgeted for. Has anyone else built something like this? Is this normal speed for web scraping agents, or did I set it up wrong? Maybe I should use a different tool for the scraping part and just use Relevance AI for structuring the data after? Any tips from more experienced users would be awesome. Still trying to figure out the best (and most cost-effective) way to do this.

Page 3 / 3 Prev

Relevance AI Reviews

Last Post by danw 2 months ago

31 Posts

31 Users

0 Reactions

169 Views

RSS

danw

(@danw)

Reputable Member

Joined: 3 months ago

Posts: 387

29/07/2026 11:19 am

Yes, that's the normal speed for a full headless browser per page. The real problem is your agent's "Extract specified elements" step. It's almost certainly feeding the *entire* page's HTML to the LLM for structuring.

You're paying for every token of that rendered DOM, including all the scripts, CSS, and footer junk. For 50 sites, that's a massive and unnecessary bill.

Skip the agent's generic extraction. Use Puppeteer's page.evaluate() right after the page loads to pull out only the pricing container HTML yourself. Then pass that 200-line snippet to the LLM for structuring. Your costs will drop 90% overnight.

ReplyQuote

Page 3 / 3 Prev

Forum Icons: Forum contains no unread posts Forum contains unread posts

Topic Icons: Not Replied Replied Active Hot Sticky Unapproved Solved Private Closed