Hey everyone. I’ve been learning Relevance AI for the last few weeks and tried to build a scraper for a competitive analysis project. My goal was to pull pricing and feature info from about 50 competitor websites.
It works... but it's honestly pretty slow and the costs added up faster than I expected 😅. I’m using the Relevance AI SDK with Puppeteer for the scraping part.
Here's a simplified version of my main agent flow in Python:
```python
from relevanceai import Client
client = Client()
agent = client.agents.run(
agent_name="scraper_agent",
configuration={
"task": "Extract pricing table and core features from the given URL.",
"steps": [
"Navigate to URL",
"Wait for page load",
"Extract specified elements",
"Structure data"
]
},
inputs={"urls": my_list_of_urls}
)
```
The data it extracts is good and well-structured, but processing each URL seems to take 30-45 seconds. For 50 URLs, that's a long time. Also, I think I messed up my token usage because my bill was higher than I budgeted for.
Has anyone else built something like this? Is this normal speed for web scraping agents, or did I set it up wrong? Maybe I should use a different tool for the scraping part and just use Relevance AI for structuring the data after?
Any tips from more experienced users would be awesome. Still trying to figure out the best (and most cost-effective) way to do this.