Just built a competitive analysis scraper with two agents, here's the code.

I've been learning AutoGen for a few weeks and wanted to share a practical project. I built a scraper that analyzes competitors' pricing pages using two agents.

One agent acts as a "Planner" that figures out which URLs to check and what data to extract. The other is a "Scraper" that uses Playwright to fetch the pages and pull the info. They coordinate to handle errors and structure the final data into a CSV. It's my first multi-agent workflow and I was surprised how clean the coordination is.

Here's the core part of the code:

```python
from autogen import AssistantAgent, UserProxyAgent

# Planner Agent
planner = AssistantAgent(
name="planner",
system_message="You are a planning agent. Analyze the target company name and product to generate a list of competitor URLs and specific data points (price, tier name, features) to scrape.",
llm_config={"config_list": [{"model": "gpt-4", "api_key": os.environ["OPENAI_API_KEY"]}]},
)

# Scraper Agent (with function calling for Playwright)
scraper = UserProxyAgent(
name="scraper",
human_input_mode="NEVER",
code_execution_config={"work_dir": "scraping", "use_docker": False},
function_map={
"fetch_page": fetch_page_function, # Custom function using Playwright
"extract_data": extract_data_function,
},
)

# Initiate the chat
planner.initiate_chat(
scraper,
message="Find pricing for cloud data platform 'Snowflake'. Identify 3 competitors and get their lowest advertised plan price and core features."
)
```

The main pitfall I hit was the scraper agent sometimes trying to execute bad selectors. I had to add more detailed error handling in the functions. Overall, it feels powerful for automating this kind of research loop. Curious if others have built similar data-gathering agents.