I needed to automate competitive analysis from multiple SaaS pricing pages. Manual review was unsustainable. Here's a functional pipeline using Claude's API with web scrapes.
First, orchestrate the data collection. I use Scrapy for structured extraction, but any method works as long as output is clean JSON.
```python
# Example item pipeline to Scrapy -> JSONL
import json
class JsonlWriterPipeline:
def open_spider(self, spider):
self.file = open('scrapes.jsonl', 'w')
def close_spider(self, spider):
self.file.close()
def process_item(self, item, spider):
line = json.dumps(dict(item)) + "n"
self.file.write(line)
return item
```
The key is preprocessing the raw scrapes into a consumable format for the API call. I chunk large pages and maintain source URLs for attribution.
```python
import json
from anthropic import Anthropic
def analyze_scrapes_with_claude(scrapes_path, output_path):
client = Anthropic(api_key=os.environ['ANTHROPIC_API_KEY'])
analyses = []
with open(scrapes_path, 'r') as f:
for line in f:
data = json.loads(line)
# Construct a focused prompt for competitive analysis
prompt = f"""Analyze this scraped content for competitive intelligence.
Content URL: {data['url']}
Scraped Text: {data['text'][:15000]} # Truncate for context limits
Extract:
1. Product features mentioned
2. Pricing models or specific price points
3. Target customer segments
4. Key differentiators claimed
Format as JSON."""
response = client.messages.create(
model="claude-3-sonnet-20240229",
max_tokens=1000,
messages=[{"role": "user", "content": prompt}]
)
analyses.append({
'url': data['url'],
'analysis': json.loads(response.content[0].text)
})
with open(output_path, 'w') as out:
json.dump(analyses, out, indent=2)
```
This gives structured JSON output from unstructured pages. For production, add error handling for JSON parsing failures and consider using batched requests if volume is high. The main cost is token usage, so aggressive truncation of scraped text is necessary. I've found it more reliable than fine-tuning a smaller model for this specific task.