Hey everyone,
I've been using You.com's product comparison feature a fair bit lately, especially when evaluating different cloud services or dev tools. While I appreciate the intent, I find the side-by-side UX a bit clunky. The way it forces you into a rigid two-column layout with truncated information means I'm constantly scrolling up and down to compare specific attributes.
For example, comparing two managed Kubernetes services, I want to see:
* Control plane pricing per hour
* Network egress costs (which are often the real budget killer)
* Supported node instance types
* Add-on costs for things like load balancers
With the current setup, I end up having to open two separate browser tabs anyway to get the full picture.
Has anyone found a better method or a workflow that works well? I'm considering building a simple local tool that scrapes or uses APIs to pull key data points into a table for my own needs. Something like a Python script that outputs a clean markdown comparison.
```python
# Pseudo-code idea
import youdotcom_api # If this existed!
services_to_compare = ["aws-eks", "gcp-gke"]
comparison_fields = ["pricing_model", "egress_cost_us", "control_plane_ha"]
for service in services_to_compare:
data = fetch_service_details(service)
# ... process and align data into a table
```
But before I go down that rabbit hole, I'd love to hear how others are handling this. Do you use a different website, a spreadsheet template, or maybe a Notion database? What's your process for an apples-to-apples comparison when researching tools?
-- Amy
Cloud cost nerd. No, I don't use Reserved Instances.
Yeah, scraping with a script is the only way it ever works for me. Those comparison sites always miss the specific data point that actually matters for your use case.
For cloud services, their own pricing calculators are a nightmare but sometimes you can get the raw JSON from the page source. Good luck though.
CRM is a means, not an end.
I hear you on the scripting workaround, but I worry that approach can get pretty fragile. Vendor sites change their DOM structure all the time, and it can break your scraper without warning.
It also puts a lot of onus on the individual user. Maybe we should be pushing the comparison tools and the vendors themselves to provide cleaner, machine-readable data exports in the first place. An API would be ideal, but even a simple CSV download option would be a huge step up from page scraping.
Keep it real, keep it kind.
You're absolutely right about scraping being fragile, that's a constant maintenance headache. I've had to fix my own API-based scrapers more times than I can count after a vendor redesigns their pricing page.
While pushing for machine-readable exports is the right long-term goal, I've found a middle ground: using structured data from review sites like G2 or PeerSpot via their own, more stable APIs. They don't have every cost detail, but they standardize a lot of the feature and capability data. You can pull that into a simple Airtable base or even a Google Sheet, then supplement it manually with the critical pricing specifics that you know are deal-breakers.
It's still a bit of a manual lift, but it's far more stable than direct website scraping.
Building a local tool is the logical next step given the limitations of generic comparison UIs. Your pseudo-code idea is on the right track, but the dependency on a hypothetical `youdotcom_api` is the core issue; it doesn't exist, and vendor APIs are inconsistent.
My benchmarking work has led me to a more hybrid approach. I use Playwright to script browser sessions for data extraction, but I don't scrape the live site UI directly. Instead, I target the network calls the pricing pages themselves make, often to internal APIs that return JSON. These endpoints are more stable than the DOM.
Here's a more realistic snippet for a single provider, based on a pattern I've seen:
```python
import asyncio
from playwright.async_api import async_playwright
async def fetch_pricing_json(url):
async with async_playwright() as p:
browser = await p.chromium.launch()
page = await browser.new_page()
prices = []
# Listen for specific network responses
page.on('response', lambda response:
prices.append(response.json()) if '/api/pricing' in response.url else None
)
await page.goto(url)
await page.wait_for_timeout(3000) # Let network calls happen
await browser.close()
return prices[0] if prices else None
```
You'd still need to map these unique JSON structures for each vendor into your common comparison fields, which is the manual, but once-per-vendor, setup cost. It's less fragile than parsing HTML, as these internal APIs change less frequently. The main caveat is you're essentially automating what a browser does, which can be against some ToS, so use it for personal analysis only.
benchmarks or bust
You're hitting on the real crux of the issue - the fragility. It's not just a technical headache; it's a time tax.
> pushing the comparison tools and the vendors themselves
This is the ideal path, but in my experience, the incentive just isn't there for the vendor. A clean CSV or API makes it *too easy* for you to leave. A clunky page that makes direct price comparisons difficult often works in their favor. It's a feature, not a bug.
That's why I tend to build my scrapers to target the underlying data services those pages use, rather than the HTML. The JSON endpoints they call for pricing tables are often more stable, but you're still at their mercy.
api first