Hey everyone! 👋
I've been deep in the weeds on a major infrastructure decision for our analytics dashboard, and I'd love to get this community's take. We're processing a massive, constantly growing log of search queries and performance metrics (think billions of rows), and our current database is groaning under the weight of our ad-hoc exploration and filtering needs.
The product team needs super fast, flexible querying for user research—like pulling all queries containing a specific phrase, filtered by date range, device type, and with a ranking above position 20, all in under a second. Our engineers are debating between two paths: going all-in on a managed **Searchable** service (like Algolia or Elasticsearch Service) versus rolling our own cluster with open-source **Elasticsearch**.
From my product lens, I care about:
* **Query speed for user-facing features** – Our power users get frustrated with any dashboard lag.
* **Development & maintenance overhead** – I need to gauge the true cost on engineering resources that could be building new features.
* **Flexibility of schema** – Our data models evolve as we learn. How painful are reindexing or mapping updates?
* **Cost predictability** – We're a SaaS startup, so scaling costs need to be somewhat forecastable.
I've heard compelling anecdotes:
* A peer at another startup swears by managed Searchable for getting complex faceted search to market in *days*, not months.
* But our lead dev argues that a well-tuned Elasticsearch cluster on AWS EC2 gives us ultimate control and better performance for our very specific, analytics-heavy queries, even if it adds operational complexity.
**My direct questions for those who have been here:**
* For complex analytical queries over huge datasets, which has proven faster/more reliable in your experience?
* What was the real-world impact on your team's velocity? Did the "time saved" with a managed service actually translate to more shipped features?
* Any horror stories or "aha!" moments around scaling, cost spikes, or query limitations with either approach?
I'll share our own decision and benchmark results once we run them—hoping to save someone else this headache!
keep building
keep building