Ran a year-long POC at my last gig. Consensus was great for the 50-person startup phase.
Tried to scale it to 2k engineers, 15k services. The seams burst.
* Alert grouping logic can't handle cloud-scale noise. You get either a flood or silence. No in-between.
* Dashboards choke on high-cardinality tags from our K8s labels. Render times measured in coffees.
* The "consensus" part breaks down when you have 10+ teams on-call. The workflow assumes a single, small ops pod.
* API rate limits feel punitive for basic automation. Had to write a queuing system just to sync our CMDB.
It's a solid v1 for getting your house in order. But when you need to move from a house to a city grid, the foundations crack. Now I see why the big players all built their own.
Status page is my homepage.
Your point about API rate limits being punitive for automation is a critical one that often gets overlooked in vendor security reviews. When you're operating at that scale, every integration becomes a data pipeline, and rate limits aren't just an inconvenience, they're a direct impediment to the CMDB accuracy that the tool itself relies on for context. Building an external queuing layer to accommodate those limits introduces a new failure domain and data latency you shouldn't have to own.
I've seen a similar fracture point with their permission model when you move beyond a single ops team. The flat, role-based access struggles to enforce least privilege across 10+ distinct on-call teams who might own different services or environments. You inevitably end up granting broad 'read' access to all alert data, which can become a data privacy and compartmentalization issue in a large enterprise.
- RayS