Having recently completed a multi-cloud deployment of CockroachDB to serve a geographically dispersed analytics workload, I've documented the precise steps and key configuration decisions. The primary objective was to achieve sub-100ms P95 read latency for users in North America, Europe, and Southeast Asia while maintaining strong consistency, using a mix of AWS, GCP, and Azure. This post will detail the architecture, the critical `cockroach` CLI commands, and the performance baseline observed post-deployment.
### **Architecture & Provider Selection**
The deployment utilized three cloud providers to mitigate vendor lock-in and leverage regional pricing advantages. The node placement was chosen based on proximity to application servers and comparative egress pricing.
* **Cloud Provider & Regions:**
* **AWS:** `us-east-1` (Virginia) - 3 nodes, `c6i.2xlarge` (8 vCPU, 16GB RAM)
* **GCP:** `europe-west4` (Netherlands) - 3 nodes, `n2-standard-8` (8 vCPU, 32GB RAM)
* **Azure:** `southeastasia` (Singapore) - 3 nodes, `Standard_D8_v5` (8 vCPU, 32GB RAM)
* **Network Considerations:** A VPC/VNet peering mesh was established using each cloud provider's native interconnect service (AWS Direct Connect Gateway, GCP Cloud Interconnect, Azure Virtual Network Gateway). This is a prerequisite; the CockroachDB nodes communicate directly on TCP port `26257`.
### **Step-by-Step Cluster Initialization**
The process begins with a single-seed node and expands to the other clouds. All commands assume you have the `cockroach` binary installed locally and SSH access configured.
1. **Start the First Node (AWS):**
```bash
# On the first AWS node
cockroach start
--insecure
--advertise-addr=
--join=,,
--locality=cloud=aws,region=us-east-1
--cache=4GiB
--max-sql-memory=4GiB
--background
```
The `--locality` flag is crucial for later data placement and latency optimization.
2. **Initialize the Cluster:**
```bash
cockroach init --insecure --host=
```
3. **Join Nodes from GCP and Azure:**
Execute a similar `cockroach start` command on each subsequent node, changing the `--advertise-addr` and `--locality` accordingly (e.g., `--locality=cloud=gcp,region=europe-west4`). The `--join` list remains identical across all nodes, enabling mutual discovery.
### **Critical Database Configuration**
Once all nine nodes are live, connect via SQL to apply operational policies.
```sql
-- Create a database and set the primary region.
CREATE DATABASE analytics;
USE analytics;
-- Configure survival goals and replication zones.
ALTER DATABASE analytics SET PRIMARY REGION "us-east1";
ALTER DATABASE analytics ADD REGION "europe-west4";
ALTER DATABASE analytics ADD REGION "southeastasia";
ALTER DATABASE analytics SURVIVE REGION FAILURE;
-- Create a table with a locality-aware partitioning key.
CREATE TABLE sensor_readings (
reading_id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
sensor_id INT NOT NULL,
recorded_at TIMESTAMPTZ NOT NULL,
value DECIMAL NOT NULL,
INDEX idx_recorded_at (recorded_at)
) LOCALITY REGIONAL BY ROW;
```
The `REGIONAL BY ROW` table locality is the key to low-latency reads; rows are automatically assigned to the region of the gateway node that performs the insert, and are then replicated to other regions for survival.
### **Observed Performance & Cost Metrics**
After a 72-hour burn-in period with a simulated workload (~10k writes/sec, ~50k reads/sec), we observed the following:
* **Latency (P95, measured from application tier in each region):**
* Reads (local region): 12-18ms
* Reads (cross-region, worst-case): 142ms
* Writes (consensus across 3 clouds): 48-67ms
* **Monthly Estimated Cost (compute only, 730h/month):**
* AWS (3x c6i.2xlarge): ~$1,050
* GCP (3x n2-standard-8): ~$1,320
* Azure (3x D8_v5): ~$1,410
* **Total:** ~$3,780
* **Key Takeaway:** The `REGIONAL BY ROW` configuration reduced read latency for the predominant local-access pattern by over 90% compared to a global `RANDOM` distribution, at the cost of slightly more complex cross-region queries. The primary trade-off is between localized performance and perfectly uniform global latency.
The configuration is now stable, but ongoing monitoring focuses on cross-region network throughput and the cost of inter-VPC data transfer, which can become a significant line item if not monitored. I can elaborate on the Prometheus/Grafana setup for this cluster if there is interest.
-- elliot
Data first, decisions later.