I have been conducting a detailed performance analysis of our hybrid cloud deployment, which utilizes an AWS Direct Connect Gateway peered with a Google Cloud Partner Interconnect in the us-east4 region. Despite following the documented configurations for a private VPC peering setup, our application monitoring over the last 72 hours indicates persistent and unacceptable latency. The observed round-trip times are averaging between 85-110ms for simple ICMP and TCP health checks between an EC2 instance (c6i.large) in us-east-1 and a GCE instance (n2-standard-2) in us-east4. This is approximately 3x higher than the expected 25-40ms based on the geographical proximity and the SLA commitments for the premium networking tiers we have procured.
Our current configuration and observations are as follows:
**Current Architecture Summary:**
* **AWS Side:** Virtual Interface (VIF) attached to a Direct Connect Gateway. BGP session established with an ASN of 64512.
* **GCP Side:** Partner Interconnect attachment of 10Gbps tier. VLAN attachment configured with a BGP peer using an ASN of 64513.
* **Routing:** Active-active BGP session with ECMP. Both clouds advertise specific, non-overlapping CIDR blocks (/24 prefixes).
* **Security:** No intermediate firewalls or proxies in the data path; security groups and firewall rules permit ICMP and application ports.
**Collected Metrics & Benchmarks:**
We used a combination of `mtr` and custom Python scripts using the `socket` library to gather granular path data. The consistency of the high latency points to a routing issue rather than transient congestion.
```bash
# Example mtr output (abbreviated) from AWS to GCP internal IP:
HOST: ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com Loss% Snt Last Avg Best Wrst StDev
1.|-- 169.254.0.1 (AWS Direct Connect Router) 0.0% 10 0.3 0.4 0.2 1.0 0.2
2.|-- 52.93.28.1 (AWS Backbone) 0.0% 10 1.2 1.5 1.1 2.3 0.4
3.|-- ??? (Provider Edge?) 0.0% 10 45.1 46.2 44.8 48.9 1.1
4.|-- 108.170.245.1 (Google Frontend) 0.0% 10 86.7 88.1 85.0 92.3 2.5
5.|-- 10.120.0.1 (GCP Internal Target) 0.0% 10 87.0 88.4 85.2 93.0 2.6
```
The significant jump (~45ms) at hop 3 suggests the traffic is taking a suboptimal geographical path, potentially traversing an intermediary aggregation point far from the intended regions. This is not aligning with the "direct" nature of the paid interconnect.
**Hypotheses & Initial Validation Attempts:**
1. **BGP Path Selection:** Verified local preference and MED values are not forcing a path through a different Google Cloud region or AWS Direct Connect location.
2. **MTU/Throughput Issues:** Conducted iPerf3 tests which confirmed full 10Gbps throughput is achievable, ruling out link saturation or TCP window scaling problems. Path MTU discovery appears functional.
3. **Cloud Provider Routing Metadata:** Reviewed AWS CloudWatch VIF metrics and Google Cloud's Interconnect metrics dashboard. Both show no errors, packet drops, or BGP session flaps during the measurement period.
My primary question for the community is whether others have encountered similar latency discrepancies with AWS-GCP peering, specifically when the advertised route appears direct but the actual data path is not. I am seeking concrete, data-backed insights on:
* Documented but non-obvious routing policies from either provider that could insert an intermediary hop for traffic inspection or billing metering.
* Empirical benchmarks for latency between us-east-1 and us-east4 (or comparable region pairs) over private peering to establish a realistic baseline.
* Effective troubleshooting steps beyond standard `traceroute`, such as using cloud-specific diagnostic tools (e.g., AWS Reachability Analyzer, GCP Network Intelligence Center) to trace the virtual path, and whether you found them accurate.
* Any procurement or configuration-level levers (e.g., selecting a different Direct Connect Location, engaging with the Partner Interconnect provider's NOC) that materially altered the latency profile.
I will be performing a parallel analysis using a VPN-over-internet connection as a control to isolate whether this is an inherent limitation of the peered path. I will share those comparative results once compiled.
show me the SLA
That's a significant delta from the expected latency. Before digging deeper into the cross-cloud path, have you ruled out any local instance-level noise? Things like CPU throttling on the health check VMs or resource contention can sometimes skew those basic ICMP/TCP tests. I'd suggest running a quick parallel test from a different instance type or even a different AZ in the same region, just to establish a cleaner baseline.
Also, for the BGP sessions, are you seeing stable routes or any flapping in the advertised prefixes? Even a few unnoticed withdrawals can cause traffic to take a suboptimal path for a bit, bumping up the average.
Keep it civil, keep it real.
Good catch on checking for BGP flapping. I had a similar headache last year that turned out to be a routing loop caused by a misconfigured local preference on the AWS side.
For your latency baseline, have you tried using a network performance tool like `iperf3` or even a traceroute with MTR? It can help isolate which hop is adding the delay. Sometimes it's not the cloud-to-cloud link, but an outbound path from your VPC to the Direct Connect gateway.
Also, double-check your Cloud Router's advertised routes in GCP. I've seen cases where a too-broad advertisement (like a default route) can cause asymmetric routing, making packets take a longer path back.
Dashboards or it didn't happen.
That's a good point about asymmetric routing. Could a too-broad route advertisement on either side also cause packet loss along with the higher latency, or would that usually just increase the delay?
CloudNewbie