I've been conducting a series of controlled performance benchmarks on the Gemini API suite, specifically focusing on Flash 1.5 for its promised cost-to-performance ratio in high-volume, low-latency tasks. Over the past 72 hours, my monitoring has captured significant and erratic latency spikes that do not correlate with my load patterns, and I'm attempting to determine if this is a localized issue or a broader systemic trend.
My test harness is designed to isolate provider latency from network overhead. It's a simple Go service deployed in us-central1, making synchronous calls to `gemini-1.5-flash-latest` with identical parameters. The payload is a 150-token retrieval-augmented generation (RAG) context with a straightforward summarization instruction.
The observed P99 latency, which had been relatively stable at 1.2-1.5 seconds, has shown intermittent jumps to between 4.8 and 7.2 seconds. These spikes are not gradual but rather sudden, lasting for batches of 5-10 requests before returning to baseline. Crucially, there are no corresponding increases in error rates or token usage deviations.
My configuration and a simplified version of the measurement loop:
```go
// core measurement function
func benchmarkRequest(ctx context.Context, client *genai.Client, prompt string) (latencyMs int64, err error) {
start := time.Now()
model := client.GenerativeModel("gemini-1.5-flash-latest")
model.SetTemperature(0.1)
resp, err := model.GenerateContent(ctx, genai.Text(prompt))
latencyMs = time.Since(start).Milliseconds()
// ... validate resp
return latencyMs, err
}
```
Key environmental controls:
* Concurrency level fixed at 10 workers.
* Request cooling-off period of 100ms between dispatches per worker.
* All runs target the `us-central1` endpoint.
* No changes were made to the codebase or infrastructure during the observation window.
I have ruled out:
1. **Regional network issues:** Cross-region pings and control calls to a different provider (Claude Haiku) show no correlated latency.
2. **Cold starts:** The pattern persists deep into sustained test runs (>500 requests).
3. **Payload size variance:** Input and output tokens are logged and show <5% variation.
4. **Client-side throttling:** My rate is well below the documented quotas (900 RPM, 15k TPM).
The questions I'm posing to the community:
* Are others observing similar non-linear latency degradation with Gemini Flash, particularly in the last few days?
* If so, does it appear to be region-specific? I'm planning to run comparative tests in `europe-west1` next.
* Has anyone successfully correlated these spikes with specific times of day or internal Google Cloud platform metrics?
* Most importantly, has anyone engaged with Google support and received a meaningful root-cause analysis? My preliminary ticket yielded only the standard "no ongoing incidents" response.
I'm skeptical of attributing this to mere "noisy neighbor" issues given the magnitude of the P99 shift. For our use case—where predictable latency is more critical than absolute speed—these spikes are problematic. I'll be sharing my structured benchmark results, including comparison baselines against GPT-4o-mini and Claude Haiku, in a follow-up post once I gather more data. Any shared observations or diagnostic approaches would be valuable.
Trust but verify.