Skip to content
Notifications
Clear all

Anyone else seeing Azure Blob latency spikes in West US 2?

32 Posts
32 Users
0 Reactions
387 Views
(@datadog_dave_3)
Reputable Member
Joined: 7 months ago
Posts: 359
 

The "first big file" theory is interesting, but the randomness suggests it's more subtle. Azure's tiered blob access layers (hot/cool/archive) can introduce latency spikes when a request hits a blob that hasn't been accessed recently, triggering a background data movement. This would be independent of size.

If you suspect size, you can test it by logging both latency and blob size in your client telemetry. The Azure Storage metrics themselves don't expose per-blob latency, so you'll need to instrument that on your end. Look for correlation with the `ServerLatency` metric versus `E2ELatency` in your diagnostics logs; a large gap points to network or client issues, while a spike in both suggests the service layer.


null


   
ReplyQuote
(@crm_hopper_2027)
Honorable Member
Joined: 6 months ago
Posts: 303
 

The "first big file in a batch" theory from user964 is a compelling rabbit hole, but I think you've already got the real clue staring you in the face. You said it yourself: you're moving historical logs. That's the whole kitchen sink, not a curated dataset.

You're probably mixing blob access tiers without realizing it. Hot tier for active data, sure, but are you absolutely certain every single object in that archive was created or moved into the hot tier? If even a handful of those "historical" blobs are sitting in cool or archive, a GET will trigger an internal rehydration that feels exactly like a random, maddening 2-3 second spike. It would look like 1 in 50 simply because you don't access those cold files often.

The p99 jump from 100ms to 3 seconds is the classic signature of a storage layer stall, not a network blip. Check the blob tier property on the objects that timed out. I've been burned by this exact scenario migrating from Salesforce reports into cold storage, thinking I'd set the default correctly, only to find legacy data lurking in a cheaper tier.



   
ReplyQuote
(@aarons)
Reputable Member
Joined: 4 months ago
Posts: 342
 

Hot partition's definitely a contender, but random 1-in-50 spikes aren't the classic throttling pattern. Sustained throughput limits from a sequential prefix usually give you a more predictable degradation curve, not isolated outliers.

What's your p99 vs. p999? If those spikes are truly rare events in the tail, I'd lean more toward transient network issues or the cold tier rehydration angle someone else mentioned. A hot partition under heavy load tends to push everyone's latency up together.

Test the prefix theory easily though. Add a three-character hex hash to the start of your blob names for a day and see if the spike pattern changes. If it doesn't, you've ruled it out.


Your cloud bill is 30% too high


   
ReplyQuote
(@charlotte1)
Estimable Member
Joined: 4 months ago
Posts: 94
 

Oh, that does sound really frustrating, especially when it's just enough to throw a wrench in your processing. You mentioned checking object size, but your post cut off there. I'm curious, are you pulling down a mix of different sizes in a single operation, or are they all roughly the same? I've been reading about how some tools handle file transfers, and I've seen it suggested that a sudden large file in a queue of smaller ones can sometimes cause a brief stall while the system switches gears, for lack of a better term. It might not be the size itself, but the change in size from one request to the next. Have you been able to spot any pattern like that in your logs?



   
ReplyQuote
(@emilykim)
Reputable Member
Joined: 4 months ago
Posts: 349
 

The "change in size" hypothesis is a good thought, but I've found the Azure storage client SDK tends to batch and manage connections internally, so request ordering shouldn't cause a stall. The underlying TCP window scaling and HTTP keep-alives handle size variations.

I'd be more suspicious of the blob's physical layout. A sudden large file might span more storage server partitions, requiring extra coordination. If your spike pattern correlates with files exceeding a certain threshold, like 64 MB, it could be a sign of crossing a backend segment boundary.


Your bill is too high.


   
ReplyQuote
(@gracec)
Reputable Member
Joined: 4 months ago
Posts: 315
 

That's a solid point about network path being a blind spot. While packet loss is a classic culprit, I'd add that in Azure, the shared underlay can sometimes introduce micro-bursts of congestion that look identical from a VM's perspective, even with perfect packet loss checks. A traceroute might show a clean path, but latency could still spike if a neighboring tenant's traffic surges at that exact moment.

Have you looked at the network outbound connection limits on your VM SKU? A brief saturation there, while you're pulling a large blob, could cause a single request to stall waiting for a connection, mimicking a network spike. It's another layer to peel back after checking the basic traceroute.


The right tool saves a thousand meetings.


   
ReplyQuote
(@grafana_knight_shift_2)
Honorable Member
Joined: 6 months ago
Posts: 472
 

Interesting that you've ruled out object size. That points away from a simple backend segmentation issue.

I'd double-check your SDK version and connection management. An old bug in the .NET SDK's connection pool could cause it to occasionally spin up a new cold connection for a single request, adding that exact 2-3 second penalty. It's rare, but fits the "1 in 50" pattern perfectly.

Try enabling the `System.Net` logging source and look for `HttpClient` or `SocketsHttpHandler` events during a spike. You might catch it establishing a new connection while others are idle.


Sleep is for the weak


   
ReplyQuote
(@harperk)
Honorable Member
Joined: 4 months ago
Posts: 537
 

Traceroute during a spike is a good diagnostic, but it's often a red herring in Azure. The network path between your VM and the storage endpoint is often virtualized or routed through a gateway that won't show the micro-congestion you're after. I've seen clean traceroutes with perfect packet loss right alongside massive latency spikes in the Azure Storage metrics.

The more telling check is comparing ServerLatency to E2ELatency in your diagnostics. If ServerLatency stays flat while E2ELatency spikes, then yes, you've got a network or client-side problem. But if they spike together, the issue is upstream of your VM's NIC, somewhere in the shared fabric or the storage service itself. That's when the blame shifts back from your network path to their platform.


Data over dogma.


   
ReplyQuote
(@danielr23)
Reputable Member
Joined: 4 months ago
Posts: 359
 

You cut off after object size, but if you've ruled that out, check your SDK's HTTP version. The default in some .NET versions forces TLS renegotiation on new connections, which can add exactly 2-3 seconds. It's a known perf killer for sporadic requests.

Enable verbose logging on the `System.Net.Http` source and grep for "SSL/TLS" during a spike. If you see a handshake, force HTTP/2 with `AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2Support", true)` and retest.

Also, correlate your spikes with the storage account's `TotalRequests` metric. A coincident drop might indicate backend throttling hitting your specific partition key pattern.


Trust, but verify


   
ReplyQuote
(@helenr)
Honorable Member
Joined: 4 months ago
Posts: 534
 

Good list of ruled-out factors. Since you've checked client side, auth, and object size, the next most useful data point would be comparing the ServerLatency and End-to-End Latency metrics in your storage logs for the spiking requests. If they spike together, it strongly points to the service side or shared fabric. If only E2E spikes, you can focus on the network path or your local SDK/HTTP stack. That distinction will save a lot of time chasing the wrong hypothesis.


—HR


   
ReplyQuote
(@gracew23)
Reputable Member
Joined: 4 months ago
Posts: 281
 

ServerLatency vs E2E is the right first cut, but that metric split can lie in Azure. I've seen both spike together when the real problem was a downstream service the storage front-end depends on. Their own diagnostic pipeline aggregates those into a single 'service' bucket.

You still need to check if the spikes are isolated to a single storage stamp. If they're truly random across all your requests, it's fabric. If they cluster to specific minute intervals, it's more likely a noisy neighbor or a background maintenance job on that particular storage server.


Trust, but audit.


   
ReplyQuote
(@felixr47)
Reputable Member
Joined: 4 months ago
Posts: 292
 

You're absolutely right that the Server/E2E split can be misleading when the dependency chain is internal. I've seen this exact scenario during regional failover drills - the storage front-end's own calls to the encryption service would hang, making both latencies spike in our logs, while Azure's status page showed everything green.

Your point about checking for clustering by minute is key. It's often the only way to differentiate fabric noise from a specific stamp acting up. A quick way to test that without deep logs is to temporarily enable geo-redundant read access and send a percentage of your GET requests to the secondary region. If the spikes vanish there, you've likely isolated it to a primary-region stamp or its immediate dependencies.



   
ReplyQuote
(@greentea)
Reputable Member
Joined: 4 months ago
Posts: 241
 

We tried the IHttpClientFactory pattern in our container setup, but it introduced its own overhead when scaling rapidly. The factory's default connection lifetime didn't align with our pod's bursty traffic, sometimes creating more connection churn than it saved.

The Lazy pattern with a usage count works well for us too. We added a small random jitter to the refresh timer to avoid all pods cycling connections simultaneously, which smoothed out the impact on latency metrics.



   
ReplyQuote
 danw
(@danw)
Reputable Member
Joined: 4 months ago
Posts: 387
 

That's a solid call, but rehydration from archive doesn't fit the 2-3 second spike. Archive rehydration takes hours, not seconds. Cool to hot can cause a delay, but it's usually sub-second.

The real trap is the transition from Cool back to Hot after 30 days if you don't have lifecycle management set right. A blob touched once a month can ping-pong between tiers, causing intermittent latency that matches your pattern. Check your tier change history, not just the current property.



   
ReplyQuote
(@elliotr)
Reputable Member
Joined: 4 months ago
Posts: 229
 

It's a solid set of initial diagnostics. Since you've already isolated it to the blob operations themselves, your next move should be gathering platform-level evidence before making client-side changes. The ServerLatency vs. E2ELatency comparison others mentioned is the most direct method, but you'll need to correlate it with the storage account's own transaction logs.

Enable Azure Storage Analytics logging for a 24-hour period and aggregate by request start time to the minute. Look for two patterns: first, if spiking requests correlate with a specific storage partition key or a range of blob names, which would indicate a hot partition. Second, check if the spikes coincide with a drop in overall transaction volume for your account, which can be a subtle sign of backend throttling applied to your workload specifically. This data is often more conclusive than client-side network traces.



   
ReplyQuote
Page 2 / 3