I've been conducting extensive load testing on Azure Durable Functions over the past quarter, specifically focusing on orchestrator scaling under sustained high-concurrency workloads, and I'm encountering concerning behavior that doesn't align with Microsoft's documented scaling promises. My hypothesis, based on empirical data, is that the internal partition management for the Durable Task Framework's storage backend becomes a significant bottleneck well before reaching the theoretical scale limits, leading to queue backlogs and cascading execution delays.
The architecture under test is a canonical fan-out/fan-in pattern. A single parent orchestrator spawns hundreds of child activity functions via `CallActivityAsync`, awaits their completion with `Task.WhenAll`, and then processes the aggregated results. The storage provider is Azure Storage (using the default queues, tables, and blobs), not the Netherite or MSSQL providers. The function app is configured for Consumption Plan scaling.
The issue manifests during a ramp-up from 0 to 500 concurrent orchestrator instances over a five-minute period. The initial 100-150 instances start promptly. However, beyond that threshold, I observe the following, captured via Application Insights and custom Durable Functions tracking queries:
* **Orchestrator Start Latency Spikes:** New orchestrator submissions experience delays of 60-120 seconds before the first execution, despite the host showing scale-out activity.
* **Partition Imbalance:** Analysis of the `control-queue-00X` queues shows severely uneven message counts. Queue 00 and 01 often contain thousands of backlogged messages, while queues 02-07 remain nearly empty. This suggests the partition distribution logic is not effectively hashing the new instance IDs.
* **Throttling-Like Behavior:** The Durable Functions extension logs show repeated warnings: "`Previous attempt to fetch work items failed. The next attempt will happen in 00:00:16.`" This occurs even with no apparent storage account throttling (HTTP 429) indicated in the storage account metrics.
Here is a simplified version of the orchestrator in question, though the pattern is standard:
```csharp
[FunctionName("BatchProcessor_Orchestrator")]
public static async Task<List> RunOrchestrator(
[OrchestrationTrigger] IDurableOrchestrationContext context)
{
var batch = context.GetInput();
var tasks = new List<Task>();
foreach (var item in batch.Items)
{
tasks.Add(context.CallActivityAsync("ProcessItem", item));
}
// Await all concurrent activities
var results = await Task.WhenAll(tasks);
return results.ToList();
}
```
My benchmark configuration uses a separate client application to trigger the orchestrators via HTTP, measuring the time from HTTP POST to the orchestration instance creation, and then to its completion.
**Key Metrics from Last Test Run:**
* Target: 500 concurrent orchestrations.
* Achieved after 10 minutes: 312 completed, 188 stuck in `Pending` or `Running` state with no forward progress.
* Average end-to-end latency for completed first 100 instances: 45 seconds.
* Average end-to-end latency for completed instances #101-312: 247 seconds.
* Storage Account: No throttling, DTU/IOPS well under limits.
My question to the community is multi-faceted: Has anyone performed similar scale testing and observed this partition skew issue with the Azure Storage backend? Is this a known, inherent limitation of the storage provider's partition algorithm that necessitates a move to the Netherite provider for high concurrency, or are there configuration nuances (e.g., `storageProvider.partitionCount`, `controlQueueBufferThreshold`) that can mitigate this? Furthermore, are there any robust monitoring strategies beyond the built-in Application Insights integration to surface this partition imbalance in real-time?
I am skeptical of the common support suggestions to simply increase the storage account limits or switch to Premium plan, as the metrics do not indicate a resource saturation problem at the Azure Storage layer itself. The evidence points to a coordination bottleneck within the Durable Task Framework's work item distribution mechanism.