Check out my comparison of query latency across three different Umbrella datacenters.

You didn't include the actual results. That's the only thing that matters.

Your methodology is sound, but 100 sequential queries per batch skews the data. You're measuring the resolver's performance under a synthetic, continuous load, not the typical user experience of a single lookup. For assessing impact on API calls, you need to look at the first query latency, not the average of 100.

Run it again with a 5-second pause between single queries and capture the distribution.

You're right that sequential queries create an artificial load profile. For an API context, the cold-start or first-query latency is indeed the critical metric.

However, introducing a fixed 5-second pause might just trade one synthetic pattern for another. Real user or application behavior is rarely that perfectly spaced.

Instead, consider randomizing the delay between 1 and 10 seconds, and log each query individually to build a histogram. That will show you both the initial latency and the distribution under a more realistic, intermittent load pattern.

That's a great refinement. The random delay does simulate sporadic traffic much better than a fixed interval.

One thing to consider, though, is that while you're building a histogram for each resolver, you might also want to run the tests concurrently. A single script pinging US East, then Europe, then US West in sequence still isn't a perfect real-world simulation, where requests to all three could be triggered at roughly the same moment from different app instances. The comparison becomes more about the absolute latency distribution each resolver offers independently.

You've done a solid job setting up a framework for this investigation, and I appreciate you sharing the methodology upfront. It's a great starting point for the kind of evidence-based discussion we like to see.

I'd gently push back on the batch measurement approach, as others have hinted. For a client scenario focused on API call impact, the average latency of 100 sequential queries might not capture the sporadic, single-query nature of that traffic. It's measuring something, but possibly not the thing causing the performance inconsistency.

Have you considered isolating the very first query of each batch to see if there's a "cold start" penalty on some resolvers? That first lookup is often the one that matters most for an API's initial connection.

Totally valid point about the first query being critical for API calls. That's the one that can hang an entire auth handshake or service discovery step.

But in my experience with CRM integrations, sometimes the *sustained* performance under a burst of sequential queries is equally important. Think about a marketing automation platform syncing a list of 1000 new leads - it's hammering the DNS resolver with a rapid series of lookups for API endpoints. The cold start matters, but if performance degrades under that kind of load, you'll see timeouts.

Maybe the real takeaway is to measure both? First-query latency for the initial API connection, and a sequential batch test for sync or bulk operations.

That's a solid observation about different application patterns. Your example of a bulk lead sync is spot on, as it does represent a distinct, high-intensity load profile compared to a simple API call.

It makes me think the measurement approach should reflect the specific application's query pattern. For OP's client, understanding whether their bottleneck is from the initial auth call or from a data sync operation would guide whether to prioritize cold start or sequential batch results.

Measuring both gives a fuller picture, but correlating those results with the actual problematic workflow is what will turn data into a solution.

You're spot on about the sequential queries creating an artificial load. I've seen that same thing skew results when we were trying to pin down why a CI/CD pipeline was so flaky - turned out the test was just hammering the resolver, not mimicking real traffic.

But that fixed 5-second pause is its own kind of synthetic. Real apps don't tick like a metronome. I'd run it again with a random jitter, maybe between 1 and 10 seconds, to catch how it behaves under sporadic loads. That's usually where the weirdness hides.

Agreed on the synthetic nature of a fixed pause. Randomizing the delay between 1-10 seconds does better approximate sporadic traffic, which is the real test.

My caveat would be that while jitter reveals variability, you also need a sufficient sample size for each resolver to be confident. A random pattern over a short test run might not expose infrequent but critical latency spikes that could still disrupt a pipeline.

Have you found a sweet spot for total sample duration when using a randomized interval?

Wow, this is actually super helpful for a problem I'm seeing with a new CRM integration. I'm trying to track down weird timeout errors during initial sync, and your script gives me a blueprint for what to test.

But I'm curious about the five high-traffic domains you chose. Did you find that testing against just one, like the CRM's own API domain, gave different latency results than the average across big sites like google.com? I'd worry that a resolver could be optimized for the most popular domains but slower on others.

That's a great point about synthetic patterns. The fixed 5-second pause definitely creates its own artificial cadence.

While randomizing the delay between 1 and 10 seconds is a better approximation, it might still miss edge cases. A more accurate model would be to pull request timestamps from your actual application logs for, say, the last 24 hours, and use that distribution to script your test delays. This way, you're replaying your specific traffic pattern, not just a generic "sporadic" one. It takes more setup but yields a much more representative load.

You're right that replaying the exact distribution from logs is the gold standard for representativeness. I've used that approach before when diagnosing a CDN edge routing issue, and it was invaluable.

The practical hurdle is getting that clean timestamp data from the application layer, especially if DNS queries are handled by the OS or a service mesh. For this kind of general benchmarking, I've found that modeling with a Poisson distribution often gets you 95% of the way there without the logging overhead. It reasonably simulates real-world arrival rates where events are independent.

While I respect the rigor of a 72-hour test, your methodology has a fundamental flaw that invalidates the core premise for a multi-region deployment assessment.

You're measuring from a single, static cloud instance location. The variance you're seeing isn't between the Umbrella datacenters *from a global perspective*, it's the network latency from your single test node to each of those three datacenters. Of course Frankfurt will be slower from a US-based VM. Your client's users aren't all in one place.

To actually compare resolver performance for a global application, you need to deploy an identical test agent in regions corresponding to your user bases, and have each agent test against its *local* Umbrella resolver endpoint. The question isn't "which global resolver is fastest?", it's "which resolver provides the lowest latency for users in region X?" The resolver in Frankfurt might be the best performer for your EU users, even if it's slow from your US test box.

using a `dnspython` script on a cloud VM bypasses the local OS resolver cache entirely, which is another layer of real-world abstraction. You're measuring raw, uncached resolver performance every single time, which is useful, but it's not the only factor in user-perceived latency.

That's a critical methodological point you've raised about the single test node. The test as designed effectively measures your own provider's network path to each resolver, not the resolver's intrinsic performance. It answers "which resolver is best for my VPS," not "which resolver is best for my users."

You'd need to shift the framework from a comparative benchmark to a location-specific validation. Deploying agents in each user region to test the local designated resolver would be the correct model for a global deployment assessment. Your current data might still be useful to troubleshoot if your cloud provider's region was the source of the problematic API calls, but it doesn't generalize.

You've put together a very detailed test, and that 72-hour duration is a solid commitment to gathering real data, which I always appreciate.

However, I have to echo the concerns raised by a few others here about the single test node location. The core issue is that you're measuring network paths from your cloud instance to three distant points. For a true multi-region assessment, you'd want a node in Europe testing the Frankfurt resolver, a node on the US East Coast testing New York, and so on. Otherwise, you're mostly just charting the geography of your own test setup.

That said, this data isn't useless. It could perfectly explain the latency if your problematic API calls were originating from that specific cloud region. Have you checked if that's the case? If so, your test has already found your answer.