I'm evaluating a core architectural decision for a new service mesh and certificate management layer, and the choice between leveraging Vault's Transit secrets engine or interacting directly with a cloud provider's KMS (AWS KMS, GCP Cloud KMS, Azure Key Vault) is a primary point of contention. The marketing and documentation from all sides are, predictably, full of unverified claims about "low latency," "high performance," and "cryptographic agility." I need to cut through that.
Our use case is high-volume, synchronous cryptographic operations: primarily signing and verification of internal service-to-service request JWTs, and secondarily encryption/decryption of small payloads (under 4KB). We're talking about sustained workloads of several thousand operations per second per service, with P99 latency requirements under 10ms. The argument for Vault Transit is centralized policy management, audit logging via Vault's built-in capabilities, and the abstraction from a specific cloud vendor. The argument for native KMS is fewer network hops and presumably a more optimized path to the hardware security module.
I have yet to find any rigorous, independent benchmarks that compare these two approaches under realistic conditions. Most blog posts are either from HashiCorp or a cloud vendor, and they inevitably show their own product in the best light. I'm looking for data on the actual overhead introduced by Vault's Transit engine.
Key dimensions I'm investigating:
* **Latency Distribution:** Mean, P50, P95, P99, and P99.9 for a simple `sign` operation. How does the network hop to the Vault cluster (plus potential Transit engine logic) compare to the KMS service latency? We must account for Vault's lease handling and token renewal overhead in a sustained scenario.
* **Throughput at Saturation:** What is the maximum operations per second a single Vault instance with a backed HSM (like AWS CloudHSM or GCP's HSM) can sustain versus a native KMS service's quotas? Does Vault become the bottleneck?
* **Cold Start / Connection Overhead:** Impact of Vault's `default_lease_ttl` on performance of intermittent operations versus KMS's typically stateless API calls.
* **Cost at Scale:** This is a FinOps question. With Vault Enterprise pricing (or even the operational cost of self-managed Vault), plus the underlying HSM cost, at what request volume does it become more expensive than native KMS pay-per-request models?
I've begun a rudimentary test myself using a simple Go service that performs ED25519 signing, running in our Kubernetes environment. The preliminary results are... interesting, but not conclusive.
```go
// Simplified test loop for latency measurement
func benchmarkSign(vaultClient *api.Client, keyName string) (latencies []time.Duration) {
data := base64.StdEncoding.EncodeToString([]byte("benchmark-payload"))
for i := 0; i < 1000; i++ {
start := time.Now()
_, err := vaultClient.Logical().Write(
fmt.Sprintf("transit/sign/%s", keyName),
map[string]interface{}{"input": data},
)
latencies = append(latencies, time.Since(start))
}
return latencies
}
```
Has anyone conducted or come across a serious, reproducible benchmark that isolates these variables? I am particularly interested in scenarios where Vault is deployed in a high-availability mode with a dedicated HSM backend, as that's our production baseline. Anecdotes about "it feels slower" are less useful than structured data. If no such benchmark exists, I'm prepared to instrument a comprehensive test and publish the results, but I'd prefer not to duplicate effort.
Measure twice, migrate once.