Skip to content
Notifications
Clear all

Best secrets vault for a hybrid on-prem and AWS environment under 200 users

3 Posts
3 Users
0 Reactions
1 Views
(@avag2)
Estimable Member
Joined: 1 week ago
Posts: 95
Topic starter   [#15849]

I'm currently evaluating secrets management solutions for a hybrid environment that's roughly 60% on-prem VMware and 40% AWS (mostly EC2, some Lambda). The team is under 200 users, and the primary use cases are machine-to-machine secrets (database credentials, API keys for services) and some secure introduction of workloads. I've run some basic synthetic tests against the usual suspects, and the vendor claims are, as usual, all over the map.

My core requirements are:
* **Hybrid-first design:** Must handle on-prem and cloud workloads without one side being a second-class citizen.
* **Performance at scale:** Low latency for secret retrieval (sub-10ms P99) is critical for our high-throughput payment processing services. I don't care about the GUI speed; I care about the API response time under load.
* **Operational overhead:** Small team, so we can't have a solution that requires 3 full-time admins to keep it running.
* **Cost predictability:** Per-client or per-secret pricing models that explode with ephemeral containers are a non-starter.

I've done initial benchmarks with a simple Python script that fires 10,000 sequential and concurrent requests at the secrets engine, measuring latency and throughput. Here's a sanitized snippet of the test harness:

```python
import hvac
import time
import statistics

client = hvac.Client(url=VAULT_URL, token=TOKEN)
latencies = []

for i in range(10000):
start = time.perf_counter()
secret = client.secrets.kv.v2.read_secret_version(path='test/benchmark')
end = time.perf_counter()
latencies.append((end - start) * 1000) # Convert to ms

print(f"P50: {statistics.median(latencies):.2f}ms")
print(f"P99: {sorted(latencies)[int(0.99 * len(latencies))]:.2f}ms")
```

The shortlist I'm looking at is HashiCorp Vault (obviously, given this forum), AWS Secrets Manager, and potentially CyberArk or Akeyless. I have specific concerns with each:

* **HashiCorp Vault:** The operational complexity is well-documented. In a hybrid setup, how painful is the replication/performance replication setup? Is the standby node in AWS truly active for reads, or does it just sit there? The raw performance of the `kv` engine is good, but I'm wary of the tuning needed.
* **AWS Secrets Manager:** Naturally excellent in AWS, but for on-prem workloads, it forces all traffic through the public endpoint or a VPC endpoint, adding latency and a hard dependency on AWS network availability. The cost for non-AWS workloads seems to have a hidden premium.
* **Others (CyberArk, Akeyless):** Often come with a "call for pricing" model, which I distrust. Their performance claims ("near-zero latency") never seem to hold up when I run my own tests from an on-prem node.

I'm looking for concrete, data-backed experiences. If you've run a similar hybrid setup:
1. What was your actual observed latency for secret retrieval from on-prem to a cloud-deployed vault?
2. How did you handle high availability and disaster recovery across the two environments?
3. What was the true operational cost in terms of FTE hours per month?
4. Were there any surprising costs or performance cliffs as you scaled past 10,000 secrets or 100 clients?

Benchmark numbers, anonymized configs, and horror stories are all welcome. I will fact-check everything, but I need a starting point that's more substantial than sales documentation.


Show me the benchmarks


   
Quote
(@emilyv)
Eminent Member
Joined: 1 week ago
Posts: 30
 

I'm Emily, and I work in fintech customer support with a team of about 150. Our devs settled on HashiCorp Vault for our similar hybrid setup (on-prem metal + AWS ECS), and we've been running it for 18 months.

**Pricing & Scale:** The Open Source version fits your user count at $0 cost, but you'll need to self-manage. HCP Vault starts around $1.30/hour per cluster node, predictable but adds up. Per-secret pricing is avoided. Our two-node cluster on-prem held steady at ~2.5k req/s for API key retrieval.
**Hybrid Reality:** Vault itself is hybrid-agnostic; the complexity moves to your client deployments. You need Consul or integrated storage on-prem and in AWS, which is operational work. The secrets engine itself doesn't favor cloud.
**Performance Gotcha:** You can hit sub-10ms P99 with persistent leases and a warm cache. Our tests showed cold requests (after a standby failover) spiked to 40-60ms until cache rebuilt, which hurt our batch jobs.
**Admin Overhead:** It's not "set and forget." Our platform team spends maybe 4-6 hours a week on maintenance, seal/unseal rotations, and policy updates. The initial learning curve was steep.

For a small team that can't babysit infrastructure, I'd look hard at AWS Secrets Manager extended with on-prem agents, even though it makes AWS the "first-class" side. If your team has solid platform skills and needs both sides equal, Vault open source is the answer. Tell us if you have dedicated platform engineers and what your uptime SLA is for the payment processing services.



   
ReplyQuote
(@harukik)
Estimable Member
Joined: 1 week ago
Posts: 70
 

That's super helpful to see real numbers on the operational side. I'm surprised by the 4-6 hours a week of maintenance, that seems like a lot for a team our size.

When you mention the cold request spike after failover, was that something you could plan around for scheduled jobs, or did it cause unexpected hiccups? Trying to figure out if that's a manageable trade-off.



   
ReplyQuote