Alright, let's cut through the usual "check the logs" advice. We've done that. Our daily full sync to an external HR system is clocking in at a solid four hours, and it's starting to push up against our compliance window for user provisioning.
We're not talking about a massive directory here—maybe 50,000 entries. The source system isn't exactly speedy, but the bottleneck seems to be on our side. PingDirectory's sync server is just... languishing. CPU is bored, memory is fine, network isn't saturated. Yet, the progress meter crawls.
We've already eliminated the low-hanging fruit: the source connector throttling looks normal, we're not syncing every attribute under the sun, and the changelog isn't bloated. I'm starting to suspect this is one of those "it's working as designed" scenarios where the default tuning parameters are absurdly conservative for anything beyond a toy deployment.
Has anyone else had to perform major surgery on the sync pipe configuration? Specifically:
- Did you have to increase the number of worker threads beyond the default, and did it actually help?
- Any gotchas with the `sync-pipe` configuration file that aren't in the docs? We're eyeing the `num-worker-threads` and `max-failed-operations` settings.
- Were you forced to batch operations differently, or did you find a particular attribute that was causing disproportionate slowdown?
I'm skeptical that throwing more hardware at it is the answer. It feels like a configuration artifact, probably in the name of "safe defaults" that assume the worst possible network and source system. I'm all for safety, but four hours is a risk in itself.
Trust but verify
Oh man, that sounds frustrating. I haven't hit that scale yet, but your point about the default tuning being for a "toy deployment" really resonates. I was watching our sync crawl with only a few thousand entries.
> Did you have to increase the number of worker threads
We did, and it made a noticeable difference for us. The trick was we had to bump up the connection pool size on the source connector side too, otherwise the extra threads just ended up waiting anyway. Is it possible your source system's connection limit is the real cap, even if the network isn't saturated? Just a thought!
Yes, increasing worker threads helps, but only to a point. You're likely hitting a different limit.
> Did you have to increase the number of worker threads
The real bottleneck is usually the source system's latency per entry, not pure throughput. More threads just create more concurrent slow requests. Default is like 10.
Two things to check:
1. `num-update-conflict-retries` - defaults to 3. With a slow source, retries kill your pace.
2. `max-simultaneous-updates` per backend connection. If you're using a single LDAP external server backend, it's probably set to 1. That's your cap.
If you're using a JDBC source, it's the fetch size. Default is often 10 rows at a time. For 50k entries, that's 5000 round trips.
Don't just crank threads. Profile the sync-pipe with a small sample using `--dry-run` and look at where it's spending time.
That connection pool point is so important, and it's an easy one to miss. I've seen teams burn hours tuning threads only to find the source connector was handing them out one at a time.
Your experience with "waiting threads" is spot on. It makes me wonder if the diagnostic data most of us check first, like overall network saturation, is misleading. A connection limit can be hit while the pipe itself looks mostly idle.
What was the giveaway for you? Did you see a specific error or just a pattern in the sync logs?
Keep it civil, keep it real.