We needed to query our Okta user and group data for dashboards without hitting API rate limits or touching prod. Built a read-only Postgres replica synced nightly.
Key parts:
* Used Okta's SCIM API with a small Python app for the initial dump.
* Set up a scheduled GitHub Actions workflow for incremental updates (handles deactivations).
* All PII is hashed in the replica for the analytics side.
The workflow:
```yaml
name: okta-directory-sync
on:
schedule:
- cron: '0 2 * * *'
jobs:
sync:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Sync Users & Groups
env:
OKTA_TOKEN: ${{ secrets.OKTA_SCIM_TOKEN }}
DB_URL: ${{ secrets.REPLICA_DB_URL }}
run: python sync_script.py --incremental
```
Now teams can query freely. Avoided the pricing trap of Okta's native analytics add-on.
YAML all the things.
The SCIM API choice for bulk extraction is pragmatic, but I'd be curious about your strategy for handling schema drift. Okta can add or deprecate attributes with little warning. Our team had to add a validation layer that compares the JSON schema from `/Schemas/Users` before each incremental run, logging discrepancies to a dedicated alert channel.
Also, while hashing PII addresses immediate privacy concerns, consider that hashed email addresses can still be correlated across systems if the salt isn't unique per downstream use case. We've moved to using blind indexing with a separate, rotated key for each analytics platform.
Regarding the cron schedule, a nightly sync can mask daytime attribution gaps. We found it useful to supplement this with a much smaller, near-real-time stream of just user status changes (active/deactivated) via event hooks, populating a separate table. This allows dashboards to accurately reflect license counts at any point in the day, not just at 2 AM.
Finally, what's your plan for schema versioning and rollback in the Postgres replica? When we introduced a similar system, we treated each sync as an idempotent migration, storing each day's snapshot in an append-only table with a `valid_from` timestamp. This allows historical point-in-time queries, which became invaluable for auditing.
Nice approach! That nightly sync hits the sweet spot for a lot of dashboard needs without overcomplicating things.
I've seen teams burn cycles building real-time streams when a daily pull was totally fine. Your point about avoiding the vendor analytics add-on is the real win here - those subscriptions get expensive fast for what's often just a scheduled query.
One thing from the email deliverability side: if you're ever using this to segment users for campaigns, the hashing is great, but watch out for timestamp drift. A user updated in Okta at 9 AM won't hit your marketing platform until after that 2 AM sync, which could matter for a same-day welcome flow. Sometimes you need a smaller, trigger-based sync for just those critical events alongside the bulk job.
don't spam bro
Great point on avoiding the vendor add-on. That cost can spiral quickly. I've been looking at a similar setup for our team.
Could you share a bit about how you handled incremental updates? Specifically, how you track which users have changed or been deactivated since the last run? Did you rely on the 'lastUpdated' timestamps from Okta, or did you need to implement a different tracking method?
Also, what's your backup plan if the SCIM API has a bad day? We had a sync fail once because Okta started paginating differently, and it broke our script for a few hours.