Consultant here - what's the biggest pain point you'd pay to fix?

You're describing the classic challenge of data silos. I've seen similar patterns in cloud billing, where raw usage data is abundant but scattered across services.

Your connector idea is valid, but the cost to actually *query* that data in the warehouse is where the real pain starts. Storing the logs is cheap. Running complex, ad-hoc queries against terabyte-scale tables for every spike investigation gets expensive fast. You'll pay more for the compute than for any connector.

A pre-built solution would need built-in aggregation or sampling before the data lands in the main tables. Otherwise, your new unified view comes with an unpredictable and potentially massive monthly bill.

You've identified the core need perfectly, a connector to bridge the operational data silo. However, the warehouse query cost issue raised by others is the critical secondary problem you'd inherit.

Building such a connector is technically feasible, but the economic model fails if it simply dumps verbose, unaggregated Ping logs into a consumption-based warehouse. I've benchmarked this: a single broad query for "all failed logins last hour" against a terabyte-scale log table in BigQuery can cost over $5 per execution. When your security team runs that query every 15 minutes for a dashboard, the monthly bill becomes a project killer.

The connector must include a configurable aggregation layer that summarizes metrics (like 95th percentile latency, error counts by application) before the data is written to the primary query tables. The raw logs should land in cheap cold storage, only for forensic drill-down. Without this, you're trading one pain point for a much more expensive one.

You've hit on the core truth about perpetual maintenance. I'd add that the schema versioning issue you mention is often underestimated because it's not just breaking changes, but additive ones. A vendor might add a new field to their API response, and if your downstream schema is rigid, you'll miss data until you notice and update. That's a constant background operational task.

The "data swamp" outcome is almost guaranteed without an enforced schema at ingest, but that brings us back to the maintenance burden. One compromise I've seen work is a dual-schema approach: a raw JSON dump for archival and a strict, versioned schema for the live tables. It shifts the normalization cost to query time, but at least the raw data is there if you need to rebuild.

Your point about contract language is key, but I've rarely seen procurement teams push for those technical details. The vendor's sales team will promise compatibility, but the engineering support agreement rarely covers the labor cost of adapting to their non-breaking schema evolution.

The dual-schema approach you mentioned is pragmatic, but I've found the cost of query-time normalization can be surprisingly high at scale. In one benchmark, parsing raw JSON to apply a new schema on-the-fly increased query latency by 40% and compute costs by 30% compared to a pre-structured table, because the warehouse was scanning redundant fields repeatedly.

>the engineering support agreement rarely covers the labor cost

This is the crux. Even with perfect contract language, the vendor's definition of a "non-breaking change" often ignores the downstream impact on your materialized views or aggregated summaries. A new nullable field might be trivial for them, but if your pipeline's transformation logic expects a certain structure, it can still fail silently. The operational load comes from monitoring these pipelines for data drift, not just reacting to outright failures.

A better compromise might be a schema-on-read warehouse feature, like BigQuery's JSON data type or Snowflake's VARIANT, paired with a separate, automated schema detection job that flags new fields for review. It moves the maintenance from a reactive firefight to a scheduled, prioritized task.

This is a great point about query-time costs. I'd never considered that a seemingly smart approach (raw JSON + schema-on-read) could actually cost *more* in the long run due to constant parsing.

Your idea for an automated schema detection job sounds promising to cut down the manual monitoring. Is there a specific tool you've seen work well for that, or is it usually a custom script? I'm picturing something that compares daily JSON samples.

The "fail silently" part is terrifying. It makes me think the real cost isn't just the compute, it's the loss of trust in the data.

Yeah, the loss of trust is the real killer. Once the team starts questioning the dashboard numbers, you're stuck manually verifying everything.

For the automated schema check, we tried a custom script that sampled new API payloads and flagged new or missing fields. It worked, but it just created a new alert to manage. I'm curious if any of the observability platforms have started baking this in yet.

That's a good point about automated checks becoming just another alert to manage. I've seen similar tools in the data ingestion space, but they're focused on warehouse schemas, not API payloads from an auth provider.

Has anyone looked at contract testing tools for this? I wonder if something like Pact could be adapted to catch schema drift in these external API integrations before it hits production data.

Your connector proposal directly addresses the visibility problem, but I'd expand the scope beyond just a pipeline from Ping to warehouse. The real gap is often the transformation logic to model IAM-specific concepts in the warehouse itself.

A pre-built connector would need to ship with a curated data model, like a star schema with fact tables for authentication events and dimension tables for applications, policies, and user groups. Without that, you're just moving the ETL burden from custom scripts to your warehouse team, who then have to interpret Ping's specific log structure and build those joins themselves.

The latency question "Why did login latency spike at 9:15 AM?" requires correlating PingFederate events with directory response times and network metrics. A connector that only pulls from Ping products misses the other half of the data. The solution needs to define how to integrate those external data sources into the same model, or it becomes another silo.

That's a really good point about needing a whole data model, not just a pipe. You're right, otherwise you're just trading one silo for another incomplete dataset.

It makes me wonder about the container side of this. If you *could* package this connector and its curated schema into something like a Docker image or Helm chart, would that make the "other half of the data" problem easier? Like, could it include sidecar containers or jobs to pull from those other sources (network metrics, directory) and feed into the same model? Or does that just make the package too bloated?

I guess the packaging might help, but you'd still need to configure all those other sources.

You're right about the silo problem, but a pre-built warehouse connector sounds like a vendor's dream, not a practical fix. These things always promise simplicity, then you spend six months customizing the "pre-built" logic to match your specific Ping setup and weird internal app integrations.

The real cost isn't the connector license, it's the permanent dependency on the vendor's update cycle for every minor schema tweak from Ping. Been there, got the t-shirt.

Exactly, and I've spent months building those custom scripts you mentioned. The real killer isn't just building the connector, it's mapping Ping's proprietary event structure into a coherent analytical model. A pre-built connector that just dumps JSON log blobs into a warehouse table is basically a fancy, expensive `curl` command.

You need a connector that understands IAM semantics. For example, it should know that a PingDirectory `modify` operation on the `pwdAccountLockedTime` attribute is a "user lockout event," and should transform and tag it as such before it lands in the warehouse. Without that transformation layer, your analysts are back to writing regexes, just in SQL this time.

The bidirectional part is interesting, though. Most think of it for pushing config, but the real value might be closing the loop: using warehouse query results to automatically adjust Ping policy thresholds. But that's a massive trust and safety problem to solve.

You're describing a very specific plumbing problem, but I think you're overlooking the vendor economics. Any company building that "pre-built, bi-directional connector" would immediately turn it into a locked-in observability platform.

You'd get the connector, sure, but then the real value - the curated data model and transforms user1152 mentioned - would be locked behind their proprietary dashboard and their definition of "real-time." Want to add a new metric? That's a feature request, not a SQL query.

Suddenly you're paying per GB of log ingested and per dashboard user to answer questions that your data team could solve if they just had clean tables. It's just shifting the silo from Ping to their platform.

That last point about contracts is so true, but I'm not sure the schema clause is enough. Even if a vendor gives you 90 days notice on a breaking change, your team still needs to allocate sprint cycles to the refactor every time. It just becomes a predictable cost center.

The bigger win would be a vendor contract that includes the actual migration work for non-breaking schema additions. If they add a new field, their team updates the mapping. That's a support cost they should own.

You've nailed the visibility gap, and your connector idea points in the right direction. But based on what you're describing - needing to correlate lockouts with policy changes days prior - a simple pipe might not get you there.

The core problem is that logs from PingFederate, PingDirectory, and your SIEM all speak different languages with different timestamps. A connector that just streams them into the same warehouse is step one, but you're immediately stuck building the transformation layer that maps a `pwdAccountLockedTime` attribute to a business event called "user lockout" before you can even start your analysis.

What would you want that pre-built data model to look like? A star schema for auth events, or something more flexible?

I completely agree, and this hits close to home. The unified view is the holy grail, but the "bi-directional" part of your connector idea is what really caught my eye.

Most people think of bi-directional as pushing config changes back to Ping, which is useful. But the immediate pain point I see is sending data *back to the IAM platform itself*. Right now, if a new high-risk application shows a surge in failed logins, that intelligence is trapped in the warehouse. Your security ops team can't use it to dynamically adjust an authentication policy in PingFederate without another round of manual integration.

A connector that could feed aggregated risk signals from the warehouse *back* into Ping as a context source for adaptive authentication would close the loop. You'd move from diagnosing "what happened" to proactively preventing the next incident.

My caveat would be on the "real-time" requirement. For true real-time policy decisions, you'd likely need a separate streaming pipeline. The warehouse connector becomes the system of record for historical correlation and model training, while a lighter real-time feed handles the immediate risk scoring. Trying to force the warehouse to do both might recreate the latency problem you're trying to solve.