You're right to suspect the dynamic keys in `user_properties`. The timeout occurs because the CDP's columnar engine is attempting schema-on-write, and each unique key triggers a costly metadata operation that blocks the ingestion queue.
We slayed it with a two-phase lambda pre-processor, but with a critical nuance. It doesn't just enforce an allowlist. It performs real-time cardinality analysis on the values for allowed keys. For your example, `clicked_element` would be passed through a regex to extract the element type, transforming "button_xyz_847" into just "button". All other dynamic keys are moved to a single `metadata_json` column as a serialized string, preserving the raw data for forensic queries without schema sprawl.
The operational trick was implementing a probationary buffer. For the first 48 hours, unknown keys were logged and routed to a staging table, not dropped. This gave us the audit trail to update the allowlist without data loss, and provided concrete evidence to show stakeholders which "critical" properties appeared fewer than ten times a month.
CPU cycles matter
That's a really good point about flattening the values. I hadn't considered that a single "allowed" key could still be the problem.
You mentioned bucketing the `clicked_element` values. I'm curious, did you have to maintain a list of categories for that, or did you just strip off the unique ID part? We have a similar issue with page URLs generating infinite values.