Alright, let's see if the collective wisdom here can spot what I'm missing. I'm evaluating OpenPipe to replace a gnarly old Airflow DAG that ingests from a Google Form into our data warehouse. The old pipeline uses the Sheets API and works, but I was promised "effortless event-driven workflows." So far, the effort is substantial and the driving is not happening.
I've set up a source connection to a Google Form, configured to watch for new submissions. According to the docs, this should emit an event that triggers my downstream pipeline—a simple transform and load into a PostgreSQL staging table. The source connection tests successfully, the destination pipeline runs fine when I trigger it manually with sample data. But when an actual form submission rolls in? Nothing. No trigger, no logs, no sign of life in the OpenPipe UI except the source showing as "active."
Here's the relevant chunk of my pipeline definition YAML. I've anonymized IDs and names.
```yaml
source:
type: google_forms
connection_id: conn_abc123
config:
form_id: "1A2B3C4D5E6F"
trigger_type: "on_new_submission"
pipeline:
- name: transform_responses
action: sql_transform
query: |
SELECT
response_id,
timestamp,
JSONB_PRETTY(response_data) as structured_data
FROM {{ input_table }}
- name: load_to_staging
action: postgres_load
target_table: raw.form_submissions
mode: append
```
I've checked the obvious:
* The form is public and accepting responses.
* The service account used in the OpenPipe connection has view permissions on the form and its linked spreadsheet.
* The event history in the OpenPipe admin panel shows no activity for this source, ever.
My suspicion is either a permissions nuance with Google Workspace (we use a custom domain) or I've misunderstood the trigger mechanism. Does OpenPipe poll the form, or does it rely on a push webhook from Google? The documentation is... optimistic in its assumptions.
Has anyone actually gotten this Google Forms source to trigger reliably? What's the magic incantation? I'm half-tempted to revert to a cron job and call it a day, but I'd like to give this "modern" tool a fair shake.
-- old salt