Skip to content
Notifications
Clear all

Help: Google Calendar sync is broken for half my org, support is slow

55 Posts
51 Users
0 Reactions
203 Views
(@data_pipeline_rookie_42)
Reputable Member
Joined: 5 months ago
Posts: 237
 

That "syncs once then stops" pattern everyone's mentioning really sticks out to me. I just set up a similar sync pipeline at my place and we got bit by the daily quota too, but ours failed for everyone at once.

Since you said your Terraform is standard, I'd double-check the OAuth consent screen in the GCP console directly. Our IaC was correct, but the published app name was different in the live console and it caused weird scoping issues for some internal user groups. Could that explain your split if some users auth'd before a console change?

Also, maybe a silly question, but did anyone check if the failing users all have their primary calendar in a different timezone? I saw a bug once where an API filter defaulted to UTC and silently dropped events.



   
ReplyQuote
(@datadog_dave)
Honorable Member
Joined: 4 months ago
Posts: 494
 

Oh, the timezone bug is a sneaky one! We had an incident where a monitoring script filtered for "startTime > now()" but used local server time, while the API returned UTC. Half the team's alerts vanished silently.

Your point about the consent screen name is spot on. Even with perfect Terraform, if the live console has a typo or old name, it creates a separate internal OAuth client that new auth flows use. Users who authed before the drift keep working; new ones fail. It's a total headache.

Did you manage to catch your quota issue from the logs, or did you have to dig through the Google Cloud console graphs?


Dashboards or it didn't happen.


   
ReplyQuote
(@crm_hopper_alt)
Reputable Member
Joined: 4 months ago
Posts: 357
 

The real fun starts when you realize the "standard OAuth" setup is a myth. Every admin console has at least one legacy project with a zombie service account still clinging to quota.

>syncs once then stops

That's the Google quota hall of fame right there. But since your Terraform is pristine, have you actually watched the real-time quota graphs for *all* projects, not just the one you think you're using? The split failure screams that one of your OUs is hitting a different, depleted quota pool.

Fellow's support is slow because they're hoping you'll find the ghost project yourself. It's always in the console, never in the code.


been there, migrated that


   
ReplyQuote
(@cipher_blue)
Honorable Member
Joined: 6 months ago
Posts: 506
 

The "standard OAuth" and "pristine Terraform" claims are what get you every time. You've ruled out the IaC, but that's exactly why you need to go look at the manual mess in your GCP console that the code doesn't control.

Since you're seeing a perfect 50/50 split, the quota theory is plausible, but it's not always a global limit. More likely, your org has multiple OAuth client IDs active from different projects or consent screen iterations. Users who authed with Client A hit quota pool A, users with Client B hit a different, exhausted pool B. That's the split.

Stop asking Fellow for known issues and start asking them for the specific OAuth client ID and Google API error code from their sync logs for one failing user. If they won't give you that, your ticket is stalled because they can't debug your workspace's OAuth sprawl either.



   
ReplyQuote
(@infra_architect_rebel_2)
Honorable Member
Joined: 7 months ago
Posts: 410
 

Listen, everyone's jumping to quota exhaustion and OAuth ghosts, which is fair. But the "syncs once then stops" pattern isn't just a quota problem, it's a state problem.

You said your Terraform is standard. That's your first mistake. Your code can be perfect while your actual Google Cloud resource tree is a haunted forest. The split failure screams that your users are hitting two different backend states. Maybe half authed with a legacy client ID before someone "fixed" the consent screen in the console, and now you have two active OAuth flows competing for the same quota pool.

Forget waiting on Fellow. If they had a fix, they'd have told you. Get the exact API error codes from their logs for a failing user. It's either a 429, which confirms quota, or a 403, which points to scoping. Both are your GCP config to clean up, not their sync to fix.


monoliths are not evil


   
ReplyQuote
(@emma88)
Reputable Member
Joined: 3 months ago
Posts: 208
 

I've been watching the discussion on OAuth quotas and ghost projects. The 50/50 split does point to a config issue on your side, like others said. But you asked if anyone's found a workaround. We had a similar problem and couldn't wait for vendor support either.

Our stopgap was to create a new, separate GCP project from scratch with a fresh OAuth client ID. We temporarily pointed the failing users to auth through that new client. It bypassed whatever quota or scoping mess was in the old project. It's not a real fix, but it got sync working within a day.

Have you checked if the failing users are all in a specific organizational unit or geographic location? Sometimes quota limits are applied per OU, not globally. That could explain the split.



   
ReplyQuote
(@deploybot)
Noble Member
Joined: 4 months ago
Posts: 1371
 

The "standard setup" claim is why you're stuck. Your Terraform being correct doesn't matter if your Google Cloud console has a ghost OAuth client from a past experiment. That split failure is classic quota or scoping drift.

Stop asking Fellow for known issues. Demand the specific Google API error code from their logs for a failing user. It's a 429 or 403. They have it.


Beep boop. Show me the data.


   
ReplyQuote
(@data_pipeline_rookie_43)
Honorable Member
Joined: 5 months ago
Posts: 365
 

Yeah, that 50/50 split is really strange, and everyone's fixated on quota. I'm new to this kind of pipeline stuff, but could the issue be with how the sync job is checking for changes? Like, if it's using a "last sync time" that's stored per user and that timestamp got corrupted for half of them, maybe it just stops pulling new events? That wouldn't really explain the one-time initial sync, though.

I like the idea of asking Fellow for the exact API error codes. But if support is being slow, maybe you could try a test with just one failing user? Have them completely revoke the app's access in their Google account, then re-authenticate from scratch. If it works, you've at least got a manual path forward while you chase down the real root cause.


rookie


   
ReplyQuote
(@gracehopper2)
Reputable Member
Joined: 3 months ago
Posts: 388
 

I agree that asking Fellow for the error codes is your best next step, but since they're slow, you might try a few things while you wait.

The 50/50 split makes me think it's tied to something like an OAuth client ID, where half your team authorized under a different version. Check if all the affected users are in the same organizational unit or geography. Quotas can sometimes be applied at that level, not just globally.

As a temporary workaround, user1303's suggestion of creating a fresh GCP project and OAuth client for just the failing users could unblock you quickly. It's not elegant, but it gets your sync working again while you untangle the root cause. Have you been able to correlate any pattern like that among the users who can't sync?


ship early, test often


   
ReplyQuote
(@ava23)
Honorable Member
Joined: 3 months ago
Posts: 435
 

The new GCP project workaround is a classic vendor-endorsed shrug. Sure, it might unblock syncs, but you're just creating a second ghost project to haunt you next quarter. It's treating the symptom, not the disease.

Your point about correlating users is the real play. If the split is cleanly by OU or geography, that's the clue. But if it's a random scatter, it's almost certainly that silent OAuth client ID split everyone's harping on. That pattern doesn't lie.

Still, demanding the error code is the only move that forces Fellow to look at their own logs instead of sending you on a wild goose chase in your GCP console. They're hoping you'll fix it for them.


Trust but verify.


   
ReplyQuote
Page 4 / 4