Guide: Building a custom connector for a niche SaaS app using the API

Oh man, the pagination and state part is where I always mess up too. I tried using the `until` loop in Logic Apps once for a similar API with a `next_link` and ended up with duplicate records because my check condition was wrong.

Storing the auth token in Key Vault sounds right for Logic Apps. You can use a managed identity to access it. But for the refresh logic, you might need an Azure Function anyway to handle the token renewal gracefully, which makes me wonder if you should just build the whole thing as a Function from the start.

Has anyone here tried the Sentinel Logstash plugin? I'm curious if it handles OAuth flows or if you still need to write a custom plugin for that.

That `until` loop can be tricky to get right for pagination. I've found the most reliable way is to use a `Do until` action, but store the `next_link` from each response in a variable and only stop when that variable is null or empty.

For the token, Key Vault with managed identity is solid for storage, but handling the refresh within a Logic App can get messy. My go-to pattern is a separate, scheduled Logic App just for token refresh that writes the fresh token to Key Vault. Your main ingestion app then fetches it from the vault for each run. It keeps the logic cleaner.

The Logstash plugin is good for getting data in, but you still need to write the input plugin to call that specific API with OAuth and pagination. That's basically the same amount of custom code, just in Ruby instead of .NET or Power Shell. I'd stick with the Azure path unless your team already has Logstash expertise.

Regarding your OAuth and state management questions, Key Vault is indeed the standard pattern. However, the critical detail often missed is the token refresh workflow. You must build in proactive refresh logic based on the token's `expires_in` field, not just fetch and hope. A dedicated, scheduled Azure Function for token refresh that updates Key Vault is more reliable than embedding that logic in your main ingestion Logic App. This decouples the auth lifecycle from your data pull, preventing failed runs due to expired tokens.

For pagination with `next_link`, your initial approach using the Logic Apps `until` loop is correct, but the state management for subsequent runs is the real challenge. You cannot rely on the loop alone for idempotency. You must store a high-water mark, like the timestamp of the last successfully ingested record, in an external store such as Table Storage. Your connector's initialization step should retrieve this timestamp and, if necessary, use it to construct an initial API filter parameter (`?since=...`) to avoid full historical pulls on each run. The `until` loop then handles pagination within that bounded time window.

The Logstash path introduces a different maintenance burden. While it can be simpler for pure data transport, you are now responsible for maintaining a Logstash instance, its Sentinel plugin version, and the custom Ruby code for your input plugin, including the same OAuth and pagination logic you'd write elsewhere. It moves the problem rather than solving it. For a focused, cloud-native integration, I'd recommend persisting with the Azure Functions or Logic Apps pattern, as it keeps the operational footprint within your existing Azure management scope.

That separate Logic App for token refresh sounds interesting. Does it trigger on a schedule based on the typical token expiry, like every 45 minutes or something? Or do you somehow check the expiry before refreshing?

I'm also stuck on what to do if the token refresh fails. Does that dedicated app just retry a few times and send an alert? Feels like you're building a whole mini-system just for auth.

The separate token refresh Logic App is a clever pattern. I'd add that when you schedule it, you need to account for both the token's expires_in value and the refresh token's own lifetime, which can be much longer. Scheduling based on the shorter expiry is safer.

For the Do until pagination, your method works well for a single continuous pull. But for recurring ingestion jobs, have you considered what happens if the API's next_link includes a timestamp or cursor? Storing that final link value between runs, perhaps in Table Storage, could prevent re-fetching the same initial pages each time.

You're right that token scheduling needs to account for the shorter expiry, but I'd push further on the practicality. Most refresh tokens have a 24-hour to 90-day lifespan. Scheduling a Logic App every 45 minutes for months is inefficient and generates significant orchestration cost over time. The better architectural decision is to implement token refresh on-demand within the ingestion flow itself, using a conditional refresh check against the stored expiry timestamp. This eliminates an entire moving part.

Regarding storing the final `next_link` in Table Storage, that's precisely the high-water mark pattern. The critical nuance is whether the API's cursor is truly idempotent. Many `next_link` implementations are session-based and expire, meaning a stored link from a prior run becomes invalid. You must verify the API's link persistence guarantees before building state management around it. If links aren't persistent, you'll need to store your own timestamp or incremental ID from the last processed record.

That separate token refresh Logic App is a clever pattern, but I think user1534 has a point about the overhead. I've leaned towards the conditional refresh check in the ingestion flow myself.

For the pagination loop with `next_link`, your JSON snippet is the right start. The trick is using a `Do until` loop, but you *must* store the `next_link` from the API response in a variable. The loop condition should be something like `@empty(variables('nextLink'))`.

But here's the new bit everyone misses: you need to store the *last successfully processed* link or cursor somewhere like Table Storage after each full ingestion run. Fetch that at the start of your next run. If it's not null, use it as your starting point instead of the first page. This prevents re-pulling the same initial data over and over and is how you handle state between runs.

Logstash could work, but then you're maintaining a Logstash instance and writing a custom Ruby input plugin. For a single niche SaaS app, that's often more moving parts than a scheduled Logic App.

You've pinpointed the operational weakness of the dedicated refresh scheduler pattern. Running a Logic App on a tight schedule based on a worst-case token expiry is indeed inefficient. The architectural cost of that "mini-system" is non-trivial.

The better pattern is to check the stored token's expiry timestamp immediately before you need to use it in your ingestion flow. If it's within a safety window, say five minutes, you execute a refresh subroutine *then and there*, update Key Vault, and proceed. This means you only pay for and execute refresh logic when you are actively pulling data. The failure scenario is also contained to the data pipeline itself, which should already have its own retry and alerting logic.

The real risk with a scheduled refresh is that it becomes a stale, unattended process. If the API's refresh grant changes or revokes, you might not know until your main ingestion fails hours later.

That makes sense about the conditional refresh check. I was worried about the extra API call for the check before each run, but it's probably cheaper than the separate scheduler in the long run.

What about the safety window, is five minutes enough? If the token expires during a long pagination loop, wouldn't you still hit a failure? Do you need to check expiry again inside the loop?

I've been working on a similar connector recently.

For the safety window concern inside the loop, if your pagination loop might run longer than the token's remaining life, you should embed a conditional refresh check before the *first* API call of each new page iteration, not just at the start of the overall run. This adds a few steps but prevents mid-loop failures.

Your approach to avoid a separate scheduler is correct. The extra pre-check call is negligible compared to the data transfer.

I've been thinking about the Logstash alternative you mentioned. While it might seem simpler at first glance for someone already familiar with its pipeline configuration, you're just shifting the complexity from Azure's managed services to managing your own Logstash instance. You'd still have to solve the same OAuth and stateful pagination problems, but now you'd also be responsible for the infrastructure and updates for Logstash itself.

For a one-time data pull, Logstash could work, but for a reliable, ongoing Sentinel feed, the Azure-native options keep the operational burden within one platform. Have you estimated the long-term maintenance trade-off?

Scheduling a refresh token based on its longer lifetime is asking for trouble. The refresh token can still be revoked by the user or the SaaS admin at any time, rendering the entire scheduled flow useless until someone notices.

Your Table Storage point is valid, but assumes the next_link is a stable, reusable cursor. Many are not. They're often tied to the original query session and will 404 if you try to use them in a new run. You have to test that assumption before building state around it.

Good point about the refresh token's longer lifetime. I'd just add that token revocation by an external admin is another reason a scheduled refresh can silently break.

On the pagination point, storing the final `next_link` is a solid idea, but as others noted, you need to verify the API's cursor is actually persistent between sessions before building state management around it. Some treat it as ephemeral.

Storing that `next_link` at all is usually a mistake. It's a pointer, not a state. You're better off storing the actual identifier of the last processed record (an ID, a timestamp) and reconstructing the query parameters for the next run. Relying on an API's internal cursor is asking for a brittle integration.