Skip to content
Notifications
Clear all

Check out my dashboard for tracking Cato tunnel health via their API.

25 Posts
25 Users
0 Reactions
109 Views
(@freddiem)
Reputable Member
Joined: 4 months ago
Posts: 295
Topic starter   [#27053]

Hey folks, I've been using Cato Networks for about a year now to handle our SASE needs, and while the portal is great, I wanted a more real-time, at-a-glance view of our tunnel health. I figured others might find this useful, so I built a simple dashboard using their API.

It pulls key metrics every 5 minutes and pushes them to a Grafana instance. The main things I track are tunnel status, latency, and packet loss for our critical locations. Here's the core Python snippet I use for the API call:

```python
import requests
import json

url = "https://.catonetworks.com/api/v1/tunnels"
headers = {
"Authorization": "CatoToken "
}
response = requests.get(url, headers=headers)
tunnel_data = response.json()

# Then I parse and extract: tunnel['name'], tunnel['status'], tunnel['latency'], tunnel['packet_loss']
```

I format the data and send it to a time-series DB. The dashboard gives me a single pane of glass and has been super helpful for spotting brownouts before they become full outages.

You'll need an API token with read permissions for tunnels from the Cato management console. The API docs are pretty straightforward. If you're already using something like Zapier or a monitoring tool that accepts webhooks, you could easily adapt this to trigger alerts.

Has anyone else built something similar? I'm curious about other metrics worth tracking or if there are clever ways to handle the alerting side.

hth



   
Quote
(@devops_grandad)
Reputable Member
Joined: 6 months ago
Posts: 354
 

Good start on the automation. Five minutes is fine for a first pass, but for spotting actual brownouts, you'll want to consider sub-minute polling for those critical tunnels. The difference between a 30-second spike and a 5-minute average can be the difference between a blip and a user revolt.

I hope you're storing that API token securely, not baking it into the script. A vault or at least an environment variable is the bare minimum. And how are you handling API failures or schema changes? That script will break silently if the response format shifts. I'd wrap the requests call in something with a dead-man's switch, maybe send a metric on the script's own health.

You're also just pulling tunnels. Don't forget to correlate with the events API endpoint. A tunnel might show as up but have a flapping event log that tells a different story. That's the data that usually explains the weird latency graphs.



   
ReplyQuote
(@ci_cd_enthusiast)
Honorable Member
Joined: 9 months ago
Posts: 382
 

Nice! I love seeing these homegrown monitoring solutions. That 5-minute interval is a solid baseline.

If you're already in Grafana, you might want to set up an alert rule based on the rate of change for latency or packet loss, not just absolute thresholds. A sudden 10ms jump on a stable tunnel can be a better early warning sign than a "high" value.

What are you using for the time-series backend? I've had good luck pairing similar scripts with InfluxDB's Telegraf agent, since it handles the batching and retries for you.


Pipeline Pilot


   
ReplyQuote
(@hannahr2)
Reputable Member
Joined: 4 months ago
Posts: 233
 

Absolutely love the suggestion about the rate of change for latency, that's a fantastic way to spot issues before users even notice. I set up something similar using the `derivative()` function in my alerts and it catches weird routing hops our ISP makes.

On the backend, I'm using Prometheus for this because our whole stack is already instrumented with it, so I just have the script expose a simple `/metrics` endpoint. But you're right, Telegraf is perfect for this use case - I use it for pulling from other cloud APIs and it really does simplify the "plumbing" so you can focus on the logic.

Have you found a sweet spot for what rate-of-change threshold works? I'm still tweaking mine between 8ms and 15ms depending on the site's baseline.


Measure twice, automate once.


   
ReplyQuote
(@georgep)
Reputable Member
Joined: 4 months ago
Posts: 298
 

Rate of change is clever, but it's still a reactive metric. By the time you see a 10ms jump, the routing change or congestion has already happened.

That five minute polling window means you're already minutes behind. The real question is whether you can get ahead of it with any telemetry Cato provides, which I doubt. You're just monitoring symptoms.

Telegraf handles batching, but it doesn't fix the core problem of pulling from an API you don't control. What's your plan when Cato's API goes down or throttles you during an incident? You'll have a gap right when you need the data most.


— geo


   
ReplyQuote
(@davidn)
Reputable Member
Joined: 4 months ago
Posts: 305
 

Completely agree on correlating with the events API. I built a separate collector for it and found that correlating tunnel metrics with even-level events was the only way to diagnose intermittent packet loss that wasn't reflected in the basic tunnel status. The events endpoint often shows upstream carrier changes minutes before latency graphs show impact.

Your point about a dead-man's switch for script health is critical. I route my script's own execution time and API response codes as separate metrics to the same time-series database. That way, a Grafana alert can fire if the metric collection stops, which has caught both token expiration and unexpected API throttling.

Have you found a reliable way to map specific event types from their API to tunnel performance degradation, or are you just flagging the presence of any event?


Measure twice, buy once.


   
ReplyQuote
(@crm_surfer_99)
Honorable Member
Joined: 6 months ago
Posts: 424
 

Correlating with the events endpoint is smart. The mapping, though, is the real challenge. Their event descriptions can be vague.

You're flagging any event, which works for an alert, but for root cause you need to categorize them. I set up a lookup table mapping their event type strings to internal severity levels. "Carrier Change" becomes a warning. "Tunnel State Change" is critical. Without that, you just get noise.

Your dead-man's switch for script health is good, but it's still after the fact. What's your play when the metrics stop? A five minute gap in polling during an outage means you're blind.


Your CRM is lying to you.


   
ReplyQuote
(@danielf)
Reputable Member
Joined: 4 months ago
Posts: 473
 

That's a solid foundation for getting better visibility. Many of us have started with a script just like this.

A key next step is adding some basic logging to that requests call. Log the response code and the time it takes. If you pipe those logs to your monitoring stack, you'll have an early warning for when the API itself becomes slow or starts to fail, which can happen independently of your tunnel health.

When you're ready to expand, as others have mentioned, pulling from the events API in parallel will give you the context you need to understand why a metric changed, not just that it did.


—daniel


   
ReplyQuote
(@harperk)
Honorable Member
Joined: 4 months ago
Posts: 537
 

Nice to see someone actually doing something useful with the API instead of just complaining about the portal. That single pane of glass is a game changer once you've had it for a week and then have to go back to the vendor console for something.

I'm with user423 on the script fragility, though. My addition: don't just log failures, but log successes too. Track the response time of that API call as a metric itself. When Cato's backend starts to slow down under load, you'll see your own collection latency creep up minutes or hours before your tunnel metrics show any issue. It's a free leading indicator.

And since you're polling, what's your strategy for alerting on the data gap when your token inevitably rolls? I've seen more than one dashboard turn into a painting on the wall because the script failed silently for a week.


Data over dogma.


   
ReplyQuote
(@benjislack)
Reputable Member
Joined: 4 months ago
Posts: 244
 

Tracking API response time is the only useful part of this. It's still just monitoring their monitoring. You're still blind if their API or auth borks.

> what's your strategy for alerting on the data gap when your token inevitably rolls?

Exactly. Nobody ever does this. They build the Rube Goldberg machine to parse the data but forget to wire up the "is this machine even on" alarm. A token expiry will kill it for weeks.


your mileage will vary


   
ReplyQuote
(@danielj)
Reputable Member
Joined: 4 months ago
Posts: 254
 

That's a really good callout. You're right, the token roll is a single point of failure that'll leave you with a silent, dead script.

My hack for this was to make the script's own runtime a metric it sends. I log the epoch timestamp of each successful run to the same TSDB. My alert rule isn't on the Cato data, it's on the *absence* of that heartbeat metric for, say, 7 minutes. That way, a broken token (or a crashed container, or a network blip) triggers a page.

It's not perfect, but it's stopped me from staring at a flatlined graph a few times.


spreadsheet ninja


   
ReplyQuote
(@gregoryt)
Reputable Member
Joined: 4 months ago
Posts: 418
 

I hadn't thought about logging the API response time itself, that's a good angle. It makes sense that if Cato's backend gets sluggish, my collector's latency would be a leading indicator.

I'm curious, where do you pipe those logs? To something like Loki, or just your standard syslog? I'm trying to keep my setup simple.



   
ReplyQuote
(@emilyf)
Reputable Member
Joined: 4 months ago
Posts: 227
 

Good point about using the API response time itself as a leading indicator. I hadn't thought of that.

Where are you sending that metric to track it, back into the same time-series database? I'm curious if you're graphing it alongside the tunnel latency or keeping it separate.



   
ReplyQuote
(@emilyl)
Honorable Member
Joined: 4 months ago
Posts: 527
 

> The difference between a 30-second spike and a 5-minute average can be the difference between a blip and a user revolt.

That's a really good point I hadn't considered at all. I was just happy to get *any* data onto a graph. If a tunnel has a quick wobble at the 5-minute mark, my current setup would average it right out, wouldn't it? Makes me wonder how other people even decide which tunnels are "critical" enough to poll that fast.

And the dead-man's switch idea for script health is brilliant. I'm still just running my script manually sometimes, so if it fails, I'd have no clue until I checked. What do you use for the switch? Like, a simple uptime check that pings another service?



   
ReplyQuote
(@ci_cd_plumber_99)
Honorable Member
Joined: 8 months ago
Posts: 426
 

Five minutes is practically geological time for a real-time dashboard. You're going to miss every interesting spike that recovers before your next scrape. You can't call it 'real-time' if your sampling rate is that slow.

What's your criteria for a 'critical' location? You'll burn through API rate limits fast if you poll everything that often, but if you're only picking a handful, you need to define why. Is it based on user count, revenue impact, or just gut feel? Without that, you're just graphing stuff and calling it critical.

Also, hardcode that token in a script and I guarantee you'll be the last person to know when it expires. The script will fail silently and your single pane of glass will just be showing yesterday's weather.


Speed up your build


   
ReplyQuote
Page 1 / 2