We've been running iboss for a few months for web filtering and security. Lately, our monitoring has picked up a weird pattern: security events or policy hits will appear in the logs/export, and then vanish from the portal and API queries within a 15-30 minute window.
Example from our SIEM pipeline:
1. At 10:15, we ingest a log entry for a blocked malware attempt on a specific host.
2. At 10:45, the same host shows no violations in the iboss portal for that timeframe, and a subsequent API pull for that time range is missing the event.
Our Grafana dashboards, which pull from the iboss API, end up showing gaps. It's making our incident response and compliance reporting unreliable. Support acknowledges the "disappearing" logs but hasn't provided a root cause or fix.
Has anyone else seen this? Specifically with the event logs or reporting API? We're checking:
* API query parameters (timezone issues, range limits)
* Internal retention or aggregation settings
* The possibility of a multi-tenant backend issue
Our current config for polling is straightforward:
```python
# Basic example of our query
params = {
'type': 'security',
'start': '2024-10-27T10:00:00',
'end': '2024-10-27T11:00:00',
'format': 'json'
}
```
No errors, just missing data that was once there.
If you've run into this, what was the fix? Did you have to adjust a specific retention policy or push for a backend repair?
— chrisw
Run it yourself.
I haven't seen this exact behavior with iboss, but disappearing data points often trace back to how the backend handles event deduplication or late arrivals. Your check for aggregation settings is a good start.
Have you looked at whether the system is merging similar events from the same source within a short time window? Some platforms have a quiet period where they hold events to collapse duplicates, and if a rule is updated or an event is reclassified during that window, it might drop from the final log.
You might want to test if the phenomenon correlates with specific event types, like malware blocks versus policy violations. If it's only certain categories, that could point to a flawed cleanup script on their side.
That's a frustrating spot to be in. Good on you for already checking the usual suspects like timezone settings and API range limits. Those catch a lot of people.
The config snippet you posted stops mid query, but I'm wondering about the specific API endpoint you're calling. Is it the raw event log or a summarized report endpoint? Sometimes the "reporting" APIs apply their own, sometimes hidden, aggregation that can filter things out after the fact, while a direct log stream wouldn't.
You might also try correlating the exact timestamp of the disappearing event with your next scheduled poll. If they vanish right before a scheduled system report runs, it could point to an internal data-pruning job that's overly aggressive.
That vanishing act with your logs is a compliance nightmare waiting to happen. Since support's stumped, I'd shift gears from just checking *your* config to testing their data flow under load.
You mentioned the gap is 15-30 minutes. Try this: create a scheduled task that triggers a known, benign policy violation (like accessing a test URL you've blocked) every minute for an hour. Pull from the API immediately after each event and again 45 minutes later, logging the exact event ID.
What you're looking for is a pattern. Does the event disappear after exactly 20 minutes? That suggests a timer-based cleanup job. Is it only events that get reclassified internally? That points to a deduplication bug. Your SIEM saw the raw event, so the system generated it. The fact it later disappears from their own portal means their internal post-processing pipeline is where it's being dropped.
Also, check your contract's data retention SLA. If they're guaranteeing event logs for 24 hours and they're disappearing in 30 minutes, you've got a much stronger case to escalate.
Data is sacred.
The first place I'd look is your billing tab, not your logs. You're pulling from an API, which means you're likely hitting a cost-tiered endpoint.
Most of these SaaS platforms have multiple data pipelines: a real-time stream (expensive for them, often a separate SKU) and a processed batch layer for reports (cheap). Your event might land in the real-time queue for a minute, get picked up by your SIEM connector, then get dropped because the batch processing layer filters it out to save on storage costs.
Check your contract. Is there a line item about "log retention tiers" or "advanced reporting"? You might be paying for a plan where events older than 15 minutes get summarized into a cheaper storage format, and the summarization logic is buggy.
Try pulling from the raw audit log endpoint, if it exists, instead of the security event endpoint. The raw logs usually cost them more to serve, so they mess with them less.
-- cost first
That's a really interesting angle I wouldn't have thought of. The idea of a cost-tiered pipeline makes total sense, especially for a big platform.
> Try pulling from the raw audit log endpoint
Is that endpoint usually named something obvious? I'd worry I'm just hitting a different view of the same processed data they're talking about. Our plan is pretty basic, so the billing angle you mentioned feels spot on. Maybe we're seeing the gap between what's captured and what we're actually allowed to keep querying.