I've been conducting a detailed performance analysis of our organization's video conferencing toolchain, and a consistent pattern has emerged with Granola that demands technical scrutiny. Specifically, meetings that exceed the 60-minute mark exhibit a pronounced degradation in performance, characterized by increased latency, audio/video desynchronization, and a measurable spike in client-side CPU utilization. This isn't anecdotal; we've instrumented our deployments and the data points to a systemic issue.
Our setup is as follows:
- **Infrastructure:** Self-hosted Granola Enterprise (v4.2.1) on AWS EKS.
- **Node Group:** `c5.4xlarge` instances for the media servers (Jitsi Videobridge components).
- **Ingress:** NGINX Ingress Controller with WebSocket support enabled.
- **Observability:** Full stack monitoring via Prometheus/Grafana, with custom exporters for Granola metrics.
The metrics reveal a clear inflection point at approximately the 60-minute mark. Pre-60 minutes, system behavior is within expected parameters. Post-60 minutes, we observe:
* A steady climb in end-to-end latency, from a baseline of ~120ms to over 450ms by the 90-minute mark.
* A corresponding increase in packet loss for video streams, jumping from <0.5% to nearly 3%.
* The JVB containers show a marked increase in heap memory usage and garbage collection cycles, suggesting a potential memory leak or accumulation of state data that isn't being efficiently managed.
* Client-side, the JavaScript console begins logging increased numbers of "congestion control" events and "packet loss" warnings from the `lib-jitsi-meet` library.
This leads me to hypothesize a few potential root causes, which I'd like the community to vet or supplement:
1. **State Accumulation in Bridges:** The Jitsi Videobridge may be accumulating negotiation state, statistics, or packet metadata for each participant over time without adequate pruning. After 60 minutes, this accumulated state could be impacting routing decisions and packet processing overhead.
2. **Resource Lease Mismanagement:** Granola's resource allocation for a meeting might be based on initial estimates that don't account for the entropy of a long-running session (e.g., participant churn, network variability), leading to resource starvation over time.
3. **Configuration-Driven Timeouts:** While not a performance issue per se, are there underlying configuration defaults (e.g., in the JVB's `sctp` or `ice` subsystems) that begin to exhibit failure modes after a certain duration? Our current `jvb.conf` includes mostly out-of-the-box settings.
```yaml
# Excerpt from our JVB deployment's environment variables
env:
- name: JVB_BRIDGE_STATISTICS_INTERVAL
value: "30000"
- name: JVB_BRIDGE_MUC_NICKNAME
value: "jvb"
- name: MAX_BANDWIDTH
value: "9000000"
- name: DISABLE_AWS_HARVESTER
value: "true"
- name: SHUTDOWN_RESTART
value: "false"
```
Has anyone else performing on-premises or cloud deployments of Granola conducted similar long-duration load tests? I am particularly interested in:
* Any known JVB flags or system properties that govern session-state cleanup.
* Experience with adjusting JVM heap and garbage collection arguments for the JVB in long-running scenarios.
* Whether migrating to the "colibri" web socket transport showed any improvement for extended sessions versus the default.
Our next step is to run a controlled test with a subset of meetings using a more aggressive statistics interval and enabling detailed GC logging, but community insight could significantly shorten the diagnostic cycle. The goal is to understand if this is a fundamental architectural constraint we must design around (e.g., implementing scheduled meeting reconnects) or a tunable parameter issue.