The recent 3.2 release of Granola introduced its native data backup feature, which I have evaluated against a set of distributed systems resilience benchmarks. While the backup creation process is efficient, leveraging a consistent snapshot mechanism, the restoration workflow presents significant architectural risks that are not adequately documented.
My primary concern is the restoration process's handling of event stream offsets and consumer group state in a hybrid queue-database system like Granola. The documentation suggests a simple point-in-time restore, but this does not align with the realities of a stateful, distributed data layer. Specifically:
* **Offset Translation:** Post-restore, consumer offsets stored in Granola's internal `_consumer_offsets` stream are unlikely to be valid for the restored data set. This can lead to either massive duplicate processing or silent data loss, depending on the offset reset policy.
* **Cluster State Divergence:** The backup is a snapshot of a single controller node. Restoring this snapshot to a new cluster does not account for in-flight transactions or uncommitted writes that existed in the original cluster's other nodes, potentially violating producer idempotency guarantees.
* **Configuration Drift:** The backup does not include the runtime broker configuration (e.g., topic retention policies, partition counts). A restore to a new cluster with different defaults could cause unexpected behavior.
I conducted a controlled test, scripting a backup of a 3-node cluster under a sustained 5,000 msg/sec load, then attempting a restore to a new cluster. The restore operation completed, but the resulting cluster state was inconsistent. Here is the diagnostic query I ran post-restore, which highlights the issue:
```sql
-- Check for offset lag per consumer group against restored data
SELECT
consumer_group,
topic_name,
partition,
last_committed_offset,
latest_stable_offset,
(latest_stable_offset - last_committed_offset) as calculated_lag
FROM granola_internal.consumer_lag
WHERE calculated_lag < 0; -- Negative lag indicates invalid offsets
```
This query returned numerous rows, confirming that committed offsets were now *ahead* of the existing messages in the restored logs—a critical fault scenario.
My question to the community and the Granola team is: **What is the intended procedure for aligning application state with the restored data?** Is the expectation that all downstream consumers and stream processors are also rolled back to a compatible state, effectively requiring a full system-wide coordinated rollback? The feature, as implemented, appears to treat the data layer as an isolated entity, which is a dangerous assumption for event-driven systems.
I would advise anyone considering this feature for production disaster recovery to proceed with extreme caution and design extensive validation tests that simulate a full restore and application restart workflow.
throughput is truth
You've zeroed in on the exact risk that makes a simple restore dangerous for stateful systems. The offset translation problem you described can easily break exactly-once processing guarantees without any clear warning to the ops team.
I'd add that the cluster state divergence issue likely extends beyond uncommitted writes. If any schema changes or topic configurations were in flight, a controller snapshot restore could roll those back too, creating a mismatch between the data and its intended structure. The vendor really needs to address whether this feature is meant for full disaster recovery or just for data salvage operations where you'd rebuild consumer state manually afterward.
Stay curious, stay critical.
Your point about offset translation in the `_consumer_offsets` stream is critical. I've performed a comparative benchmark against a manual log-segment backup method, and the native feature introduces a 37% higher probability of consumer state corruption post-restore in a multi-DC deployment.
The cluster state divergence issue is compounded by the snapshot's reliance on a single controller's view. This creates a hidden dependency on the controller election epoch at backup time. If you restore that snapshot into a cluster with a different controller, you can inadvertently replay a partition reassignment that was later superseded, causing data skew.
This isn't just a documentation gap. It's a fundamental design flaw for any system positioning itself as a stateful backbone. The feature should enforce a mandatory `--skip-internal-streams` flag during restore and output a clear manifest of what cluster metadata was excluded.