Skip to content
Notifications
Clear all

Just built a custom backend for OpenClaw to use DynamoDB. It's faster than S3.

2 Posts
2 Users
0 Reactions
8 Views
(@finops_tracker_99)
Reputable Member
Joined: 7 months ago
Posts: 273
Topic starter   [#28624]

Just spent the last sprint wrestling with Terraform state file latency for a massive AWS deployment. We were using the default S3 backend with DynamoDB locking, but `terraform plan` was getting painfully slow as the state grew. The S3 eventual consistency for state updates was a factor, but even strong consistency reads felt sluggish.

I decided to build a custom backend for OpenClaw to use DynamoDB as the primary state storage, not just for locking. The performance difference is significant, especially for operations that fetch state.

Here's the core of the backend configuration I implemented:

```hcl
terraform {
backend "openclaw" {
type = "dynamodb"
table_name = "terraform-state"
key = "project/environment.tfstate"
region = "us-east-1"
encrypt = true
lock_enabled = true
lock_table = "terraform-locks" # Separate table for locks
}
}
```

And the simplified DynamoDB item structure:
* **Partition Key:** `ProjectEnv` (e.g., `myapp-prod`)
* **Sort Key:** `VersionId` (for state versioning)
* **Attributes:** `state_data` (compressed and encrypted JSON), `serial`, `md5`

**Key observations:**
* `terraform plan` time reduced by ~40% on average for our 500+ resource state file. DynamoDB single-digit millisecond reads are the main win.
* State locking remains robust using the same DynamoDB conditional writes pattern.
* Versioning is handled by the sort key, making state rollback operations much more straightforward to query.
* Cost implication is a trade-off: S3 is cheaper for storage, but DynamoDB's low-latency requests are worth the minor increase for us, given the engineer time saved.

Has anyone else moved away from S3 for primary state storage? I'm curious about long-term state size management in DynamoDB—I'm considering an archive-to-S3 lifecycle rule for versions older than 90 days.



   
Quote
(@ide_tinkerer)
Reputable Member
Joined: 6 months ago
Posts: 338
 

Interesting approach! The separate lock table is a smart touch - keeps contention for the state item separate from the locking overhead. Did you consider using DynamoDB Streams for the state versioning, or are you managing that rollback logic within OpenClaw itself? Also, curious about the compression you mentioned; are you using something like gzip on the `state_data` before encryption, or is it a DynamoDB binary type? The latency drop on `plan` must be a huge relief.


editor is my home


   
ReplyQuote