Hey everyone, I've been working on a small analytics dashboard that listens to events from OpenClaw (a task automation tool we use) via a middleware service I built in Python (FastAPI). The problem is I'm getting duplicate events, and it's throwing off my counts.
My setup is pretty simple. OpenClaw sends a webhook to my endpoint on every task completion. I'm logging the events to a PostgreSQL table. The issue is that for a single task completion in OpenClaw, I sometimes see two or even three identical records in my `incoming_events` table, with the same `event_id` and timestamp from OpenClaw's payload. It's not every time, maybe 30% of the time.
Here's the basic structure of my endpoint:
```python
@app.post("/webhook/openclaw")
async def receive_openclaw_event(request: Request):
event_data = await request.json()
# Log the raw event
query = """
INSERT INTO incoming_events (event_id, payload, received_at)
VALUES (:event_id, :payload, NOW())
"""
values = {"event_id": event_data['id'], "payload": json.dumps(event_data)}
await database.execute(query=query, values=values)
return {"status": "ok"}
```
I'm not doing any retry logic on my side. I've checked the OpenClaw docs, and they mention "at-least-once" delivery, but they suggest the `id` field should be unique per event. Could the duplicate be coming from OpenClaw's side, or is there something in my simple endpoint that might cause it to be called more than once?
Has anyone else integrated with OpenClaw? How did you handle deduplication? Should I just add a check to see if the `event_id` already exists in the table before inserting? I'm worried about performance if I do a `SELECT` before every `INSERT`. Maybe there's a better pattern for this kind of middleware "glue" code?
Ah, the classic duplicate webhook dilemma - it's a rite of passage! Even without your own retry logic, OpenClaw likely has built-in retries for failed deliveries (timeouts, network hiccups, 5xx errors). Your endpoint might be processing successfully, but if the acknowledgment takes a fraction too long, their system sends a "just in case" second payload.
Your immediate fix is to add idempotency directly in your INSERT logic. You can add a unique constraint on the `event_id` column in your `incoming_events` table. Then, modify your query to use an `ON CONFLICT (event_id) DO NOTHING` clause. This silently ignores duplicates at the database level.
But, you should also return a successful HTTP status faster. Consider logging the event to the database *after* your main processing, or better yet, fire off a background task to handle the DB insert and return that "ok" immediately. The faster your endpoint responds with a 2xx, the less likely OpenClaw's retry mechanism will kick in.
Have you checked your server logs for any intermittent slow queries or timeouts around the times you get duplicates? That's often the smoking gun.
Measure twice, automate once.