My workflow: How we manage exception requests without chaos

Managing exception requests—like one-off data corrections or overriding a validation rule—is a notorious source of chaos. It often starts with a Slack message, escalates to a database `UPDATE` run directly in production, and leaves no audit trail. Our team has built a workflow using a simple internal service that brings order to this process, heavily leaning on idempotency and state machines.

At its core, the service is a Go service with a Postgres backend and Redis for caching request state. We treat each exception as a "ticket" with a strict lifecycle. The key was designing a schema that captures intent, approval, execution, and result.

Here's the simplified core of our `exception_requests` table:

```sql
CREATE TABLE exception_requests (
id UUID PRIMARY KEY,
request_type VARCHAR(50) NOT NULL, -- e.g., 'user_data_correction'
entity_id VARCHAR(100) NOT NULL, -- The affected record
requested_by VARCHAR(100) NOT NULL,
requested_at TIMESTAMPTZ NOT NULL,
payload JSONB NOT NULL, -- The actual change
status VARCHAR(20) NOT NULL DEFAULT 'pending', -- pending, approved, rejected, executed, failed
approved_by VARCHAR(100),
approved_at TIMESTAMPTZ,
execution_result JSONB,
idempotency_key VARCHAR(255) UNIQUE -- Critical for retries
);
```

The workflow:

* **Submission:** An engineer POSTs to our internal API with an `idempotency_key`. This prevents duplicate requests from being processed.
* **Approval:** A lead reviews a queue (a simple UI querying this table). Approval updates `status`, `approved_by`, and `approved_at`.
* **Execution:** A separate, idempotent worker process picks up `approved` records. It executes the logic specific to the `request_type` against our services.
* **Result:** The `execution_result` and `status` are updated. We cache the final state in Redis for 24 hours for quick lookup by other systems.

We avoid chaos by:
- Never allowing direct database writes for these; all changes funnel through this service.
- Using the `idempotency_key` to make the entire flow retry-safe.
- Logging every state transition for a clear audit trail.
- Caching the final decision in Redis to prevent repeated expensive joins for common lookup patterns (e.g., "was exception X for entity Y applied?").

The result is a system where exceptions are trackable, repeatable, and non-destructive. It's not glamorous, but it eliminated the "who changed what and why?" panic.

-- latency