We've been using Arize for about nine months to track model performance drift and data quality. One of the more operationally useful patterns we've built is automatically tagging production inference data with an `incident` flag whenever we have a known deployment issue or data pipeline break. This gives us a clean way to segment and later analyze what happened during that specific window, separate from general model drift.
Our system is built on a Kubernetes batch job that writes to Arize's Python SDK, triggered by our existing incident management process. The core idea is simple: when a P1/P2 incident is declared in our alerting system (we use PagerDuty), a webhook also fires to a small service we built. This service knows the time range of the incident and the affected model. It then does two things:
1. It fetches the relevant inference IDs from our data warehouse for that model and time window. We log all inferences with a UUID and timestamp to BigQuery.
2. It calls the Arize API to apply a tag named `production_incident` with a value describing the issue (e.g., `feature_store_latency_high`).
Here's the guts of the tagging script. It runs as a Kubernetes `Job` with the incident details passed as environment variables.
```hcl
# terraform for the k8s job that runs the tagger
resource "kubernetes_job" "arize_incident_tagger" {
metadata {
name = "arize-incident-tagger-${var.incident_id}"
namespace = "arize-ops"
}
spec {
template {
spec {
container {
name = "tagger"
image = "${var.container_registry}/arize-tagger:latest"
env {
name = "INCIDENT_START_UTC"
value = var.incident_start
}
env {
name = "INCIDENT_END_UTC"
value = var.incident_end
}
env {
name = "INCIDENT_TYPE"
value = var.incident_type
}
env {
name = "MODEL_ID"
value = var.model_id
}
# Secrets mounted for Arize API keys & DB credentials
}
restart_policy = "Never"
}
}
backoff_limit = 2
}
}
```
```python
# core section of the tagger script (arize-tagger)
import os
from datetime import datetime
from arize.api import Client
# Fetch inference IDs from warehouse (pseudo-code)
inference_ids = bigquery_client.query(f"""
SELECT inference_id FROM `prod_model_logs.table`
WHERE model_id = '{model_id}'
AND timestamp BETWEEN '{start}' AND '{end}'
""").to_list()
# Initialize Arize client
arize_client = Client(api_key=os.environ['ARIZE_API_KEY'], space_key=os.environ['ARIZE_SPACE_KEY'])
# Tag the records
response = arize_client.tag_records(
model_id=model_id,
inference_ids=inference_ids,
tags=[{"production_incident": incident_type}]
)
if response.status_code != 200:
raise Exception(f"Tagging failed: {response.text}")
```
The main benefits we've seen from this workflow:
* **Post-Incident Analysis:** In Arize, we can filter any chart (drift, performance, data quality) by the `production_incident` tag. This lets us isolate the impact. Was the drop in accuracy *only* during the incident period? If yes, we can likely attribute it to the data issue. If no, we have a separate, longer-term drift problem.
* **Reduced Alert Noise:** We set up our Arize monitors to exclude periods tagged with major incidents. This prevents a cascade of alerting while we're already fighting a fire, and lets us focus on the root cause.
* **Audit Trail:** The tags serve as a record linking operational events to model behavior, which is useful for retrospectives and for explaining performance graphs to stakeholders.
The pitfalls we had to work around:
* Rate limiting on the Arize API when tagging hundreds of thousands of records at once. We had to implement batch sizing and retries with exponential backoff in the script.
* Ensuring the tagging job is idempotent. If the incident is updated or the job fails and retries, we don't want duplicate tags or partial tags. Our script now checks for existing tags on the inference IDs before proceeding.
* Latency between the inference event and it being available for tagging in our warehouse. We had to build in a buffer period (like +30 minutes after the incident end) to ensure we capture all relevant records.
Overall, this has moved Arize from being just a monitoring dashboard to being an integrated part of our incident response and analysis loop. Curious if others have built similar automated tagging workflows and how you handle the data pipeline dependencies.
Automate everything. Twice.