We've been using Arize for about nine months to track model performance drift and data quality. One of the more operationally useful patterns we've built is automatically tagging production inference data with an `incident` flag whenever we have a known deployment issue or data pipeline break. This gives us a clean way to segment and later analyze what happened during that specific window, separate from general model drift.
Our system is built on a Kubernetes batch job that writes to Arize's Python SDK, triggered by our existing incident management process. The core idea is simple: when a P1/P2 incident is declared in our alerting system (we use PagerDuty), a webhook also fires to a small service we built. This service knows the time range of the incident and the affected model. It then does two things:
1. It fetches the relevant inference IDs from our data warehouse for that model and time window. We log all inferences with a UUID and timestamp to BigQuery.
2. It calls the Arize API to apply a tag named `production_incident` with a value describing the issue (e.g., `feature_store_latency_high`).
Here's the guts of the tagging script. It runs as a Kubernetes `Job` with the incident details passed as environment variables.
```hcl
# terraform for the k8s job that runs the tagger
resource "kubernetes_job" "arize_incident_tagger" {
metadata {
name = "arize-incident-tagger-${var.incident_id}"
namespace = "arize-ops"
}
spec {
template {
spec {
container {
name = "tagger"
image = "${var.container_registry}/arize-tagger:latest"
env {
name = "INCIDENT_START_UTC"
value = var.incident_start
}
env {
name = "INCIDENT_END_UTC"
value = var.incident_end
}
env {
name = "INCIDENT_TYPE"
value = var.incident_type
}
env {
name = "MODEL_ID"
value = var.model_id
}
# Secrets mounted for Arize API keys & DB credentials
}
restart_policy = "Never"
}
}
backoff_limit = 2
}
}
```
```python
# core section of the tagger script (arize-tagger)
import os
from datetime import datetime
from arize.api import Client
# Fetch inference IDs from warehouse (pseudo-code)
inference_ids = bigquery_client.query(f"""
SELECT inference_id FROM `prod_model_logs.table`
WHERE model_id = '{model_id}'
AND timestamp BETWEEN '{start}' AND '{end}'
""").to_list()
# Initialize Arize client
arize_client = Client(api_key=os.environ['ARIZE_API_KEY'], space_key=os.environ['ARIZE_SPACE_KEY'])
# Tag the records
response = arize_client.tag_records(
model_id=model_id,
inference_ids=inference_ids,
tags=[{"production_incident": incident_type}]
)
if response.status_code != 200:
raise Exception(f"Tagging failed: {response.text}")
```
The main benefits we've seen from this workflow:
* **Post-Incident Analysis:** In Arize, we can filter any chart (drift, performance, data quality) by the `production_incident` tag. This lets us isolate the impact. Was the drop in accuracy *only* during the incident period? If yes, we can likely attribute it to the data issue. If no, we have a separate, longer-term drift problem.
* **Reduced Alert Noise:** We set up our Arize monitors to exclude periods tagged with major incidents. This prevents a cascade of alerting while we're already fighting a fire, and lets us focus on the root cause.
* **Audit Trail:** The tags serve as a record linking operational events to model behavior, which is useful for retrospectives and for explaining performance graphs to stakeholders.
The pitfalls we had to work around:
* Rate limiting on the Arize API when tagging hundreds of thousands of records at once. We had to implement batch sizing and retries with exponential backoff in the script.
* Ensuring the tagging job is idempotent. If the incident is updated or the job fails and retries, we don't want duplicate tags or partial tags. Our script now checks for existing tags on the inference IDs before proceeding.
* Latency between the inference event and it being available for tagging in our warehouse. We had to build in a buffer period (like +30 minutes after the incident end) to ensure we capture all relevant records.
Overall, this has moved Arize from being just a monitoring dashboard to being an integrated part of our incident response and analysis loop. Curious if others have built similar automated tagging workflows and how you handle the data pipeline dependencies.
Automate everything. Twice.
That's a solid, practical use of a monitoring platform. Tagging data during known incidents seems like the one thing these expensive tools should be doing well from the start. Surprised you need a whole custom Kubernetes job and external service to make it happen, though.
Does Arize not have a native integration for this, or is their PagerDuty/webhook setup gated behind the "Enterprise Platinum" tier? I've seen that pattern before where the useful automations are the paid add-ons. Feels like you're building the feature they're selling.
—DW
That's a clever way to tie your incident management directly into your observability data. I especially like that you're using it to segment analysis later - makes post-mortems much more straightforward.
On the tooling side, I've seen a few teams use the monitoring platform's API for tagging, like you did, but others just add the flag as a column when they log inferences initially. It depends how quickly you need that tag to appear in your dashboards. Your batch job method is probably cleaner for retroactive tagging after an incident is declared.
Do you find the added latency of the batch job affects how your team responds during the incident itself, or is it purely for historical analysis?
Raise the signal, lower the noise.
Good question about latency. I'd guess it's mostly for post-incident analysis - if you're in the middle of a fire, you're probably looking at your alert dashboard, not waiting for tags to populate.
It made me think about how we'd handle this in a smaller setup. We log to a database directly, so we'd probably just run a one-off UPDATE query after an incident to tag the records. Same result, fewer moving parts than a K8s job. But the API/batch approach is better if you can't write to the data store directly.
Do you know if there's a cost to having those extra tags in Arize, like counting against your data points?
Integrating incident management with your observability data is a solid pattern. However, the architecture you've described raises immediate questions about operational cost and complexity that may not be apparent on a per-incident basis.
You're introducing a Kubernetes Job, a custom service, and API calls to Arize for each tagging event. The compute cost for spinning up that Job pod, even briefly, and the data egress cost from BigQuery for fetching inference IDs could be non-trivial at scale. More critically, this creates a new failure domain - if the tagging service or the Job fails, your incident data is incomplete, which defeats the purpose.
I'd challenge the need for a separate batch process. Why not tag the inferences at ingestion by having your serving application check a low-latency cache (like Redis) for an active incident flag? This would be real-time, eliminate the batch dependency, and likely reduce the total cost of ownership for this feature. The batch retroactive tagging could remain as a fallback, but making it the primary path seems to add unnecessary moving parts and latency for post-mortem analysis.
Always check the data transfer costs.
Agree on the complexity and failure domain. The caching approach at ingestion is ideal, but requires service-level changes many teams can't deploy quickly.
A simpler compromise: keep the batch job but trigger it from a cron that polls your incident system's status API every minute. That removes the custom service dependency and webhook failure point. The pod cost is negligible if you run it on a shared pool.
Your point about egress cost from BigQuery is valid if they're scanning full tables. They should be using partitioned queries on the incident time window.
You're doing this backwards. The batch job and extra fetch from BigQuery adds latency and a point of failure. You should tag the data at the source when it's logged.
If your serving layer can't check incident state before logging, you're already at a disadvantage during the incident itself. The batch job just creates historical markers you should already have.
And what happens when the batch job fails? Your tagged data is incomplete, which is worse than having no tag at all. You're building a system to document failures that can fail.
Don't panic, have a rollback plan.
Hard disagree, but only because you assume a serving layer that's even capable of checking an incident state. That's a luxury.
Most teams I've seen are logging from a dozen different services, some legacy, some third-party. Coordinating a real-time flag across all of them is a six-month refactor project. The batch job, for all its flaws, is something you can deploy next Tuesday.
You're right that it's a secondary failure domain, but so is everything you build. The question is whether an incomplete tag is worse than no tag. I'd argue incomplete is still useful - you'd see the gap in the data and know something went wrong with the tagging process itself, which is its own kind of diagnostic.
You're absolutely right about the practical reality of legacy and third party services. The six-month refactor estimate is often optimistic.
My concern with the "incomplete tag as diagnostic" argument is it creates a meta-problem. You now have two layers of incident data to reconcile: the original production issue and the subsequent tagging failure. In a post-mortem, you're explaining why your incident analysis is incomplete because a separate automation broke. That can dilute focus from the root cause.
A polling-based cron job, as user1366 suggested, is a decent middle ground that reduces the custom service dependency. It trades some immediacy for a simpler, more auditable failure mode.
Plan the exit before entry.
> low-latency cache (like Redis)
That assumes you can modify the logging logic in all your serving apps. Many of us can't.
You're right about cost and failure domains for the batch job. But your proposed fix is a major architectural change. The batch job's failure domain is at least contained and observable. A failed Redis check means you lose the tag entirely, with zero trace in your observability platform.
Cost is real. You can mitigate by running the job on a shared spot node pool and using strict time-window queries. The operational burden of maintaining another caching layer for all services is often higher than a simple, fault-aware batch process.
Metrics don't lie.
That meta-problem point is really sharp, I hadn't thought of it that way. Trying to debug the tagger during an incident post-mortem sounds like a nightmare.
But doesn't the cron job polling idea just shift the failure domain? If the polling fails silently, you're still left with an incomplete tag, and now you have to check your cron system's logs instead of a service's. It seems like the observability burden just moves around.
You make a good point about the failure domain, and the Redis suggestion is ideal for a greenfield system. But I've seen teams try that and stumble on cache invalidation - knowing exactly when an incident ends to clear the flag is surprisingly tricky. That's partly why some opt for retroactive tagging.
The cost factor is real, especially egress from BigQuery. If you're scanning huge time windows, you'll feel it. But you can mitigate a lot by making the job smart about query boundaries and running it on preemptible infrastructure. The tradeoff is engineering time versus cloud spend, and the math varies for every shop.
Keep it real, keep it kind.
Your script is incomplete. You don't explain how the Job authenticates to BigQuery and Arize. Are you passing raw service account keys in the Job spec? That's a serious credential exposure on its own.
Also, you haven't mentioned idempotency. What happens if the webhook fires twice and the Job runs concurrently? You'll get duplicate tags or an API error. Your tagging logic needs to handle that.
Least privilege is not a suggestion.
Absolutely right about the credential exposure - embedding service account keys directly in the Job spec is asking for trouble. That should be handled via environment variables sourced from a secrets manager or, better yet, using workload identity federation if you're on GKE. It's a hidden operational risk that's easy to overlook when you're focused on the data pipeline logic.
The idempotency point is critical, too. A simple deduplication check using the incident ID as a key in a small, ephemeral table would solve that. But that adds another failure mode if the deduplication store flakes out. It becomes a question of which failure you'd rather handle: duplicate API calls or a more complex dependency.
throughput first
You're correct about the source tagging being architecturally superior, but I think you're underestimating the cumulative cost penalty of real-time checks in high-volume systems.
If every log line or API call needs to query an external state store, you're adding latency and paying for that network overhead millions of times per hour. A batch job, while flawed, aggregates that cost into a single, potentially spot-instance operation. The cost difference isn't trivial; I've seen real-time flag checks increase Lambda or container invocations by 15-20% during traffic surges, which is precisely when you'd be declaring an incident.
The batch failure risk is real, but it's a single, monitorable process. A distributed failure in your real-time flagging logic, where one service can't reach Redis but others can, creates an inconsistent data state that's far harder to detect and repair.
every dollar counts