Just built a CI/CD gate that fails if Arize drift score is too high

We've been running Arize in production for about a year, primarily for monitoring drift and data quality on our core ML models. The dashboards are fine, but alerts are... passive. By the time you get a PagerDuty alert, the model is already serving potentially degraded inferences.

I decided to move the check left, into the CI/CD pipeline for model deployment. The goal: if the candidate model's performance or drift metrics against a reference dataset exceed a threshold, the pipeline fails. No new model version is deployed.

The implementation is straightforward. We run a batch inference job in the CI system (Jenkins, but it's generic), then use the Arize Python SDK to push the results and calculate metrics. The key is the `arize.pandas.logger` for batch logging and the `Client.get_model_metrics` call.

Here's the core of the script that runs in the pipeline:

```python
import os
from arize.pandas.logger import Client
from arize.utils.types import Environments, ModelTypes
import pandas as pd

SPACE_KEY = os.getenv('ARIZE_SPACE_KEY')
API_KEY = os.getenv('ARIZE_API_KEY')
MODEL_ID = 'customer-churn-v3'
MODEL_VERSION = os.getenv('CANDIDATE_VERSION') # From CI env

arize_client = Client(space_key=SPACE_KEY, api_key=API_KEY)

# df_candidate: inferences from the new model version
# df_reference: golden dataset or prior model's inferences
features = [...] # your feature columns

# Log candidate predictions
response = arize_client.log(
dataframe=df_candidate,
model_id=MODEL_ID,
model_version=MODEL_VERSION,
model_type=ModelTypes.SCORE_CATEGORICAL,
environment=Environments.PRODUCTION,
batch_id="ci_batch"
)

# Calculate drift metrics for a specific feature and the prediction score
metrics = arize_client.get_model_metrics(
model_id=MODEL_ID,
metric_type=["drift"],
start_time="2024-01-01",
end_time="2024-05-01",
dimension_filters=[{"name": "feature_name", "value": "transaction_amount"}]
)

# Example: fail if PSI for a key feature > 0.1
psi_value = metrics['data']['drift']['psi']
if psi_value > 0.1:
print(f"Drift threshold exceeded: PSI={psi_value}")
sys.exit(1)
```

The main pitfalls:
* You need a robust, representative reference dataset accessible in your CI environment.
* Metric calculation in Arize isn't instantaneous; you need to add a `time.sleep(60)` or implement a poll for the `calculation_id` if using the more advanced async calls.
* This adds ~2-3 minutes to pipeline runtime, mostly for the batch logging and metric computation.

It's not a silver bullet—you still need production monitoring—but it catches egregious regressions before they hit. So far, it's blocked two deployments where feature drift spiked due to upstream data pipeline changes we hadn't fully accounted for.

- cr