Anyone actually using Arize AI in production for a 200-user retail app?

Given our team's focus on rigorous A/B testing and funnel analysis for a retail application with approximately 200 daily active users, I am investigating production-grade tooling for LLM evaluation. Our primary use case involves a customer support chatbot that handles product inquiries, return policies, and order status. We require a framework capable of monitoring performance drift, detecting hallucination in product descriptions, and scoring response relevance against our knowledge base.

I have conducted a preliminary evaluation of Arize AI's Phoenix platform, specifically its integration for LLM observability. My analysis centers on its applicability for a small-to-midsize production environment like ours, rather than a large-scale deployment. The key considerations are implementation overhead, actionable insights for a sub-500 user base, and cost-effectiveness for limited inference volume.

Below is a summary of my configuration and findings, structured for clarity:

**Implemented Monitoring Dimensions:**
* **Response Relevance:** Using a custom `arize.python.llm_evaluation` scorer that compares the chatbot's response to the retrieved context chunks.
* **Hallucination Detection:** Leveraging the built-in `hallucination` scorer for claims made about product specifications not present in the provided context.
* **Toxicity & Refusal Rates:** Basic sentiment and safety scoring as a baseline.
* **Latency & Token Usage:** Infrastructure metrics collected via the tracing integration.

**Example Configuration Snippet for a Scorer:**
```python
from arize.python.llm_evaluation import RelevanceScorer, HallucinationScorer

relevance_scorer = RelevanceScorer(
model="gpt-4",
criteria="How directly does the response address the customer's query using only the provided product catalog context?",
parameters={
"response": "response",
"query": "user_query",
"context": "retrieved_documents"
}
)

hallucination_scorer = HallucinationScorer(
model="gpt-4",
context_column="retrieved_documents"
)
```

**Preliminary Results & Open Questions:**
* The automated evals provide a consistent quantitative baseline, which is valuable for detecting regression after model updates.
* However, for 200 users, the volume of unique queries is relatively low. This raises questions about statistical significance when tracking metric shifts week-over-week.
* The cost of using GPT-4 as the judge model for all evals may be disproportionate to the scale of our operation.
* I have yet to determine if the platform can effectively segment evaluations by user cohort (e.g., new vs. returning customers) to tie LLM performance directly to business funnel stages.

My core question for the community is whether anyone operates at a similar scale and has derived clear ROI from Arize or a comparable framework. Specifically:
* Have you successfully tied LLM evaluation metrics (e.g., improved relevance scores) to concrete business outcomes like reduced support tickets or increased conversion in a specific funnel stage?
* For low-volume applications, do you find more value in automated batch evals on a sample of queries versus real-time monitoring of all inferences?
* Are there alternative or complementary lightweight approaches you would recommend for a team with strong analytics capabilities but limited ML engineering bandwidth?

I am particularly interested in methodologies for designing a scoring rubric that aligns with retail-specific user behavior, such as the ability to accurately convey promotional terms or inventory availability.

— Amanda