What's the best way to log all AI-agent API calls to our data warehouse?

Having recently implemented several agentic workflows that orchestrate calls across multiple providers (OpenAI, Anthropic, various OSS models via LiteLLM), I've found the logging and subsequent analysis of these API calls to be a non-trivial challenge. The payloads are often large, the costs are significant, and understanding the latency patterns and token usage across different agent steps is critical for both optimization and debugging.

My current stack uses a combination of Python's logging library and some manual instrumentation, but this feels brittle and doesn't easily facilitate joining the agent call data with our existing application logs or business events in Snowflake. I'm looking to architect a more robust, centralized logging pipeline specifically for this AI-agent traffic.

The core requirements I'm trying to solve for are:
* **Fidelity:** Capture the full request and response, including headers, for every outbound call to an LLM provider. This includes streaming responses.
* **Structured Data:** The logs should be parsed into a queryable schema (provider, model, prompt tokens, completion tokens, latency, cost estimate, etc.).
* **Minimal Latency Impact:** The logging mechanism should not materially slow down the agent's execution loop.
* **Centralized Destination:** All data must flow into our cloud data warehouse (Snowflake) for analysis alongside other product data.

I've considered a few patterns and would like to dissect the trade-offs:

**Pattern A: Sidecar Proxy**
Deploy a dedicated proxy service (e.g., a configured Nginx or a custom Go service) that all agent traffic flows through. The proxy handles the logging asynchronously.
```yaml
# Conceptual proxy logging config
log_format llm_json escape=json
'{'
'"timestamp":"$time_iso8601",'
'"provider":"$upstream_http_x_provider",'
'"model":"$upstream_http_x_model",'
'"request_body":"$request_body",'
'"response_body":"$upstream_http_response_body",'
'"latency":"$upstream_response_time"'
'}';
```
* **Pros:** Language-agnostic, centralizes configuration, can buffer and batch writes.
* **Cons:** Adds a network hop, parsing JSON payloads in proxy logs can be messy, streaming responses are tricky.

**Pattern B: Decorator/Wrapper Library**
Implement a universal client wrapper in our primary application language (Python) that instruments all `httpx` or `requests` calls to known LLM endpoints.
```python
class LoggingLLMClient:
def __init__(self, sync_client, logging_queue):
self.client = sync_client
self.queue = logging_queue # Async queue to a background worker

def post(self, url, json, **kwargs):
start = time.perf_counter()
response = self.client.post(url, json, **kwargs)
latency = time.perf_counter() - start

log_entry = self._construct_entry(url, json, response, latency)
self.queue.put_nowait(log_entry) # Non-blocking
return response
```
* **Pros:** High fidelity, can easily parse and structure data before emission, can handle streaming.
* **Cons:** Language-locked, requires modifying application code, risk of interfering with client logic.

**Pattern C: Asynchronous Fire-and-Forget from Application**
Instrument the agent directly to publish a log event to an internal message bus (like Kafka or Redis Pub/Sub) immediately after each API call, then have a consumer write to Snowflake.
* **Pros:** Decouples logging from the main app flow completely, allows for multiple consumers (e.g., also alert on high latency).
* **Cons:** Significant architectural complexity, introduces a new moving part (the message bus).

I am leaning towards **Pattern B** combined with a background thread that consumes from the internal queue and pushes batches to a cloud object store (S3) for Snowpipe ingestion into Snowflake. This seems to balance control, fidelity, and performance. Has anyone implemented a similar pipeline? I'm particularly interested in how you handled schema evolution, idempotency in the log writes, and whether you found worthwhile open-source libraries that already provide this instrumentation layer.

testing all the things