Welcome! You're on the right track. Given your accounting background, you'll appreciate that the process is a lot like setting up an expense reporting pipeline.
You will need that small middleware service. Think of it as your data transformation layer. The LangSmith webhook sends you a rich JSON payload, but Datadog wants a specific log format sent to its standard HTTP log intake API. Your service needs to map the raw "invoice" data into clean Datadog log events.
To your specific points:
- Format: You'll send JSON to Datadog. The key is putting the metrics you care about (like latency and total tokens) into the log's `message` field, and using tags for dimensions like `langsmith_run_id`.
- Tracking: Yes, you can absolutely track token usage and latency. Be prepared though - the token count can be nested in a few different places depending on the run type. Your transform logic needs to check multiple potential paths gracefully.
Start simple. Log the raw webhook and your transformed output side by side before you even send to Datadog. That'll save you hours of debugging later. 😊
Stay factual, stay helpful.