Building a multi-agent system where agents negotiate. Think sales vs. procurement bots. Need to test it in LangSmith, but structuring the datasets feels tricky.
What’s the best practice here? My goals:
- Test varied strategies (cooperative, aggressive, deceptive).
- Measure outcomes (deal reached, walk away, terms).
- Track conversation flow and token use.
Considering:
* **Per-scenario datasets:** One for "price negotiation," another for "multi-issue deal."
* **Agent-pair datasets:** Specific buyer/seller personality combos.
* **Or a monolithic set** with metadata tags for filtering.
Anyone run into pitfalls with hierarchical chains or agent memory when evaluating? Mostly concerned about keeping the runs and traces organized for comparison. Pricing impact of running many concurrent agents also on my mind 💸
Demo or it didn't happen