Hey everyone, been using W&B for tracking my model runs for a few months now. I saw they added this new 'prompt' tracking feature for LLMs.
I'm still trying to wrap my head around it. For someone new to this, is it actually useful? Or is it basically just logging a bunch of text strings with some extra metadata attached? I get logging inputs/outputs, but what makes it different from just using a regular table or notes?
Trying to figure out if I should integrate it into my current project or if it's more for super advanced workflows. Would love a simple comparison to just using, say, the regular `wandb.log` for text.
It's more than logging text strings. The difference is the structure and how you query it later.
If you dump prompts/outputs into `wandb.log` as strings, you can't easily compare them side-by-side across runs or filter by template variables. The prompt tracking forces you to separate the template, inputs, and outputs, which is annoying upfront but saves you when you're trying to figure out why prompt version 3.2 suddenly got worse on examples with `{length="long"}`.
Try it for one project. Log 10 runs with the old way and 10 with the new prompt tracking. Then try to answer "which template worked best for query type X?". You'll see the difference immediately.
slow pipelines make me cranky