Hi everyone! I'm really excited to be here and to dive into DeepSeek Chat. I've been hearing a lot about it, especially for technical and data-related tasks. My main goal is to evaluate it for creating and maintaining technical documentation—think data pipeline specs, Looker/Tableau guide updates, and dbt model descriptions.
As someone still getting their feet wet in data analytics, I'd love some guidance on where to begin. I'm looking for a structured approach to test it out. Could you share what specific tasks or prompts you started with? For example:
* Summarizing a complex SQL query into plain English for a wiki?
* Generating a step-by-step guide for a new ETL process?
* Explaining a data model to non-technical stakeholders?
I'm also curious about any "gotchas" you've encountered. Does it handle niche data tool terminology (like dbt tests, LookML explores) well, or does it sometimes miss the mark? Any tips on crafting prompts for the best, most accurate output would be incredibly helpful! 😊
Basically, I'd love a beginner-friendly walkthrough on how you'd put it through its paces for documentation work. What would your evaluation checklist look like?
Welcome, and it's great to see such a thoughtful approach to your evaluation. Your proposed starter tasks are spot-on for documentation work. I'd actually begin with the SQL query summary - it's a concrete, bounded test that immediately shows you how well it translates technical logic into clear prose.
On your point about niche terminology: it's generally strong, but I've seen it occasionally be too generic with specific Looker or dbt concepts. A tip is to "show, don't just tell" - provide a short example of a dbt test or a LookML explore snippet in your prompt and ask it to explain that specific pattern. That usually yields more accurate, relevant output than just asking for a general definition.
For a structured checklist, I'd add evaluating how it handles incremental updates. Try feeding it an existing, slightly outdated section of a pipeline spec and ask for a refresh based on new notes. Does it preserve the correct structure while making the needed changes? That's a huge time-saver in real documentation maintenance. Good luck
— isabel
Your proposed tests are fine for a first impression, but they miss the real gotcha for documentation: consistency over time. A single query explanation is a parlor trick.
The real test is feeding it a 200-page evolving spec and asking for the 10-page update six months later. Will it remember the style, the naming conventions, the assumed audience? Or will it give you something that reads like a different product?
Start with your *worst*, most jargon-filled internal wiki page. Ask it to rewrite for a new hire. Then feed that output back in and ask for a version suitable for a VP. That's where the generic advice usually falls apart.
Doubt everything