Skip to content
Notifications
Clear all

Hot take: DeepSeek is a great brainstorming tool but a poor data analyst.

5 Posts
5 Users
0 Reactions
2 Views
(@data_pipeline_tinker)
Estimable Member
Joined: 3 months ago
Posts: 122
Topic starter   [#20602]

Having spent considerable time evaluating various AI assistants for augmenting data workflows, I've arrived at a nuanced, perhaps controversial, conclusion regarding DeepSeek Chat. Its performance is highly dependent on the phase of the analytical process. For initial ideation and problem decomposition, it is remarkably proficient. However, when tasked with the concrete, rigorous work of data analysis—particularly the translation of business questions into precise, executable code and the correct interpretation of outputs—it frequently falters.

My testing methodology involved presenting it with a series of realistic data pipeline and analysis scenarios, from simple aggregations to more complex window function logic and join strategies typical in a modern data warehouse like BigQuery. The pattern that emerged was consistent.

**Where DeepSeek Excels: Brainstorming & Pipeline Design**
* When presented with a vague goal like "I need to track user engagement metrics," it successfully generates a comprehensive list of potential metrics (DAU, MAU, session duration, feature adoption rates).
* It can propose high-level data model designs and suggest appropriate ETL stages. For instance, it will outline a sensible flow from raw event logs to a cleaned `user_sessions` table, and finally to an aggregated `user_engagement_daily` fact table.
* It offers multiple architectural approaches (e.g., debate between a wide fact table vs. normalized dimensions) with reasonable pros and cons, which is excellent for sparking team discussion.

**Where DeepSeek Struggles: Concrete Analysis & Code**
The issues arise in implementation. Given a specific table schema and a business question, the generated SQL often contains subtle logical errors or misunderstandings of function behavior.

Consider this simplified test. I provided a schema for a `sales` table (`sale_id`, `customer_id`, `sale_date`, `amount`) and asked: *"Calculate the running total of sales for each customer, ordered by sale_date."*

A correct answer requires an understanding of window functions. DeepSeek's first attempt was often close but flawed:

```sql
-- A typical flawed attempt from DeepSeek
SELECT
customer_id,
sale_date,
amount,
SUM(amount) OVER (PARTITION BY customer_id ORDER BY sale_date) AS running_total
FROM sales;
```

While this seems plausible, it fails to account for multiple sales on the same date. A more robust solution would specify the window frame or at least consider the possibility of duplicate `sale_date` values for a single customer. More critically, when I then asked it to *"modify this to show only the most recent running total for each customer,"* it frequently pivoted incorrectly to a `LAST_VALUE` function without a proper frame specification, or attempted to use a subquery with `MAX(sale_date)` in a way that would not yield the desired running total.

Furthermore, its interpretation of output is sometimes confidently incorrect. When presented with a sample output dataset and asked to draw an insight, it may correctly describe the *trend* in the data it sees but miss a glaring data quality issue (like a negative `amount` value) that a human analyst would immediately flag.

In essence, DeepSeek functions as a powerful **divergent thinking** tool for the early, conceptual stages of data work. It widens the solution space. However, it lacks the **convergent thinking** precision required for correct, production-grade analysis. It cannot reliably replace the critical, detail-oriented scrutiny of a seasoned data analyst or engineer. For this reason, I now use it exclusively for brainstorming and initial design documents, but I never accept its analytical code or interpretations without rigorous, line-by-line validation.


Extract, transform, trust


   
Quote
(@hannahg)
Estimable Member
Joined: 1 week ago
Posts: 71
 

You're right about it being a great brainstorming partner. I use it similarly for early-stage UX research planning. Throwing a vague problem like "how might we measure dashboard usability?" at it yields a fantastic, broad list of methods - from RTA questions to potential analytics events.

But that's where the magic stops for me too. When I ask it to turn one of those methods into an actual, testable Figma prototype flow or a specific survey question set, the quality drops off a cliff. It's like it can generate the "ingredients list" but can't actually cook the meal.

Makes me wonder if its strength is just recombination of known concepts rather than genuine, applied problem-solving.



   
ReplyQuote
(@grafana_guardian)
Trusted Member
Joined: 3 months ago
Posts: 57
 

That's a great analogy about the ingredients versus the cooking. It lines up with something I've noticed when asking it about observability metrics. It can list every golden signal under the sun, but if you ask for a specific PromQL query to detect a particular failure mode in your unique topology, the results are often syntactically correct but logically flawed.

Your point about recombination vs. genuine problem-solving hits home. I think the gap shows in tasks requiring real synthesis - taking those recombined concepts and adapting them to the messy constraints of a real system. That's where human context is irreplaceable.


- GG


   
ReplyQuote
(@averyt)
Eminent Member
Joined: 4 days ago
Posts: 21
 

Totally feel this with API integrations. It'll give you a perfect-looking Zapier step setup for, say, "when a new Stripe customer, add to HubSpot." But the second your workflow needs a conditional branch because your free trials *don't* go to HubSpot, the logic it suggests falls apart. It can't handle the messy, undocumented rules of your actual business.

It's like you said: synthesis is the hard part. The AI has the components, but gluing them together correctly requires understanding the *why* behind the exception. That's our job, and honestly, I'm okay with that split. Let it draft the boilerplate, I'll handle the nuance.


Automate all the things


   
ReplyQuote
(@henryp)
Trusted Member
Joined: 5 days ago
Posts: 38
 

Exactly. The boilerplate it drafts is often the vendor-lock-in boilerplate. It'll give you a perfect Zapier flow because that's what it's seen a million times. Ask it for the equivalent in n8n or a self-hosted alternative and the quality drops even faster.

You're happy to handle the nuance, but are you happy to inherit the architecture? That's the real cost of letting it draft.


Doubt everything


   
ReplyQuote