Switched from a custom Luigi pipeline to LangGraph. Regret it. Too much abstraction.

Having spent the last 18 months architecting and maintaining a custom data orchestration pipeline built on Luigi for our NLP feature engineering workflows, I was intrigued by the promises of LangGraph. The premise of a native Python framework designed for cyclic, stateful workflows, particularly for LLM applications, seemed like a logical evolution. We undertook a significant migration effort, and after three months in production, I must conclude the abstraction cost has severely outweighed the benefits for our specific use case. The cognitive and operational overhead introduced by LangGraph's paradigm has been substantial.

Our original Luigi pipeline, while verbose, offered explicit, linear control. Each `Task` class was a self-contained unit of work with clear input/output dependencies. Migrating to LangGraph required a complete mental model shift into a state-driven, cyclic graph. The core issue is that our pipeline is not truly cyclic in the LLM-agent sense; it's a directed acyclic graph (DAG) of data transformation steps, with some conditional branching. Forcing this into LangGraph's `StateGraph` with a `State` schema felt like an unnecessary indirection.

Consider a simple data validation and routing step. In Luigi, this was a straightforward conditional in the `run()` method:

```python
class ValidateAndRouteTask(luigi.Task):
def run(self):
data = self.input().load()
if data['quality_score'] > THRESHOLD:
route = 'high_quality_path'
else:
route = 'review_path'
self.output().write({'data': data, 'route': route})
```

In LangGraph, this becomes a node function that must mutate a shared state dictionary, with routing logic deferred to graph edges:

```python
def validation_node(state: State):
if state["data"]["quality_score"] > THRESHOLD:
state["route"] = "high_quality_path"
else:
state["route"] = "review_path"
return state

graph_builder = StateGraph(State)
graph_builder.add_node("validate", validation_node)
graph_builder.add_conditional_edges(
"validate",
lambda state: state["route"],
{"high_quality_path": "process_high", "review_path": "send_for_review"}
)
```

While more "declarative," this fragments logic. The routing decision is split between the node's state mutation and the edge definition's conditional lambda. Debugging a data flow issue now requires tracing mutations through a shared state object across multiple nodes, rather than inspecting discrete task outputs. The abstraction layer obscures the actual data lineage.

Furthermore, the operational clarity of Luigi's visualizer, which shows task statuses and dependencies directly, has been replaced by LangGraph's graph visualization, which emphasizes topology over execution state. For monitoring, we've lost fine-grained, per-step logging and metrics that were built into our Luigi task lifecycle. LangGraph's focus is on the flow of *state* rather than the execution of *tasks*, which proves less intuitive for batch data processing.

In summary, for our batch-oriented, non-conversational ML pipeline:
* **Abstraction Overhead:** The stateful graph model introduced complexity where a simple task DAG sufficed.
* **Debugging Difficulty:** Tracing data transformations through a mutable state object is more challenging than inspecting immutable task artifacts.
* **Loss of Operational Transparency:** Built-in tooling for monitoring and visualization is less aligned with batch processing needs than traditional orchestrators.
* **Over-engineering:** The powerful constructs for cycles and human-in-the-loop interactions are features we do not use, yet we pay the conceptual and implementation tax.

LangGraph appears excellently suited for building persistent, multi-session LLM agentic workflows. For migrating a traditional, complex-but-linear data pipeline, it has been a regression in clarity and maintainability. We are currently evaluating a move to more fit-for-purpose orchestration like Prefect, which might offer a better balance of abstraction and explicit control.