Our data science team is migrating from a loose collection of Jupyter notebooks to a more structured codebase with proper modules and CI/CD. We need to enforce some code quality standards, but the tools should be pragmatic for data work—where you often see exploratory pandas code, scientific libraries, and scripts that are run, not deployed as long-lived services.
I've been evaluating the top contenders and wanted to share my notes. The main goal is catching bugs and enforcing consistency without being overly burdensome.
**My shortlist:**
* **flake8**: The classic. Fast, extensible with plugins, and great for enforcing PEP 8.
* **pylint**: Very thorough. Catches a wider range of issues but can be noisy.
* **ruff**: The new, incredibly fast tool written in Rust. It's becoming the default for many teams.
* **black** (with **isort**): While primarily a formatter, Black's strict style eliminates all formatting debates. Often paired with a linter.
For our use case, I'm leaning towards **ruff** for linting and **black** for formatting. Ruff is fast enough to run on every save, and it replaces flake8, isort, and several other plugins. Here's a sample config that's been working well for us in `pyproject.toml`:
```toml
[tool.ruff]
line-length = 88
target-version = "py311"
select = [
"E", # pycodestyle errors
"W", # pycodestyle warnings
"F", # pyflakes
"I", # isort
"B", # flake8-bugbear
]
[tool.black]
line-length = 88
target-version = ['py311']
```
The speed is the real win. Running `ruff check .` on our codebase takes under a second, whereas pylint can take 20+ seconds. For data scientists who might be less patient with tooling, this lower friction is key.
What's your stack? Does anyone have experience integrating these into a CI pipeline specifically for a data science team, or recommendations for rules to disable for notebook-derived code?
-- Amy
Cloud cost nerd. No, I don't use Reserved Instances.