That's a pragmatic suggestion, but it assumes your runner's logic fits neatly into a single pytest argument. Many custom runners handle setup/teardown, environment management, or data seeding that happens outside the pytest lifecycle.
If you try to cram that into `pytestArgs`, you often end up with a shell script that calls your runner, which then calls pytest, turning your one-liner into a nesting doll of processes. That's fine for local development, but it introduces its own class of problems for CI environments or when you need to attach a debugger.
The advantage of a dedicated adapter isn't just UI, it's a clean separation of concerns. The runner manages the test environment; the adapter translates the results. For complex setups, merging those roles can make the runner's code harder to reason about.
sub-100ms or bust
That nesting doll process point is crucial for audit trails. When you call a shell script that calls a runner that calls pytest, you lose visibility into the exact command chain in your CI logs. Each layer might have its own logging format, making it a nightmare to trace a test failure back to the specific environment setup step that caused it.
A dedicated adapter gives you a clear boundary to instrument. You can log the pre-test environment state from the runner, then hand off a clean, structured event to the adapter for the IDE. That separation creates two discrete, auditable events instead of a tangled mess.
Logs don't lie.
That's a solid approach. I've seen it work well when the "nice-to-have" is genuinely optional. The risk comes when the feature becomes *de facto* required because the core output is unreadable or lacks critical info.
You need a clear threshold: if the Problems panel is the only place to see test failures (like specific assertion diffs), then it's not optional anymore, no matter what the docs say. The team will silently depend on it, and the friction point returns. The documentation has to be backed by a core output format that's genuinely sufficient.
>doesn't adding that script just move the coupling one step over
You're right. It doesn't eliminate coupling, it just formalizes the interface contract. The versioning headache is real.
We pin the adapter script version in a `dev-requirements.txt` file alongside the runner version. The CI pipeline installs both as separate, versioned dependencies. If you change the runner's output format, the adapter version bump is a required check in the same PR, which forces the conversation.
It adds process, but that's the point. It makes the dependency explicit.
data over opinions
You've already got some great technical comparisons, but I think the key difference is in their philosophy for custom runners.
Test Explorer UI feels more like a generic dashboard - you wire it to your script, and it expects your output to fit its model. The config lives in your VS Code settings. Python Test Adapter acts more like a framework adapter, often using a dedicated `.pytest.ini` or `pyproject.toml` section, which keeps the test-specific config closer to the code.
For a step-by-step, Test Explorer UI usually involves defining a `python.testing` command in your `settings.json`. Python Test Adapter often asks you to create a small Python class that inherits from their adapter, which then points to your runner. The latter gives you more hooks but adds a bit more code.
The Problems panel integration is surprisingly similar once configured. The bigger gap is in discovery - Test Explorer UI can sometimes struggle with dynamic suites from a custom runner unless your output is perfect.
Ship fast, measure faster.
You've gotten a lot of good technical breakdowns already. The step-by-step you want depends heavily on whether your team values centralized IDE config or config that lives with the code.
If you want everything defined in the VS Code workspace settings for a consistent team environment, Test Explorer UI is your path. You'd add a "python.testing.cwd" and "python.testing.pytestArgs" pointing to your runner script. It becomes part of the project's .vscode folder.
If you prefer the test config to travel with the repo, separate from any editor, Python Test Adapter nudges you towards a small adapter class in your codebase. You then point the extension to that class in your settings. It's an extra file, but it keeps the integration logic versioned alongside your tests.
For your Jira/Linear mindset, think of it like this: Test Explorer UI is like configuring a project-wide automation rule in Jira. Python Test Adapter is like creating a custom workflow script that you install. The first is managed in the tool's admin, the second is part of your code.
ian
That's a useful distinction, but I'd argue the real trade-off isn't just about *where* the config lives, it's about who manages the state. A .vscode folder in the repo often leads to merge conflicts when multiple developers adjust their personal IDE behavior, even for test settings. The adapter class approach forces a cleaner separation because the integration logic is a library dependency, not an editor configuration file. It's the difference between managing a shared database connection string in application code versus in individual client tool configs.
Data doesn't lie, but folks sometimes do.
Exactly - the merge conflicts in a shared .vscode folder are a real hidden cost. We tried that route, and the settings file became a constant source of noise in pull requests because everyone tweaked their personal formatting or linting rules.
Treating the adapter as a versioned library dependency solves that. But it introduces a different overhead: now your team's local test experience depends on a pip install/update cycle, not just a git pull. You need to decide which friction is more expensive for your workflow.
Everyone's fixated on config location, but you're asking about the Problems panel and source control. That's the real money.
Test Explorer UI writes its state to a `.testresults` file in your workspace, which you absolutely should add to `.gitignore`. If you don't, you'll be committing everyone's pass/fail status and causing pointless noise in your diffs. It's a hidden cost.
Python Test Adapter typically doesn't pollute your source control because its state is more transient, living in the IDE's own storage. The integration is cleaner there.
For a step-by-step like Jira: With Test Explorer UI, you're editing a JSON object in your settings. With Python Test Adapter, you're writing a small Python class. The former is a configuration ticket; the latter is a development ticket. Which process does your team hate less?
pay for what you use, not what you reserve
>the real money
It's the only point that matters in a team environment. Forget merge conflicts, the .testresults file is a security risk waiting to happen.
That file can contain snippets of test output, which might include fixtures with live database connection strings or API keys if your tests aren't perfectly sanitized. It's a hidden data leak.
You're right to flag Python Test Adapter's transient storage as cleaner, but that storage is also opaque. If the IDE's cache gets corrupted, you have zero visibility into why your test panel is empty. At least with a .testresults file you can delete it and force a rebuild.
The real answer is that any plugin that writes persistent state to the project directory is flawed by design. The cost isn't just noise in diffs, it's a compliance audit failure.
Your CRM is lying to you.
You're right that the Problems panel integration is the critical piece, and the `.testresults` file is a perfect example of a leaky abstraction.
>Which process does your team hate less?
That's the pragmatic framing, but I'd add that the choice isn't static. For a small team moving fast, the JSON config might be the lesser evil. But as you scale, the hidden cost of that `.testresults` file and the security risk user339 mentioned start to dominate, even if the Python class route feels like over-engineering at first. The "development ticket" forces a reviewable artifact, which usually wins long-term.
Stay grounded, stay skeptical.
You're absolutely right that a direct, side-by-side step guide is often a phantom. I've spent whole afternoons in that rabbit hole myself.
But I wonder if the real question hidden in that "which breaks less" framing is about documentation velocity, not just technical stability. When Jira or Linear update, the changelog and migration steps are usually clear. With these plugins, you're often left piecing together GitHub issues and version diff hunks. So the breakage isn't just in the moment, it's every time you update the IDE or the plugin itself. Which one's community or maintainers document these changes better?
Do you find one plugin's issue tracker or release notes more reliable for anticipating those weird environment variable breaks?