Step-by-step: add a custom test runner plugin to your IDE

I haven't tried Python Test Adapter, but I've been using Test Explorer UI for a few weeks with a custom setup. For pointing to a non-standard script, I had to edit the python.testing.pytestArgs setting in my workspace config. It wasn't too bad once I found the right path format.

One thing I'm still figuring out is how the results display ties into the Problems panel. Does Test Explorer UI show failures there automatically, or do you need another extension for that? I'd love to hear what you find out. Good luck

Having extensive experience integrating custom test runners in VS Code for complex Python projects, I can provide a detailed comparison of these two extensions. My observations are based on a project where we used a custom wrapper script that invoked pytest with specific environment variables and filtering logic.

For your first point regarding configuration for a non-standard script, the approach differs significantly. With Test Explorer UI and its companion Python Test Explorer, configuration is typically done via the `python.testing.pytestArgs` setting in `.vscode/settings.json`. You would add the path to your runner script there. Python Test Adapter, conversely, often requires you to define a custom "test framework" in its configuration, specifying the path to your script and any arguments in a more structured JSON object within its own settings. I've found the Python Test Adapter method to be more explicit for complex runners, as it cleanly separates the runner executable from the pytest arguments.

Regarding test discovery and result display, Test Explorer UI excels at presenting a hierarchical, interactive tree view of tests, which is superb for large suites. Python Test Adapter integrates results more tightly into the native VS Code Testing sidebar, which some teams prefer for a unified interface. For integration with the Problems panel, neither extension pushes failures there directly; you typically rely on VS Code's Python extension to parse test output for that. The key difference is that Test Explorer UI's results feel more like a separate application pane, while Python Test Adapter's output is more streamlined into the core IDE experience. If your workflow involves frequent source control operations, the native Testing panel's context menus might offer slightly smoother integration for running tests on changed files.

Hey, while I don't have deep experience with those specific VS Code extensions, I've run into similar "custom runner" scenarios when configuring data pipeline integration tests.

One thing I'd suggest checking is the extension's handling of environment variables. For custom scripts, you often need to set API keys, database URLs, or other configs that are external to the test logic itself. Some adapters make this easy through their UI or config files, others force you to bake it into your wrapper script. That integration detail can become a real time-saver (or headache) day-to-day.

Good luck with the setup! Curious which one you end up going with.

You're absolutely right about environment variables being a critical detail that can break the whole setup. It's the kind of thing you don't discover until you're debugging a failing test in the IDE.

From my experience, Test Explorer UI inherits the environment from the VS Code terminal, which can be inconsistent. Python Test Adapter, however, lets you define environment variables directly in its `.pytester.json` configuration file. That's a cleaner separation, but it also means managing config in another location. For a team, you then have to decide if that file gets committed or if it's a local developer setup artifact.

Which approach is "better" really depends on whether your team uses a centralized secrets manager or expects each dev to manage their own `.env` file.

>Does Test Explorer UI show failures there automatically

It doesn't. The Test Explorer UI extension populates its own panel, full stop. Your test failures won't appear in the Problems panel unless you use a separate linter or static analysis tool that flags the same lines.

If you want that integration, you need the "Test Explorer UI - Problem Matcher" extension. It's a separate install. It watches test runs and converts failures into diagnostic entries for the Problems panel. I've used it, and it's decent, but it adds another layer of config. You have to define patterns for it to parse your runner's specific output format.

That's a solid breakdown of the Problem Matcher add-on. It's exactly the kind of extra, fragmented setup that can trip up team adoption later. You get everyone excited about the integrated test panel, then someone asks why their failures aren't in Problems, and you have to explain it's a whole separate plugin with its own config.

For a team, I'd almost lean towards living without the Problems panel integration, just to keep the initial onboarding simpler. One less moving part to document and maintain across versions. What's been your experience rolling this out to others?

Totally agree about the extra plugin being a team friction point. I've seen this exact thing with a PR template - everyone's jazzed about the new CI pipeline, then someone asks why the test results aren't in the PR status check, and you realize you need another workflow step. 😅

Maybe a middle ground? For our team, we documented that Problems panel integration is an optional, personal productivity boost for those who want it, but the core test runner setup works without it. It's a documented "nice-to-have" vs. a required part of the dev environment.

Totally get that frustration with extra plugins. 😅 I've seen teams adopt the "optional productivity boost" approach you mentioned, and it can work, but it sometimes leads to a weird split where some folks' workflows are subtly different.

One thing that helped us was baking the decision into the team's `devcontainer.json` or a shared setup script. We made the base config include *just* the core test runner, and then added a commented-out section for the problem matcher with a link to the internal wiki explaining the trade-offs. That way, the "extra step" is at least visible and documented right in the setup file, not a separate hunt.

>I'm used to tools like Jira and Linear where the setup steps are very clear

That's the core of the issue, isn't it? IDE plugin configuration is rarely that linear or well-documented. You're chasing a phantom.

The detailed comparison you're asking for is going to lead you down a rabbit hole of JSON settings and undocumented edge cases. Both extensions will fight you on the custom runner. Test Explorer UI feels bolted together from three different components. Python Test Adapter's config schema is obscure.

Instead of a step-by-step contrast, I'd suggest a different question: which one breaks *less* when you inevitably have to pass a weird environment variable or a custom path to a coverage tool? In my experience, that's the real test, and neither wins outright. You'll end up writing more wrapper scripts to appease the plugin, which defeats the whole point.

Pick one, suffer through the config for an afternoon, and standardize that for your team. The productivity loss from endless plugin tweaking outweighs any feature difference.

Yeah, that comparison is exactly what I was looking for when I integrated a custom runner last year. Your point about clear setup steps from tools like Linear rings so true.

I actually ended up using a hybrid approach after trying both. For the config, Test Explorer UI was easier for me to get started because I could just drop my script path into the standard `pytestArgs`. But Python Test Adapter gave me finer control over the test discovery pattern later, which mattered for our monorepo setup.

One thing that might help your decision: check how your team already handles secrets for local dev. If you use a `.env` file loaded automatically, Test Explorer UI might work smoother. If you need per-test environment variables defined in the IDE config, Python Test Adapter's separate config file is actually an advantage. That was the deciding factor for us.

That's a really solid point about letting your existing secrets management strategy guide the tool choice. It's the kind of practical constraint that often overrides pure feature lists.

I've seen teams get stuck because they forced a complex IDE config for environment variables when they already had a simple, working `.env` file pattern. Adding a second, IDE-specific config just created a divergence and more places for things to go stale.

Your hybrid approach makes sense. Starting simple with Test Explorer UI, then layering in the more granular control only when you actually need it for something like monorepo discovery is a pragmatic way to avoid premature complexity.

Exactly. The divergence you describe between a `.env` file and an IDE-specific config isn't just about staleness - it's a configuration drift problem. One source of truth becomes the actual environment, the other becomes a documentation artifact that's always six months out of date.

That "hybrid approach" still risks creating two configs. If you start with Test Explorer UI using the project's `.env`, then later add Python Test Adapter for monorepo patterns, you now have `.env` AND `.pytester.json`. The moment a new required variable appears, where does it go? Someone updates the `.env` and the tests break in the IDE because they forgot the JSON config. Now you need a synchronization script, which defeats the whole purpose of keeping it simple.

Pragmatism means picking one config method and sticking to it, even if the tool isn't perfect for every edge case.

You've zeroed in on the crucial team decision there: whether that config file gets committed. I've seen teams go both ways, and the commit decision often gets made *after* the tool choice, which is backwards.

If you commit the `.pytester.json` with environment variables, you risk leaking secrets unless you're very disciplined with a secrets manager. But if you don't commit it, then every new team member has to recreate it from a template, which is a friction point you were trying to avoid in the first place. It's a tough spot.

You're right about the rabbit hole, but I think the "which one breaks less" question still leads to plugin-specific tweaking. The real shift is accepting that the IDE config *itself* is the wrapper script. Once you see the plugin's JSON schema as just another, poorly-documented scripting interface, you stop expecting a linear setup and start treating it like a devops problem - version the config that works, and update it when the toolchain changes, not when the plugin updates.

My team standardized on one, as you suggest, but we also added a lint check that compares the plugin's config file against a known-good template. It flags drifts, so the "suffering for an afternoon" becomes a one-time cost captured in code, not a recurring team burden.