Skip to content

Add test filtering to e2e tests#7707

Draft
eddyashton wants to merge 1 commit intomicrosoft:mainfrom
eddyashton:py_e2e_test_filter
Draft

Add test filtering to e2e tests#7707
eddyashton wants to merge 1 commit intomicrosoft:mainfrom
eddyashton:py_e2e_test_filter

Conversation

@eddyashton
Copy link
Member

Our current end-to-end Python test runner is hand-written, and invoked from ctest. This makes it hard to run single specific tests - they don't have standalone entry points, and we can't pass extra args to ctest for filtering.

The current solution is to comment out specific tests. That's manual and error-prone. I think we should make the runner richer, and able to apply this filtering.

There was a regex -R option for when you called python ../tests/... ... directly, but I don't think that's a common use case. This approach uses an envvar instead, which isn't ideal, but is a sensible way to plumb something past ctest. So it works whether you call python ... or ctest ....

Opening for discussion now. For full utility, we'd want to improve our @reqs.description coverage (this only filters the test functions that are correctly decorated!), but I think we should do that as a separate pass.

@achamayou
Copy link
Member

I think I would like to find a way to drop the tests.sh wrapper, by either using uv or some other auto-venv setup feature, and cut ctest out as well.

Now that we have one set of cross-platform binaries, I think the value of ctest is essentially zero or mildly negative. We may still want to create a test config with directories, but that's a one-off job that does not require a runner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants