Skip to content

Add --DRT-testfilter and --DRT-testmode=list to the default unittest runner - #23981

Open
atilaneves wants to merge 2 commits into
dlang:masterfrom
atilaneves:unittest-filter-upstream
Open

atilaneves wants to merge 2 commits into
dlang:masterfrom
atilaneves:unittest-filter-upstream

Conversation

@atilaneves

@atilaneves atilaneves commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds exact unittest selection and listing to the default per-unittest runner.

--DRT-testmode=list                 → print available unittest names
--DRT-testfilter=name1,name2         → run exactly those unittests

Names consist of the module name followed by the compiler-generated unittest name, as printed by list:

./app --DRT-testfilter=my.pkg.mod.__unittest_L9_C1
./app --DRT-testfilter=my.pkg.mod.__unittest_L9_C1,other.mod.__unittest_L5_C1

The list accepts exact names only. Package/module selection, prefix matching, and exclusions are not supported. Names are not mangled symbols; user-defined names are intended as separate future work.

Unknown names and empty entries are errors, reported before any tests run. An empty option value means no filtering. Duplicate names run the test once, in the runner's usual order.

--DRT-testmode=list prints one name per line on stdout without running tests or main. Combining it with --DRT-testfilter lists only the selected tests. Filtering cannot be combined with run-main, test-only, or test-or-main.

Modules without individual unittest metadata cannot be listed or selected; unfiltered execution retains its existing fallback. Custom handlers set through Runtime.moduleUnitTester or Runtime.extendedModuleUnitTester are unaffected.

Evidence

  • Before: The proposed filter accepted package/module prefixes and exclusions.
    After: It accepts only comma-separated exact unittest names. Tests cover single and multiple names, duplicate names, failures, listing, empty values, malformed lists, unknown names, and incompatible modes.
  • Druntime's test/unittest/.run suite passes with release and debug runtime builds on Linux x86-64.

Merge Danger

Door: Two-way for implementation changes; the new command-line interface introduces a compatibility commitment.

Blast Radius: Druntime. Selection and listing affect the default unittest runner when explicitly requested.

@github-actions

github-actions Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

DMD perf check

No differences above the noise thresholds.

All measurements
Metric Base PR Δ
compile hello.d (instr) 203.6 M 203.6 M +0.003%
compile hello.d -O -release (instr) 221.2 M 221.2 M +0.003%
compile Phobos (instr) 4,703.6 M 4,704.1 M +0.011%
compile Phobos codegen (instr) 1,419.3 M 1,419.6 M +0.017%
compile vibe.d (instr) 13,558.7 M 13,560.4 M +0.013%
compile ctRegex tests (instr) 29,184.5 M 29,184.7 M +0.001%
dmd binary size (stripped) 8.17 MB 8.17 MB 0.00%
hello binary size (stripped) 0.72 MB 0.72 MB 0.00%
peak RSS (compile hello.d) 44.29 MB 44.25 MB -0.10%
peak RSS (compile Phobos) 610.5 MB 610.6 MB +0.01%
peak RSS (compile vibe.d) 1881 MB 1882 MB +0.04%
peak RSS (compile ctRegex tests) 2041 MB 2040 MB 0.00%
page faults (compile hello.d) 8,729 8,729 0.00%
page faults (compile Phobos) 152,322 152,323 0.00%
page faults (compile vibe.d) 470,516 470,210 -0.07%
page faults (compile ctRegex tests) 497,882 497,883 0.00%
compile dmd itself (wall) 8.3 s 8.2 s -1.04%
compile ctRegex tests (wall) 2,735 ms 2,706 ms -1.09%
compile hello.d (wall) 43.9 ms 43.9 ms +0.02%
compile Phobos (wall) 934 ms 942 ms +0.85%

dc8b891 vs merge-base f10507a · about these metrics · historical dashboard

@rikkimax

rikkimax commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

I would suggest mangling and not this: ./app --DRT-testfilter=my.pkg.mod.__unittest_L9_C1 # a single unittest

Due to things like template instantiations, it's going to be useless, and all ways to get the name anyway should provide you with the mangling as well.

@dkorpel

dkorpel commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

How is this intended to be used in a real world project?

@atilaneves

Copy link
Copy Markdown
Contributor Author

I would suggest mangling and not this: ./app --DRT-testfilter=my.pkg.mod.__unittest_L9_C1 # a single unittest

Due to things like template instantiations, it's going to be useless, and all ways to get the name anyway should provide you with the mangling as well.

The idea is to turn on support for running tests by name, and then to add support for actually naming them something reasonable separately.

@atilaneves

Copy link
Copy Markdown
Contributor Author

How is this intended to be used in a real world project?

Like in unit-threaded: users give their tests names and use those names to run specific tests. Having testing support in the language is great, but the out-of-the-box experience isn't. There's a reason silly and unit-threaded are two of the most popular dub projects, and that a DConf 2026 talk had a Google Test inspired test runner.

@dkorpel

dkorpel commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

Having testing support in the language is great, but the out-of-the-box experience isn't

I get that, I should have been more precise in my question. I'm asking specifically about the user experience and integration with tools. Are users supposed to manually type --DRT=xxx flags in the console and copy-pasting mangled names? That feels rather clumsy. If instead this is supposed to be driven by dub, then the information can be passed simpler than this bespoke filter syntax.

@dkorpel

dkorpel commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

An unknown --DRT-testmode value is now reported as an error before anything runs. It used to run the unittests and then hit an assert(0), which is a bare halt (SIGILL) in a release druntime.

Can this be done in a separate PR closing an issue?

@nordlow

nordlow commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

What about adding a flag that outputs execution time of individual unittests upon exit?

@atilaneves

Copy link
Copy Markdown
Contributor Author

Split out the unknown --DRT-testmode fix into #24030 (fixes #24029); this PR no longer contains it.

@atilaneves

Copy link
Copy Markdown
Contributor Author

Having testing support in the language is great, but the out-of-the-box experience isn't

I get that, I should have been more precise in my question. I'm asking specifically about the user experience and integration with tools. Are users supposed to manually type --DRT=xxx flags in the console and copy-pasting mangled names? That feels rather clumsy. If instead this is supposed to be driven by dub, then the information can be passed simpler than this bespoke filter syntax.

Are users supposed to manually type --DRT=xxx flags in the console

I don't know how else to do it. Suggestions welcome.

copy-pasting mangled names?

Temporarily, the idea is to merge this first then another PR would allow users to give their tests "proper" names.

If instead this is supposed to be driven by dub, then the information can be passed simpler than this bespoke filter syntax.

Aren't those orthogonal? By which I mean: if dub is going to pass these anyway, does it matter that much how ergonomic the flags are?

@atilaneves

Copy link
Copy Markdown
Contributor Author

An unknown --DRT-testmode value is now reported as an error before anything runs. It used to run the unittests and then hit an assert(0), which is a bare halt (SIGILL) in a release druntime.

Can this be done in a separate PR closing an issue?

#24030

@nordlow

nordlow commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

What about adding a flag that outputs execution time of individual unittests upon exit?

Key question is output formatting and whether timings of individual unittests should be reported.

See master...nordlow:dmd:unit-test-duration-report. No tests added yet.

@thewilsonator

Copy link
Copy Markdown
Contributor

Dependant PR merged. This needs a rebase now.

@atilaneves
atilaneves force-pushed the unittest-filter-upstream branch from cb1bde1 to 688bb8d Compare October 14, 2026 07:49

@dkorpel dkorpel left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know how else to do it. Suggestions welcome.

A comma separated list of mangles of unittests to run.

Aren't those orthogonal? By which I mean: if dub is going to pass these anyway, does it matter that much how ergonomic the flags are?

It matters because this PR adds 200 lines of new flag parsing code in druntime, exposed as a public API so it will have to be maintained for at least a deprecation/edition cycle. I prefer to keep druntime small, and I'm not sure we need to commit this filter logic as one of druntime's responsibilities. If experience shows this is desirable, then we can always add it. But I can already see this spiral into "oops we now have a duplicate regex engine in druntime for test filtering, if only we left filter syntax as the build system's responsibility". That's why I'd like to see rationale for specifically this PR's --DRT flag api, or alternatively, label it explicitly as experimental so we can change it later.

atilaneves and others added 2 commits October 26, 2026 17:05
…runner

--DRT-testfilter takes a comma-separated list of package, module or
unittest names, matched on whole dot-separated components, with a `-`
prefix to exclude. A name matching nothing, an empty selection, or use
with a per-module --DRT-testmode is an error.

--DRT-testmode=list prints the names of the (selected) unittests
without running them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@atilaneves
atilaneves force-pushed the unittest-filter-upstream branch from 0042528 to 4898c6a Compare October 26, 2026 16:06
@atilaneves

Copy link
Copy Markdown
Contributor Author

I don't know how else to do it. Suggestions welcome.

A comma separated list of mangles of unittests to run.

I simplified it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants