Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_runner,cli: mark test isolation as stable #56298

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 21 additions & 16 deletions doc/api/cli.md
Original file line number Diff line number Diff line change
@@ -1025,20 +1025,6 @@ generated as part of the test runner output. If no tests are run, a coverage
report is not generated. See the documentation on
[collecting code coverage from tests][] for more details.

### `--experimental-test-isolation=mode`

<!-- YAML
added: v22.8.0
-->

> Stability: 1.0 - Early development

Configures the type of test isolation used in the test runner. When `mode` is
`'process'`, each test file is run in a separate child process. When `mode` is
`'none'`, all test files run in the same process as the test runner. The default
isolation mode is `'process'`. This flag is ignored if the `--test` flag is not
present. See the [test runner execution model][] section for more information.

### `--experimental-test-module-mocks`

<!-- YAML
@@ -2238,8 +2224,8 @@ added:
-->

The maximum number of test files that the test runner CLI will execute
concurrently. If `--experimental-test-isolation` is set to `'none'`, this flag
is ignored and concurrency is one. Otherwise, concurrency defaults to
concurrently. If `--test-isolation` is set to `'none'`, this flag is ignored and
concurrency is one. Otherwise, concurrency defaults to
`os.availableParallelism() - 1`.

### `--test-coverage-branches=threshold`
@@ -2323,6 +2309,23 @@ added:
Configures the test runner to exit the process once all known tests have
finished executing even if the event loop would otherwise remain active.

### `--test-isolation=mode`

<!-- YAML
added: v22.8.0
changes:
- version: REPLACEME
pr-url: https://github.com/nodejs/node/pull/56298
description: This flag was renamed from `--experimental-test-isolation` to
`--test-isolation`.
-->

Configures the type of test isolation used in the test runner. When `mode` is
`'process'`, each test file is run in a separate child process. When `mode` is
`'none'`, all test files run in the same process as the test runner. The default
isolation mode is `'process'`. This flag is ignored if the `--test` flag is not
present. See the [test runner execution model][] section for more information.

### `--test-name-pattern`

<!-- YAML
@@ -3058,6 +3061,7 @@ one is included in the list below.
* `--experimental-shadow-realm`
* `--experimental-specifier-resolution`
* `--experimental-strip-types`
* `--experimental-test-isolation`
* `--experimental-top-level-await`
* `--experimental-transform-types`
* `--experimental-vm-modules`
@@ -3128,6 +3132,7 @@ one is included in the list below.
* `--test-coverage-functions`
* `--test-coverage-include`
* `--test-coverage-lines`
* `--test-isolation`
* `--test-name-pattern`
* `--test-only`
* `--test-reporter-destination`
6 changes: 3 additions & 3 deletions doc/node.1
Original file line number Diff line number Diff line change
@@ -180,9 +180,6 @@ Use this flag to enable ShadowRealm support.
.It Fl -experimental-test-coverage
Enable code coverage in the test runner.
.
.It Fl -experimental-test-isolation Ns = Ns Ar mode
Configures the type of test isolation used in the test runner.
.
.It Fl -experimental-test-module-mocks
Enable module mocking in the test runner.
.
@@ -455,6 +452,9 @@ Require a minimum threshold for line coverage (0 - 100).
Configures the test runner to exit the process once all known tests have
finished executing even if the event loop would otherwise remain active.
.
.It Fl -test-isolation Ns = Ns Ar mode
Configures the type of test isolation used in the test runner.
.
.It Fl -test-name-pattern
A regular expression that configures the test runner to only execute tests
whose name matches the provided pattern.
2 changes: 1 addition & 1 deletion lib/internal/test_runner/utils.js
Original file line number Diff line number Diff line change
@@ -241,7 +241,7 @@ function parseCommandLine() {
}

if (isTestRunner) {
isolation = getOptionValue('--experimental-test-isolation');
isolation = getOptionValue('--test-isolation');
timeout = getOptionValue('--test-timeout') || Infinity;

if (isolation === 'none') {
9 changes: 5 additions & 4 deletions src/node_options.cc
Original file line number Diff line number Diff line change
@@ -145,7 +145,7 @@ void EnvironmentOptions::CheckOptions(std::vector<std::string>* errors,
debug_options_.allow_attaching_debugger = true;
} else {
if (test_isolation != "process") {
errors->push_back("invalid value for --experimental-test-isolation");
errors->push_back("invalid value for --test-isolation");
}

#ifndef ALLOW_ATTACHING_DEBUGGER_IN_TEST_RUNNER
@@ -684,10 +684,11 @@ EnvironmentOptionsParser::EnvironmentOptionsParser() {
"the line coverage minimum threshold",
&EnvironmentOptions::test_coverage_lines,
kAllowedInEnvvar);

AddOption("--experimental-test-isolation",
AddOption("--test-isolation",
"configures the type of test isolation used in the test runner",
&EnvironmentOptions::test_isolation);
&EnvironmentOptions::test_isolation,
kAllowedInEnvvar);
AddAlias("--experimental-test-isolation", "--test-isolation");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably add a TODO – unless we're fine keeping the undocumented alias, it's very low maintenance for us after all

Suggested change
AddAlias("--experimental-test-isolation", "--test-isolation");
AddAlias("--experimental-test-isolation", "--test-isolation"); // TODO(cjihrig): remove this alias in a semver major.

AddOption("--experimental-test-module-mocks",
"enable module mocking in the test runner",
&EnvironmentOptions::test_runner_module_mocks);
4 changes: 2 additions & 2 deletions test/parallel/test-runner-cli-concurrency.js
Original file line number Diff line number Diff line change
@@ -26,14 +26,14 @@ test('concurrency of two', async () => {
});

test('isolation=none uses a concurrency of one', async () => {
const args = ['--test', '--experimental-test-isolation=none'];
const args = ['--test', '--test-isolation=none'];
const cp = spawnSync(process.execPath, args, { cwd, env });
assert.match(cp.stderr.toString(), /concurrency: 1,/);
});

test('isolation=none overrides --test-concurrency', async () => {
const args = [
'--test', '--experimental-test-isolation=none', '--test-concurrency=2',
'--test', '--test-isolation=none', '--test-concurrency=2',
];
const cp = spawnSync(process.execPath, args, { cwd, env });
assert.match(cp.stderr.toString(), /concurrency: 1,/);
2 changes: 1 addition & 1 deletion test/parallel/test-runner-cli-timeout.js
Original file line number Diff line number Diff line change
@@ -21,7 +21,7 @@ test('timeout of 10ms', async () => {

test('isolation=none uses the --test-timeout flag', async () => {
const args = [
'--test', '--experimental-test-isolation=none', '--test-timeout=10',
'--test', '--test-isolation=none', '--test-timeout=10',
];
const cp = spawnSync(process.execPath, args, { cwd, env });
assert.match(cp.stderr.toString(), /timeout: 10,/);
18 changes: 9 additions & 9 deletions test/parallel/test-runner-cli.js
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@ for (const isolation of ['none', 'process']) {
// File not found.
const args = [
'--test',
`--experimental-test-isolation=${isolation}`,
`--test-isolation=${isolation}`,
'a-random-file-that-does-not-exist.js',
];
const child = spawnSync(process.execPath, args);
@@ -27,7 +27,7 @@ for (const isolation of ['none', 'process']) {
// Default behavior. node_modules is ignored. Files that don't match the
// pattern are ignored except in test/ directories.
const args = ['--test', '--test-reporter=tap',
`--experimental-test-isolation=${isolation}`];
`--test-isolation=${isolation}`];
const child = spawnSync(process.execPath, args, { cwd: join(testFixtures, 'default-behavior') });

assert.strictEqual(child.status, 1);
@@ -46,7 +46,7 @@ for (const isolation of ['none', 'process']) {
{
// Should match files with "-test.(c|m)js" suffix.
const args = ['--test', '--test-reporter=tap',
`--experimental-test-isolation=${isolation}`];
`--test-isolation=${isolation}`];
const child = spawnSync(process.execPath, args, { cwd: join(testFixtures, 'matching-patterns') });

assert.strictEqual(child.status, 0);
@@ -64,7 +64,7 @@ for (const isolation of ['none', 'process']) {
for (const type of ['strip', 'transform']) {
// Should match files with "-test.(c|m)(t|j)s" suffix when typescript support is enabled
const args = ['--test', '--test-reporter=tap', '--no-warnings',
`--experimental-${type}-types`, `--experimental-test-isolation=${isolation}`];
`--experimental-${type}-types`, `--test-isolation=${isolation}`];
const child = spawnSync(process.execPath, args, { cwd: join(testFixtures, 'matching-patterns') });

if (!process.config.variables.node_use_amaro) {
@@ -91,7 +91,7 @@ for (const isolation of ['none', 'process']) {
'--require', join(testFixtures, 'protoMutation.js'),
'--test',
'--test-reporter=tap',
`--experimental-test-isolation=${isolation}`,
`--test-isolation=${isolation}`,
];
const child = spawnSync(process.execPath, args, { cwd: join(testFixtures, 'default-behavior') });

@@ -112,7 +112,7 @@ for (const isolation of ['none', 'process']) {
const args = [
'--test',
'--test-reporter=tap',
`--experimental-test-isolation=${isolation}`,
`--test-isolation=${isolation}`,
join(testFixtures, 'index.js'),
];
const child = spawnSync(process.execPath, args, { cwd: testFixtures });
@@ -129,7 +129,7 @@ for (const isolation of ['none', 'process']) {
const args = [
'--test',
'--test-reporter=tap',
`--experimental-test-isolation=${isolation}`,
`--test-isolation=${isolation}`,
join(testFixtures, 'default-behavior/node_modules/*.js'),
];
const child = spawnSync(process.execPath, args);
@@ -143,7 +143,7 @@ for (const isolation of ['none', 'process']) {

{
// The current directory is used by default.
const args = ['--test', `--experimental-test-isolation=${isolation}`];
const args = ['--test', `--test-isolation=${isolation}`];
const options = { cwd: join(testFixtures, 'default-behavior') };
const child = spawnSync(process.execPath, args, options);

@@ -165,7 +165,7 @@ for (const isolation of ['none', 'process']) {
const args = [
'--test',
'--test-reporter=tap',
`--experimental-test-isolation=${isolation}`,
`--test-isolation=${isolation}`,
'test/fixtures/test-runner/default-behavior/index.test.js',
'test/fixtures/test-runner/nested.js',
'test/fixtures/test-runner/invalid-tap.js',
2 changes: 1 addition & 1 deletion test/parallel/test-runner-coverage.js
Original file line number Diff line number Diff line change
@@ -260,7 +260,7 @@ test.skip('coverage works with isolation=none', skipIfNoInspector, () => {
'--experimental-test-coverage',
'--test-reporter',
'tap',
'--experimental-test-isolation=none',
'--test-isolation=none',
];
const result = spawnSync(process.execPath, args, {
env: { ...process.env, NODE_TEST_TMPDIR: tmpdir.path },
2 changes: 1 addition & 1 deletion test/parallel/test-runner-extraneous-async-activity.js
Original file line number Diff line number Diff line change
@@ -52,7 +52,7 @@ const { spawnSync } = require('child_process');
{
const child = spawnSync(process.execPath, [
'--test',
'--experimental-test-isolation=none',
'--test-isolation=none',
fixtures.path('test-runner', 'async-error-in-test-hook.mjs'),
]);
const stdout = child.stdout.toString();
2 changes: 1 addition & 1 deletion test/parallel/test-runner-force-exit-failure.js
Original file line number Diff line number Diff line change
@@ -10,7 +10,7 @@ for (const isolation of ['none', 'process']) {
'--test',
'--test-reporter=spec',
'--test-force-exit',
`--experimental-test-isolation=${isolation}`,
`--test-isolation=${isolation}`,
fixture,
];
const r = spawnSync(process.execPath, args);
8 changes: 4 additions & 4 deletions test/parallel/test-runner-no-isolation-filtering.js
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@ test('works with --test-only', () => {
const args = [
'--test',
'--test-reporter=tap',
'--experimental-test-isolation=none',
'--test-isolation=none',
'--test-only',
fixture1,
fixture2,
@@ -35,7 +35,7 @@ test('works without --test-only', () => {
const args = [
'--test',
'--test-reporter=tap',
'--experimental-test-isolation=none',
'--test-isolation=none',
fixture1,
fixture2,
];
@@ -57,7 +57,7 @@ test('works with --test-name-pattern', () => {
const args = [
'--test',
'--test-reporter=tap',
'--experimental-test-isolation=none',
'--test-isolation=none',
'--test-name-pattern=/test one/',
fixture1,
fixture2,
@@ -75,7 +75,7 @@ test('works with --test-skip-pattern', () => {
const args = [
'--test',
'--test-reporter=tap',
'--experimental-test-isolation=none',
'--test-isolation=none',
'--test-skip-pattern=/one/',
fixture1,
fixture2,
2 changes: 1 addition & 1 deletion test/parallel/test-runner-no-isolation-hooks.mjs
Original file line number Diff line number Diff line change
@@ -4,7 +4,7 @@ import { test } from 'node:test';

const testArguments = [
'--test',
'--experimental-test-isolation=none',
'--test-isolation=none',
];

const testFiles = [
6 changes: 3 additions & 3 deletions test/parallel/test-runner-snapshot-tests.js
Original file line number Diff line number Diff line change
@@ -349,7 +349,7 @@ test('snapshots from multiple files (isolation=none)', async (t) => {
await t.test('fails prior to snapshot generation', async (t) => {
const args = [
'--test',
'--experimental-test-isolation=none',
'--test-isolation=none',
fixture,
fixture2,
];
@@ -370,7 +370,7 @@ test('snapshots from multiple files (isolation=none)', async (t) => {
await t.test('passes when regenerating snapshots', async (t) => {
const args = [
'--test',
'--experimental-test-isolation=none',
'--test-isolation=none',
'--test-update-snapshots',
fixture,
fixture2,
@@ -391,7 +391,7 @@ test('snapshots from multiple files (isolation=none)', async (t) => {
await t.test('passes when snapshots exist', async (t) => {
const args = [
'--test',
'--experimental-test-isolation=none',
'--test-isolation=none',
fixture,
fixture2,
];
2 changes: 1 addition & 1 deletion test/parallel/test-runner-watch-mode.mjs
Original file line number Diff line number Diff line change
@@ -47,7 +47,7 @@ async function testWatch({
const ran2 = Promise.withResolvers();
const child = spawn(process.execPath,
['--watch', '--test', '--test-reporter=spec',
isolation ? `--experimental-test-isolation=${isolation}` : '',
isolation ? `--test-isolation=${isolation}` : '',
file ? fixturePaths[file] : undefined].filter(Boolean),
{ encoding: 'utf8', stdio: 'pipe', cwd: tmpdir.path });
let stdout = '';
Loading