Test Infrastructure

INFO

This article serves as an invitation for sharing ideas to improve our test infrastructure, feel free to contact us on Discord.

In Oxc, correctness and reliability are taken extremely seriously.

We spend a great deal of time strengthening the test infrastructure to prevent problems from propagating to downstream tools.

Parser

Conformance

Parser tests from Test262, Babel, and TypeScript are used to test JavaScript, TypeScript, and JSX syntax.

The shared tasks/coverage harness reuses these fixture collections across parser, semantic analysis, code generation, transformation, minification, and ESTree tests.

For Test262, all stage 4 and regular expression tests are included.

All conformance results are stored in snapshot files for tracking changes:

• test262.snap.
• babel.snap.
• typescript.snap.

All syntax errors are written to these snapshot files for diffing changes.

Fuzzing

To ensure that the parser does not panic when encountering random data, three fuzzers are used:

1. cargo-fuzz for sending random data to the parser.
2. shift-fuzzer-js by bakkot for producing random but valid ASTs.
3. Automated-Fuzzer by qarmin, which actively reports crashes.

Memory Safety

Oxc uses an arena allocator as the memory allocator for its AST and other data. None of the AST node types have a Drop implementation. This is enforced at compile time by Oxc's allocator, which causes a compile-time error if any code attempts to allocate types in the arena which are Drop. This statically ensures that types which own heap-allocated data cannot be stored in the arena, which would result in memory leaks.

Unsafe code

Oxc uses unsafe code for performance optimizations. We aim to contain unsafe to within self-contained data structures which present safe APIs externally. Miri runs on relevant changes to the crates containing these structures.

Linter

Snapshot Diagnostics

All linter diagnostics are written to snapshot files for testing against regressions.

For example:

 ⚠ typescript-eslint(adjacent-overload-signatures): All "foo" signatures should be adjacent.
  ╭─[adjacent_overload_signatures.tsx:3:18]
2 │         function foo(s: string);
3 │         function foo(n: number);
  ·                  ───
4 │         type bar = number;
5 │         function foo(sn: string | number) {}
  ·                  ───
6 │       }
  ╰────

Ecosystem CI

oxc-ecosystem-ci runs Oxlint and Oxfmt against large repositories. Oxlint checks for false positives, regressions, and panics. Oxfmt checks for formatting differences, dropped code, errors, and panics. The repositories tested include:

• rolldown/rolldown
• napi-rs/napi-rs
• toeverything/affine
• preactjs/preact
• microsoft/vscode
• bbc/simorgh
• elastic/kibana
• DefinitelyTyped/DefinitelyTyped

Idempotency

Idempotency testing is used for integration tests and end-to-end tests on all tools.

An idempotency test follows this procedure:

let sourceText = "foo";
let printed = tool(sourceText);
let printed2 = tool(printed);
assert(printed == printed2);

For example, idempotently minifying a piece of code should yield the same result.

Codegen, transformer, minifier, and formatter are idempotently tested on Test262, Babel, and TypeScript test files.

Integration Tests

Integration tests are preferred over unit tests.

codecov currently reports line coverage.

End to End

The repository monitor-oxc performs end-to-end tests against the top 3000 npm packages from npm-high-impact.

It runs codegen, transformation, compression, mangling, whitespace removal, and formatting over the package files.

Its package.json lists the packages as dependencies:

"devDependencies": {
  "@aashutoshrathi/word-wrap": "latest",
  "@actions/http-client": "latest",
  "@adobe/css-tools": "latest",
  "@alloc/quick-lru": "latest",
 ...
  "zip-stream": "latest",
  "zod": "latest",
  "zone.js": "latest",
  "zustand": "latest"
}

And a test file that imports these packages and asserts the import:

src/dynamic.test.mjs

import test from "node:test";
import assert from "node:assert";
test("@aashutoshrathi/word-wrap", () => import("@aashutoshrathi/word-wrap").then(assert.ok));
test("@actions/http-client", () => import("@actions/http-client").then(assert.ok));
test("@adobe/css-tools", () => import("@adobe/css-tools").then(assert.ok));
test("@alloc/quick-lru", () => import("@alloc/quick-lru").then(assert.ok));
...
test("zod", () => import("zod").then(assert.ok));
test("zone.js", () => import("zone.js").then(assert.ok));
test("zustand", () => import("zustand").then(assert.ok));
test("zwitch", () => import("zwitch").then(assert.ok));

This test file is run after each of the tools (codegen, transformer, minifier, etc.) rewrites all the files in node_modules.

The workflow runs every three hours.

Isolated declarations are also tested against Vue.

---

If you have any ideas on how to improve our test infrastructure, feel free to contact us on Discord.