Proposal: Introduce Mutation Testing to Enhance Test Coverage

[IamLizu]

I wanted to suggest introducing mutation testing to help strengthen our test suite. It adds another layer of confidence by checking how well our tests catch real issues, not just that they pass. This could help us ensure our tests are more effective in spotting potential bugs.

I'm happy to help explore how we could integrate mutation testing and improve our overall test coverage.

I know the team already has a solid testing strategy in place, so I'd love to hear your thoughts and see if this idea fits with the project's direction.

[ljharb]

What is “mutation testing”?

[IamLizu]

Hi @ljharb 👋

Sure, I'd be happy to explain! Mutation testing is a technique where small changes (mutations) are deliberately introduced to the codebase to test how well our existing test suite catches them. It’s essentially a way to assess whether our tests are effective in identifying potential issues. Somehow, I feel you know that already 😅.

In this context, I was referring to using a tool like Stryker, which mutates source files automatically and helps us find gaps in our test coverage. I noticed we currently don't have any Stryker configuration in the repo, so I was wondering if we are perhaps using another tool or approach for mutation testing?

In the meantime, I tried setting up Stryker myself to see if it could add value. Here's the initial configuration I used:

module.exports = {
  mutate: ["lib/**.js"],
  testRunner: "mocha",
  reporters: ["html", "clear-text"],
  mochaOptions: {
    spec: ["test/**/*.js"],
  },
  coverageAnalysis: "off",
  disableTypeChecks: "/examples/**/**/**.html",
};

While testing, I found that some of our test cases may need a bit of updating—particularly around how fixture paths are referenced in a few places. Currently, the tests pass on their own, but they fail with Stryker due to these path issues.

I didn't take a deeper look yet, but it seems we might need to rewrite a number of test files to address these issues. I wanted to bring it up here first before putting in that effort, just to ensure it aligns with what we want to do moving forward.

I'd be more than happy to help resolve these issues if mutation testing is something we'd like to explore further. Looking forward to your thoughts on this!

cc: @expressjs/express-tc

[UlisesGascon]

Let's discuss this in a future meeting, seems interesting 👍

[ljharb]

I've literally never heard of this or seen it used in any codebase in my entire career - it does seem interesting, though.

[wesleytodd]

We should probably clarify the label meaning? I removed these because meeting is the label for meeting agendas and top priority was intended to be the ones we really for sure wanted to do.

[IamLizu]

I've literally never heard of this or seen it used in any codebase in my entire career - it does seem interesting, though.

It is comparatively a new thing for sure. I guess we will talk about this in future, but in the meantime here's something you can read which includes references from research papers.

Edit: The practice is new but it has existed for more than 50 years 😆

[ljharb]

It seems awesome, but I'm very unclear on how it could be done in a practical sense for JavaScript.

[kibertoad]

@ljharb see https://www.npmjs.com/package/@stryker-mutator/core