Benchmarks¶

Method¶

• Machine. WSL Ubuntu on Windows 11, 16 cores, Python 3.14.
• Synthetic project. Forty independent functions, one test each, 200 mutants. A conftest.py sleep sets the suite length, so the same mutant pool can be measured against a fast suite and a slow one.
• Control. The score. Every configuration must produce the same one.
• Single runs, not averages. Treat any difference under about 10 percent as noise.

The feature matrix¶

Two hundred mutants, eight workers.

10.8x on the slow suite, 7.4x on the fast one. The score never moved.

Where a run's time goes¶

One selected single test run, measured by isolating each step:

pie showData
    title One 327ms selected run
    "interpreter start" : 18
    "import pytest" : 155
    "collect and configure" : 139
    "the actual test" : 15

About 95 percent is overhead. This is why warm workers exist, and why test selection alone reaches a ceiling.

Which feature pays when¶

The rule of thumb: selection removes test time, warm workers remove startup time. A project with a slow suite wants the first. A project with many cheap tests wants the second. Batching helps both, though it partly competes with selection.

A real codebase, where the numbers do not hold¶

The synthetic result does not survive contact with click. 957 mutants, a 4.21 second suite, eight workers.

The optimisations bought essentially nothing, and test selection was slightly slower than no selection at all. Every configuration lands near 1.2 seconds per mutant.

The likely cause is visible in the verdicts. Between 60 and 76 mutants time out on every run. A timeout costs the full timeout budget, here about 22 seconds, and no amount of batching or test selection shortens it. On click, waiting for timeouts dominates everything the optimisations save.

The synthetic project has no timeouts at all, which is precisely why it showed 10.8x.

Verdicts moved slightly, and why that is not the batching bug¶

The three rows disagree: 540, 540 and 541 detected; 336, 336 and 335 survived.

This is timeout classification, not misattribution. A mutant sitting near the timeout threshold is detected on a loaded machine and survives on an idle one, because timeout_factor is a wall clock budget. The kill and timeout columns also trade against each other between runs for the same reason.

The verdict matrix that runs in continuous integration uses a fixture with no timeouts, so it cannot catch this class of variation. That is a known gap rather than a passing grade.

An invalid fourth row¶

A fourth configuration was measured, but the operator set was expanded while the benchmark was running, so it planted 4596 mutants instead of 957. It is excluded here rather than reported, because it compares two different tools.

flask¶

Skipped. Its suite is not green on this machine out of the box, and Angelo will not run against a red baseline.

Comparison against mutmut¶

Same machine, same project, both at eight workers.

mutmut is about 1.2x cheaper per mutant. It uses schemata plus fork(); Angelo comes close with neither. The operator sets differ, so per mutant cost is the only fair column.

mutmut cannot run on Windows at all, because it requires fork().

The bug that made earlier numbers false¶

Four configurations that must agree reported 77, 73, 69 and 78 kills.

A .pyc file is reused when the source's recorded modification time in whole seconds and its byte size both still match. Same size replacements, such as + becoming -, written in the same second as the previous one, therefore ran the old bytecode. The mutant survived for free.

• Invisible on Windows, where a 1.9 second suite pushes writes into different seconds.
• Obvious on Linux, where a 0.2 second suite lands many runs inside one second.
• Fixed by never writing bytecode. Note that the environment variable alone is not sufficient if a .pyc already exists, because Python still reads it.

The lesson is worth stating plainly: a mutation tester's characteristic failure is inventing test gaps that do not exist. The verdict matrix in continuous integration exists because of this bug.

What these numbers do not prove¶

• The synthetic project has independent functions with disjoint tests, which is the best possible case for batching. Real code shares tests, so expect less.
• Single runs on one machine, with no confidence intervals.
• Windows process spawning is far slower than Linux. Only compare within a platform.
• A tool that dies instantly on every mutant looks fast and scores zero. Check the error count before trusting any score.

Reproducing¶

bash scripts/verdict-matrix.sh          # the correctness gate, runs in CI
bash scripts/setup-extra.sh             # a virtualenv per repository in extra/
bash scripts/bench-repo.sh extra/click  # the feature matrix on a real project

extra/ holds gitignored shallow clones of click, flask, httpx, requests, fastapi and django.

Each needs its own virtualenv. Real projects pin pytest plugins in pyproject.toml. Run them against a global Python and pytest exits 3, an internal error, before collecting anything. Angelo then refuses to start, correctly, but the fix is dependencies rather than Angelo.

django is cloned but Angelo cannot mutate it, because it uses its own test runner rather than pytest.