Skip to content

Angelo

Python mutation testing is slow and expensive, but what if it wasn't?

Angelo measures how good your tests actually are. It breaks your code on purpose, one small change at a time, and checks whether your test suite notices. It is a single Rust binary that drives your ordinary pytest suite.

On a project with 200 planted faults and a two second suite, Angelo takes 4.5 seconds where the standard approach takes 48 seconds. The score is identical.

Run your first mutation test See the evidence

Two commands and an existing pytest suite. Nothing to configure to get started.

What mutation testing measures

Test coverage tells you a line ran. It does not tell you whether anything checked the result. A test suite can execute every line of a function and still pass when that function is wrong.

Mutation testing answers the harder question. Change the code so it is definitely broken, then run the tests:

Term Meaning
Mutant One deliberate small change, such as + becoming -
Killed A test failed. Your suite caught the bug. Good.
Survived Every test passed. Your suite missed a real bug.
Score killed / (killed + survived)

A survivor is the useful output. It points at a line where you can break the behaviour and nobody complains.

def is_adult(age):
    return age >= 18

Change >= to > and the function now rejects eighteen year olds. If your tests still pass, you never tested the boundary. Angelo tells you that.

Why it is normally slow

The standard approach runs the whole test suite once per mutant. Five hundred mutants means five hundred runs. Worse, most of each run is not testing at all. Measured on a selected single test:

Step Time
Start the interpreter 18 ms
Import pytest 155 ms
Collect and configure 139 ms
Run the actual test 15 ms

Roughly 95 percent of the work is setup. That is the problem Angelo attacks.

How Angelo is fast

Four techniques, each measured, each documented in its own note.

Technique Idea Speedup
Batch mutating Test several mutants in one run 4.6x
Test selection Run only the tests that touch the mutant 2.8x
Warm workers Keep one pytest process alive 2.5x
Diff mode Only mutate lines you changed varies

Those figures are an upper bound

They come from a synthetic project with independent functions and no timeouts, which is the best case for all three. Measured against the real click codebase, where about seven percent of mutants hang until the timeout, the same techniques bought almost nothing. Waiting for timeouts dominates whatever they save.

Batching in one picture

Batching is the central idea, so it is worth understanding before anything else.

Normally each mutant gets its own run:

flowchart LR
    M1[mutant 1] --> R1[full test run]
    M2[mutant 2] --> R2[full test run]
    M3[mutant 3] --> R3[full test run]

Angelo puts several mutants into the same run:

flowchart LR
    M1[mutant 1] --> R[one test run]
    M2[mutant 2] --> R
    M3[mutant 3] --> R
    R --> V[three verdicts]

The catch is obvious. If the run fails, which mutant caused it? Angelo solves this by only grouping mutants that no single test can reach at the same time. It learns which test touches which line from one coverage run, before any mutating starts. Two mutants that share a test are kept apart. A failing test therefore points at exactly one mutant.

Read the full argument, including what happens when a result cannot be explained.

The rule that governs everything

Batching, test selection and warm workers are speed features only. None of them is allowed to change a verdict.

This is enforced, not promised. scripts/verdict-matrix.sh runs eight configurations against the same project and fails the build if any of them disagrees about the score. It runs in continuous integration on every push. A speedup that changes a result is treated as a bug.

Where to go next

Point Angelo at a project you already have and see what your tests are missing.

Get started