How does mutation testing work in EK9?

← Fuzzing and Mutation Testing · Ref: Q753

Mutation testing measures the quality of your tests by introducing small, deliberate bugs into your source code and checking whether your tests detect them.

BASIC USAGE

  ek9 -fuzzmutate source.ek9 mutations/

This analyses source.ek9, generates mutated variants, and writes them to the mutations/ directory. Each variant has exactly one semantic change that compiles but alters behaviour.

MUTATION CATEGORIES

The mutator applies these categories of change:

  ARITH_SWAP: + becomes -, * becomes / (and vice versa).
  CMP_FLIP: < becomes >=, == becomes <>.
  GUARD_STRIP: removes if-guard conditions (if x <- expr becomes unconditional).
  CONST_BUMP: changes literal values (0 becomes 1, boundary shifts).
  NEGATE_BOOL: flips Boolean expressions (true becomes false).
  SWAP_LINES: reorders independent statements.
  DUP_OPERAND: duplicates an operand (x + y becomes x + x).
  REMOVE_CALL: removes a method call statement.

MANIFEST FILE

The mutations/ directory contains a manifest.txt listing each variant with its mutation category, location, and description. This file is machine-readable for CI integration.

KILL CHECK

Run your tests against each mutant to check detection:

  ek9 -fuzzmutate source.ek9 mutations/ -test tests.ek9

Mutants that survive (tests still pass) reveal gaps in your test suite.

OPTIONS

  -n 50          Generate at most 50 mutants (default 100).
  -seed 42       Reproducible mutation selection.
  -overwrite     Replace existing mutations/ directory.

See Q749 for fuzzing overview. See Q754 for test generation. See Q756 for the quality loop. See Q155 for writing tests. See Q206 for test coverage.

Example

defines module qa.fuzzingandmutation.mutationtesting

  <?-
    Code designed to show how mutation testing reveals weak tests.
    Each operator and comparison is a mutation target.
    If tests do not check boundary conditions, mutants survive.
  -?>

  defines function

    <?-
      A mutation tester would flip < to >=, changing the behaviour.
      If your test only checks the happy path, the mutant survives.
    -?>
    isWithinRange() as pure
      ->
        low as Integer
        high as Integer
        candidate as Integer
      <- inRange as Boolean: false

      if candidate >= low and candidate <= high
        inRange: true

    <?-
      ARITH_SWAP would change + to - in the discount calculation.
      CMP_FLIP would change > to <= in the threshold check.
    -?>
    applyDiscount() as pure
      ->
        price as Float
        discountRate as Float
      <- finalPrice as Float: price

      minimumRate <- 0.0
      maximumRate <- 1.0
      if discountRate > minimumRate and discountRate <= maximumRate
        reduction <- price * discountRate
        finalPrice: price - reduction

  defines program

    MutationTestingDemo()
      stdout <- Stdout()

      low <- 1
      high <- 10
      for candidate in [0, 1, 5, 10, 11]
        result <- isWithinRange(low, high, candidate)
        stdout.println(`${candidate} in [${low}..${high}]: ${result}`)

      price <- 100.0
      rate <- 0.25
      discounted <- applyDiscount(price, rate)
      stdout.println(`Price ${price} at ${rate} discount: ${discounted}`)
Other ways to ask this
  • What is the ek9 -fuzzmutate flag for?
  • How do I assess test quality with mutation testing in EK9?
  • What mutation categories does EK9 apply?

Coming from another language?

Java: PIT (Pitest) for mutation testing, separate Maven/Gradle plugin, slow on large codebases. Python: mutmut or cosmic-ray (external tools). Rust: no mainstream mutation testing tool. Go: no built-in, experimental tools like go-mutesting. EK9: built-in ek9 -fuzzmutate with manifest output, kill-check via -test flag, and reproducible seeds.

Keywords: fuzzmutate, test, kill, swap, strip, mutant, survive, mutation, testing, flip, mutate, quality, guard