Test Generation in EK9

EK9 generates edge-case tests and mutation variants for your code from the compiler's own symbol table. The generator already knows your types, methods, operators and constructors, so there is nothing to configure: it knows what to test.

Looking for compiler fuzzing? To test the compiler itself, see Compiler Fuzzing. This page is about generating tests for your code.

• The Problem - why generate tests
• Two Modes Overview - test generation and mutation testing
• Mode 1: Test Generation - generate edge-case @Test programs
• Mode 2: Mutation Testing - generate mutation variants
• The AI-Compiler Feedback Loop - the workflow
• CLI Reference - every flag
• Comparison with Industry Tools - EK9 beside PIT, Hypothesis and AFL

The Problem

The tests pass and the coverage looks adequate, but edge cases are still untested: boundary values, empty collections, unset values, the extremes of a type. Developers write tests for the expected cases; the unexpected ones are where the bugs hide.

Tools such as PIT/Pitest (mutation testing), Hypothesis/QuickCheck (property-based testing) and AFL/libFuzzer (coverage-guided fuzzing) address this, but each needs its own setup, configuration and integration. EK9 puts it in the compiler: the generator has the symbol table, knows your types, and writes EK9 test files that compile.

Two Modes Overview

Both modes compile every candidate through the PRE_IR_CHECKS phase before writing it, and keep only the candidates that compile.

Test Generation

Usage

The second argument is the output file, where the generated tests are written. It must be a dev source (under dev/), because @Test programs compile nowhere else; any other path is refused before anything is generated. If the output file already exists, EK9 refuses to overwrite it and reports an error, so a curated test file is never destroyed by accident. Add -overwrite to replace it:

$ ek9 -fuzztest mylib.ek9 dev/mylib_tests.ek9  # Generate tests to dev/mylib_tests.ek9
$ ek9 -fuzztest mylib.ek9 dev/mylib_tests.ek9 -overwrite  # Replace existing output
$ ek9 -fuzztest mylib.ek9 dev/mylib_tests.ek9 -n 500 -seed 42  # Up to 500 candidates, reproducible

What It Analyses

The generator compiles your source through PRE_IR_CHECKS, then reads the symbol table for what can be tested:

• Constructors - default and parameterised constructors of every non-abstract type
• Methods - instance methods, with their parameter types
• Operators - the operators you define (+, <, ?, $ and so on)
• Functions - module-level functions, with their parameter types

For each parameter, the generator takes values from that type's edge-case value pool. Every pool includes an unset value (for example String()), so the tests check that your code handles the unset state of EK9's tri-state semantics.

Edge-Case Value Pools

Output Format

Given this small library:

#!ek9
defines module site.testgeneration.account

  defines class

    Account
      balance as Integer: 0

      Account()
        -> initialBalance as Integer
        balance: initialBalance

      deposit()
        -> amount as Integer
        if amount > 0
          balance += amount

      current() as pure
        <- rtn as Integer: balance

      operator $ as pure
        <- rtn as String: `Account(${balance})`

      default operator ?

Download testGeneration_account.ek9

The output file (the second argument) is a .ek9 file of @Test programs that compiles: scaffolding. The generator writes the calls, the edge values and the wiring; each //AI: line marks an expectation that you set from the requirement, never from what the code returns today. The edge values are named constants in one shared dev/constants.ek9:

$ ek9 -fuzztest testGeneration_account.ek9 dev/account_tests.ek9 -seed 42 && cat dev/account_tests.ek9
#!ek9
<?-
  Fuzz-generated tests for testGeneration_account.ek9
  Generated: date time
  Candidates: 34, Survivors: 33 (97%)
  Source: testGeneration_account.ek9
  Seed: 42

  SCAFFOLDING, not finished tests. The generator wrote the calls, the edge values
  and the wiring. It did NOT decide what the right answer is.

  Each //AI: line marks an expectation YOU must set, from the REQUIREMENT the
  code is meant to satisfy — never from what the implementation currently returns.
  Running these and keeping whichever assertion passes cements today's behaviour,
  bugs included, as green forever. A RED test is this tool working: decide which is
  wrong, the expectation or the code.

  Keep the useful ones, delete the rest.
  These @Test programs live in site.testgeneration.account.tests and reference the source module (site.testgeneration.account); add this file to your project's dev sources, then run: ek9 -t
-?>
defines module site.testgeneration.account.tests

  references
    site.testgeneration.account::Account

  defines program

    // Construct Account with Integer edge: Integer()
    @Test
    testFuzz_Account_construct_Integer_0()
      instance <- Account(initialBalance: Integer())
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: 0
    @Test
    testFuzz_Account_construct_0_1()
      instance <- Account(initialBalance: EDGE_0)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: 1
    @Test
    testFuzz_Account_construct_1_2()
      instance <- Account(initialBalance: EDGE_1)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: -1
    @Test
    testFuzz_Account_construct_1_3()
      instance <- Account(initialBalance: EDGE_NEG_1)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: 9223372036854775807
    @Test
    testFuzz_Account_construct_9223372036854775807_4()
      instance <- Account(initialBalance: EDGE_9223372036854775807)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: -9223372036854775808
    @Test
    testFuzz_Account_construct_9223372036854775808_5()
      instance <- Account(initialBalance: EDGE_NEG_9223372036854775808)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: 9223372036854775806
    @Test
    testFuzz_Account_construct_9223372036854775806_6()
      instance <- Account(initialBalance: EDGE_9223372036854775806)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: -9223372036854775807
    @Test
    testFuzz_Account_construct_9223372036854775807_7()
      instance <- Account(initialBalance: EDGE_NEG_9223372036854775807)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: 2147483647
    @Test
    testFuzz_Account_construct_2147483647_8()
      instance <- Account(initialBalance: EDGE_2147483647)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: -2147483648
    @Test
    testFuzz_Account_construct_2147483648_9()
      instance <- Account(initialBalance: EDGE_NEG_2147483648)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: 2147483646
    @Test
    testFuzz_Account_construct_2147483646_10()
      instance <- Account(initialBalance: EDGE_2147483646)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: -2147483647
    @Test
    testFuzz_Account_construct_2147483647_11()
      instance <- Account(initialBalance: EDGE_NEG_2147483647)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: 42
    @Test
    testFuzz_Account_construct_42_12()
      instance <- Account(initialBalance: EDGE_42)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: 100
    @Test
    testFuzz_Account_construct_100_13()
      instance <- Account(initialBalance: EDGE_100)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with Integer edge: 255
    @Test
    testFuzz_Account_construct_255_14()
      instance <- Account(initialBalance: EDGE_255)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Construct Account with default constructor
    @Test
    testFuzz_Account_defaultConstruct_15()
      instance <- Account()
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: Integer()
    @Test
    testFuzz_Account_deposit_Integer_16()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: Integer())
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: 0
    @Test
    testFuzz_Account_deposit_0_17()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_0)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: 1
    @Test
    testFuzz_Account_deposit_1_18()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_1)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: -1
    @Test
    testFuzz_Account_deposit_1_19()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_NEG_1)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: 9223372036854775807
    @Test
    testFuzz_Account_deposit_9223372036854775807_20()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_9223372036854775807)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: -9223372036854775808
    @Test
    testFuzz_Account_deposit_9223372036854775808_21()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_NEG_9223372036854775808)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: 9223372036854775806
    @Test
    testFuzz_Account_deposit_9223372036854775806_22()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_9223372036854775806)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: -9223372036854775807
    @Test
    testFuzz_Account_deposit_9223372036854775807_23()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_NEG_9223372036854775807)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: 2147483647
    @Test
    testFuzz_Account_deposit_2147483647_24()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_2147483647)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: -2147483648
    @Test
    testFuzz_Account_deposit_2147483648_25()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_NEG_2147483648)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: 2147483646
    @Test
    testFuzz_Account_deposit_2147483646_26()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_2147483646)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: -2147483647
    @Test
    testFuzz_Account_deposit_2147483647_27()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_NEG_2147483647)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: 42
    @Test
    testFuzz_Account_deposit_42_28()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_42)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: 100
    @Test
    testFuzz_Account_deposit_100_29()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_100)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.deposit() with Integer edge: 255
    @Test
    testFuzz_Account_deposit_255_30()
      instance <- Account(initialBalance: EDGE_0)
      instance.deposit(amount: EDGE_255)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance?` or flip to `assert not instance?`
      assert instance?

    // Call Account.current() with no args
    @Test
    testFuzz_Account_current_noArgs_31()
      instance <- Account(initialBalance: EDGE_0)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert instance.current()?` or flip to `assert not instance.current()?`
      assert instance.current()?

    // Apply operator ? on Account
    @Test
    testFuzz_Account_op_isSet_32()
      a <- Account(initialBalance: EDGE_0)
      //AI: set this expectation from the REQUIREMENT, not from what the code returns
      //    keep `assert a?` or flip to `assert not a?`
      assert a?

//EOF

$ ek9 -fuzztest testGeneration_account.ek9 dev/account_tests.ek9 -seed 42 && cat dev/constants.ek9
#!ek9
<?-
  Edge-value constants for site.testgeneration.account.tests.
  Generated by ek9_generate_edge_case_tests and SHARED by every generated test
  file in this module - they declare none of their own, so this module always has
  exactly one file carrying constants (E11066/E11067 count them per module).
-?>
defines module site.testgeneration.account.tests

  defines constant

    EDGE_0 <- 0
    EDGE_1 <- 1
    EDGE_NEG_1 <- -1
    EDGE_9223372036854775807 <- 9223372036854775807
    EDGE_NEG_9223372036854775808 <- -9223372036854775808
    EDGE_9223372036854775806 <- 9223372036854775806
    EDGE_NEG_9223372036854775807 <- -9223372036854775807
    EDGE_2147483647 <- 2147483647
    EDGE_NEG_2147483648 <- -2147483648
    EDGE_2147483646 <- 2147483646
    EDGE_NEG_2147483647 <- -2147483647
    EDGE_42 <- 42
    EDGE_100 <- 100
    EDGE_255 <- 255
    EDGE_0_0 <- 0.0
    EDGE_1_0 <- 1.0
    EDGE_NEG_1_0 <- -1.0
    EDGE_0_0001 <- 0.0001
    EDGE_NEG_0_0001 <- -0.0001
    EDGE_1_7976931348623157E308 <- 1.7976931348623157E308
    EDGE_NEG_1_7976931348623157E308 <- -1.7976931348623157E308
    EDGE_4_9E_324 <- 4.9E-324
    EDGE_999999_999 <- 999999.999
    EDGE_NEG_999999_999 <- -999999.999
    EDGE_3_14159 <- 3.14159
    EDGE_0_5 <- 0.5
    EDGE_EMPTY <- ""
    EDGE_SPACE <- " "
    EDGE_A <- "a"
    EDGE_TEST <- "test"
    EDGE_HELLO_WORLD <- "Hello World"
    EDGE_SPECIAL <- "special!@#%"
    EDGE_VERY_LONG_STRING_THAT_TE <- "very long string that tests buffer handling capacity"
    EDGE_LOREM_IPSUM_DOLOR_SIT_AM <- "Lorem ipsum dolor sit amet consectetur adipiscing elit Lorem ipsum dolor sit amet consectetur adipiscing elit Lorem ipsum dolor sit amet consectetur adipiscing elit Lorem ipsum dolor sit amet consectetur adipiscing elit Lorem ipsum dolor sit amet consectetur adipiscing elit "
    EDGE_TRUE <- true
    EDGE_FALSE <- false
    EDGE_A_1 <- 'a'
    EDGE_Z <- 'z'
    EDGE_A_2 <- 'A'
    EDGE_Z_1 <- 'Z'
    EDGE_0_1 <- '0'
    EDGE_9 <- '9'
    EDGE_VAL <- ' '
    EDGE_1970_01_01 <- 1970-01-01
    EDGE_2024_01_15 <- 2024-01-15
    EDGE_2099_12_31 <- 2099-12-31
    EDGE_2024_02_29 <- 2024-02-29
    EDGE_2000_01_01 <- 2000-01-01
    EDGE_00_00_00 <- 00:00:00
    EDGE_23_59_59 <- 23:59:59
    EDGE_12_00_00 <- 12:00:00
    EDGE_00_00_01 <- 00:00:01
    EDGE_10_30_00 <- 10:30:00
    EDGE_1970_01_01T00_00_00Z <- date-time
    EDGE_2024_01_15T10_30_00Z <- date-time
    EDGE_2099_12_31T23_59_59Z <- date-time
    EDGE_2024_02_29T12_00_00Z <- date-time
    EDGE_PT0S <- PT0S
    EDGE_PT1S <- PT1S
    EDGE_PT1H <- PT1H
    EDGE_PT24H <- PT24H
    EDGE_PT30S <- PT30S
    EDGE_PT9223372036854775807S <- PT9223372036854775807S
    EDGE_P1D <- P1D
    EDGE_P365D <- P365D
    EDGE_0MS <- nms
    EDGE_1MS <- nms
    EDGE_500MS <- nms
    EDGE_1000MS <- nms
    EDGE_999999MS <- nms
    EDGE_9223372036854775807MS <- nms
    EDGE_000000 <- #000000
    EDGE_FFFFFF <- #FFFFFF
    EDGE_FF0000 <- #FF0000
    EDGE_00FF00 <- #00FF00
    EDGE_0000FF <- #0000FF
    EDGE_0MM <- 0mm
    EDGE_1MM <- 1mm
    EDGE_10CM <- 10cm
    EDGE_100M <- 100m
    EDGE_1KM <- 1km
    EDGE_0_00_USD <- 0.00#USD
    EDGE_0_01_USD <- 0.01#USD
    EDGE_100_00_USD <- 100.00#USD
    EDGE_999999_99_USD <- 999999.99#USD
    EDGE_1_00_GBP <- 1.00#GBP
    EDGE_0B0 <- 0b0
    EDGE_0B1 <- 0b1
    EDGE_0B1010 <- 0b1010
    EDGE_0B11111111 <- 0b11111111
    EDGE_0B0000000000000000 <- 0b0000000000000000
    EDGE_VAL_1 <- /./
    EDGE_VAL_2 <- /^$/
    EDGE_D <- /\d+/
    EDGE_VAL_3 <- /.*/
    EDGE_5000 <- 5000
    EDGE_1_0E_12 <- 1.0E-12
    EDGE_5000_0 <- 5000.0
    EDGE_XXXXXXXXXXXXXXXXXXXXXXXX <- "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    EDGE_7 <- '7'
    EDGE_0CM <- 0cm
    EDGE_1000KM <- 1000km
    EDGE_ABC <- /abc/

//EOF

What to do with the output: review the generated tests, keep the ones that expose interesting edge cases and delete the rest. Run what is left with ek9 -t.

Console Output

$ ek9 -fuzztest testGeneration_account.ek9 dev/account_tests.ek9 -seed 42
EK9 Test Generation: analyzing testGeneration_account.ek9
Seed: 42, Max candidates: 200
Found 1 types, 0 functions
Tests will be declared in module site.testgeneration.account.tests (source module is site.testgeneration.account) — emitting a references block
Generated 34 test candidates
Survivors: 33 / 34 (97%)
Constants written to: project-directory/dev/constants.ek9
Output written to: dev/account_tests.ek9

Mutation Testing

Usage

The second argument is the output directory, where the mutation variants are written. If the directory already exists, EK9 refuses to overwrite it and reports an error, so a curated mutation set is never destroyed by accident. Add -overwrite to replace it:

$ ek9 -fuzzmutate myapp.ek9 mutations/  # Generate mutations to mutations/ directory
$ ek9 -fuzzmutate myapp.ek9 mutations/ -n 200  # Up to 200 candidates
$ ek9 -fuzzmutate myapp.ek9 mutations/ -seed 123  # Reproducible mutations
$ ek9 -fuzzmutate myapp.ek9 mutations/ -overwrite  # Replace existing directory

How It Works

Mutation testing asks of your tests: if I introduce a small bug, do they catch it? The generator makes variants of your source, each with one mutation. If your tests still pass against a variant (a mutant), that mutation is an undetected fault: a gap in your tests.

Each mutation is a single-point change to one line of your source, so any test failure traces back to exactly one change.

Mutation Categories

The categories follow EK9's semantics. There are no break, continue or return mutations, because EK9 does not have those statements.

Output Format

The mutations are written to a directory: one .ek9 file per mutant that compiles, plus a manifest.txt summary. On the library above:

$ ek9 -fuzzmutate testGeneration_account.ek9 mutations/ -seed 123
EK9 Mutation Testing: analyzing testGeneration_account.ek9
Seed: 123, Max candidates: 100
Generated 6 mutation candidates
Survivors: 6 / 6 (100%)
Output written to: mutations/

Reading the Manifest

$ ek9 -fuzzmutate testGeneration_account.ek9 mutations/ -seed 123 && cat mutations/manifest.txt
# Mutation manifest for testGeneration_account.ek9
# Generated: date time
# Candidates: 6, Survivors: 6 (100%), Seed: 123
#
# File                                     Category        Line   Original             Mutated
mutation_000_boundary_L7.ek9               BOUNDARY        7      0                    -1
mutation_001_boundary_L7.ek9               BOUNDARY        7      0                    1
mutation_002_cmp_flip_L15.ek9              CMP_FLIP        15     >                    <
mutation_003_cmp_flip_L15.ek9              CMP_FLIP        15     >                    >=
mutation_004_boundary_L15.ek9              BOUNDARY        15     0                    -1
mutation_005_boundary_L15.ek9              BOUNDARY        15     0                    1

The Category column is the label each mutation's file name carries. Every mutant is distinct: the generator lists every single-line mutation the source admits and drops any that produce the same source; -n (with -seed for a reproducible choice) picks at most that many of them, listed in source order.

Interpreting Results

Run your existing tests against each mutant. A mutant your tests kill (make fail) is detected, which is good. A mutant that survives (the tests still pass) shows a gap:

• ARITH_SWAP survivors - the tests do not check the arithmetic
• CMP_FLIP survivors - the tests do not check boundary conditions
• BOOL_LOGIC survivors - the tests do not cover both branches
• GUARD survivors - the tests do not exercise the difference between unset and set
• METHOD_REMOVE survivors - the tests do not check side effects

The mutation score is the share of mutants killed; higher is better. A low score shows gaps that line-based coverage metrics may not reveal.

The AI-Compiler Feedback Loop

Test generation and mutation testing work best alongside an AI assistant:

1. Write code - implement the feature with its first tests
2. Generate volume - run -fuzztest for hundreds of edge-case candidates, in seconds
3. Compile-check - the compiler drops the candidates that do not compile
4. AI curates - an AI assistant reviews what is left, picks the most valuable tests and discards the noise
5. Mutate - run -fuzzmutate for mutation variants
6. Assess - run the tests against the mutants to find the gaps
7. Strengthen - write targeted tests for the surviving mutants

CLI Reference

Comparison with Industry Tools

What sets EK9's approach apart is that there is nothing to configure. The compiler already has your symbol table, type information and method signatures; external tools have to rediscover them through instrumentation or configuration.

See Also

• Compiler Fuzzing - grammar-based fuzzing of the compiler itself
• Testing - test types, assertions, test runner commands
• Code Coverage - threshold enforcement, quality metrics, HTML reports
• Code Quality - quality enforcement rules and thresholds
• Command Line - all flags and exit codes
• For AI Assistants - machine-readable output schemas