Why does EK9 use compile-check-discard for generated tests?

← Fuzzing and Mutation Testing · Ref: Q755

EK9 test generation uses a compile-check-discard pattern: generate a candidate test, compile it, keep it only if compilation succeeds. This is fundamentally different from every other test generation tool.

THE COMPILER IS THE FILTER

Other tools generate tests and hope they are valid. EK9 generates candidates and uses its own compiler as an oracle. A candidate that fails type checking, violates purity, uses an invalid operator, or triggers any quality check is silently discarded. Only programs that pass ALL compiler checks survive.

WHY THIS WORKS IN EK9

EK9 has no warnings. Every check is pass or fail. This means the compiler gives a definitive answer: this candidate is either fully valid or it is not. There is no ambiguity, no 'valid with warnings' gray area.

TYPE-AWARE GENERATION

Because the generator knows the type system, it generates calls with correct types. But edge-case combinations (empty string to a function expecting non-empty, zero to a divider) may fail purity or data-flow checks. Compile-check-discard handles this automatically.

PRACTICAL BENEFIT

The output file contains ONLY tests that:

  1. Parse correctly (valid EK9 syntax).
  2. Pass type checking (correct argument types).
  3. Pass data flow analysis (no uninitialised variables).
  4. Pass quality checks (no magic literals, no dead code).

This is a guarantee no other test generator provides.

NO OTHER LANGUAGE DOES THIS

AFL, libFuzzer, EvoSuite, Hypothesis, and cargo-fuzz all generate inputs, not full programs. EK9 generates complete @Test programs and validates them against the full compiler pipeline.

See Q749 for fuzzing overview. See Q754 for test generation. See Q753 for mutation testing. See Q310 for quality at compile time.

Example

defines module qa.fuzzingandmutation.compilecheckdiscard

  <?-
    Demonstrates functions whose edge-case combinations
    could fail quality checks. The compile-check-discard
    pattern automatically handles these cases.
  -?>

  defines function

    <?-
      Test generator might try dividing by zero.
      The guard prevents a crash, but a candidate test
      calling this with zero would still compile because
      the guard handles it. Compile-check-discard keeps it.
    -?>
    safeDivide() as pure
      ->
        numerator as Integer
        divisor as Integer
      <- result as Integer: 0

      zero <- 0
      if divisor <> zero
        result: numerator / divisor

    <?-
      Test generator might try empty strings.
      The guard prevents accessing an empty string.
      Generated tests with empty input compile and exercise the guard.
    -?>
    safeFirstChar() as pure
      -> text as String
      <- found as Character: Character()

      if text?
        found :=? text.first()

    validateAge() as pure
      -> age as Integer
      <- valid as Boolean: false

      minimumAge <- 0
      maximumAge <- 150
      if age >= minimumAge and age <= maximumAge
        valid: true

  defines program

    CompileCheckDiscardDemo()
      stdout <- Stdout()

      ten <- 10
      three <- 3
      zero <- 0
      stdout.println(`10 / 3 = ${safeDivide(ten, three)}`)
      stdout.println(`10 / 0 = ${safeDivide(ten, zero)}`)

      greeting <- "Hello"
      emptyText <- String()
      stdout.println(`first of Hello: ${safeFirstChar(greeting)}`)
      stdout.println(`first of empty: ${safeFirstChar(emptyText)}`)

      for age in [0, 25, 150, -1, 200]
        stdout.println(`age ${age} valid: ${validateAge(age)}`)

Coming from another language?

Java: EvoSuite generates tests that may not compile or may have warnings. Python: Hypothesis generates inputs, not full test programs. Rust: proptest generates values, not complete test functions. Go: go test -fuzz generates corpus entries, not test functions. EK9: generates complete @Test programs validated by the full compiler pipeline, discarding any that fail any check.

Keywords: test, aware, check, guarantee, generation, filter, discard, compile, type, quality, oracle, valid