Testing in EK9

Tests check that your code works. When you change the code later, running them confirms you have not broken what already worked.

Quality Enforcement Applies to Tests: test code meets the same standards as production code. A test function has the same limits as any other function (cyclomatic complexity 45, cognitive 35, nesting depth 6, 150 statements), needs descriptive variable names (no temp, flag, data) and must meet the cohesion and coupling limits. See Compile-Time Validation and Code Quality.

Overview - Testing as a language feature
Your First Test
Test Types - Assert-Based, Black-Box, Parameterized
The @Test Directive - Grouping tests, expected exit code
Assertion Statements - assert, assertThrows, assertDoesNotThrow
The require Statement - Uncatchable preconditions (panic-like)
Test Runner - Commands and output formats
Output Placeholders - Matching dynamic values
Compile-Time Validation - Quality enforcement on test code
Quick Reference

Overview: Testing as a Language Feature

In EK9 you do not import a testing library such as JUnit or pytest: testing is part of the language grammar. That gives you:

Compile-time test validation - an empty test, an orphan assertion and an assertion in production code are compiler errors, not runtime surprises
Quality enforcement on test code - tests meet the same standards as production code (complexity limits, naming quality, cohesion)
Always-on coverage - an 80% threshold enforced automatically; exit code 12 if below
Zero imports - no test framework dependency, so no version conflicts

Your First Test

Create a project with two files:

first/
├── testing_first.ek9          # Your code (the function to test)
└── dev/
    └── testing_first_tests.ek9   # Your tests

Files in the dev/ directory are compiled only when tests run, which keeps test code separate from production code.

testing_first.ek9 - The Code to Test

#!ek9
defines module site.testing.first

  defines function

    add() as pure
      ->
        a as Integer
        b as Integer
      <- result as Integer: a + b

Download testing_first.ek9

dev/testing_first_tests.ek9 - The Test

#!ek9
defines module site.testing.first.tests

  references
    site.testing.first::add

  defines program

    @Test
    AdditionWorks()
      result <- add(2, 3)
      assert result == 5

Download testing_first_tests.ek9

references imports symbols from other modules, defines program declares an entry point, @Test marks it for the test runner, and assert checks a condition.

Run it:

$ ek9 -t testing_first.ek9
ℹ️ Found 1 test:
  1 assert (unit tests with assertions)

Executing 1 test...

✅ PASS site.testing.first.tests::AdditionWorks [Assert] (nms)

Summary: 1 passed, 0 failed (1 total)
Types: 1 assert
Duration: nms

The test file can have any name, and dev/ can hold as many .ek9 files as you need, each with as many @Test programs as you need. The test runner finds them all.

When Tests Fail

Suppose add multiplied by mistake:

#!ek9
defines module site.testing.failing

  defines function

    add() as pure
      ->
        a as Integer
        b as Integer
      <- result as Integer: a * b

Download testing_failing.ek9

The same test, in dev/:

#!ek9
defines module site.testing.failing.tests

  references
    site.testing.failing::add

  defines program

    @Test
    AdditionWorks()
      result <- add(2, 3)
      assert result == 5

Download testing_failing_tests.ek9

EK9 shows exactly what failed, and ek9 -t exits 11:

$ ek9 -t testing_failing.ek9
ℹ️ Found 1 test:
  1 assert (unit tests with assertions)

Executing 1 test...

❌ FAIL site.testing.failing.tests::AdditionWorks [Assert] (nms)
    Assertion failed: `result == 5` at ./dev/testing_failing_tests.ek9:12:7

Summary: 0 passed, 1 failed (1 total)
Types: 1 assert
Duration: nms
ek9: tests failed (exit 11)

The expression (result == 5), file, line and column are captured from the source automatically. There is no stack trace to read and no custom message to write.

What You Get Out of the Box

ek9 -t6p main.ek9 generates a full HTML report with coverage, quality metrics and performance profiling:

Coverage dashboard showing status banner, quality metric dials, coverage donut charts, and module list

The source views show coverage line by line, with complexity and profiling for each function:

Source code view with coverage highlighting, complexity badges, and profiling badges

Profiling includes flame graphs and a hot function table with percentile metrics:

Flame graph and hot function table with timing percentiles

All of this comes from one command, with no external tools and no configuration. See Code Coverage and Profiling for details.

Tests Run Automatically

When you package (ek9 -P) or deploy (ek9 -D), the tests run first, and EK9 will not package code whose tests fail or whose coverage is below the threshold. Testing is part of the workflow, not a separate step to remember.

Test Types

EK9 supports three kinds of test:

Assert-Based - unit testing individual functions: call them and check the results with assert
Black-Box - regression testing a program's output: compare stdout with expected_output.txt
Parameterized - one test run with several sets of input, one per pair of files

1. Assert-Based Tests

Use assert, assertThrows and assertDoesNotThrow inside test code.

Project Structure

asserts/
├── testing_asserts.ek9           # Production code (functions to test)
└── dev/
    └── testing_asserts_tests.ek9  # Test programs with @Test directive

Production Code (testing_asserts.ek9)

#!ek9
defines module site.testing.asserts

  defines function

    add() as pure
      ->
        a as Integer
        b as Integer
      <- result as Integer: a + b

    multiply() as pure
      ->
        a as Integer
        b as Integer
      <- result as Integer: a * b

Download testing_asserts.ek9

Test Code (dev/testing_asserts_tests.ek9)

#!ek9
defines module site.testing.asserts.tests

  references
    site.testing.asserts::add
    site.testing.asserts::multiply

  defines program

    @Test
    AdditionTest()
      result <- add(2, 3)
      assert result?
      assert result == 5

    @Test
    MultiplicationTest()
      result <- multiply(4, 5)
      assert result?
      assert result == 20

    @Test
    CombinedOperationsTest()
      sum <- add(10, 20)
      product <- multiply(sum, 2)
      assert product == 60

Download testing_asserts_tests.ek9

$ ek9 -t testing_asserts.ek9
ℹ️ Found 3 tests:
  3 assert (unit tests with assertions)

Executing 3 tests...

✅ PASS site.testing.asserts.tests::AdditionTest [Assert] (nms)
✅ PASS site.testing.asserts.tests::CombinedOperationsTest [Assert] (nms)
✅ PASS site.testing.asserts.tests::MultiplicationTest [Assert] (nms)

Summary: 3 passed, 0 failed (3 total)
Types: 3 assert
Duration: nms

2. Black-Box Tests

A black-box test checks a program's output against a file. For a test without command line arguments, the file must be named exactly expected_output.txt and sit in the same directory as the test. This suits regression testing and AI-generated tests.

Project Structure

blackbox/
├── testing_blackbox.ek9            # Production code
└── dev/
    ├── testing_blackbox_tests.ek9  # Test program
    └── expected_output.txt         # Expected stdout output

Production Code (testing_blackbox.ek9)

#!ek9
defines module site.testing.blackbox

  defines function

    greet() as pure
      -> name as String
      <- greeting as String: `Hello, ${name}!`

Download testing_blackbox.ek9

Test Code (dev/testing_blackbox_tests.ek9)

#!ek9
defines module site.testing.blackbox.tests

  references
    site.testing.blackbox::greet

  defines program

    @Test
    GreetingOutputTest()
      stdout <- Stdout()
      stdout.println(greet("World"))
      stdout.println(greet("EK9"))

Download testing_blackbox_tests.ek9

Expected Output (dev/expected_output.txt)

Hello, World!
Hello, EK9!

$ ek9 -t testing_blackbox.ek9
ℹ️ Found 1 test:
  1 blackbox (output validated against expected_output.txt)

Executing 1 test...

✅ PASS site.testing.blackbox.tests::GreetingOutputTest [BlackBox] (nms)

Summary: 1 passed, 0 failed (1 total)
Types: 1 blackbox
Duration: nms

When Output Doesn't Match

When the output differs, EK9 compares it line by line. Suppose the greeting ended with ? by mistake:

#!ek9
defines module site.testing.blackboxfail

  defines function

    greet() as pure
      -> name as String
      <- greeting as String: `Hello, ${name}?`

Download testing_blackboxfail.ek9

With the same test and the same expected_output.txt:

#!ek9
defines module site.testing.blackboxfail.tests

  references
    site.testing.blackboxfail::greet

  defines program

    @Test
    GreetingOutputTest()
      stdout <- Stdout()
      stdout.println(greet("World"))
      stdout.println(greet("EK9"))

Download testing_blackboxfail_tests.ek9

$ ek9 -t testing_blackboxfail.ek9
ℹ️ Found 1 test:
  1 blackbox (output validated against expected_output.txt)

Executing 1 test...

❌ FAIL site.testing.blackboxfail.tests::GreetingOutputTest [BlackBox] (nms)
    Output mismatch

    Mismatched line(s): 1, 2

      Line 1:
        Expected: Hello, World!
        Actual:   Hello, World?
      Line 2:
        Expected: Hello, EK9!
        Actual:   Hello, EK9?

Summary: 0 passed, 1 failed (1 total)
Types: 1 blackbox
Duration: nms
ek9: tests failed (exit 11)

3. Parameterized Tests

To run one test with several inputs, give each case a pair of files: commandline_arg_{id}.txt and expected_case_{id}.txt.

Project Structure

parameterized/
├── testing_parameterized.ek9            # Production code
└── dev/
    ├── testing_parameterized_tests.ek9  # Test program with parameters
    ├── commandline_arg_simple.txt       # Case "simple": input arguments
    ├── expected_case_simple.txt         # Case "simple": expected output
    ├── commandline_arg_edge.txt         # Case "edge": input arguments
    └── expected_case_edge.txt           # Case "edge": expected output

Production Code (testing_parameterized.ek9)

#!ek9
defines module site.testing.parameterized

  defines function

    processArg() as pure
      -> text as String
      <- result as String: `Processed: ${text}`

Download testing_parameterized.ek9

Test Code (dev/testing_parameterized_tests.ek9)

#!ek9
defines module site.testing.parameterized.tests

  references
    site.testing.parameterized::processArg

  defines program

    @Test
    ArgProcessor()
      ->
        first as String
        second as String

      stdout <- Stdout()
      stdout.println(processArg(first))
      stdout.println(processArg(second))

Download testing_parameterized_tests.ek9

Test Case "simple"

commandline_arg_simple.txt (one argument per line):

hello
world

expected_case_simple.txt:

Processed: hello
Processed: world

Test Case "edge"

commandline_arg_edge.txt:

single
only

expected_case_edge.txt:

Processed: single
Processed: only

One test program, two results - one per case:

$ ek9 -t testing_parameterized.ek9
ℹ️ Found 1 test:
  1 parameterized (2 cases total)

Executing 1 test...

✅ PASS site.testing.parameterized.tests::ArgProcessor [Param:edge] (nms)
✅ PASS site.testing.parameterized.tests::ArgProcessor [Param:simple] (nms)

Summary: 2 passed, 0 failed (2 results from 1 programs)
Types: 2 parameterized
Duration: nms

The @Test Directive

The @Test directive marks a program as a test. The test runner finds and runs only programs that carry it.

Ungrouped vs Grouped Tests

Tests run in parallel by default. When tests must run one after another, put them in a group: typically tests that use a database, the file system, or another shared resource where order matters.

The syntax is @Test: "groupname". Tests in the same group run in sequence; different groups run in parallel with each other.

Project Structure

grouped/
├── testing_grouped.ek9            # Production code (Counter class)
└── dev/
    └── testing_grouped_tests.ek9  # Test programs

Production Code (testing_grouped.ek9)

#!ek9
defines module site.testing.grouped

  defines class

    Counter
      ticks as Integer: 0

      count() as pure
        <- rtn as Integer: ticks

      increment()
        ticks++

      default operator ?

Download testing_grouped.ek9

Test Code (dev/testing_grouped_tests.ek9)

#!ek9
defines module site.testing.grouped.tests

  references
    site.testing.grouped::Counter

  defines program

    @Test: "counter"
    CounterIncrementTest()
      counter <- Counter()
      counter.increment()
      assert counter.count() == 1

    @Test: "counter"
    CounterMultipleIncrementTest()
      counter <- Counter()
      counter.increment()
      counter.increment()
      counter.increment()
      assert counter.count() == 3

    @Test
    IndependentTest()
      counter <- Counter()
      assert counter.count() == 0

Download testing_grouped_tests.ek9

CounterIncrementTest and CounterMultipleIncrementTest are both in the "counter" group, so they run in sequence. IndependentTest has no group and runs in parallel with the other ungrouped tests. The runner names each group before it starts, and shows a grouped test's group in braces:

$ ek9 -t testing_grouped.ek9
ℹ️ Found 3 tests:
  3 assert (unit tests with assertions)

Group "counter": 2 tests (sequential execution)

Executing 3 tests...

✅ PASS site.testing.grouped.tests::CounterIncrementTest [Assert] {counter} (nms)
✅ PASS site.testing.grouped.tests::CounterMultipleIncrementTest [Assert] {counter} (nms)
✅ PASS site.testing.grouped.tests::IndependentTest [Assert] (nms)

Summary: 3 passed, 0 failed (3 total)
Types: 3 assert
Duration: nms

Expected Exit Code

A program's returned Integer is its exit code (see Command Line), and a test passes only when its program exits with the code the test expects. That is 0 unless the @Test declares another with "EXIT:<n>", where n is a whole number from 0 to 255.

A program that exits 3 without declaring it fails.
A program that declares "EXIT:3" but exits 0 or 4 fails.
The code must be the program's own. An uncaught Exception or Panic, which writes to stderr as it ends the program, is reported as an error, never as the expected exit.

A @Test may carry a requirement id, a group name and an expected exit code, in any order. Each part is recognised by what it contains, so a misspelled one - "EXT:3", "EXIT:abc", "EXIT:", "EXIT:256", or two EXIT parts - is a compile error, never a silently ignored expectation.

#!ek9
defines module site.testing.exitcode

  defines function

    exitCodeFor() as pure
      -> input as String
      <- rtn as Integer: 0
      if input is empty
        rtn: 3

Download testing_exitcode.ek9

#!ek9
defines module site.testing.exitcode.tests

  references
    site.testing.exitcode::exitCodeFor

  defines constant
    validOrder <- "order-42"
    noOrder <- ""

  defines program

    @Test
    ValidInputExitsZero()
      <- rtn as Integer: exitCodeFor(validOrder)
      assert rtn?

    @Test: "EXIT:3"
    EmptyInputExitsThree()
      <- rtn as Integer: exitCodeFor(noOrder)
      assert rtn?

    @Test: "REQ:input-validation": "validation": "EXIT:3"
    AnchoredGroupedExitsThree()
      <- rtn as Integer: exitCodeFor(noOrder)
      assert rtn?

Download testing_exitcode_tests.ek9

$ ek9 -t testing_exitcode.ek9
ℹ️ Found 3 tests:
  3 assert (unit tests with assertions)

Group "validation": 1 test (sequential execution)

Executing 3 tests...

✅ PASS site.testing.exitcode.tests::AnchoredGroupedExitsThree [Assert] {validation} (nms)
✅ PASS site.testing.exitcode.tests::EmptyInputExitsThree [Assert] (nms)
✅ PASS site.testing.exitcode.tests::ValidInputExitsZero [Assert] (nms)

Summary: 3 passed, 0 failed (3 total)
Types: 3 assert
Duration: nms

All three pass. Remove "EXIT:3" from EmptyInputExitsThree and it fails with Test program exited with code 3; give ValidInputExitsZero an "EXIT:3" and it fails with Test program exited with code 0 but its @Test declares "EXIT:3".

Assertion Statements

An EK9 assertion reports its own failure precisely: the source location (file, line, column), the expression that failed and the context, captured from the source without a custom message.

The examples below test one production function. Integer division by zero does not throw in EK9 - it gives an unset Integer - so a function that must fail on a zero divisor throws explicitly:

checkedDivide() as pure
  ->
    dividend as Integer
    divisor as Integer
  <- quotient as Integer?

  //Integer division by zero does not throw: it gives an unset Integer.
  //So a function that must fail says so explicitly.
  if divisor == 0
    throw Exception("division by zero")
  quotient: dividend / divisor

Full source: testing_assertions.ek9 - Download testing_assertions.ek9

assert

Checks that a condition is true:

@Test
CheckDivision()
  result <- checkedDivide(10, 2)
  assert result?          //Check result is set (not unset)
  assert result == 5      //Check result equals the expected value

Full source: testing_assertions_tests.ek9 - Download testing_assertions_tests.ek9

assertThrows

Checks that an expression throws a specific exception type, and gives back the exception it caught so the test can inspect it (an Exception carries its message as reason()):

@Test
CheckDivisionByZero()
  assertThrows(Exception, checkedDivide(10, 0))

@Test
CheckAndInspectException()
  caught <- assertThrows(Exception, checkedDivide(10, 0))
  assert caught.reason() == "division by zero"

Full source: testing_assertions_tests.ek9 - Download testing_assertions_tests.ek9

assertDoesNotThrow

Checks that an expression completes without throwing any exception. It is a statement and gives nothing back, so a test that also checks the value asserts it separately:

@Test
CheckSafeDivision()
  assertDoesNotThrow(checkedDivide(10, 2))

@Test
CheckSafeDivisionResult()
  //assertDoesNotThrow gives nothing back: check the value with its own assert
  assertDoesNotThrow(checkedDivide(10, 2))
  result <- checkedDivide(10, 2)
  assert result == 5

Full source: testing_assertions_tests.ek9 - Download testing_assertions_tests.ek9

All five pass:

$ ek9 -t testing_assertions.ek9
ℹ️ Found 5 tests:
  5 assert (unit tests with assertions)

Executing 5 tests...

✅ PASS site.testing.assertions.tests::CheckAndInspectException [Assert] (nms)
✅ PASS site.testing.assertions.tests::CheckDivision [Assert] (nms)
✅ PASS site.testing.assertions.tests::CheckDivisionByZero [Assert] (nms)
✅ PASS site.testing.assertions.tests::CheckSafeDivision [Assert] (nms)
✅ PASS site.testing.assertions.tests::CheckSafeDivisionResult [Assert] (nms)

Summary: 5 passed, 0 failed (5 total)
Types: 5 assert
Duration: nms

When an Assertion Fails

Each assertion reports what it checked and where. These three tests of the same function are wrong on purpose, one per assertion:

#!ek9
defines module site.testing.assertionsfail.tests

  references
    site.testing.assertionsfail::checkedDivide

  defines program

    @Test
    WrongQuotient()
      result <- checkedDivide(10, 2)
      assert result == 6

    @Test
    ExpectsAThrowThatNeverComes()
      assertThrows(Exception, checkedDivide(10, 2))

    @Test
    DoesNotExpectTheThrow()
      assertDoesNotThrow(checkedDivide(10, 0))

Download testing_assertionsfail_tests.ek9

$ ek9 -t testing_assertionsfail.ek9
ℹ️ Found 3 tests:
  3 assert (unit tests with assertions)

Executing 3 tests...

❌ FAIL site.testing.assertionsfail.tests::DoesNotExpectTheThrow [Assert] (nms)
    assertDoesNotThrow FAILED
      Location: ./dev/testing_assertionsfail_tests.ek9:20:7
      Expression: checkedDivide(10, 0)
      Expected: No exception
      Actual: org.ek9.lang::Exception
      Message: Exception: division by zero
❌ FAIL site.testing.assertionsfail.tests::ExpectsAThrowThatNeverComes [Assert] (nms)
    assertThrows FAILED
      Location: ./dev/testing_assertionsfail_tests.ek9:16:7
      Expression: checkedDivide(10, 2)
      Expected: org.ek9.lang::Exception
      Actual: No exception was thrown
❌ FAIL site.testing.assertionsfail.tests::WrongQuotient [Assert] (nms)
    Assertion failed: `result == 6` at ./dev/testing_assertionsfail_tests.ek9:12:7

Summary: 0 passed, 3 failed (3 total)
Types: 3 assert
Duration: nms
ek9: tests failed (exit 11)

The production function they test is the same checkedDivide:

#!ek9
defines module site.testing.assertionsfail

  defines function

    checkedDivide() as pure
      ->
        dividend as Integer
        divisor as Integer
      <- quotient as Integer?

      if divisor == 0
        throw Exception("division by zero")
      quotient: dividend / divisor

Download testing_assertionsfail.ek9

Why EK9 does it this way. With a testing library such as JUnit or pytest, you write a custom message for each assertion, read a stack trace to find where it failed, or guess which of several assertions in a test failed. Because assertions are part of the grammar, EK9 captures the expression text, the source location and the context at compile time. The output is structured, so an AI assistant or a CI pipeline can read it (and open an issue at the exact line), and debugging needs no stack trace.

require vs assert

EK9 separates production preconditions from test assertions:

require - a production precondition, checked on any code path
assert - a test check, valid only in @Test programs

An assert outside a dev source file is E81015, and an assert that production code can reach is E81012. A failed assert fails its test and ek9 -t exits 11; a failed require ends a program with exit status 1. Test and production code both meet EK9's quality standards.

The require Statement

require checks a precondition in production code. When its condition is false or unset, the language raises a Panic - not an Exception, so no catch can see it - and the program ends with exit status 1 once its finally blocks and teardown have run (see the program's exit status). Error Handling Strategy shows how require fits beside throw/try-catch, Result and Optional.

Panic-like Behavior: a require failure cannot be caught, and, unlike panic in Go or Rust, there is no way to recover from it. A failed require means a serious, unrecoverable defect in your program.

Why EK9 does it this way. A require failure cannot drive control flow:

No exception-based control flow - you cannot catch a require failure and carry on
Fail fast - a broken precondition is a programming error, not a condition to recover from
A clear contract - if a require fails, the caller broke the contract: that is a bug, not a situation to handle

When to Use require

Order
  customerId as String: String()
  items as List of OrderItem: List()
  status as OrderStatus: OrderStatus.Created

  //Constructor with preconditions - if these fail, the caller has a bug
  Order()
    ->
      customerId as String
      initialItems as List of OrderItem
    require customerId?
    require initialItems is not empty
    this.customerId: customerId
    items += initialItems

  //Method with preconditions
  addItem()
    -> item as OrderItem
    require item?
    require status == OrderStatus.Created
    items += item

  //Method ensuring an invariant
  submit()
    require items is not empty
    require status == OrderStatus.Created
    status: OrderStatus.Submitted

  default operator ?

Full source: testing_require.ek9 - Download testing_require.ek9

Every precondition holds in this run of the full source's program:

Order submitted

require vs Validation

require is for programming errors, not for validating user input:

Use `require`	Use Validation (catchable)
An unset argument where a value is required	User entered invalid email format
Collection empty when must have items	User entered age outside valid range
Method called in wrong object state	File not found (recoverable error)
Internal invariant violated	Network timeout (can retry)

If the condition could fail because of user input or something outside the program, validate it and use a catchable exception.
If its failure means the programmer made a mistake, use require.

Comparison with Other Languages

Language	Equivalent	Behavior
EK9	`require`	Uncatchable Panic; finally blocks run, then the program exits 1
Go	`panic`	Uncatchable (unless `recover` used)
Rust	`panic!`	Unwinds stack, terminates thread
Java	`assert`	Off unless the JVM runs with `-ea` (weak)
C/C++	`assert()`	Can be compiled out with NDEBUG (weak)

EK9's require is stronger than a Java or C assertion because it cannot be disabled: the precondition is always checked in production code.

Test Runner

Run tests with the -t flag:

ek9 -t main.ek9         # Run tests (human output; -t1 is the same)
ek9 -t0 main.ek9        # Terse output: one summary line
ek9 -t2 main.ek9        # JSON output (for CI/AI)
ek9 -t3 main.ek9        # JUnit XML output
ek9 -t4 main.ek9        # Coverage detail: lists uncovered items, writes .ek9/coverage-detail.json
ek9 -t5 main.ek9        # Verbose coverage: lists covered AND uncovered items, writes .ek9/coverage-verbose.json
ek9 -t6 main.ek9        # Interactive HTML coverage report
ek9 -tL main.ek9        # List tests without running them
ek9 -tg database main.ek9   # Run only the "database" group
ek9 -tC main.ek9        # Show coverage after the results (with -t2/-t3: writes .ek9/test-results and .ek9/coverage files)

# With profiling (append 'p' to any format; 'p' implies -tC):
ek9 -tp main.ek9        # Human output + coverage + hot-function summary
ek9 -t2p main.ek9       # JSON, with profiling written to .ek9/coverage.json
ek9 -t6p main.ek9       # Full HTML report with flame graph

-tL lists what the runner found, by kind, without running anything (this is the directory structure example below):

$ ek9 -tL testing_layout.ek9
ℹ️ Found 4 tests:
  2 assert (unit tests with assertions)
  1 blackbox (output validated against expected_output.txt)
  1 parameterized (2 cases total)

Tests:
  [Param/2] site.testing.layout.calculator::DoubleEachArgument
  [BlackBox] site.testing.layout.greeting::GreetingOutput
  [Assert] site.testing.layout.unit::GreetingIsPolite
  [Assert] site.testing.layout.unit::DoublingWorks

Exit codes:

0 - all tests passed and coverage meets the threshold
11 - one or more tests failed
12 - all tests passed but code coverage is below 80%

So a CI/CD pipeline can tell a failing test from insufficient coverage. See Command Line for all exit codes and E83001 for the coverage threshold error.

Why `ek9 -t` Sees the Whole Tree

ek9 -t main.ek9 names one file, yet it also compiles the other files beside and below it, and the dev/ directories that hold the tests. That is because -t is a development build, and a development build always adds the standard includes (every .ek9 file below the directory of the named file, plus .properties, image, .js and .css resources) and always includes dev/, whatever the defines package block says. Only the package's excludeFiles (and the standard excludes, if the package sets applyStandardExcludes <- true) can take files out of it again.

An ordinary build (ek9 main.ek9, -c, packaging) is narrower. It compiles the named file, plus the files that the package's includeFiles globs match, plus every file below the directory when the package says applyStandardIncludes <- true; it then removes whatever excludeFiles matches (and the standard excludes, when applyStandardExcludes <- true), and it never includes dev/. Tests are not part of the program you ship, and the program you ship does not depend on them.

Output Formats

Each output format suits a different use. The formats below are runs of the same project: two functions, one of them wrong, and four tests.

#!ek9
defines module site.testing.formats

  defines function

    add() as pure
      ->
        a as Integer
        b as Integer
      <- result as Integer: a + b

    //A defect, for the failing tests to find: it subtracts
    subtract() as pure
      ->
        a as Integer
        b as Integer
      <- result as Integer: b - a

Download testing_formats.ek9

#!ek9
defines module site.testing.formats.tests

  references
    site.testing.formats::add
    site.testing.formats::subtract

  defines program

    @Test
    AdditionWorks()
      assert add(2, 3) == 5

    @Test
    AdditionIsCommutative()
      assert add(2, 3) == add(3, 2)

    @Test
    SubtractionWorks()
      result <- subtract(5, 3)
      assert result == 2

    @Test
    SubtractionOfZero()
      assert subtract(7, 0) == 7

Download testing_formats_tests.ek9

Human Format (-t or -t1)

Visual output with icons, for a terminal. A grouped test also shows its group in braces (see the grouped example):

$ ek9 -t testing_formats.ek9
ℹ️ Found 4 tests:
  4 assert (unit tests with assertions)

Executing 4 tests...

✅ PASS site.testing.formats.tests::AdditionIsCommutative [Assert] (nms)
✅ PASS site.testing.formats.tests::AdditionWorks [Assert] (nms)
❌ FAIL site.testing.formats.tests::SubtractionOfZero [Assert] (nms)
    Assertion failed: `subtract(7, 0) == 7` at ./dev/testing_formats_tests.ek9:25:7
❌ FAIL site.testing.formats.tests::SubtractionWorks [Assert] (nms)
    Assertion failed: `result == 2` at ./dev/testing_formats_tests.ek9:21:7

Summary: 2 passed, 2 failed (4 total)
Types: 4 assert
Duration: nms
ek9: tests failed (exit 11)

Terse Format (-t0)

Minimal output for scripts and CI pass/fail checks:

$ ek9 -t0 testing_formats.ek9
4 tests: 2 passed, 2 failed (4 assert)
ek9: tests failed (exit 11)

JSON Format (-t2)

Structured output for AI/LLM integration and custom tooling:

$ ek9 -t2 testing_formats.ek9
{
  "version": "1.0",
  "timestamp": "date-time",
  "architecture": "JVM",
  "summary": {
    "total": 4,
    "passed": 2,
    "failed": 2,
    "skipped": 0,
    "duration_ms": n,
    "types": {
      "assert": 4,
      "blackbox": 0,
      "blackbox_hybrid": 0,
      "parameterized": 0,
      "parameterized_hybrid": 0
    }
  },
  "tests": [
    {
      "name": "AdditionIsCommutative",
      "fqn": "site.testing.formats.tests::AdditionIsCommutative",
      "source_file": "file://project-directory/./dev/testing_formats_tests.ek9",
      "test_type": "assert",
      "case_id": null,
      "status": "passed",
      "duration_ms": n,
      "group": null,
      "output": ""
    },
    {
      "name": "AdditionWorks",
      "fqn": "site.testing.formats.tests::AdditionWorks",
      "source_file": "file://project-directory/./dev/testing_formats_tests.ek9",
      "test_type": "assert",
      "case_id": null,
      "status": "passed",
      "duration_ms": n,
      "group": null,
      "output": ""
    },
    {
      "name": "SubtractionOfZero",
      "fqn": "site.testing.formats.tests::SubtractionOfZero",
      "source_file": "file://project-directory/./dev/testing_formats_tests.ek9",
      "test_type": "assert",
      "case_id": null,
      "status": "failed",
      "duration_ms": n,
      "group": null,
      "failure": {
        "type": "assertion",
        "message": "Assertion failed: `subtract(7, 0) == 7` at ./dev/testing_formats_tests.ek9:25:7"
      },
      "output": ""
    },
    {
      "name": "SubtractionWorks",
      "fqn": "site.testing.formats.tests::SubtractionWorks",
      "source_file": "file://project-directory/./dev/testing_formats_tests.ek9",
      "test_type": "assert",
      "case_id": null,
      "status": "failed",
      "duration_ms": n,
      "group": null,
      "failure": {
        "type": "assertion",
        "message": "Assertion failed: `result == 2` at ./dev/testing_formats_tests.ek9:21:7"
      },
      "output": ""
    }
  ]
}
ek9: tests failed (exit 11)

JUnit XML Format (-t3)

The standard format for CI/CD systems (Jenkins, GitHub Actions, GitLab):

$ ek9 -t3 testing_formats.ek9
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="site.testing.formats.tests" tests="4" failures="2" errors="0" skipped="0" time="n">
  <properties>
    <property name="ek9.types.assert" value="4"/>
    <property name="ek9.types.blackbox" value="0"/>
    <property name="ek9.types.blackbox_hybrid" value="0"/>
    <property name="ek9.types.parameterized" value="0"/>
    <property name="ek9.types.parameterized_hybrid" value="0"/>
  </properties>
  <testcase name="AdditionIsCommutative" classname="site.testing.formats.tests" time="n">
    <properties>
      <property name="ek9.fqn" value="site.testing.formats.tests::AdditionIsCommutative"/>
      <property name="ek9.source_file" value="file://project-directory/./dev/testing_formats_tests.ek9"/>
      <property name="ek9.test_type" value="assert"/>
    </properties>
  </testcase>
  <testcase name="AdditionWorks" classname="site.testing.formats.tests" time="n">
    <properties>
      <property name="ek9.fqn" value="site.testing.formats.tests::AdditionWorks"/>
      <property name="ek9.source_file" value="file://project-directory/./dev/testing_formats_tests.ek9"/>
      <property name="ek9.test_type" value="assert"/>
    </properties>
  </testcase>
  <testcase name="SubtractionOfZero" classname="site.testing.formats.tests" time="n">
    <properties>
      <property name="ek9.fqn" value="site.testing.formats.tests::SubtractionOfZero"/>
      <property name="ek9.source_file" value="file://project-directory/./dev/testing_formats_tests.ek9"/>
      <property name="ek9.test_type" value="assert"/>
    </properties>
    <failure message="Assertion failed: `subtract(7, 0) == 7` at ./dev/testing_formats_tests.ek9:25:7" type="AssertionError">
Assertion failed: `subtract(7, 0) == 7` at ./dev/testing_formats_tests.ek9:25:7
    </failure>
  </testcase>
  <testcase name="SubtractionWorks" classname="site.testing.formats.tests" time="n">
    <properties>
      <property name="ek9.fqn" value="site.testing.formats.tests::SubtractionWorks"/>
      <property name="ek9.source_file" value="file://project-directory/./dev/testing_formats_tests.ek9"/>
      <property name="ek9.test_type" value="assert"/>
    </properties>
    <failure message="Assertion failed: `result == 2` at ./dev/testing_formats_tests.ek9:21:7" type="AssertionError">
Assertion failed: `result == 2` at ./dev/testing_formats_tests.ek9:21:7
    </failure>
  </testcase>
</testsuite>
ek9: tests failed (exit 11)

Code Coverage

EK9 collects code coverage whenever tests run, and always enforces it: below the 80% threshold, the test runner exits 12. Code Coverage covers the threshold, the output formats (JSON, JaCoCo XML, HTML), the quality metrics (complexity, readability) and the interactive HTML reports.

Performance Profiling

Profiling is built in. Append p to any test format flag (for example -t6p) to collect call counts, timing and percentile metrics with the test results and coverage. Profiling explains the flame graph, the hot function table and the per-function profiling badges.

Output Placeholders

A black-box test often prints values that change between runs, such as dates, timestamps or IDs. In the expected output file, use a placeholder named after the EK9 type to match any value of that type.

Example: Testing a Report Generator

testing_placeholders.ek9

#!ek9
defines module site.testing.placeholders

  defines function

    generateReport()
      -> itemCount as Integer
      <- report as String: `Report generated on ${Date().today()} with ${itemCount} items`

Download testing_placeholders.ek9

dev/testing_placeholders_tests.ek9

#!ek9
defines module site.testing.placeholders.tests

  references
    site.testing.placeholders::generateReport

  defines program

    @Test
    ReportIncludesDate()
      stdout <- Stdout()
      stdout.println(generateReport(42))

Download testing_placeholders_tests.ek9

dev/expected_output.txt

Report generated on {{Date}} with {{Integer}} items

$ ek9 -t testing_placeholders.ek9
ℹ️ Found 1 test:
  1 blackbox (output validated against expected_output.txt)

Executing 1 test...

✅ PASS site.testing.placeholders.tests::ReportIncludesDate [BlackBox] (nms)

Summary: 1 passed, 0 failed (1 total)
Types: 1 blackbox
Duration: nms

The test passes whatever the date or item count, because {{Date}} matches any valid date (for example 2025-12-31) and {{Integer}} matches any integer.

Available Placeholders

Placeholder	Matches	Example
`{{String}}`	Any non-empty text	`hello world`
`{{Integer}}`	Whole numbers	`42`, `-17`
`{{Float}}`	Decimal numbers	`3.14`, `-2.5`
`{{Boolean}}`	true or false	`true`
`{{Date}}`	ISO date	`2025-12-31`
`{{Time}}`	Time of day	`14:30:45`
`{{DateTime}}`	ISO datetime with timezone	`2025-12-31T14:30:45+00:00`
`{{Duration}}`	ISO duration	`PT1H30M`, `P1Y2M3D`
`{{Millisecond}}`	Milliseconds	`5000ms`
`{{Money}}`	Currency amount	`10.50#USD`
`{{Colour}}`	Hex colour	`#FF5733`
`{{Dimension}}`	Measurement with unit	`10.5px`, `100mm`
`{{GUID}}`	UUID format	`550e8400-e29b-41d4-...`
`{{FileSystemPath}}`	File/directory path	`/path/to/file.txt`, `C:\dir\file`

E81010 lists all 18 valid placeholders. The test runner reports an invalid placeholder name as that error.

Compile-Time Validation

The compiler checks test code in two ways: rules specific to tests, and the same quality enforcement as all other code.

Test-Specific Validation

The compiler's call graph knows which code each @Test can reach, and these checks come from it:

E81007 - an empty @Test (no assertions, no expected files)
E81011 - an orphan assertion (not reachable from any @Test)
E81012 - a production assertion (an assert on a non-test code path)

Quality Enforcement on Test Code

All quality enforcement applies to test code, not only to production code:

Naming Quality (E11026, E11030, E11031)

E11026: Reference Ordering - references in test modules must be in alphabetical order
E11030: Similar Names - test variable names cannot be confusingly similar (one edit apart, a swap included, same type)
E11031: Non-Descriptive Names - test variables cannot use generic names such as temp, flag or data

Complexity Limits (E11010-E11013)

E11010: Cyclomatic Complexity - a test function has the same limit as any function, 45
E11011: Nesting Depth - nesting cannot exceed 6
E11012: Statement Count - at most 150 statements in a function, 100 in a method
E11013: Expression Complexity - complex expressions must be broken down

Cohesion and Coupling (E11014-E11016)

E11014: Low Cohesion - test classes must keep their cohesion (the LCOM4 metric)
E11015: Efferent Coupling - each test construct must respect the limit on the types it depends on
E11016: Module Coupling - test modules must respect the limit on the modules they depend on

Why EK9 does it this way. Tests are code. Badly structured tests with confusing names and high complexity cause the same problems as production code with those issues, so EK9 keeps tests readable and maintainable.

Example: Quality Violation in Test Code

Fails with E11031: non-descriptive variable name - use meaningful identifier

#!ek9
defines module bad.site.testing.qualitynames

  defines function

    fetchUser()
      <- rtn as String: "Ada"

    processUser()
      -> userName as String
      <- rtn as String: `Processed ${userName}`

  defines program

    @Test
    UserIsProcessed()
      temp <- fetchUser()
      data <- processUser(temp)
      assert data?

Download testing_qualitynames_bad.ek9

This test won't compile. Fix the naming first:

#!ek9
defines module site.testing.quality

  defines function

    fetchUser()
      <- rtn as String: "Ada"

    processUser()
      -> userName as String
      <- rtn as String: `Processed ${userName}`

  defines program

    @Test
    UserIsProcessed()
      user <- fetchUser()
      processedUser <- processUser(user)
      assert processedUser?

Download testing_quality_good.ek9

Test code that fails a quality check does not compile, so its tests do not run: fix the quality issue first. See Code Quality for every enforcement rule and the Error Index for each error.

Test Directory Structure

Test files live in the dev/ directory, which is included only when tests run (the -t flag).

layout/
├── testing_layout.ek9                  # Production code
└── dev/                                # Test source directory
    ├── testing_layout_unit.ek9         # Assert-based tests
    ├── greeting/                       # Black-box test (one per directory)
    │   ├── testing_layout_greeting.ek9
    │   └── expected_output.txt
    └── calculator/                     # Parameterized test
        ├── testing_layout_calculator.ek9
        ├── commandline_arg_basic.txt
        ├── expected_case_basic.txt
        ├── commandline_arg_edge.txt
        └── expected_case_edge.txt

A black-box test sits in a directory of its own because expected_output.txt belongs to the directory; a parameterized test likewise owns its case files. The sources:

#!ek9
defines module site.testing.layout

  defines function

    greet() as pure
      -> name as String
      <- greeting as String: `Hello, ${name}!`

    double() as pure
      -> amount as Integer
      <- result as Integer: amount * 2

Download testing_layout.ek9

#!ek9
defines module site.testing.layout.unit

  references
    site.testing.layout::double
    site.testing.layout::greet

  defines program

    @Test
    GreetingIsPolite()
      assert greet("Ada") == "Hello, Ada!"

    @Test
    DoublingWorks()
      assert double(21) == 42

Download testing_layout_unit.ek9

#!ek9
defines module site.testing.layout.greeting

  references
    site.testing.layout::greet

  defines program

    @Test
    GreetingOutput()
      stdout <- Stdout()
      stdout.println(greet("World"))

Download testing_layout_greeting.ek9

#!ek9
defines module site.testing.layout.calculator

  references
    site.testing.layout::double

  defines program

    @Test
    DoubleEachArgument()
      -> amount as Integer
      stdout <- Stdout()
      stdout.println(double(amount))

Download testing_layout_calculator.ek9

The runner finds all three kinds, and runs each case of the parameterized test as its own result:

$ ek9 -t testing_layout.ek9
ℹ️ Found 4 tests:
  2 assert (unit tests with assertions)
  1 blackbox (output validated against expected_output.txt)
  1 parameterized (2 cases total)

Executing 4 tests...

✅ PASS site.testing.layout.calculator::DoubleEachArgument [Param:basic] (nms)
✅ PASS site.testing.layout.calculator::DoubleEachArgument [Param:edge] (nms)
✅ PASS site.testing.layout.greeting::GreetingOutput [BlackBox] (nms)
✅ PASS site.testing.layout.unit::DoublingWorks [Assert] (nms)
✅ PASS site.testing.layout.unit::GreetingIsPolite [Assert] (nms)

Summary: 5 passed, 0 failed (5 results from 4 programs)
Types: 2 assert, 1 blackbox, 2 parameterized
Duration: nms

Test Configuration Errors

The Error Index documents test configuration errors (E81xxx), test execution errors (E82xxx) and coverage threshold errors (E83xxx).

Quick Reference

Task	Command / Syntax
Run all tests	`ek9 -t main.ek9`
Run with JSON output	`ek9 -t2 main.ek9`
Run with JUnit XML	`ek9 -t3 main.ek9`
List tests only	`ek9 -tL main.ek9`
Run specific group	`ek9 -tg groupname main.ek9`
Coverage details	See Code Coverage (`-t4`, `-t5`, `-t6`, `-tC`)
Profiling	See Profiling (append `p` to any format, e.g. `-t6p`)
Mark as test	`@Test` before program
Mark as grouped test	`@Test: "groupname"`
Expect an exit code	`@Test: "EXIT:3"` (default 0)
Assert condition	`assert condition`
Assert throws	`assertThrows(ExceptionType, expr)`
Assert no throw	`assertDoesNotThrow(expr)`
Precondition (uncatchable)	`require condition`
Black-box expected file	`dev/expected_output.txt`
Parameterized args	`dev/commandline_arg_{id}.txt`
Parameterized expected	`dev/expected_case_{id}.txt`

Overview: Testing as a Language Feature

Your First Test

testing_first.ek9 - The Code to Test

dev/testing_first_tests.ek9 - The Test

When Tests Fail

What You Get Out of the Box

Tests Run Automatically

Test Types

1. Assert-Based Tests

Project Structure

Production Code (testing_asserts.ek9)

Test Code (dev/testing_asserts_tests.ek9)

2. Black-Box Tests

Project Structure

Production Code (testing_blackbox.ek9)

Test Code (dev/testing_blackbox_tests.ek9)

Expected Output (dev/expected_output.txt)

When Output Doesn't Match

3. Parameterized Tests

Project Structure

Production Code (testing_parameterized.ek9)

Test Code (dev/testing_parameterized_tests.ek9)

Test Case "simple"

Test Case "edge"

The @Test Directive

Ungrouped vs Grouped Tests

Project Structure

Production Code (testing_grouped.ek9)

Test Code (dev/testing_grouped_tests.ek9)

Expected Exit Code

Assertion Statements

assert

assertThrows

assertDoesNotThrow

When an Assertion Fails

require vs assert

The require Statement

When to Use require

require vs Validation

Comparison with Other Languages

Test Runner

Why ek9 -t Sees the Whole Tree

Output Formats

Human Format (-t or -t1)

Terse Format (-t0)

JSON Format (-t2)

JUnit XML Format (-t3)

Code Coverage

Performance Profiling

Output Placeholders

Example: Testing a Report Generator

testing_placeholders.ek9

dev/testing_placeholders_tests.ek9

dev/expected_output.txt

Available Placeholders

Compile-Time Validation

Test-Specific Validation

Quality Enforcement on Test Code

Naming Quality (E11026, E11030, E11031)

Complexity Limits (E11010-E11013)

Cohesion and Coupling (E11014-E11016)

Example: Quality Violation in Test Code

Test Directory Structure

Test Configuration Errors

See Also

Quick Reference

Why `ek9 -t` Sees the Whole Tree