Testing in EK9

Tests verify your code works correctly. When you change code later, running tests confirms you haven't broken existing functionality.

• Your First Test
• What Makes EK9 Different
• Test Types - Assert-Based, Black-Box, Parameterized
• The @Test Directive - Grouping tests
• Assertion Statements - assert, assertThrows, assertDoesNotThrow
• Test Runner - Commands and output formats
• Code Coverage - Always-on collection, -tC output
• Output Placeholders - Matching dynamic values
• Compile-Time Validation
• Quick Reference

Your First Test

Create a simple project with two files:

myproject/
├── main.ek9        # Your code (the function to test)
└── dev/
    └── tests.ek9   # Your tests

The dev/ directory is special - files here are only included when running tests. This keeps test code separate from production code.

main.ek9 - The Code to Test

#!ek9
defines module my.first.test

  defines function

    add() as pure
      -> a as Integer, b as Integer
      <- result as Integer: a + b

//EOF

dev/tests.ek9 - The Test

#!ek9
defines module my.first.test.tests

  references
    my.first.test::add

  defines program

    @Test
    AdditionWorks()
      result <- add(2, 3)
      assert result == 5

//EOF

Key concepts: references imports symbols from other modules, defines program declares an entry point, @Test marks it for the test runner, and assert validates conditions.

Run it:

$ ek9 -t main.ek9
[i] Found 1 test:
  1 assert (unit tests with assertions)

Executing 1 test...

[OK] PASS my.first.test.tests::AdditionWorks [Assert] (2ms)

Summary: 1 passed, 0 failed (1 total)

The file tests.ek9 can be named anything you like, and you can have as many .ek9 files in dev/ as you need - each can contain multiple @Test programs. The test runner discovers all of them.

When Tests Fail

EK9 shows exactly what failed:

[X] FAIL my.first.test.tests::AdditionWorks [Assert] (2ms)
    Assertion failed: `result==5` at ./dev/tests.ek9:14:7

Summary: 0 passed, 1 failed (1 total)

The expression (result==5), file, line, and column are captured automatically from the AST. No stack traces to parse, no custom messages to write.

What Makes EK9 Testing Different

Unlike frameworks that require imports and setup, EK9's testing is built into the language grammar. This enables compile-time validation:

• Empty test = compile error - A @Test with no assertions won't compile
• Orphan assertion = compile error - An assert not reachable from any @Test won't compile
• Production assertion = compile error - Using assert in non-test code won't compile

In JUnit or pytest, an empty test passes silently. An orphan assertion is never discovered. EK9 catches these mistakes before you run.

Comparison with Other Languages

Tests Run Automatically

When you package (ek9 -P) or deploy (ek9 -D) your code, tests are executed automatically. You don't need to remember to run them - EK9 won't package code with failing tests. Testing isn't a separate manual step; it's woven into the development workflow.

Test Types

EK9 supports three complementary testing approaches:

• Assert-Based - Unit testing individual functions; call functions and check results with assert
• Black-Box - Regression testing program output; compare stdout to expected_output.txt
• Parameterized - Testing with multiple input sets; run same test with different input files

1. Assert-Based Tests

Use assert, assertThrows, and assertDoesNotThrow for internal validation within test code.

Project Structure

simpleAssertTest/
├── main.ek9           # Production code (functions to test)
└── dev/
    └── tests.ek9      # Test programs with @Test directive

Production Code (main.ek9)

#!ek9
defines module simple.assert.test

  defines function

    add() as pure
      ->
        a as Integer
        b as Integer
      <- result as Integer: a + b

    multiply() as pure
      ->
        a as Integer
        b as Integer
      <- result as Integer: a * b

//EOF

Test Code (dev/tests.ek9)

#!ek9
defines module simple.assert.test.tests

  references
    simple.assert.test::add
    simple.assert.test::multiply

  defines program

    @Test
    AdditionTest()
      result <- add(2, 3)
      assert result?
      assert result == 5

    @Test
    MultiplicationTest()
      result <- multiply(4, 5)
      assert result?
      assert result == 20

    @Test
    CombinedOperationsTest()
      sum <- add(10, 20)
      product <- multiply(sum, 2)
      assert product == 60

//EOF

2. Black-Box Tests

Validate program output against expected files. For tests without command line arguments, the file must be named exactly expected_output.txt in the same directory as the test. Ideal for regression testing and AI-generated tests.

Project Structure

blackBoxTest/
├── main.ek9                # Production code
└── dev/
    ├── tests.ek9           # Test program
    └── expected_output.txt # Expected stdout output

Production Code (main.ek9)

#!ek9
defines module blackbox.test

  defines function

    greet() as pure
      -> name as String
      <- greeting as String: "Hello, " + name + "!"

//EOF

Test Code (dev/tests.ek9)

#!ek9
defines module blackbox.test.tests

  references
    blackbox.test::greet

  defines program

    @Test
    GreetingOutputTest()
      stdout <- Stdout()
      stdout.println(greet("World"))
      stdout.println(greet("EK9"))

//EOF

Expected Output (dev/expected_output.txt)

Hello, World!
Hello, EK9!

When Output Doesn't Match

If the actual output differs from expected, EK9 shows a line-by-line comparison:

[X] FAIL blackbox.test.tests::GreetingOutputTest [BlackBox] (3ms)
    Output mismatch at line 2:
      Expected: Hello, EK9!
      Actual:   Hello, EK9?

3. Parameterized Tests

Run the same test with multiple inputs using commandline_arg_{id}.txt and expected_case_{id}.txt file pairs. Each file pair defines a test case.

Project Structure

parameterizedTest/
├── main.ek9                       # Production code
└── dev/
    ├── tests.ek9                  # Test program with parameters
    ├── commandline_arg_simple.txt # Case "simple": input arguments
    ├── expected_case_simple.txt   # Case "simple": expected output
    ├── commandline_arg_edge.txt   # Case "edge": input arguments
    └── expected_case_edge.txt     # Case "edge": expected output

Production Code (main.ek9)

#!ek9
defines module parameterized.test

  defines function

    processArg() as pure
      -> arg as String
      <- result as String: "Processed: " + arg

//EOF

Test Code (dev/tests.ek9)

#!ek9
defines module parameterized.test.tests

  references
    parameterized.test::processArg

  defines program

    @Test
    ArgProcessor()
      ->
        arg0 as String
        arg1 as String

      stdout <- Stdout()
      stdout.println(processArg(arg0))
      stdout.println(processArg(arg1))

//EOF

Test Case "simple"

commandline_arg_simple.txt:

hello
world

expected_case_simple.txt:

Processed: hello
Processed: world

Test Case "edge"

commandline_arg_edge.txt:

single
only

expected_case_edge.txt:

Processed: single
Processed: only

The @Test Directive

Mark programs as tests using the @Test directive. Only programs with this directive are discovered and executed by the test runner.

Ungrouped vs Grouped Tests

By default, tests run in parallel for faster execution. Use groups when tests need sequential execution - typically for database tests, file system tests, or tests that share external resources where order matters.

Syntax: @Test: "groupname" - tests in the same group run sequentially, while different groups run in parallel with each other.

Project Structure

groupedTests/
├── main.ek9        # Production code (Counter class)
└── dev/
    └── tests.ek9   # Test programs

Production Code (main.ek9)

#!ek9
defines module grouped.tests

  defines class

    Counter
      value as Integer: 0

      getValue() as pure
        <- rtn as Integer: value

      increment()
        value: value + 1

//EOF

Test Code (dev/tests.ek9)

#!ek9
defines module grouped.tests.tests

  references
    grouped.tests::Counter

  defines program

    @Test: "counter"
    CounterIncrementTest()
      c <- Counter()
      c.increment()
      assert c.getValue() == 1

    @Test: "counter"
    CounterMultipleIncrementTest()
      c <- Counter()
      c.increment()
      c.increment()
      c.increment()
      assert c.getValue() == 3

    @Test
    IndependentTest()
      c <- Counter()
      assert c.getValue() == 0

//EOF

CounterIncrementTest and CounterMultipleIncrementTest are both in the "counter" group and run sequentially. IndependentTest has no group and runs in parallel with other ungrouped tests.

Assertion Statements

Unlike traditional testing frameworks that require parsing stack traces, EK9's assertions provide structured, precise error information automatically captured from the AST. This includes the exact source location (file, line, column), the expression that failed, and contextual details - all without writing custom error messages.

assert

Validates that a condition is true:

@Test
CheckAddition()
  result <- 2 + 3
  assert result?          // Check result is set (not unset)
  assert result == 5      // Check result equals expected value

Failure Output

When an assertion fails, EK9 shows the exact expression and location:

Assertion failed: `result==5` at ./dev/tests.ek9:28:7

assertThrows

Validates that an expression throws a specific exception type:

@Test
CheckDivisionByZero()
  assertThrows(Exception, 10 / 0)

@Test
CheckAndInspectException()
  caught <- assertThrows(Exception, 10 / 0)
  assert caught.message()?

Failure Output - No Exception Thrown

If the expression doesn't throw:

assertThrows FAILED
  Location: ./dev/tests.ek9:5:3
  Expression: 10 / 2
  Expected: org.ek9.lang::Exception
  Actual: No exception was thrown

assertDoesNotThrow

Validates that an expression completes without throwing any exception:

@Test
CheckSafeDivision()
  assertDoesNotThrow(10 / 2)

@Test
CheckAndCaptureResult()
  result <- assertDoesNotThrow(10 / 2)
  assert result == 5

Failure Output

If the expression throws unexpectedly:

assertDoesNotThrow FAILED
  Location: ./dev/tests.ek9:5:3
  Expression: 10 / 0
  Expected: No exception
  Actual: org.ek9.lang::Exception
  Message: Division by zero

Why This Matters

Traditional testing frameworks like JUnit or pytest require you to either:

• Write custom assertion messages manually
• Parse stack traces to find the failure location
• Guess which assertion failed when there are multiple in a test

EK9's grammar-level assertions automatically capture the expression text, source location, and all relevant context at compile time. This is especially valuable for:

• AI/LLM integration - Structured output is easily parsed
• CI/CD pipelines - Precise locations enable automated issue creation
• Debugging - No stack trace parsing required

require vs assert

EK9 distinguishes between production preconditions and test assertions:

• require - Production preconditions, checked in any code path
• assert - Test validation, only valid in @Test programs

Using assert in production code paths produces compile-time error E81012.

Test Runner

Run tests using the -t flag:

ek9 -t myproject.ek9        # Run tests (human output)
ek9 -t0 myproject.ek9       # Terse output
ek9 -t2 myproject.ek9       # JSON output (for CI/AI)
ek9 -t3 myproject.ek9       # JUnit XML output
ek9 -tL myproject.ek9       # List tests without running
ek9 -tg database myproject.ek9  # Run only "database" group

Exit codes: The test runner returns exit code 0 when all tests pass, and exit code 11 when any test fails. This enables CI/CD pipelines to detect failures automatically. See Command Line for all exit codes.

Output Formats

EK9 provides four output formats optimized for different use cases:

Human Format (-t or -t1)

Visual output with icons for terminal use:

[i] Found 4 tests:
  4 assert (unit tests with assertions)

Executing 4 tests...

[OK] PASS myapp.tests::AdditionWorks [Assert] (3ms)
[X] FAIL myapp.tests::DivisionFails [Assert] (2ms)
    Assertion failed: `result==5` at ./dev/tests.ek9:28:7
[X] FAIL myapp.tests::AnotherFailure [Assert] (1ms)
    Assertion failed: `1==2` at ./dev/tests.ek9:33:7
[OK] PASS myapp.tests::MultiplicationWorks [Assert] (2ms)

Summary: 2 passed, 2 failed (4 total)
Types: 4 assert
Duration: 8ms

Grouped tests show their group name:

[OK] PASS myapp.tests::CounterTest [Assert] {counter} (2ms)

Terse Format (-t0)

Minimal output for scripting and CI pass/fail checks:

4 tests: 2 passed, 2 failed (4 assert)

JSON Format (-t2)

Structured output for AI/LLM integration and custom tooling:

{
  "version": "1.0",
  "timestamp": "2025-12-31T14:30:00+00:00",
  "architecture": "JVM",
  "summary": {
    "total": 4,
    "passed": 2,
    "failed": 2,
    "types": { "assert": 4 }
  },
  "tests": [
    {
      "name": "AdditionWorks",
      "fqn": "myapp.tests::AdditionWorks",
      "status": "passed",
      "duration_ms": 3
    },
    {
      "name": "DivisionFails",
      "fqn": "myapp.tests::DivisionFails",
      "status": "failed",
      "failure": {
        "type": "assertion",
        "message": "Assertion failed: `result==5` at ./dev/tests.ek9:28:7"
      }
    }
  ]
}

JUnit XML Format (-t3)

Standard format for CI/CD systems (Jenkins, GitHub Actions, GitLab):

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="myapp.tests" tests="4" failures="2" errors="0" time="0.008">
  <testcase name="AdditionWorks" classname="myapp.tests" time="0.003"/>
  <testcase name="DivisionFails" classname="myapp.tests" time="0.002">
    <failure message="Assertion failed" type="AssertionError">
Assertion failed: `result==5` at ./dev/tests.ek9:28:7
    </failure>
  </testcase>
  <testcase name="AnotherFailure" classname="myapp.tests" time="0.001">
    <failure message="Assertion failed" type="AssertionError">
Assertion failed: `1==2` at ./dev/tests.ek9:33:7
    </failure>
  </testcase>
  <testcase name="MultiplicationWorks" classname="myapp.tests" time="0.002"/>
</testsuite>

Code Coverage

EK9 automatically collects code coverage data during test execution. Coverage probes are inserted at compile time at all control flow points:

• Function/method entry and exit
• Branch decisions - if/else branches, switch cases
• Loop body execution - for, for-range, while, do-while
• Exception handlers - catch/handle blocks

Coverage is always collected when running tests with -t. Use the -tC flag to display or output coverage results.

Coverage Output with -tC

The coverage output format follows the test output format:

ek9 -t -tC main.ek9         # Human-readable coverage summary
ek9 -t0 -tC main.ek9        # Terse coverage: Coverage: 87.5% (7/8)
ek9 -t2 -tC main.ek9        # JSON output to files
ek9 -t3 -tC main.ek9        # JUnit XML + JaCoCo XML to files

File Output for Machine Formats

When using machine formats (-t2 or -t3) with -tC, results are written to files in the .ek9/ directory:

A human-readable summary with file paths is printed to stdout:

Summary: 3 passed, 0 failed (3 total)
Types: 3 assert
Duration: 13ms

Test results written to: .ek9/test-results.json
Coverage results written to: .ek9/coverage.json

The JaCoCo XML format (.ek9/coverage.xml) is compatible with coverage tools like SonarQube, Codecov, and Coveralls.

Coverage JSON Format

{
  "coverage": {
    "methods": { "percentage": 100.00, "covered": 3, "total": 3 },
    "lines": { "percentage": 100.00, "covered": 5, "total": 5 },
    "branches": { "percentage": 100.00, "covered": 5, "total": 5 },
    "overall": 100.00,
    "probesHit": 5,
    "probesTotal": 5,
    "modules": {
      "my.module.tests": { "coverage": 100.00, "hit": 3, "total": 3 },
      "my.module": { "coverage": 100.00, "hit": 2, "total": 2 }
    }
  }
}

JaCoCo XML Format

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE report PUBLIC "-//JACOCO//DTD Report 1.1//EN" "report.dtd">
<report name="EK9 Coverage">
  <counter type="METHOD" missed="0" covered="3"/>
  <counter type="LINE" missed="0" covered="5"/>
  <counter type="BRANCH" missed="0" covered="5"/>
  <package name="my/module/tests">
    <counter type="BRANCH" missed="0" covered="3"/>
  </package>
</report>

Output Placeholders

Black-box tests often produce dynamic values like dates, timestamps, or IDs that change between runs. Use type-based placeholders in expected output files to match these values. Placeholder names match EK9 type names - if you know EK9 types, you know the placeholders.

Example: Testing a Report Generator

main.ek9

#!ek9
defines module report.generator

  defines function

    generateReport() as pure
      -> itemCount as Integer
      <- report as String: `Report generated on ` + $Date() + ` with ` + $itemCount + ` items`

//EOF

dev/tests.ek9

#!ek9
defines module report.generator.tests

  references
    report.generator::generateReport

  defines program

    @Test
    ReportIncludesDate()
      stdout <- Stdout()
      stdout.println(generateReport(42))

//EOF

dev/expected_output.txt

Report generated on {{Date}} with {{Integer}} items

The test passes regardless of which date or item count is used, because {{Date}} matches any valid date (e.g., 2025-12-31) and {{Integer}} matches any integer.

Available Placeholders

See E81010 for the complete list of 18 valid placeholders. Using an invalid placeholder name produces a compile-time error.

Compile-Time Validation

EK9's call graph analysis detects testing issues at compile time:

• E81007 - Empty @Test (no assertions, no expected files)
• E81011 - Orphan assertion (not reachable from any @Test)
• E81012 - Production assertion (assert in non-test code path)

Test Directory Structure

Test files live in the dev/ directory, which is only included when running tests (-t flag).

myproject/
├── main.ek9                    # Production code
└── dev/                        # Test source directory
    ├── unitTests.ek9           # Assert-based tests
    ├── greetingTest/           # Black-box test (one per directory)
    │   ├── test.ek9
    │   └── expected_output.txt
    └── calculatorTest/         # Parameterized test
        ├── test.ek9
        ├── commandline_arg_basic.txt
        ├── expected_case_basic.txt
        ├── commandline_arg_edge.txt
        └── expected_case_edge.txt

Test Configuration Errors

See the Error Index for complete documentation of test configuration errors (E81xxx) and test execution errors (E82xxx).

Quick Reference