EK9 in Depth

Testing the Compiler Trusts

Testing as You Go in Volume 1 made three promises about this chapter: "the second volume goes into testing in depth", how coverage for publishing is measured "is covered in the second volume", and of test-driven development, "The second volume returns to it". It also left the loop of the guessing game untested, and Dependency Injection and Applications has since put that loop under a test. This chapter is the rest. The universal problem it answers is one you have met in JUnit and pytest: a green suite can be hollow. A test that asserts nothing passes. An assertion in a helper that no test calls never runs, and nothing says so. Coverage is a plugin that someone must remember to configure. The EK9 runner is part of the compiler, and the compiler refuses to call that suite green.¹

Three kinds of test, one runner

Every example in this chapter is one project: a small pricing and checkout module with a dev/ folder beside it. The program under test is complete, and it is the one you have seen the shape of already: a component behind an interface, a class that injects it, and a few pure functions.

#!ek9
defines module book.testing.shop

  defines constant
    PERCENT_BASE <- 100
    BULK_QUANTITY <- 10
    BULK_PERCENT <- 20

  defines class

    OrderService
      notifier as Notifier!

      placeOrder()
        ->
          item as String
          price as Integer
          percent as Integer

        notifier.notify(receipt(item, discounted(price, percent)))

      default operator ?

  defines function

    discounted() as pure
      ->
        price as Integer
        percent as Integer
      <- rtn as Integer: Integer()

      if percent > PERCENT_BASE
        throw Exception(`a discount of ${percent} percent is more than the price`)
      rtn: price - price * percent / PERCENT_BASE

    bulkPrice() as pure
      ->
        quantity as Integer
        unitPrice as Integer
      <- rtn as Integer: quantity * unitPrice

      if quantity >= BULK_QUANTITY
        rtn: discounted(quantity * unitPrice, BULK_PERCENT)

    receipt() as pure
      ->
        item as String
        price as Integer
      <- rtn as String: `${item}: ${price}p`

  defines component

    Notifier as abstract
      notify() as abstract
        -> message as String

    ConsoleNotifier is Notifier
      override notify()
        -> message as String
        Stdout().println(message)

  defines application

    shopApp
      register ConsoleNotifier() as Notifier

  defines program

    Checkout() with application of shopApp
      orders <- OrderService()
      orders.placeOrder("kettle", 2500, 20)
      orders.placeOrder("mug", 400, 0)

It prints the two receipts:

kettle: 2000p
mug: 400p

A test is a program marked @Test, in a source file under dev/. There is no test framework to add, and no test class to extend. The runner recognises three kinds, and it treats them as equals.

An assertion test states what must be true. assert is for tests, as require is for production code, and assertThrows and assertDoesNotThrow cover exceptions. assertThrows hands back the exception it caught, so you can look at the reason:

    @Test
    MoreThanFullDiscountIsRefused()
      assertThrows (Exception, discounted(400, 101))

    @Test
    ReasonNamesThePercent()
      refused <- assertThrows (Exception, discounted(400, 150))
      assert refused.reason() == "a discount of 150 percent is more than the price"

    @Test
    SensibleDiscountThrowsNothing()
      assertDoesNotThrow (discounted(400, 50))

A black-box test has no assertion. It prints, and the runner compares everything it printed to a file named expected_output.txt beside it. Where the program talks to the world, here through Notifier, the test brings its own stand-in and a test application that registers it: TaggedNotifier is the whole of what a mocking framework would have done, and it is an ordinary component.

defines module book.testing.shop.checkout

  references
    book.testing.shop::Notifier
    book.testing.shop::OrderService

  defines component

    TaggedNotifier is Notifier
      override notify()
        -> message as String
        Stdout().println(`[test] ${message}`)

  defines application

    taggedShop
      register TaggedNotifier() as Notifier

  defines program

    @Test
    CheckoutOutput() with application of taggedShop
      orders <- OrderService()
      orders.placeOrder("kettle", 2500, 20)
      orders.placeOrder("mug", 400, 0)

with expected_output.txt beside it holding:

[test] kettle: 2000p
[test] mug: 400p

A parameterized test takes program arguments, and the runner runs it once per case: a commandline_arg_<case>.txt file supplies the arguments, one per line, and expected_case_<case>.txt pins that case's output. The table is files, not a data provider:

defines module book.testing.shop.receipts

  references
    book.testing.shop::discounted
    book.testing.shop::receipt

  defines program

    @Test
    ReceiptForArguments()
      ->
        item as String
        price as String
        percent as String

      stdout <- Stdout()
      stdout.println(receipt(item, discounted(Integer(price), Integer(percent))))

Two cases are sale (kettle, 2500, 20, expecting kettle: 2000p) and free (mug, 400, 100, expecting mug: 0p). If you know Go's table-driven tests or golden files, these are the same ideas with a file convention instead of a library. Here is the whole project under ek9 -t -tC:

ℹ️ Found 13 tests:
  10 assert (unit tests with assertions)
  2 blackbox (output validated against expected_output.txt)
  1 parameterized (2 cases total)

Executing 13 tests...

✅ PASS book.testing.shop.checkout::CheckoutOutput [BlackBox] (8ms)
✅ PASS book.testing.shop.production::ProductionCheckout [BlackBox] (8ms)
✅ PASS book.testing.shop.receipts::ReceiptForArguments [Param:free] (6ms)
✅ PASS book.testing.shop.receipts::ReceiptForArguments [Param:sale] (1ms)
✅ PASS book.testing.shop.unit::EveryPartIsSet [Assert] (6ms)
✅ PASS book.testing.shop.unit::FullDiscountIsFree [Assert] (6ms)
✅ PASS book.testing.shop.unit::MoreThanFullDiscountIsRefused [Assert] (6ms)
✅ PASS book.testing.shop.unit::NineItemsPayInFull [Assert] (6ms)
✅ PASS book.testing.shop.unit::NoDiscountKeepsThePrice [Assert] (6ms)
✅ PASS book.testing.shop.unit::ReasonNamesThePercent [Assert] (6ms)
✅ PASS book.testing.shop.unit::ReceiptShowsItemAndPrice [Assert] (6ms)
✅ PASS book.testing.shop.unit::SensibleDiscountThrowsNothing [Assert] (6ms)
✅ PASS book.testing.shop.unit::TenItemsGetTwentyPercentOff [Assert] (6ms)
✅ PASS book.testing.shop.unit::TenPercentOff [Assert] (6ms)

Summary: 14 passed, 0 failed (14 results from 13 programs)
Types: 10 assert, 2 blackbox, 2 parameterized
Duration: 89ms

📊 Coverage Summary
  ✅ Methods:  90.9% (10/11)
  ✅ Lines:    88.6% (31/35)
  ✅ Branches: 100.0% (2/2)

  ✅ Overall: 88.89% (32/36 probes hit)

📦 By Module:
  ✅ book.testing.shop: 88.89% (32/36)

The labels [Assert], [BlackBox] and [Param:free] are the three kinds, counted together in one summary. The tests are listed in alphabetical order, not source order. A test that fails says where and why. Here are two that were written wrongly, one asserting the wrong total and one expecting an exception that is never thrown:

❌ FAIL book.testing.pricing.failing::NothingThrown [Assert] (3ms)
    assertThrows FAILED
      Location: ./dev/testFailingAssertions.ek9:15:7
      Expression: discounted(2000, 10)
      Expected: org.ek9.lang::Exception
      Actual: No exception was thrown
❌ FAIL book.testing.pricing.failing::WrongExpectation [Assert] (3ms)
    Assertion failed: `discounted(2000, 10) == 1900` at ./dev/testFailingAssertions.ek9:11:7

Summary: 0 passed, 2 failed (2 total)
Types: 2 assert
Duration: 7ms
ek9: tests failed (exit 11)

The exit code is 11 for a failed test, which is what a CI step needs to see.

Test first, and what red looks like

Testing as You Go cited Kent Beck for writing the test before the code. In Java or Python the first red is a failing assertion, after you have written a stub that returns the wrong thing. In EK9 it can come sooner. Write the bulk-order test before bulkPrice exists, and the project does not compile:

defines module book.testing.pricing.first

  references
    book.testing.pricing::bulkPrice

  defines program

    @Test
    TenItemsGetTwentyPercentOff()
      assert bulkPrice(10, 200) == 1600

EK9     : Error   : E03030: 'bulkPrice' on line 5 position 26: 'book.testing.pricing::bulkPrice' reference does not resolve
             See: ek9 -h E03030

That is the red step: the test names a function, and the compiler says there is no such thing. Write bulkPrice (it is in the program above), run ek9 -t, and the same test is green. The loop is short because -t compiles the program and its dev/ files together, so a test that does not yet have something to test cannot be mistaken for one that passes.

What the compiler refuses to call a test

The three rules that follow are what make the word "green" mean something. Each is checked by the compiler, before any test runs, and each has a code. The first is a test that cannot fail because it checks nothing:

    @Test
    TenPercentOff()
      stdout <- Stdout()
      stdout.println($discounted(2000, 10))

EK9     : Error   : E81007: 'TenPercentOff' on line 10 position 4: 'book.testing.pricing.empty::TenPercentOff' has no assertions and no expected output files: @Test program has no assertions and no expected output files
             See: ek9 -h E81007

The second half of the message matters. A test with no assert is legal when it has an expected_output.txt or its expected_case files, because then its output is the assertion; that is a black-box test. With neither, it checks nothing.

The second is an assertion that no test reaches. A helper that holds an assert is only useful if a @Test calls it:

  defines function

    checkNeverNegative()
      -> price as Integer
      assert discounted(price, 10) >= 0

  defines program

    @Test
    TenPercentOff()
      assert discounted(2000, 10) == 1800

EK9     : Error   : E81011: 'checkNeverNegative' on line 9 position 4: 'book.testing.pricing.orphan::checkNeverNegative' contains assertions but is not reachable from any @Test program: assertion is not reachable from any @Test program
             See: ek9 -h E81011

This is the rule that retires the dead assertion. In JUnit the helper compiles, is never called, and the suite is green with the check silently absent; here an orphan assertion does not compile.

The third is the opposite fault, an assertion that production code can reach. If a plain program also calls the helper, the assertion would run in production:

    @Test
    TenPercentOff()
      checkNeverNegative(2000)

    Report()
      checkNeverNegative(400)

EK9     : Error   : E81012: 'checkNeverNegative' on line 9 position 4: 'book.testing.pricing.production::checkNeverNegative' contains assertions but is reachable from production program (use 'require' instead): assertion is reachable from non-test program
             See: ek9 -h E81012

The message names the remedy: production code that must check something uses require, the other half of the pair from Testing as You Go. Last, assert and @Test are only legal in dev/ source. In the production file both are the same code, E81015, from different phases:

EK9     : Error   : E81015: 'assert' on line 15 position 6: 'assert': test construct only allowed in dev source files
             See: ek9 -h E81015
EK9     : Error   : E81015: 'TenPercentOff' on line 18 position 4: '@Test': test construct only allowed in dev source files
             See: ek9 -h E81015

Keep the two codes apart. E81015 is about where the file is: an assert or a @Test in production source. E81012 is about what reaches an assertion: it fires inside dev/, on a helper a production program also calls. E81011 and E81007 need the dev/ folder compiled with the program, which is why they appear under ek9 -t and not under a plain compile.

The gate, and what coverage cannot see

Coverage is not an add-on here. ek9 -t collects it on every run, and it gates: below 80% the run ends with E83001 and exit 12 even if every test passed. The same 80% is the bar ek9 -P applies before it will package a program, and there is no flag to lower it. Take the project above and leave out its last two additions, the production black-box test and EveryPartIsSet. All twelve tests are still green, and the build still fails:

Summary: 12 passed, 0 failed (12 results from 11 programs)
Types: 9 assert, 1 blackbox, 2 parameterized
Duration: 79ms

📊 Coverage Summary
  ❌ Methods:  36.4% (4/11)
  ⚠️ Lines:    65.7% (23/35)
  ✅ Branches: 100.0% (2/2)

  ⚠️ Overall: 63.89% (23/36 probes hit)

📦 By Module:
  ⚠️ book.testing.shop: 63.89% (23/36)

⚠️  Coverage below 80% - detailed report written to: .ek9/coverage-detail.json

❌ E83001: Code coverage 63.9% is below required threshold 80%
   Methods:  36.4% (4/11)
   Lines:    65.7% (23/35)
   Branches: 100.0% (2/2)

   Uncovered items:
     7 method(s)
       • book.testing.shop::OrderService.? (./Shop.ek9:22)
       • book.testing.shop::Notifier.? (./Shop.ek9:53)
       • book.testing.shop::ConsoleNotifier.? (./Shop.ek9:57)
       ... and 4 more

   Use -t4 or -t5 for detailed coverage report.
ek9: coverage below the required threshold (exit 12)

Two of the uncovered methods are worth explaining, because you will meet them in every small project. The ? operators are compiler-written, one for each aggregate, and each counts as a method; assert OrderService()? in EveryPartIsSet runs them. And the production wiring, shopApp, only counts as exercised if some test runs against it, which is what the production test does with the real ConsoleNotifier. The Checkout program itself stays uncovered, because a test cannot call a program; it is the thin part that talks, and the bar leaves room for it.

Coverage is a fact about lines, though, and it can be 100% while the tests asked only the easy questions. ek9 -t5 adds a report of input variety: for each callable, how many kinds of value the tests passed in. These are the five callables of the project above:

Input-variety (numerator, measurement-only):
  book.testing.shop::ConsoleNotifier.notify          driven 2 cell(s), asserted 2: p0: SET  p1: SPECIAL_CHARS
  book.testing.shop::OrderService.placeOrder         driven 6 cell(s), asserted 6: p0: SET  p1: TYPICAL  p2: POSITIVE,TYPICAL  p3: POSITIVE,ZERO
  book.testing.shop::bulkPrice                       driven 2 cell(s), asserted 2: p0: POSITIVE  p1: POSITIVE
  book.testing.shop::discounted                      driven 4 cell(s), asserted 4: p0: POSITIVE,TYPICAL  p1: POSITIVE,ZERO
  book.testing.shop::receipt                         driven 4 cell(s), asserted 4: p0: TYPICAL  p1: POSITIVE,TYPICAL,ZERO

Every line of discounted ran, but its two Integer parameters were only ever given positive numbers and zero: no negative price, no unset one. ek9 -t6 writes an HTML report whose Input Variety dial puts a number on it, with the denominator, which the -t5 table does not print:

Input classes driven
17.0%
18 / 106 cells across 7 construct(s)

For discounted alone the report's JSON, input-variety.json, gives a floor of 18 classes between its two parameters, of which the tests drove 4. The report's own text states its status: "The floor is 1-wise: each input value-class needs driving at least once. It is measurement only -- nothing here fails a build." That is the whole of its authority. Input variety is the exposure half of HEX, the advisory change-risk score (ek9 -h hex), and Quality Built In is where the two halves meet. It never gates; coverage gates. A suite validated only by expected-output files records driven classes and none asserted, because only an assert opens the window in which a result is observed, so read a low asserted figure beside the count of output-validated tests before concluding that testing is thin.

The cost

The compiler policing your tests is not free, and you will feel it in three places. A test that exists only to print something useful while you debug is E81007 the moment it is committed without an expected-output file, so a scratch test has to be a scratch program, outside dev/. A black-box test compares all of the output, byte for byte, so a changed message or an unstable value, a timestamp or a random number, fails it, and you either control the source of the value, as the fixed secret did in Dependency Injection and Applications, or you assert on part of it instead. And the 80% bar does not distinguish a thorough test from assert OrderService()?: the ? assertion above exists to run a compiler-written method, and the gate is satisfied by it.

What the cost buys is narrower than "your code is correct", and the chapter has tried not to say more. It buys a suite in which every green test has either an assertion that production code cannot reach or an output file that pins what was printed, every assertion is reachable from a test, coverage is measured without a plugin, and a stand-in for the outside world is a component you write in a dozen lines. What it does not buy is a judgement about whether the right questions were asked. Input variety is the nearest the compiler comes to that, and it only measures.