How do I test that my program exits with a particular exit code?

← Testing · Ref: Q1540

Declare it on the @Test directive with "EXIT:<n>", where n is a whole number from 0 to 255.

A program's returned Integer is its exit code, and under 'ek9 -t' the test judges it: a test passes only when its program exits with exactly the code the test expects. Without an EXIT part that is 0, so a test program that returns 3 FAILS with 'Test program exited with code 3'. Declare the 3 and the same program passes:

  @Test: "EXIT:3"
  EmptyOrderIsRejected()
    <- rtn as Integer: exitCodeFor(noOrder)
    assert rtn?

The declared code is checked both ways: a test declaring "EXIT:3" whose program exits 0 or 4 also fails ('Test program exited with code 0 but its @Test declares "EXIT:3"').

IT MUST BE THE PROGRAM'S OWN CODE

Return the code from the program. An uncaught Exception or Panic also ends the program with a non-zero code, but it writes to stderr as it does, and the test runner reports that as an error, never as the expected exit.

COMBINING WITH A REQUIREMENT AND A GROUP

A @Test carries up to three parts, in any order: a requirement id, an execution group and the exit code.

  @Test: "REQ:input-validation": "validation": "EXIT:3"

Each part is told apart by what it contains, so a mistake is a compile error rather than a silently ignored expectation: "EXIT:abc", "EXIT:", "EXIT:256", a second EXIT part, and a misspelled prefix such as "EXT:3" (it contains a colon, so it cannot quietly become a group name) are all rejected with E50200.

See Q1437 for how a program's exit code differs from the ek9 command's. See Q806 for test groups. See Q155 for writing tests.

Example

defines module qa.testdeep.expectedexitcode

  defines function

    exitCodeFor() as pure
      -> order as String
      <- rtn as Integer: 0
      if order is empty
        rtn: 3

  defines constant
    noOrder <- ""

  defines program

    //Returns 3 for an empty order, and declares that 3 is what it must exit with.
    @Test: "EXIT:3"
    EmptyOrderIsRejected()
      <- rtn as Integer: exitCodeFor(noOrder)
      assert rtn?

Common mistakes

E50200 — The expected exit code is written EXIT:<n>. A misspelled prefix contains a colon, so it is rejected rather than quietly becoming an execution group and dropping the expectation. See ek9 -h E50200 for details.

Incorrect:

    @Test: "EXT:3"

Correct:

    @Test: "EXIT:3"

Coming from another language?

Java: JUnit has no declared exit status; testing System.exit needs a SecurityManager workaround or a subprocess and a hand-written assertion. Python: pytest.raises(SystemExit) and then checking .code. Go: run the binary with os/exec and inspect ExitCode(). EK9: the expected exit code is part of the @Test directive and is checked when the source compiles.

Keywords: exit code, test, @Test, exit status, non-zero exit, expected exit code, returned Integer, EXIT