How do @Test programs work with command-line arguments, timeouts, and expected output?

← Testing · Ref: Q953

@Test programs accept TYPED PARAMETERS (not just String) — the EK9 runtime introspects the program's parameter signature and parses commandline_arg lines into the declared types. The E81xxx codes flag config issues (paired-file naming, arg counts, type conversion); the E82xxx code flags an execution issue (a test still running at its time limit).

TEST PROGRAM SIGNATURES

Tests ARE programs. They CAN have typed parameters:

  @Test
  NumberProcessor()
    ->
      num1 as Integer
      num2 as Integer
    assert num1?
    assert num2?

The runtime auto-parses each commandline_arg line into the declared type — Integer, Date, Duration, DateTime, Money, Boolean, etc. Per parameterizedHybrid.ek9 corpus example.

COMPANION FILES (in the same directory as the @Test program's source file, usually under dev/)

  expected_output.txt — expected stdout (a test with no parameters)
  commandline_arg_<id>.txt + expected_case_<id>.txt — one case of a parameterised test: its arguments, one per line, and its expected stdout
  envvars.txt — KEY=VALUE lines added to the test's environment; envvars_arg_<id>.txt for one case (else envvars.txt)
  stdin.txt — what the test reads from Stdin; stdin_arg_<id>.txt for one case (else stdin.txt)
  expected_stderr.txt — expected stderr (a test with no parameters); stderr_case_<id>.txt for one case

Expected stdout and stderr may hold placeholders such as {{DateTime}} and {{Millisecond}}.
The runner reads no other file: a misspelt name (env_vars.txt, expected_ouput.txt) is simply not read, so a test left with no assertion and no expected output is E81007.

E81xxx CONFIG ERRORS (per TestConfigurationError.java)

  - E81001 MISSING_EXPECTED_OUTPUT — commandline_arg_X.txt without expected_case_X.txt
  - E81002 ORPHAN_EXPECTED_OUTPUT — expected_case_X.txt without commandline_arg_X.txt
  - E81003 ARGUMENT_COUNT_MISMATCH — program declares N params, file provides not-N args
  - E81004 UNEXPECTED_ARGUMENTS — commandline file exists but program has no params
  - E81005 MISSING_ARGUMENTS — program declares params but no commandline files found
  - E81006 TYPE_CONVERSION_ERROR — string can't parse as declared parameter type
  - E81007 EMPTY_TEST — @Test program has no asserts AND no expected_output
  - E81008 DUPLICATE_CASE_ID — two case ids in one directory that differ only in letter case (commandline_arg_Big.txt and commandline_arg_big.txt are one file on macOS)
  - E81010 INVALID_PLACEHOLDER — bad {{...}} syntax in expected output
  - E81013 ENVVARS_PARSE_ERROR — a line in an envvars file with no =
  - E81014 ORPHAN_ENVVARS — envvars_arg_<id>.txt without a matching commandline_arg_<id>.txt
  - E81016 ORPHAN_STDIN — stdin_arg_<id>.txt without a matching commandline_arg_<id>.txt

E82xxx EXECUTION ERRORS

  - E82001 TEST_TIMEOUT — the test was still running at its time limit; the failure names the limit, and the other tests in the run still run

THE TIME LIMIT

Each test run (each case of a parameterised test) may take at most EK9_TEST_TIMEOUT_MS milliseconds: 30000 (30 seconds) unless you set it. Set it in the environment (export EK9_TEST_TIMEOUT_MS=60000) or in ~/.ek9/ek9.env (a line EK9_TEST_TIMEOUT_MS=60000; the IDE's settings page edits that file): one value governs ek9 -t and the IDE's test runs. A value that is not a whole number greater than 0 is reported once, and the default is used. At the limit a native test is killed with every process it started. Under ek9 -t a JVM test's run is interrupted and abandoned, and ek9 -t still ends when the other tests are done; a test run from the IDE or MCP has a JVM of its own, which is killed with every process it started, and the run's other tests go on in a fresh one.

STDOUT MISMATCH

Note: stdout that does not match the expected output is NOT a numbered error - and that includes a test that prints nothing where expected_output.txt expects output. It is reported in the test runner's pass/fail report as an output mismatch.

See Q1296 for capture isolation. See Q204 for black-box patterns. See Q205 for parameterized tests.

Example

#!ek9
defines module qa.testing.programargs

  defines program

    <?-
      Demonstrates the @Test program pattern with typed parameters
      and the companion-file model. In a real test, prefix with @Test.
      Programs (and tests) accept typed parameters that the EK9 runtime
      auto-parses from commandline_arg files.
    -?>
    TestProgramArgsDemo()
      stdout <- Stdout()

      stdout.println("@Test program facts:")
      stdout.println("  Tests ARE programs (defines program + @Test directive)")
      stdout.println("  Typed parameters supported (Integer, Date, Money, etc.)")
      stdout.println("  Runtime auto-parses commandline_arg lines into types")
      stdout.println("  E81xxx codes flag config issues (paired files, arg counts, type conversion)")
      stdout.println("  E82001 = TEST_TIMEOUT (still running at the time limit)")
      stdout.println("  The time limit: EK9_TEST_TIMEOUT_MS, default 30000 ms")
      stdout.println("  A test that prints nothing where output is expected: an output mismatch")
Other ways to ask this
  • What triggers E81001 MISSING_EXPECTED_OUTPUT?
  • What triggers E81002 ORPHAN_EXPECTED_OUTPUT?
  • What triggers E81003 ARGUMENT_COUNT_MISMATCH?
  • What triggers E81004 UNEXPECTED_ARGUMENTS?
  • What triggers E81005 MISSING_ARGUMENTS?
  • What triggers E81006 TYPE_CONVERSION_ERROR?
  • What triggers E81007 EMPTY_TEST?
  • What triggers E81008 DUPLICATE_CASE_ID?
  • What triggers E82001 TEST_TIMEOUT?
  • How long can a @Test run before it times out?
  • Can @Test programs accept typed parameters?
  • How do I pass arguments to test programs?

Coming from another language?

Java JUnit: @Test annotation, @ParameterizedTest with @ValueSource for typed params. Python pytest: fixtures + parametrize for typed params. Go: testing.T with table-driven tests. Rust: #[test] with proptest for parameterised. EK9: @Test directive on a program that accepts typed parameters; the runtime auto-parses commandline_arg files into the declared types, no manual parsing or annotations needed.

Keywords: E81001, E81002, E81003, E81004, E81005, E81006, E81007, E81008, E82001, EK9_TEST_TIMEOUT_MS, argument, commandline_arg, envvars, expected, expected_case, output, program, stdin, test, timeout, typed parameters