How do I benchmark two approaches in EK9?

← Profiling · Ref: Q631

EK9 provides two complementary approaches to benchmarking: the built-in profiler for real-world measurement, and Millisecond/SystemClock for manual timing.

APPROACH 1: PROFILER COMPARISON

Write two test functions that exercise the two approaches, then profile:

  ek9 -t2p myproject.ek9

The JSON output shows self-time and call count for each function. Compare the self-time values directly.

APPROACH 2: MANUAL TIMING

Use SystemClock and Millisecond for explicit before/after measurement:

  clock <- SystemClock()
  before <- clock.millisecond()
  // ... code to measure ...
  after <- clock.millisecond()
  elapsed <- after - before

This gives wall-clock elapsed time as a Millisecond value.

BEST PRACTICES

1. Warm up: run the code several times before measuring. JVM needs time to JIT-compile hot paths.
2. Repeat: measure multiple iterations and compute the average. Single measurements are noisy.
3. Isolate: benchmark one thing at a time. Do not mix the two approaches being compared in the same run.
4. Use profiler for real code: manual timing is good for quick checks but the profiler provides p50/p95/p99 distributions.
5. Avoid optimisation: profiling forces -O0 automatically. For manual timing, use ek9 -cd (debug mode, no optimisation) to prevent dead code elimination.

WHEN TO USE EACH

- Profiler (-tp/-t2p): for production-representative measurement with statistical detail.
- Manual timing: for quick A/B comparison during development.
- Both together: profile to find hot methods, then manually time the specific methods you want to optimise.

See Q627 for profiling programs. See Q628 for profiling tests. See Q630 for identifying hot methods. See Q544 for elapsed time and benchmarking with Millisecond. See Q41 for the Millisecond type.

Example

defines module qa.profilingdeep.benchmark

  defines function

    <?-
      Two approaches to summing a range, to demonstrate benchmarking.
      Approach 1: iterative loop.
      Approach 2: mathematical formula.
    -?>
    sumIterative() as pure
      -> limit as Integer
      <- total as Integer: 0

      for i in 1 ... limit
        total: total + i

    sumFormula() as pure
      -> limit as Integer
      <- total as Integer: limit * (limit + 1) / 2

  defines program

    BenchmarkDemo()
      stdout <- Stdout()

      testSize <- 1000

      clock <- SystemClock()

      beforeIterative <- clock.millisecond()
      iterativeResult <- sumIterative(testSize)
      afterIterative <- clock.millisecond()
      iterativeElapsed <- afterIterative - beforeIterative

      beforeFormula <- clock.millisecond()
      formulaResult <- sumFormula(testSize)
      afterFormula <- clock.millisecond()
      formulaElapsed <- afterFormula - beforeFormula

      stdout.println(`Iterative: ${iterativeResult} (${iterativeElapsed})`)
      stdout.println(`Formula: ${formulaResult} (${formulaElapsed})`)

Common mistakes

E50001 — Removing the variable declaration means later references to the variable become unresolved, triggering E50001. See ek9 -h E50001 for details.

Incorrect:

sumIterative(testSize)

Correct:

iterativeResult <- sumIterative(testSize)

Coming from another language?

Java: JMH (Java Microbenchmark Harness) with @Benchmark annotations, System.nanoTime() for manual timing. Python: timeit module, time.perf_counter() for manual timing. Rust: criterion crate for statistical benchmarks, std::time::Instant for manual. Go: testing.B benchmark framework, time.Now() for manual. JavaScript: performance.now() in browser, benchmark.js (external). EK9: built-in profiler with p95/p99 statistics via -tp/-t2p, SystemClock + Millisecond for manual timing, no external framework needed.

Keywords: profile, performance, benchmark, flame-graph, compare, elapsed, timing, approach, Millisecond, SystemClock, measure, warm