EK9 Standard Types

The Built in Types and Collection Types are designed to be reused, by composition. The standard types on this page are utility API classes.

The Types

These types are most useful for CLI (command line) applications, though you can use them in any application. EK9 is aimed at a range of application sizes, and the CLI, sidelined for years in favour of GUI applications, has come back into favour. The approach follows Unix/Linux/macOS, especially for signal handling.

What's needed

An effective CLI application needs:

Why EK9 does it this way. Some consider this anachronistic (too Unix?), but for people in 'DevOps' roles knocking up a quick script for a one-off task is essential. The AWS CLI, which drives an entire cloud infrastructure, shows what the CLI approach can do, and it is still in use decades after it was invented.

Choose the language with care:

  • Shell first. On Unix/Linux, KSH/BASH and sed, awk, cut get you a long way; use them before writing any EK9 if you know them. At size, shell scripts get complex, and because they are interpreted (like Python) you have to test every path.
  • Past a certain size or complexity you need a programming language. When you adopt EK9 depends on your background (coder first or sys admin first).
  • For very high performance use C, GoLang or Rust; for highly mathematical or scientific work, perhaps Python.

EK9 is designed as an intermediate language: not as fast as C/Golang/Rust, but performant. It is quick to write, reliable and readable; it is strongly typed and compiled, so many errors are found before execution; and it gives you more reuse. Computing resources are cheap compared with people, so software that is good/performant enough, written sooner with fewer defects, is often the better choice. Cloud servers run up for one-off jobs make other languages viable too.

A CLI application almost always reads, writes and processes text files, so you need to locate them by path. Unix/Linux/macOS separates directories with '/' and Windows with '\'; FileSystemPath handles both.

A FileSystemPath is a path; the file need not exist. Useful methods:

Worked Example

Earlier sections describe each type on its own. This section uses one worked example to show the standard types working together. It is the first full application on the site (a simple one, about 400 lines), kept in a single source file because none of it is reused elsewhere.

Main Requirements

Discard blank lines and lines starting with #.

Additional Requirements

On platforms that fully support signals (macOS, Unix and Linux, but not Windows):

Signal support varies significantly between operating systems.

Sample Inputs

There are two inputs, 'Standard Input' and the 'Named File':

Run on these two inputs as they stand, the program prints nothing on 'Standard Output': the invalid line in the named file is reported on 'Standard Error' and the program exits with code 2, as the requirements ask.

Desired Output

With that invalid line removed from the named file, the output is as below (dateCustomer is not required, so the program never reads or checks it):

This is a typical data migration application: extract data, clean it up, validate it and transform it. Any programming language could do it; this is how EK9 does it.

How the solution is structured

The example uses these EK9 constructs:

It does not define any:

Having a rich set of constructs does not mean you have to use them all. As with tools in a tool box or ingredients for cooking, you won't always need everything. The example blends pragmatic Functional and Object Oriented techniques, and uses the standard types listed at the top of this page. Each section explains its design decisions.

The application could have been designed in many ways: a single monolithic program, only functions, or only classes. It could also have used an application with the program, one component holding the OptionsHandler and command line flag and another holding the NamedFileProcessor, injected into the functions. That reduces parameter passing and uses IOC (inversion of control); Components And Applications solves the same problem that way.

Why EK9 does it this way. Architectural solutions vary by business area and by the experience of the team: a finance team, a telecommunications team and a military software team each produce a different solution, as do teams with mathematics, science or engineering backgrounds. EK9 aims to provide the constructs that let any of these solution architectures be used. This example is not the only one right way; no such way exists.

Start at the entry point, the main program:

The Main Program
defines program

  DataCorrelation()
    -> argv as List of String

    stdin <- Stdin()
    stdout <- Stdout()
    stderr <- Stderr()

    optionsHandler <- OptionsHandler()
    options <- optionsHandler.processCommandLine(argv)

    verboseMode <- options contains "-v"
    debugLevel <- optionsHandler.processDebugLevel(verboseMode, options.getOrDefault("-d", ""))

    //Use NamedFileProcessor for clean separation of concerns
    filename <- Optional(options.getOrDefault("-f", ""))
    namedFileContents <- NamedFileProcessor(verboseMode, debugLevel).processNamedFile(filename)

    if verboseMode
      stderr.println(`Process id is [${OS().pid()}]`)
    if verboseMode or debugLevel > 0
      stderr.println(`Loaded ${$ length namedFileContents} records from named file`)

    setupSignalHandling(verboseMode, debugLevel)

    validLine <- createLineValidator(stderr, debugLevel)
    toEntry <- createStdinLineHandler(stderr, debugLevel)
    byMerging <- createCustomerRecordMerger(stderr, debugLevel, namedFileContents)

    if debugLevel > 1
      stderr.println("Ready to start processing Standard Input")

    outputHeader(verboseMode, debugLevel, stdout)
    cat stdin | filter by validLine | map toEntry | filter by validEntry | map byMerging > stdout

    if debugLevel > 1
      stderr.println("Standard Input processing complete")

Full source: standardTypes_data_correlation.ek9 - Download standardTypes_data_correlation.ek9

Incoming Parameters

A program that declares a single parameter of type List of String receives the command line arguments in it; here it is named argv (C tradition). Programs shows how EK9 maps command line arguments directly to typed parameters. This example wants full control of a range of flexible, optional arguments, and shows the GetOpt class.

Stdin

To read 'Standard Input' (the content 'piped' into the application), declare a variable of type Stdin. Use it like an iterator with hasNext() and next(), or as the source of a stream pipeline, as at the end of the program above.

Stdout

For 'Standard Out', declare a variable of type Stdout and send String values with print and println. The program above uses it as the sink at the end of a stream pipeline.

Stderr

Stderr works like Stdout but writes to the error channel. The example uses stderr widely for verbose and debug information.

The Developed Classes

OptionsHandler, a class shown later, handles the command line options and produces a Dictionary of them.

NamedFileProcessor, also a class, processes the named file and produces a Dictionary of CustomerRecords (a record shown later).

OS - Operating System

The OS class and its method pid() give the process id to print to 'Standard Error'.

Standard Functions

setupSignalHandling is a standard function that registers the signal handlers. outputHeader outputs the header (commented csv). Both are shown later.

Higher Order Functions

Higher order functions create the functions validLine, toEntry, byMerging used in the stream pipeline that processes stdin; see below.

Main Processing Pipeline

This pipeline drives the program. You could write it with for/while loops, but the stream pipeline construct joins processing steps together in a readable, reusable and testable way.

Why EK9 does it this way. Some find pipelines too 'functional'. In my experience they bring clarity to stages of processing that get lost in nested loops: the decision logic nested in a loop can be pulled out, used separately and tested in isolation. It is also the same pattern of development everywhere you use it. It looks strange at first (unless you come from a Unix shell, where it looks familiar), but you soon start thinking in pipelines. 

cat stdin | filter by validLine | map toEntry | filter by validEntry | map byMerging > stdout

Full source: standardTypes_data_correlation.ek9 - Download standardTypes_data_correlation.ek9

Cat

That single line holds every stage of processing the incoming data, in an abstract but readable way. cat catenates a collection or iterator; here stdin is a collection of Strings.

Filter

The first stage filters out blank lines and lines starting with #. The higher order function createLineValidator (shown later) creates the validator function; for now, take it that only valid lines get through.

Map

The next stage maps each String to a DictEntry of (String, CustomerRecord): a String key and a CustomerRecord value. The higher order function createStdinLineHandler (shown later) returns a function that extends the open function lineToCustomerRecord, whose signature accepts a String and returns that DictEntry.

Filter

The data on a line may be wrong. The requirement is to carry on but not output invalid data, so this stage filters out the invalid DictEntry of (String, CustomerRecord)s.

Map

The penultimate stage merges each DictEntry of (String, CustomerRecord) from stdin with the correlated entry from the named file, and produces the output String. The higher order function createCustomerRecordMerger makes this mapping function.

Collect

The final stage sends each String to stdout. An invalid (un set) String is ignored by Stdout and not output.

Discussion

Why this approach?

The first version was only a program. Bigger blocks of code were pulled out into functions, then those were broken into smaller functions: all functions and no classes. Breaking them down further made it simpler to pull some into classes as methods, many of them private. You could go further and encapsulate some classes/data in components.

The next step would be to pull every output String into interpolated Strings and then refactor them into Text blocks, so the application can be ported to other spoken languages.

Development Process

Start writing code, refactor when you see complexity or reuse, and use a construct when it feels right (if it never does, don't use it). The example's test cases (driver code) are not shown; they would make it too large. Design note: start with code, then refactor sets out this way of working, and why EK9 has evolved to support it.

The Functions
defines function

  lineValidator() as pure abstract
    -> line as String
    <- rtn as Boolean?

  lineToCustomerRecord() as pure open
    -> line as String
    <- rtn as DictEntry of (String, CustomerRecord)?
    require line?
    rtn: DictEntry() of (String, CustomerRecord)

  customerRecordMerger() as abstract
    -> entry as DictEntry of (String, CustomerRecord)
    <- rtn as String?

  createLineValidator()
    ->
      stderr as Stderr
      debugLevel as Integer
    <-
      validator as lineValidator?

    validator: (stderr, debugLevel) is lineValidator as pure function
      rtn :=? line is not empty and #<line != '#'
      if debugLevel > 1 and not rtn
        stderr.println(`Discarding [${line}]`)

  createStdinLineHandler()
    ->
      stderr as Stderr
      debugLevel as Integer
    <-
      processor as lineToCustomerRecord?

    processor: (stderr, debugLevel) is lineToCustomerRecord as pure function
      rtn: DictEntry() of (String, CustomerRecord)
      traceLevel <- 3
      expectedFieldCount <- 4

      splitLine <- line.split(/,/)
      if debugLevel == traceLevel
        stderr.println(`About to split stdin line [${line}]`)

      if length of splitLine == expectedFieldCount
        id <- splitLine.getOrDefault(0, "").trim()
        firstname <- splitLine.getOrDefault(1, "").trim()
        lastname <- splitLine.getOrDefault(2, "").trim()
        dobStr <- splitLine.getOrDefault(3, "").trim()
        if id is not empty and firstname is not empty and lastname is not empty and dobStr is not empty
          dob <- $getAsDateTime(id, false, dobStr)
          if dob?
            entryValue <- CustomerRecord(id: id, firstName: firstname, lastName: lastname, email: String(), dob: dob, lastPurchase: String())
            if debugLevel == traceLevel
              stderr.println(`Line [${line}] processed`)
            rtn :=? DictEntry(id, entryValue)

      if not rtn?
        stderr.println(`Invalid line [${line}]`)

  getAsDateTime() as pure
    ->
      id as String
      throwException as Boolean
      aDateInput as String //expecting YYYYMMDD
    <-
      rtn as DateTime: DateTime()

    expectedDateParts <- 3
    group <- aDateInput.group(/(\d{4})(\d{2})(\d{2})/)
    if length group == expectedDateParts
      dateStr <- cat group | join with dashSeparated | collect as String
      rtn := DateTime(Date(dateStr))
    else if throwException
      throw Exception(`Id [${id}] Invalid date [${aDateInput}]`, 2)
    else
      Stderr().println(`Id [${id}] Invalid date [${aDateInput}]`)

  validEntry() as pure
    -> entry as DictEntry of (String, CustomerRecord)
    <- rtn as Boolean: entry?

  dashSeparated() as pure
    ->
      firstPart String
      secondPart String
    <-
      rtn as String: firstPart? and secondPart? <- `${firstPart}-${secondPart}` : String()

  outputFormatSeparated() as pure
    ->
      firstPart String
      secondPart String
    <-
      rtn as String: firstPart? and secondPart? <- `${firstPart}, ${secondPart}` : String()

  setupSignalHandling()
    ->
      verboseMode as Boolean
      debugLevel as Integer

    terminationHandler <- (verboseMode, debugLevel) is SignalHandler as function
      result: 1 //The error code to terminate with
      if verboseMode or debugLevel > 0
        Stderr().println(`Handled Terminal Signal [${signal}]`)

    terminations <- Signals().register(["HUP", "ABRT"], terminationHandler)
    if terminations not contains "HUP"
      Stderr().println("HUP Signal not supported")
    if terminations not contains "ABRT"
      Stderr().println("ABRT Signal not supported")

    debugHandler <- (verboseMode, debugLevel) is SignalHandler as function
      maxDebugLevel <- 3
      result: Integer() //Note the Integer is not set so process will not terminate
      if signal == "USR1" and debugLevel < maxDebugLevel
        debugLevel++
      else if signal == "USR2" and debugLevel > 0
        debugLevel--
      if verboseMode or debugLevel > 0
        Stderr().println(`Handled Info Signal [${signal}] Debug Level now [${debugLevel}]`)

    debugs <- Signals().register(["USR1", "USR2"], debugHandler)
    if debugs not contains "USR1"
      Stderr().println("USR1 Signal not supported")
    if debugs not contains "USR2"
      Stderr().println("USR2 Signal not supported")

  outputHeader()
    ->
      verboseMode as Boolean
      debugLevel as Integer
      stdout as Stdout

    detailedDebugLevel <- 2
    enGB <- Locale("en_GB")
    if verboseMode or debugLevel > detailedDebugLevel
      Stderr().println(`About to create output header with locale ${enGB}`)

    envVars <- EnvVars()
    user <- envVars contains "USER" <- envVars.get("USER") else envVars.get("USERNAME")
    dateTime <- enGB.mediumFormat(SystemClock().dateTime())

    stdout.println(`#Created by ${user} on ${dateTime}`)
    stdout.println("#id, firstName, lastName, emailAddress, dateOfBirth, dateLastPurchase")

  createCustomerRecordMerger()
    ->
      stderr as Stderr
      debugLevel as Integer
      namedFileContents as Dict of (String, CustomerRecord)
    <-
      merge as customerRecordMerger?

    merge: (stderr, debugLevel, namedFileContents) is customerRecordMerger
      traceLevel <- 3
      rtn: String() //Output will be unset by default and there for not output
      id <- entry.key()
      if debugLevel == traceLevel
        stderr.println(`Merging id [${id}]`)
      if namedFileContents contains id
        stdinEntry <- entry.value()
        namedFileEntry <- namedFileContents.getOrDefault(id, CustomerRecord())
        stdinEntry :~: namedFileEntry
        rtn := $stdinEntry
      else
        stderr.println(`Not merging [${id}] as named file does not contain id`)

Full source: standardTypes_data_correlation.ek9 - Download standardTypes_data_correlation.ek9

Abstract Functions

The first three are function signatures for dynamic functions to extend: lineValidator and customerRecordMerger are abstract, and lineToCustomerRecord is open, with a body that returns an un set DictEntry. createLineValidator, the first higher order function, creates and returns a dynamic function validator that is a lineValidator; it checks whether the line is empty or starts with #.

createStdinLineHandler is the main higher order function. It creates the function that parses each line from stdin, validates the content and builds a CustomerRecord inside a DictEntry (which can then be processed further). If processing fails, it returns an un set DictEntry and outputs a note of the line to stderr.

Random

Random is a trait that models a source of random values. It follows the same injectable pattern as Clock/SystemClock: code depends on the Random trait, and you inject the non-deterministic SystemRandom in production or the reproducible SeededRandom under test. It provides:

Because the behaviour is behind a trait, tests can inject a SeededRandom (or a mock) to make anything that uses randomness deterministic, the same idea that makes Clock injectable.

SystemRandom

SystemRandom is the production implementation of Random, backed by the platform thread-local generator (the same source as OS().random()). It is non-deterministic: each run produces a different sequence. Create it with SystemRandom(). Its next(lower, upper) bounds are inclusive and match OS().random(lower, upper) exactly.

SeededRandom

SeededRandom is the reproducible implementation of Random. Given a seed, it produces a pinned sequence that is the same across runs and JVMs: the determinism seam for tests and replayable simulations. Create it with SeededRandom(seed); an un set seed gives an un set SeededRandom whose methods all return un set values. 

#!ek9
defines module site.standardTypes.seededRandom

  defines program

    SeededRandomExample()
      stdout <- Stdout()

      //Same seed -> same sequence every run, so output is reproducible
      rnd <- SeededRandom(42)

      dice1 <- rnd.next(1, 6)
      dice2 <- rnd.next(1, 6)
      stdout.println(`Rolled ${dice1} and ${dice2}`)

Download standardTypes_seeded_random.ek9

Every run prints: 

Rolled 2 and 5
EnvVars

The function outputHeader outputs the header comment, reading the user name with EnvVars. Environment variables differ by platform and can be altered before your program runs (which can be good or bad). EnvVars behaves much like a dictionary: contains, get(), and keys() for the names of all the entries. 

envVars <- EnvVars()
user <- envVars contains "USER" <- envVars.get("USER") else envVars.get("USERNAME")

Full source: standardTypes_data_correlation.ek9 - Download standardTypes_data_correlation.ek9

Most of the other functions are straightforward; setupSignalHandling and createCustomerRecordMerger need more explanation.

setupSignalHandling

This function sets up two SignalHandlers. The first terminates the application on signals HUP/ABRT: it returns an Integer value of 1, so the whole program exits with code 1. Signal handling depends on the platform (Windows has limited support).

Signal Handler

The second SignalHandler handles the non-terminating signals USR1/USR2, which increment and decrement the debugLevel.

How can altering debugLevel here affect the whole application? If debugLevel were a primitive, it could not. But EK9 has only Objects: the program passes debugLevel to 'setupSignalHandling', and the dynamic function below captures it: 

debugHandler <- (verboseMode, debugLevel) is SignalHandler as function
  maxDebugLevel <- 3
  result: Integer() //Note the Integer is not set so process will not terminate
  if signal == "USR1" and debugLevel < maxDebugLevel
    debugLevel++
  else if signal == "USR2" and debugLevel > 0
    debugLevel--
  if verboseMode or debugLevel > 0
    Stderr().println(`Handled Info Signal [${signal}] Debug Level now [${debugLevel}]`)

Full source: standardTypes_data_correlation.ek9 - Download standardTypes_data_correlation.ek9

Because debugLevel is an Object it is passed by reference everywhere, and because the debugHandler dynamic function has 'captured' it, every reference points to the same object. So debugLevel++ and debugLevel-- change the one value everything sees.

This is very unlike an int primitive, and it allows data to be modified in several functions (not very pure).

createCustomerRecordMerger

The second significant higher order function. It creates the merge function, which looks up and correlates ids and merges the two partial CustomerRecords, one from 'Standard Input' and one from the 'Named File'. The merge uses the operator :~: and the conversion to a String uses the $ operator, both on the CustomerRecord record (shown later). 

stdinEntry :~: namedFileEntry
rtn := $stdinEntry

Full source: standardTypes_data_correlation.ek9 - Download standardTypes_data_correlation.ek9

The namedFileContents object (the dictionary of the 'named file' contents) is also 'captured', from the main program down through the higher order function createCustomerRecordMerger into the dynamic function merge as customerRecordMerger.

If several applications deal with comma or colon separated data, pull the abstract and concrete functions into a separate utility module so they can be reused. The same goes for signal handling: all your applications could use the same mechanism and reuse setupSignalHandling. A range of small, reusable functions builds an internal library of small, reliable software 'chunks'.

The Classes

There are two classes. OptionsHandler, below, processes the command line options the user entered when running the application. 

defines class

  OptionsHandler

    processCommandLine() as pure
      -> arguments as List of String
      <- options as Dict of (String, String)?

      getopts <- setupGetOpt(":")
      options :=? getopts.options(arguments)

    private setupGetOpt() as pure
      -> rqParam as String
      <- getopts as GetOpt?

      supportedOptions <- setupSupportedOptions(rqParam)
      usage <- setupUsage()
      getopts :=? GetOpt(rqParam, supportedOptions, usage)

    private setupSupportedOptions() as pure
      -> rqParam as String
      <- rtn as Dict of (String, String)?

      rtn :=? {
        "-v": String(),
        "-f": rqParam,
        "-d": rqParam
        }

    private setupUsage() as pure
      <- rtn as String?

      desc <- "Invalid option, only those list below are supported:"
      verbose <- "-v, verbose"
      filename <- "-f filename, use file of filename (mandatory option)"
      debug <- "-d level, use of debugging"
      rtn :=? `${desc}\n${verbose}\n${filename}\n${debug}`

    processDebugLevel()
      ->
        verboseMode as Boolean
        level as String
      <-
        rtn as Integer: 0 //default to zero

      maxDebugLevel <- 3
      debugLevel <- Integer(level)

      rtn := (debugLevel in 0 ... maxDebugLevel) <- debugLevel else 0

      if verboseMode and (debugLevel not in 0 ... maxDebugLevel)
        Stderr().println(`Debug level ${debugLevel} not supported - reverting to debug level '0'`)

Full source: standardTypes_data_correlation.ek9 - Download standardTypes_data_correlation.ek9

GetOpt

The processCommandLine method on OptionsHandler uses private methods to set up and configure the GetOpt class. Each private method does one task: configuring the supported options, or configuring the help text.

The main program calls the public method processDebugLevel to get the debug level the user entered on the command line, if any. Integer(level) may be unset, less than zero or greater than 3; the ternary operator keeps the result between 0 and 3.

TextFile

The second class, NamedFileProcessor, has a single responsibility: loading the 'named file' into a Dict of (String, CustomerRecord). It is much stricter about errors. 

NamedFileProcessor
  verboseMode as Boolean: false
  debugLevel as Integer: Integer()

  NamedFileProcessor()
    ->
      verboseMode as Boolean
      debugLevel as Integer
    this.verboseMode = verboseMode
    this.debugLevel = debugLevel

  processNamedFile()
    -> filename as Optional of String
    <- rtn <- Dict() of (String, CustomerRecord)

    stderr <- Stderr()
    if filename?
      namedFile <- getNamedFile(filename.get())
      validLine <- createLineValidator(stderr, debugLevel)
      toEntry <- createNamedFileLineHandler(stderr, debugLevel)

      if verboseMode or debugLevel > 0
        stderr.println(`About to start processing [${namedFile}]`)

      cat namedFile.input() | filter by validLine | map toEntry > rtn

      if debugLevel > 1
        stderr.println(`Processing [${namedFile}] complete`)
    else
      throw Exception("Filename of 'named file' is required", 2)

  private getNamedFile()
    -> filename as String
    <- rtn as TextFile: TextFile()

    filePath <- FileSystemPath(filename)
    rtn := filePath.isAbsolute() <- TextFile(filePath) else TextFile(FileSystem().cwd() + filePath)
    if not rtn?
      throw Exception(`${filePath} is not an existing file`, 2)
    if not rtn.isReadable()
      throw Exception(`${rtn} is not readable`, 2)

  private createNamedFileLineHandler() as pure
    ->
      stderr as Stderr
      debugLevel as Integer
    <-
      processor as lineToCustomerRecord?

    processor :=? (stderr, debugLevel) is lineToCustomerRecord as pure function
      traceLevel <- 3
      expectedFieldCount <- 4
      splitLine <- line.split(/:/)
      if debugLevel == traceLevel
        stderr.println(`About to split line [${line}]`)

      if length of splitLine != expectedFieldCount
        throw Exception(`Invalid line [${line}]`, 2)
      id <- splitLine.getOrDefault(0, "").trim()
      if id is empty
        throw Exception(`Invalid line [${line}] empty ID`, 2)

      emailAddress <- splitLine.getOrDefault(1, "").trim()
      if emailAddress not matches /[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$/
        throw Exception(`Id [${id}] Invalid email address [${emailAddress}]`, 2)

      dateLastPurchase <- $getAsDateTime(id, true, splitLine.getOrDefault(3, "").trim())

      entryValue <- CustomerRecord(id: id, firstName: String(), lastName: String(), email: emailAddress, dob: String(), lastPurchase: dateLastPurchase)
      if debugLevel == traceLevel
        stderr.println(`Named file line [${line}] processed`)

      rtn :=? DictEntry(id, entryValue)

  default operator ?

Full source: standardTypes_data_correlation.ek9 - Download standardTypes_data_correlation.ek9

The public method processNamedFile() first checks that a file name was provided, and throws an Exception with exit code 2 if not. It then gets a TextFile with the method getNamedFile(), creates a couple of pipeline functions and starts the main pipeline.

private getNamedFile()

This private method creates a FileSystemPath from the String filename. A ternary operation decides whether the path is relative or absolute. It then checks that the file exists and is readable, and returns the TextFile.

Current Working Directory

A relative path needs the user's current working directory, which the FileSystem class provides.

createLineValidator

This higher order function is covered above.

createNamedFileLineHandler

This higher order method creates the function that splits each line on the : separator, checks the data and throws an Exception if any of it is invalid. It uses the standard function getAsDateTime to process the date.

Processing Pipeline

This pipeline has almost the same structure as the main pipeline in the program, and uses the same higher order function createLineValidator to validate lines. The difference is the final stage: it collects each DictEntry into the return variable rtn. 

cat namedFile.input() | filter by validLine | map toEntry > rtn

Full source: standardTypes_data_correlation.ek9 - Download standardTypes_data_correlation.ek9

Using functions rather than only classes gives more reuse, as createLineValidator and the abstract function signatures show, and less need for deep class hierarchies and inheritance. The smaller and tighter a 'chunk' of code, and the fewer its side effects, the more reusable and reliable it is.

The same pattern of processing applies again and again, and a pipeline is easier to read than nested loops. The final part of the program is the data item at the heart of the merging and output: the record.

The Record

CustomerRecord is mainly a data object, with a few key operators for processing its data. 

defines record

  CustomerRecord
    id <- String()
    firstName <- String()
    lastName <- String()
    email <- String()
    dob <- String()
    lastPurchase <- String()

    default CustomerRecord() as pure

    CustomerRecord() as pure
      ->
        id as String
        firstName as String
        lastName as String
        email as String
        dob as String
        lastPurchase as String
      this.id :=? String(id)
      this.firstName :=? String(firstName)
      this.lastName :=? String(lastName)
      this.email :=? String(email)
      this.dob :=? String(dob)
      this.lastPurchase :=? String(lastPurchase)

    override operator ? as pure
      <- rtn as Boolean: id?

    operator :~:
      -> other as CustomerRecord

      id :=? String(other.id)
      firstName :=? String(other.firstName)
      lastName :=? String(other.lastName)
      email :=? String(other.email)
      dob :=? String(other.dob)
      lastPurchase :=? String(other.lastPurchase)

    operator $ as pure
      <- rtn as String: cat [id, firstName, lastName, email, dob, lastPurchase]
        | join with outputFormatSeparated
        | collect as String

Full source: standardTypes_data_correlation.ek9 - Download standardTypes_data_correlation.ek9

It holds the output data that must be converted to a single comma separated String.

operator ?

The is set operator checks whether the record is valid, for filtering and processing.

operator :~:

The merge operator merges the stdin partial record and the named file partial record into one fully populated record. It uses the assign-if-unset operator (:=?), which assigns a property/field only if it is currently un set, coalescing the field with the argument's field.

operator $

The string conversion operator puts each property/field of the record in a List of Strings, and another pipeline joins them with a 'comma' and collects them into a rtn String: 

operator $ as pure
  <- rtn as String: cat [id, firstName, lastName, email, dob, lastPurchase]
    | join with outputFormatSeparated
    | collect as String

Full source: standardTypes_data_correlation.ek9 - Download standardTypes_data_correlation.ek9

Joining the fields with the + operator would be a chain of eleven parts, which EK9 does not allow: more than two parts joined with + is E11068.

The alternative is String interpolation: 

rtn := `${id}, ${firstName}, ${lastName}, ${email}, ${dob}, ${lastPurchase}`

Full source: standardTypes_record_string.ek9 - Download standardTypes_record_string.ek9

That is slightly shorter but more intricate. To add fields or change their order, the List approach is easier and more reliable. And if the output format changes, it is encapsulated in a single function, outputFormatSeparated.

Summary

The functionality could have been written as a single program with no other constructs; the first version was, and it was much shorter. But decision logic was mixed in with format logic, and there was repetition and duplication.

Why EK9 does it this way. Whether the example is over engineered depends on your background and experience; those from a functional background would say it is under engineered, and should use much more pure immutability to be more robust. You have to find the balance between engineering and bloat/complexity or plain 'dirty' code.

Justification of Approach

These are all encapsulated:

And there is room to improve:

So when there is a defect, or a need to improve the software, it is obvious where to look. Breaking a monolithic program into functions and classes also makes it possible to make them pure and to move them into separate modules for reuse.

What can be reused has been reused:

If new fields were required, or another date format had to be accepted, it would be obvious where to look.

Common design patterns are used and reused:

You may or may not like stream pipelines, but to alter the processing you look at the pipeline, not deep inside nested loops.

Sample of the Looping approach

As a contrast, the 'named file processing' could have been written directly in the program, as below. Some developers find this more natural (especially from a C or Python background). I find all that low level utility code distracting high up in the main part of the program.

Fails with E11021: cognitive complexity exceeds limit - extract functions

defines program

  DataCorrelation()
    -> argv as List of String

    stdin <- Stdin()
    stdout <- Stdout()
    stderr <- Stderr()

    optionsHandler <- OptionsHandler()
    options <- optionsHandler.processCommandLine(argv)

    verboseMode <- options contains "-v"
    debugLevel <- optionsHandler.processDebugLevel(verboseMode, options.getOrDefault("-d", ""))

    debugTraceLevel <- 3
    expectedFieldCount <- 4

    //All the named file processing inlined here, rather than in NamedFileProcessor
    filename <- options.getOrDefault("-f", "")
    if not filename?
      throw Exception("Filename of 'named file' is required", 2)
    filePath <- FileSystemPath(filename)
    namedFile <- filePath.isAbsolute() <- TextFile(filePath) else TextFile(FileSystem().cwd() + filePath)
    if not namedFile?
      throw Exception(`${filePath} is not an existing file`, 2)
    if not namedFile.isReadable()
      throw Exception(`${namedFile} is not readable`, 2)

    if verboseMode or debugLevel > 0
      stderr.println(`About to start processing [${namedFile}]`)

    namedFileContents <- Dict() of (String, CustomerRecord)
    try
      -> input <- namedFile.input()
      while input?
        line <- input.next()
        if line is not empty and #<line != '#'
          splitLine <- line.split(/:/)
          if debugLevel == debugTraceLevel
            stderr.println(`About to split line [${line}]`)

          if length of splitLine != expectedFieldCount
            throw Exception(`Invalid line [${line}]`, 2)
          id <- splitLine.getOrDefault(0, "").trim()
          if id is empty
            throw Exception(`Invalid line [${line}] empty ID`, 2)

          emailAddress <- splitLine.getOrDefault(1, "").trim()
          if emailAddress not matches /[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+$/
            throw Exception(`Id [${id}] Invalid email address [${emailAddress}]`, 2)

          dateLastPurchase <- $getAsDateTime(id, true, splitLine.getOrDefault(3, "").trim())

          entryValue <- CustomerRecord(id: id, firstName: String(), lastName: String(), email: emailAddress, dob: String(), lastPurchase: dateLastPurchase)
          if debugLevel == debugTraceLevel
            stderr.println(`Named file line [${line}] processed`)

          namedFileContents += DictEntry(id, entryValue)

        else if debugLevel > 1
          stderr.println(`Discarding [${line}]`)

    if verboseMode
      stderr.println(`Process id is [${OS().pid()}]`)
    if verboseMode or debugLevel > 0
      stderr.println(`Loaded ${$ length namedFileContents} records from named file`)

    setupSignalHandling(verboseMode, debugLevel)

    validLine <- createLineValidator(stderr, debugLevel)
    toEntry <- createStdinLineHandler(stderr, debugLevel)
    byMerging <- createCustomerRecordMerger(stderr, debugLevel, namedFileContents)

    if debugLevel > 1
      stderr.println("Ready to start processing Standard Input")

    outputHeader(verboseMode, debugLevel, stdout)
    cat stdin | filter by validLine | map toEntry | filter by validEntry | map byMerging > stdout

    if debugLevel > 1
      stderr.println("Standard Input processing complete")

Full source: standardTypes_data_correlation_inlined.ek9 - Download standardTypes_data_correlation_inlined.ek9

EK9 does not leave this as a matter of taste: the inlined program has a cognitive complexity of 37, over the limit of 35, so it does not compile (E11021). The structured version above does.

The inlined code also needs comments to explain it, whereas breaking it into classes and functions gave each piece a meaningful name.

It does show the alternative loop and try with resource approach to resources such as a TextFile: 

try
  -> input <- namedFile.input()
  while input?
    line <- input.next()

Full source: standardTypes_data_correlation_inlined.ek9 - Download standardTypes_data_correlation_inlined.ek9

Because 'namedFile.input()' is created in the try's incoming parameter, the try block closes the input when processing finishes (much as Java does). The input is then used like an iterator to get each line. The concept will be familiar, though the syntax differs.

Having written the same functionality both ways, you can see how to convert 'procedural code' into a 'functional pipeline'. Take the check on the line content: 

if line is not empty and #<line != '#'

Full source: standardTypes_data_correlation_inlined.ek9 - Download standardTypes_data_correlation_inlined.ek9

With no meaningful 'else' (other than the output to stderr), it can become a filter function. The next part of the code takes the incoming String, processes and validates its parts and makes a partial DictEntry of (String, CustomerRecord): it maps one type of content to another, so it is a map function.

One more class, not used in the example, is useful in a range of applications.

Mutex Lock and Key

Use a MutexLock only out of necessity: when the application runs several concurrent Threads that must share read/write data. Typically that is with async in pipelines, TCP, UDP or other HTTP server type constructs. Here be Dragons.

This example shows the syntax and the design pattern for MutexLocks: 

#!ek9
defines module site.standardTypes.mutexLock

  defines class

    <?-
      MutexLock lives as a field on this class — the canonical field-rooted-home
      pattern that EK9 requires for compile-time lock identity tracking. Methods
      on the class expose nameHolder operations; the lock itself is never handed
      out as a local or a return value.
    -?>
    ProtectedName
      lockedName as MutexLock of String: MutexLock("Steve")

      <?-
        Read the nameHolder value by copying it out via :=:
      -?>
      read()
        <- rtn as String?

        rtn: String()

        accessKey <- (rtn) is MutexKey of String as function
          rtn :=: lockedItem

        require lockedName.enter(accessKey)

      <?-
        Update the nameHolder value by copying the new value in via :=:
        Note: :=: (copy), not := (reassignment) — reassignment would
        leave an outside reference to the original value, bypassing the lock.
      -?>
      update()
        -> newName as String

        accessKey <- (newName) is MutexKey of String as function
          lockedItem :=: newName

        require lockedName.enter(accessKey)

      <?-
        Non-blocking update — returns false if the lock was held by
        another thread; the caller can decide whether to retry.
      -?>
      tryUpdate()
        -> newName as String
        <- updated as Boolean: Boolean()

        accessKey <- (newName) is MutexKey of String as function
          lockedItem :=: newName

        updated: lockedName.tryEnter(accessKey)

      default operator ?

  defines program

    LockExample()
      stdout <- Stdout()

      //Construct the nameHolder wrapper — the MutexLock is created
      //by the class's field initialiser, not here in user code.
      nameHolder <- ProtectedName()

      stdout.println(`Initial: [${nameHolder.read()}]`)

      //Blocking update via the class's API.
      nameHolder.update("Stephen")
      stdout.println(`After blocking update: [${nameHolder.read()}]`)

      //Non-blocking update — the Boolean tells us whether the lock was free.
      success <- nameHolder.tryUpdate("Stevie")
      stdout.println(`Non-blocking update succeeded? ${success}: [${nameHolder.read()}]`)

Download standardTypes_mutex_lock.ek9

The MutexLock is a property of the 'ProtectedName' class, created by its initialiser; it is never created in a function or handed out as a value. The program uses only the class's methods, and prints: 

Initial: [Steve]
After blocking update: [Stephen]
Non-blocking update succeeded? true: [Stevie]

The code looks straightforward, but multi-threaded access to a single data structure is fraught with difficulties, race conditions and deadlock. Avoid it if at all possible, and look hard for a solution without concurrent access to shared data before going down this path. When you cannot avoid it, this is what the Mutex lock and key are for.

An EK9 MutexLock:

Inside the MutexKey function, do the minimum processing, and avoid out of process or long running operations.

Why EK9 does it this way. Why build such a big object/process around access to data, instead of synchronized or similar? Because multi-threaded access to shared data is a big deal. As with remote calls to other systems over TCP/UDP/HTTP, EK9 does not hide the nasty details; it gets them out in the open. These are not normal method calls: they are costly, risky calls that need 'focus'.

The data must stay in the lock. In complex data structures such as trees of lists, holding a reference to the protected data from elsewhere would let it be altered without the lock. So the compiler rejects a reference assignment of the locked item inside a MutexKey (E08258): you copy, merge or replace (:=:, :~:, :^:) instead. That is why the example gives "Steve" to the lock in its initialiser and copies "Stephen" in and out with :=:.

That rule covers assignment only. The compiler does not stop the locked item being added to, or passed into, another structure: in a MutexKey, rtn += lockedItem on a captured list of lists compiles, and the locked list can then be changed outside the lock. So you still need the discipline to copy data rather than hold references to it.

With more than one MutexLock, long-lived code tends to acquire locks in different orders over time, and that deadlocks. The compiler rejects a nested enter() on a different lock that forms a lock-order cycle (E08252), and two nested locks of the same type whose order it cannot prove (E08255). Even so, you have made your life much harder.

Avoid MutexLocks of data if at all possible; if you must use them, employ pure with rigour. See a modification of a MutexLocked variable as a very rare and very significant event!

Estimate multi-threaded work generously: it takes much longer than you expect, longer still when data is shared, and longer again with several shared data structures, and you should expect live operational issues. However hard you think it will be, doing it correctly with multiple threads is harder.

If you never use this class, consider that a major achievement! You have saved yourself a whole world of hate.

Security Types

The InputSanitizer class sanitizes input. It implements EK9's "Rejection at the Source" security model, which detects and blocks common web attacks including SQL injection, XSS, command injection, path traversal, XXE and SSTI. Input Sanitization has the full documentation.

It is secure by default with no configuration, and supports enterprise logging formats through an environment variable.

Two approaches:

InputSanitizer

The core threat detection class. It logs every detected threat to stderr; only the SILENT log format (for testing) suppresses that. 

userInput <- "Some user supplied text"
sanitizer <- InputSanitizer()

//Sanitize input - returns unset if threat detected
result <- sanitizer.sanitize(userInput)
if result?
  processData(result)

//With context for richer logging
context <- SanitizationContext(
  timestamp: SystemClock().dateTime(),
  service: "UserService",
  operation: "getUser",
  fieldName: "userId",
  fieldSource: "PATH",
  sourceIp: "127.0.0.1",
  traceId: "trace-0001"
  )
richResult <- sanitizer.sanitize(userInput, context)
if richResult?
  processData(richResult)

Full source: standardTypes_security.ek9 - Download standardTypes_security.ek9

SanitizationContext

A built-in type that captures metadata for security event logging; ek9 -h SanitizationContext shows its seven-value constructor and accessors: 

$ ek9 -h SanitizationContext
EK9 Help for 'SanitizationContext'
========================================


--- TYPE API (org.ek9.lang) ---

    SanitizationContext as open
      SanitizationContext() as pure

      SanitizationContext() as pure
        ->
          timestamp as DateTime
          service as String
          operation as String
          fieldName as String
          fieldSource as String
          sourceIp as String
          traceId as String

      fieldName() as pure
        <- rtn as String?

      fieldSource() as pure
        <- rtn as String?

      operation() as pure
        <- rtn as String?

      service() as pure
        <- rtn as String?

      sourceIp() as pure
        <- rtn as String?

      timestamp() as pure
        <- rtn as DateTime?

      traceId() as pure
        <- rtn as String?

      operator #? as pure
        <- rtn as Integer?

      operator #^ as pure
        <- rtn as String?

      operator $ as pure
        <- rtn as String?

      operator :=:
        -> other as SanitizationContext

      operator :^:
        -> other as SanitizationContext

      operator :~:
        -> other as SanitizationContext

      operator <> as pure
        -> other as SanitizationContext
        <- rtn as Boolean?

      operator == as pure
        -> other as Any
        <- rtn as Boolean?

      operator == as pure
        -> other as SanitizationContext
        <- rtn as Boolean?

      override operator ? as pure
        <- rtn as Boolean?
Log Format Configuration

The EK9_SANITIZER_LOG_FORMAT environment variable sets the log format: 

export EK9_SANITIZER_LOG_FORMAT=JSON    # Simple JSON (default)
export EK9_SANITIZER_LOG_FORMAT=ECS     # Elastic Common Schema
export EK9_SANITIZER_LOG_FORMAT=CEF     # Common Event Format
export EK9_SANITIZER_LOG_FORMAT=SIMPLE  # Simple bracket format
export EK9_SANITIZER_LOG_FORMAT=SILENT  # Suppress output (testing only)

Input Sanitization describes the security model in detail.

Sensitive

Sensitive is a black-box wrapper for application credentials (API keys, passwords, tokens). The type system protects against accidental secret exposure. InputSanitizer and sanitized protect what comes in; Sensitive protects secrets going out.

Key constraints:

envVars <- EnvVars()

//Load secret from environment variable
if dbPassword <- envVars.sensitiveGet("DB_PASSWORD")
  //dbPassword is set, pass to database driver
  database.connect(host: "localhost", port: 5432, user: "app", password: dbPassword)

  //This prints "***REDACTED***" - never the real password
  stdout.println($dbPassword)
else
  stdout.println("DB_PASSWORD is not set, so there is nothing to connect with")

Full source: standardTypes_security.ek9 - Download standardTypes_security.ek9

Run with no DB_PASSWORD in its environment (as the site's examples are), it prints: 

DB_PASSWORD is not set, so there is nothing to connect with

Operators: ? (isSet), == / <> (constant-time comparison), $ and #^ (redacted string), $$ (redacted JSON), :=: (copy). No <=>, #?, + or length.

Privileged

Privileged is a marker trait declaring that a class (or another trait) is trusted to handle raw secret values. Only a method of a Privileged class or trait can call Sensitive.reveal(); anywhere else, reveal() is compile error E11090. Privilege is inherited: a class that extends a Privileged class, and a class or trait with Privileged anywhere in its trait chain, may also call reveal(). 

DatabaseDriver with trait of Privileged

  connect()
    ->
      host as String
      port as Integer
      user as String
      password as Sensitive
    //reveal() is legal here - DatabaseDriver has the Privileged trait
    rawPassword <- password.reveal()
    internalConnect(host: host, port: port, user: user, secret: rawPassword)

  // ... internals

  default operator ?

Full source: standardTypes_security.ek9 - Download standardTypes_security.ek9

Audit trail: search a codebase for trait of Privileged, then follow each of those types' subclasses and sub-traits, to find every class and trait that can access raw secrets.

Conclusion

This longer example has real functionality, and shows different approaches to development, in general and with EK9, as concrete snips of code you can use. It shows techniques you may not have used before, and that EK9 works for CLI development.

Next Steps

For networking with EK9, read network types next.