Getting Started with EK9 · Chapter 22

Programs, Input and Output

Most programs in this book have taken a value or two, worked something out and printed the answer. A real program also deals with the outside world: it reads what you type on the command line, keeps things in files that last beyond a single run, and writes back to your screen. This chapter is about that boundary, between your typed code and a world that is not.

Everything that arrives from outside starts as plain text: a word you typed, a line in a file, an option you may or may not have given. It might be missing, or the wrong shape. The boundary is where you check, and EK9 makes sure you do.

We will build a small notes-keeper: a program called Notes that keeps lines of text in a file. Add a note today, and it is still there tomorrow.

Giving a program its instructions

In Making Decisions and Repeating Work a program took a single typed value:

    Weather()
      -> temperature as Integer

and you ran it with ek9 Weather.ek9 12. That is the simplest way in: one value, named and typed, handed over as the program starts.

Real tools usually want more, and want it to be optional. Should the program report what it is doing? Which file should it use? Is there a note to add this time, or are we just looking? These are options — short flags like -v, -f, -a — and they may appear in any order, or not at all. For that, a program takes the whole command line as a list:

    Notes()
      -> argv as List of String

argv is every word after the program name, in order, as Strings. The name is short for argument vector: "argument" for each value you give the program as it starts, and "vector", an old word for a list. It comes from the C language and Unix and has been passed down ever since.¹ You could call the parameter arguments or commandLine, but argv is the name most programmers recognise, so we use it too.

You could pick through that list yourself, but options are fiddly — some are on/off, some carry a value, they come in any order — so EK9 provides a tool for it.

Reading options with GetOpt

GetOpt takes a short description of the options you accept and returns what was given. Here is how Notes sets it up:

      usage <- "Usage: Notes [-v] [-f file] [-a note]"
      supported <- {
        "-v": String(),
        "-f": ":",
        "-a": ":"
      }
      options <- GetOpt(":", supported, usage).options(argv)

The supported dictionary says which options exist and what each expects. "-v": String() is an on/off flag: it carries no value. "-f": ":" and "-a": ":" are options that take a value; the ":" marks them. The usage line is the message shown if someone passes something wrong. options(argv) reads the actual command line against that description and returns a Dict mapping each option that appeared to its value.

You read the answer with the Dict methods from Collections:

      verbose <- options contains "-v"
      filename <- options.getOrDefault("-f", "notes.txt")

contains asks whether a flag was present — right for an on/off option like -v. getOrDefault reads a value, with a fallback for when the option was left out: no -f, and we use notes.txt.

The choice of fallback matters, and it is the three states at work. When we come to the note itself:

      newNote <- options.getOrDefault("-a", String())
      if newNote?
        notes += newNote
        writeNotes(file, notes)

the fallback is String() — an unset String — not "". An empty string "" is set: it would pass the if newNote? check, and we would add a blank note. String() is unset — it means "no -a was given" — so the check is false and nothing is added. The right fallback is what lets the program tell "an empty note" apart from "no note at all". Choose the wrong one and the bug is silent.

Two ways out: Stdout and Stderr

A program has more than one output channel; use each for its purpose:

      stdout <- Stdout()
      stderr <- Stderr()

Stdout — standard output — is for the program's result: here, the list of notes. Stderr — standard error — is for everything else: progress messages, warnings, and things that went wrong. Keeping them apart is an old and useful convention.² It means someone can capture just the notes — save them to another file, or feed them to another program — without the message "using notes.txt" mixed in. So Notes sends the notes to stdout and its other messages to stderr:

      if verbose
        stderr.println(`using ${filename}`)

(Reading input as you type it, from standard input, is what the game in the next chapter does. Here, everything the program needs arrives on the command line and in the file.)

A path is just a path

Now the file. A FileSystemPath is only a path: a name for a place on disk. It does not open or read anything, or decide what kind of file is there. It is an address.

      path <- FileSystemPath().inCurrentWorkingDirectory(filename)

inCurrentWorkingDirectory attaches our filename to the directory the program is running in, so notes.txt becomes a full path in the right place. (Give it a name that is already a full path and it comes back unchanged.)

Making the file — and being made to check

Creating a file can fail. The disk might be full, or the directory might not be writable. So the method that creates a file does not return a plain yes or no that you could ignore; it returns a Result — the type from Optional and Result that holds either success or an error message — and the compiler will not let you ignore it:

      //Create the file only if it is not there yet; an existing one is left alone.
      //createFileIfAbsent() hands back a Result the compiler will not let us ignore.
      outcome <- path.createFileIfAbsent()
      if outcome.isError()
        stderr.println(`cannot create ${filename}: ${outcome.error()}`)

This is the safe-access rule from Optional and Result. We may read outcome.error() only inside a block that has checked outcome.isError(), and if we never checked at all, the program would not compile. A step that can fail, at the edge of the program, is where that rule matters most.

FileSystemPath has two ways to create a file, and the choice matters. createFileIfAbsent() means "make it, or keep the one that is there": a new file and an existing one are both success, and only a real failure is an error. That is what Notes wants — on the second run, yesterday's notes must be read, not replaced. createFile() is the strict one: it means create a new file, and if one is already there, even an empty one, it reports an error. Use it when an existing file would mean something has gone wrong — when a program that believes it is making a fresh file must not quietly use, or later overwrite, one that was there before.

After the attempt, the program carries on only if the file really exists now:

      if path.exists()
        if verbose
          stderr.println(`using ${filename}`)

If creating it failed, the file is not there, so the work is skipped: no crash, no half-written file, just a message on stderr and a quiet stop.

Reading and writing the notes

Only now, with a path to a file that exists, do we open it, and that is TextFile's job, not the path's. A TextFile is made from a path:

        file <- TextFile(path)

Reading and writing both happen inside a try. Here is the read:

    readNotes()
      -> file as TextFile
      <- rtn as List of String: List() of String
      try
        -> input <- file.input()
        while input.hasNext()
          line <- input.next()
          if line?
            rtn += line

The first line inside the try is -> input <- file.input(). The -> is the same "this comes in" arrow as on every function: the try is given a resource — the reader that file.input() opens — and names it input for the length of the block. hasNext() and next() then walk through it line by line, like the iterators you have used before. Each line is checked with if line? before it joins the list. For an ordinary file every line is set, so every line goes in; but reading text can also give an unset line (one rejected by the reader's sanitisation, its check on incoming text for content that could be harmful), and the check skips it rather than putting a gap in the list.

Why give the reader to the try, rather than open it into an ordinary variable? Because of what happens at the end. An open file uses a resource: the operating system gives your program a file descriptor — a small numbered handle on the open file — and it has only so many to give. Open files and never close them, and the handles run out. That is a resource leak: invisible in a quick test, and fatal in a long-running program that eventually cannot open anything more. In many languages the cure is to call close() on every file when you are done — easy to forget, especially on the path where something has already gone wrong.

EK9 takes the close out of your hands. Because input was given to the try, the try owns it, and closes it when the block ends — whether it ended normally or because something failed partway through. You do not write the close, and you cannot: writing input.close() is a compile error (E81060), because EK9 generates every close itself. There is nothing to forget, so the leak cannot happen.³

Writing is the mirror image, with one more thing to know:

    writeNotes()
      ->
        file as TextFile
        notes as List of String

      try
        -> output <- file.output()
        for note in notes
          output.println(note)

Opening a file for output starts it afresh; it does not add to the end. So "adding a note" is not "write one more line": it is read the whole list, add to it in memory, and write the whole list back. That is why Notes reads first and writes the complete list every time. It is a good example of checking what an operation at the boundary actually does, rather than assuming.

Running it

Put together, the program adds notes, and they stay. Each command below is a separate run in the same directory.

$ ek9 Notes.ek9 -a milk
notes (1):
  milk

$ ek9 Notes.ek9 -a bread
notes (2):
  milk
  bread

$ ek9 Notes.ek9
notes (2):
  milk
  bread

The first run creates notes.txt and adds milk. The second finds the file already there — which createFileIfAbsent() treats as success — reads milk back, adds bread, and writes both. The third adds nothing (-a was left out, so the note was unset) and lists what is stored. With -v, the progress message goes to standard error, separate from the notes:

$ ek9 Notes.ek9 -v -a eggs
notes (3):
  milk
  bread
  eggs

with using notes.txt printed separately, on stderr.

Running it as its own command

One more thing explains a line at the top of every example in this book:

#!ek9

That first line is a shebang. On macOS and Linux it tells the system "to run this file, hand it to ek9." So far we have run programs by naming the tool first, ek9 Notes.ek9. You can also run the file on its own. Mark it executable, once:

chmod u+x Notes.ek9

chmod changes a file's permissions; u+x reads as "user: add execute", so you may now run this file, not only read it. From then on you run it directly, like any other command:

./Notes.ek9 -a bread

There is no ek9 in front: the shebang points the system at ek9 for you. The leading ./ means "the file here, in this directory". A note with spaces in it works too; put it in quotes so the shell keeps it as one piece:

./Notes.ek9 -a "buy fresh milk"

The result is the same as running it the other way; only the typing is shorter. The shebang is a Unix convention, so macOS and Linux honour it and Windows does not — there you keep typing ek9 Notes.ek9. Either way, Notes is now a command you made.

A setting from the environment: a web address

Some values belong to the place a program runs, not to the program itself. The address of a server is the usual example. Suppose the notes later live on a web server as well: its address is different on your own computer, on a test machine and on the real service, and it changes years after the code was written. A setting like that is kept in an environment variable, as the language was in Speaking Every Language.

For a web address EK9 goes further than for a language: you cannot write one into your code at all. A web address, a URL, comes from the environment, through Networking:

#!ek9
defines module book.notelink

  defines program
    NoteLink()
      -> noteNumber as Integer
      stdout <- Stdout()

      server <- Networking().urlFromEnv("NOTES_SERVER")
      if server?
        link <- server.withPath("/notes").withPathSegment($noteNumber)
        stdout.println($link)
      else
        stdout.println("NOTES_SERVER is not set to a web address")

Networking().urlFromEnv("NOTES_SERVER") reads the variable NOTES_SERVER and checks that it holds a real web address, starting http:// or https://. If the variable is missing, or holds something that is not a web address, the result is unset, and if server? asks the usual question: has something been put in the box?

withPath("/notes") and withPathSegment($noteNumber) build a new address from the one that came in. They keep its server, so values in the program can change where on the server a link points, never which server. withPathSegment also encodes its text, so a / or a space in it cannot change the shape of the address. $noteNumber turns the number into text, as in Text, Numbers, Time and Money.

On macOS and Linux you set the variable for one run by writing it in front of the command:

$ NOTES_SERVER=https://notes.example.com ek9 NoteLink.ek9 7
https://notes.example.com/notes/7

$ NOTES_SERVER=HTTPS://Notes.Example.COM:443 ek9 NoteLink.ek9 7
https://notes.example.com/notes/7

$ ek9 NoteLink.ek9 7
NOTES_SERVER is not set to a web address

The second spelling, in capitals and with the standard secure port 443 written out, is the same address, so EK9 writes it the same way. In Windows PowerShell, set the variable first with $env:NOTES_SERVER = "https://notes.example.com", then run ek9 NoteLink.ek9 7.

Try writing the address into the program instead:

      server <- Networking().urlOf("https://notes.example.com")

EK9     : Error   : E08275: 'urlOf' on line 9 position 29: '"https://notes.example.com"': the text given to 'Networking().urlOf()' is known when the program is compiled - an endpoint in the source is configuration hard-coded: read it with 'Networking().urlFromEnv(name)', and derive other URLs from it with 'withPath', 'withPathSegment', 'withQueryParam' or 'resolve'; 'urlOf' is for URL text that arrives while the program runs
             See: ek9 -h E08275

urlOf exists for an address that arrives while the program runs: one a person types into a form, or one inside a message from another program. Text the compiler can see when it compiles is refused, whether it is written in place, kept in a constant or copied into a variable first. There are two reasons. The people who run a program need to change its addresses without changing its code. And an address buried in code is hidden from anyone checking what the program connects to; a setting outside the code is in plain view.⁴

Try it yourself

1. Add a -c option that clears the list: when it is given, replace notes with an empty list and write it out before any new note is added. Add -c to the usage text as well.
2. Keep two lists with -f: add notes to shopping.txt and to todo.txt, and check that each run shows only its own file's notes.
3. With -v, also report on stderr how many notes were read. Run it with 2> /dev/null on macOS or Linux and see that the notes still appear while the progress messages do not.
4. Give NoteLink a second argument, a word to search for, and add it to the link with withQueryParam("search", word). Run it with a word containing a space and see how it is written.

Recap

This chapter dealt with the edge of a program, where it meets the outside world. A program takes the command line as argv, and GetOpt turns a short description of your options into a Dict you read with contains (was the flag there?) and getOrDefault (what value, or what fallback?). The fallback should be the truthful one: String(), not "". There are two output channels: Stdout for results and Stderr for everything else. A file comes in layers, each doing one job: a FileSystemPath is only an address; createFileIfAbsent() and createFile() return a Result the compiler makes you check, the strict createFile() refusing a file that is already there; and a TextFile, made from the path, reads and writes — inside a try that closes it for you, so you never write a close yourself. A web address is a setting, not code: Networking().urlFromEnv reads a URL from an environment variable, unset if it is missing or not a web address, and withPath and withPathSegment build new addresses on the same server; writing an address into the code is a compile error. The thread through all of it: at the boundary, input is untrusted and operations can fail, and that is exactly where EK9 makes you look.

Next, all of this — options, output and everything else in this book — goes into a program you would want to run: a game.

In the EK9 reference: Packaging in EK9, Programs in EK9