Why does 'cat items >> items' not compile, and how do I append a List to itself?

← Streams and Pipelines · Ref: Q2250

A stream pulls from its source one element at a time. '>>' appends to its target as each element arrives, so 'cat items >> items' (or a 'tee' into 'items' under 'cat items') adds to the List while it is still being read, and the read never reaches the end. GNU cat refuses the same thing on a file: "input file is output file". The compiler reports it as E08277, on the target, whenever the same variable - by identity, not by name - is a 'cat' source and an appending target ('>>' or 'tee') of one statement; 'cat a, items >> items' and 'this.items' against plain 'items' are the same variable too.

Read a COPY ('copy += items' on a new List, then 'cat copy >> items'), or REPLACE with '>', which fills a fresh List and swaps it in when the stream is finished ('cat items | sort > items' is fine), or collect into a new variable. The rule has no list of types: a Set, a Dict, a PriorityQueue or a JSON that is read and appended to is E08277 too, so read a copy of those as well. A variable held by another object ('other.items') is another variable.

Where the compiler cannot see two names for one List (or JSON) - 'alias <- items' then 'cat items >> alias', or one List passed to a function as both its source and its target - the program runs, and the stream stops before it reads anything with an Exception whose reason is "<Type> changed while it was being read", for a List, a Set, a Dict, a PriorityQueue or a JSON alike (it was "No such element" for a List, and nothing at all for a Set or a Dict): the statement compares the objects it reads with the objects it appends to. A List is never silently copied for you: the error is the answer. The same holds for a 'for' loop that adds to the List it is looping over.

Example

#!ek9
defines module qa.streams.append.list.to.itself

  defines function

    appendAll()
      ->
        source as List of Integer
        target as List of Integer
      cat source >> target

  defines program

    AppendListToItself()
      stdout <- Stdout()

      //Read a copy while '>>' appends to the original.
      items <- [1, 2, 3]
      copy <- List() of Integer
      copy += items
      cat copy >> items
      stdout.println(`appended ${items} from ${copy}`)

      //'>' fills a fresh List and swaps it in when the stream is finished, so reading what it held is fine.
      sorted <- [3, 1, 2]
      cat sorted | sort > sorted
      stdout.println(`replaced ${sorted}`)

      //Or build the result in a new variable.
      original <- [1, 2, 3]
      doubled <- cat original, original | collect as List of Integer
      stdout.println(`collected ${doubled}`)

      //E08277 is for every type: a Set read and appended to is rejected too, so read a copy of it.
      unique <- Set() of Integer
      unique += 1
      unique += 2
      snapshot <- Set() of Integer
      snapshot += unique
      cat snapshot >> unique
      stdout.println(`Set ${unique}`)

      //The compiler cannot see that source and target are one List here, so the List reports it as it is read.
      same <- [1, 2]
      try
        appendAll(same, same)
        stdout.println(`appended ${same}`)
      catch
        -> ex as Exception
        stdout.println(`alias: ${ex.reason()}`)

Common mistakes

E08277 — The stream reads 'items' while '>>' appends to it, so the read never ends. Read a copy of the List, use '>' to replace it, or collect into a new variable.

Incorrect:

      cat items >> items

Correct:

      cat copy >> items

Coming from another language?

Unix shell: 'cat file >> file' fails with 'input file is output file'. Java: adding to an ArrayList inside a for-each or stream over it throws ConcurrentModificationException. Python: 'for x in items: items.append(x)' never ends. EK9: E08277 at compile time where the compiler can see it, an Exception 'List changed while it was being read' where it cannot.

Keywords: >>, No such element, append, cat, changed while read, copy, input is output, same list, tee