Why can't I sort or group a stream from Stdin, UDP or a TCP connection?

← Streams and Pipelines · Ref: Q1301

Buffering operations (sort, group, tail, uniq) must see or accumulate over ALL of the input before producing output, so they require a BOUNDED (finite) source. Stdin, UDP and TCPConnection are UNBOUNDED — infinite IO streams with no defined end — so buffering one would consume memory without bound and never emit. EK9 rejects this at compile time (E10050).

ERROR

  stdin <- Stdin()
  cat stdin | sort > stdout        // ERROR: E10050

FIX WITH head N (head bounds the stream)

  cat stdin | head 1000 | sort > stdout   // OK

FIX WITH bounded windows

  while active
    batch <- cat udpConnection | head 100 | collect as List of Packet
    cat batch | sort > processor

STREAMING OPERATIONS ARE FINE ON UNBOUNDED SOURCES

  cat stdin | filter by validLine | map with toEntry > stdout   // OK
  cat udpConnection | map by handler > stdout                   // OK

head N is the mechanism that converts an unbounded stream to a bounded one.

Example

defines module qa.streams.unboundedbuffer

  defines function

    nonEmpty() as pure
      -> line as String
      <- rtn as Boolean: length line > 0

  defines program

    UnboundedStreamDemo()
      stdin <- Stdin()
      stdout <- Stdout()

      //Streaming operations (filter/map) compose freely with an unbounded Stdin source.
      cat stdin | filter by nonEmpty > stdout

      //To sort/group/tail/uniq, 'head N' must bound the unbounded stream first.
      first10Sorted <- List() of String
      cat stdin | head 10 | sort > first10Sorted

      stdout.println(`Captured: ${length first10Sorted}`)

Common mistakes

E10050 — Buffering operators like sort need a bounded source; stdin is unbounded, so add 'head N' before sort. See ek9 -h E10050 for details.

Incorrect:

      cat stdin | sort > first10Sorted

Correct:

      cat stdin | head 10 | sort > first10Sorted

Coming from another language?

Java: Stream.sorted() on an infinite stream hangs at runtime, no compile check. Python: sorted(infinite_generator) hangs. Rust: collecting an infinite iterator to sort hangs. EK9: the compiler proves at compile time that buffering an unbounded source is impossible and requires 'head N' to bound it first.

Keywords: Stdin, backpressure, tail, E10050, sort, UDP, uniq, stream, TCP, unbounded, group, head, bounded