How do I use tee and uniq in stream pipelines in EK9?

← Collections and Data Structures · Ref: Q133

EK9 provides tee for capturing intermediate pipeline state and uniq for removing duplicate elements. Both are pipeline operations that integrate with cat | collect syntax.

TEE FOR SIDE EFFECTS

Tee copies each element to a variable while passing it through:

  captured <- List() of String
  copy <- cat items | tee in captured | collect as List of String

After the pipeline, captured contains exactly the items that flowed through that point. Useful for debugging or capturing intermediate state.

TEE IS A LAZY PEEK

A head bounds the PULL, so a tee before a head records only the items the head demanded: cat items | tee in seen | head 3 leaves 3 items in seen, and cat items | sort | tee in seen | head 3 also leaves 3 (the sort sees everything, the tee only what is emitted past it). To capture every item, put the tee before a stage that must see everything (sort, tail, join) or drop the head.

TEE FOR DEBUGGING

Insert tee at any point in a pipeline to observe values:

  cat items
    | filter by isValid
    | tee in afterFilter
    | map with transform
    | collect as List of String

afterFilter shows what passed the filter before transformation.

UNIQ KEEPS THE FIRST OF EACH

uniq keeps the first occurrence of each element anywhere in the stream and drops every later one that is == to it - global, not just consecutive repeats:

  cat items | uniq | collect as List of String

No sort is needed; add one only to get the result in order:

  cat items | sort | uniq | collect as List of String

The element's #? (hashcode) only narrows the search; == decides.

UNIQ BY KEY EXTRACTOR

Remove duplicates based on a specific field:

  cat items | uniq by extractKey | collect as List of Item

The key extractor function determines what counts as a duplicate.

COMBINING TEE AND UNIQ

Capture state before deduplication:

  cat items
    | sort
    | tee in beforeUniq
    | uniq
    | collect as List of String

beforeUniq has all sorted items; the result has duplicates removed.

See Q89 for basic stream pipelines. See Q96 for hashcode operator (#?). See Q120 for sort (optional before uniq, only to order the result). See Q124 for group by. See Q235 for complete stream operations reference. See Q237 for streams vs loops decision guide.

Example

defines module qa.collections.teeuniq

  defines function

    firstChar() as pure
      -> word as String
      <- rtn as String: $#<word

    longEnough() as pure
      -> word as String
      <- rtn <- Boolean()
      expectedCount <- 4
      rtn: length word >= expectedCount

  defines program

    TeeUniqDemo()
      stdout <- Stdout()

      words <- ["banana", "apple", "cherry", "apple", "banana", "date", "cherry", "elderberry"]

      // === TEE: CAPTURE INTERMEDIATE STATE ===

      afterFilter <- List() of String
      result <- cat words
        | filter by longEnough
        | tee in afterFilter
        | sort
        | collect as List of String
      stdout.println(`After filter (via tee): ${afterFilter}`)
      stdout.println(`Final sorted: ${result}`)

      // === UNIQ: REMOVE DUPLICATES ===

      // uniq is global first-seen; the sort only orders the result
      unique <- cat words | sort | uniq | collect as List of String
      stdout.println(`Unique: ${unique}`)

      // === UNIQ BY KEY EXTRACTOR ===

      // One word per first character
      uniqueByFirst <- cat words | sort | uniq by firstChar | collect as List of String
      stdout.println(`Unique by first char: ${uniqueByFirst}`)

      // === TEE + UNIQ TOGETHER ===

      beforeUniq <- List() of String
      deduped <- cat words
        | sort
        | tee in beforeUniq
        | uniq
        | collect as List of String
      stdout.println(`Before uniq: ${beforeUniq}`)
      stdout.println(`After uniq: ${deduped}`)

Common mistakes

E50060 — EK9 has no distinct() or stream() methods. Use | uniq in a pipeline to keep the first of each equal element (add | sort to order the result). See ek9 -h E50060 for details.

Incorrect:

unique <- words.stream().distinct().collect()

Correct:

unique <- cat words | sort | uniq | collect as List of String

E07830 — The collect type must match the pipeline's element type. A String pipeline cannot collect into List of Integer. See ek9 -h E07830 for details.

Incorrect:

unique <- cat words | sort | uniq | collect as List of Integer

Correct:

unique <- cat words | sort | uniq | collect as List of String

E07830 — The collect type must match the pipeline's element type. After uniq by firstChar, the stream still contains String elements, not Integer. See ek9 -h E07830 for details.

Incorrect:

uniqueByFirst <- cat words | sort | uniq by firstChar | collect as List of Integer

Correct:

uniqueByFirst <- cat words | sort | uniq by firstChar | collect as List of String

E50060 — EK9 has no distinct() or stream() methods. Use | uniq in a pipeline to keep the first of each equal element (add | sort to order the result). See ek9 -h E50060 for details.

Incorrect:

unique <- words.stream().distinct().collect()

Correct:

unique <- cat words | sort | uniq | collect as List of String

E07830 — The collect type must match the pipeline's element type. A String pipeline cannot collect into List of Integer. See ek9 -h E07830 for details.

Incorrect:

| tee in afterFilter
        | sort
        | collect as List of Integer

Correct:

| tee in afterFilter
        | sort
        | collect as List of String

E07830 — The tee and collect must match the pipeline type. After uniq the stream still contains Strings, so collecting into List of Integer fails. See ek9 -h E07830 for details.

Incorrect:

| tee in beforeUniq
        | uniq
        | collect as List of Integer

Correct:

| tee in beforeUniq
        | uniq
        | collect as List of String

E50060 — EK9 List has no toString() method. Use the $ prefix operator or string interpolation for string conversion. See ek9 -h E50060 for details.

Incorrect:

stdout.println(unique.toString())

Correct:

stdout.println(`Unique: ${unique}`)

Coming from another language?

Unix: cmd | tee file.txt | next. Java: stream.peek() for side effects (like tee), stream.distinct() for unique elements. Python: no built-in tee (itertools.tee is different), set() for unique. JavaScript: no built-in pipeline tee, [...new Set(arr)] for unique. Rust: iter.inspect() for tee (lazy, like EK9 tee before head); iter.dedup() only drops CONSECUTIVE repeats, unlike EK9 uniq, which is global first-seen like Java distinct(). Go: manual implementation for both. EK9: | tee in variable for capture, | uniq for global dedup by ==, | uniq by func for dedup by key.

Keywords: inspect, data-structure, observe, collection, copy, stream, unique, intermediate, debug, duplicate, snapshot, uniq, effect, tee, capture, side, side list, pipeline, tap, intercept