How do I use tee and uniq in stream pipelines in EK9?

← Collections and Data Structures · Ref: Q133

EK9 provides tee for capturing intermediate pipeline state and uniq for removing duplicate elements. Both are pipeline operations that integrate with cat | collect syntax.

TEE FOR SIDE EFFECTS

Tee copies each element to a variable while passing it through:

  captured <- List() of String
  cat items | tee in captured | collect as List of String

After the pipeline, captured contains all items that flowed through that point. Useful for debugging or capturing intermediate state.

TEE FOR DEBUGGING

Insert tee at any point in a pipeline to observe values:

  cat items
    | filter by isValid
    | tee in afterFilter
    | map with transform
    | collect as List of String

afterFilter shows what passed the filter before transformation.

UNIQ BY HASHCODE

Remove consecutive duplicates using the #? (hashcode) operator:

  cat items | sort | uniq | collect as List of String

Sort first to ensure duplicates are adjacent. Uses the element's #? operator for comparison.

UNIQ BY KEY EXTRACTOR

Remove duplicates based on a specific field:

  cat items | uniq by extractKey | collect as List of Item

The key extractor function determines what counts as a duplicate.

COMBINING TEE AND UNIQ

Capture state before deduplication:

  cat items
    | sort
    | tee in beforeUniq
    | uniq
    | collect as List of String

beforeUniq has all sorted items; the result has duplicates removed.

See Q89 for basic stream pipelines. See Q96 for hashcode operator (#?). See Q120 for sort (needed before uniq). See Q124 for group by. See Q235 for complete stream operations reference. See Q237 for streams vs loops decision guide.

Example

defines module qa.collections.teeuniq

  defines function

    firstChar() as pure
      -> word as String
      <- rtn as String: $#<word

    longEnough() as pure
      -> word as String
      <- rtn <- Boolean()
      expectedCount <- 4
      rtn: length word >= expectedCount

  defines program

    TeeUniqDemo()
      stdout <- Stdout()

      words <- ["banana", "apple", "cherry", "apple", "banana", "date", "cherry", "elderberry"]

      // === TEE: CAPTURE INTERMEDIATE STATE ===

      afterFilter <- List() of String
      result <- cat words
        | filter by longEnough
        | tee in afterFilter
        | sort
        | collect as List of String
      stdout.println(`After filter (via tee): ${afterFilter}`)
      stdout.println(`Final sorted: ${result}`)

      // === UNIQ: REMOVE DUPLICATES ===

      // Sort first to make duplicates adjacent
      unique <- cat words | sort | uniq | collect as List of String
      stdout.println(`Unique: ${unique}`)

      // === UNIQ BY KEY EXTRACTOR ===

      // One word per first character
      uniqueByFirst <- cat words | sort | uniq by firstChar | collect as List of String
      stdout.println(`Unique by first char: ${uniqueByFirst}`)

      // === TEE + UNIQ TOGETHER ===

      beforeUniq <- List() of String
      deduped <- cat words
        | sort
        | tee in beforeUniq
        | uniq
        | collect as List of String
      stdout.println(`Before uniq: ${beforeUniq}`)
      stdout.println(`After uniq: ${deduped}`)

Common mistakes

E50060 — EK9 has no distinct() or stream() methods. Use | sort | uniq in a pipeline to remove consecutive duplicates by hashcode. See ek9 -h E50060 for details.

Incorrect:

unique <- words.stream().distinct().collect()

Correct:

unique <- cat words | sort | uniq | collect as List of String

E07830 — The collect type must match the pipeline's element type. A String pipeline cannot collect into List of Integer. See ek9 -h E07830 for details.

Incorrect:

unique <- cat words | sort | uniq | collect as List of Integer

Correct:

unique <- cat words | sort | uniq | collect as List of String

E07830 — The collect type must match the pipeline's element type. After uniq by firstChar, the stream still contains String elements, not Integer. See ek9 -h E07830 for details.

Incorrect:

uniqueByFirst <- cat words | sort | uniq by firstChar | collect as List of Integer

Correct:

uniqueByFirst <- cat words | sort | uniq by firstChar | collect as List of String

E50060 — EK9 has no distinct() or stream() methods. Use | sort | uniq in a pipeline to remove consecutive duplicates by hashcode. See ek9 -h E50060 for details.

Incorrect:

unique <- words.stream().distinct().collect()

Correct:

unique <- cat words | sort | uniq | collect as List of String

E07830 — The collect type must match the pipeline's element type. A String pipeline cannot collect into List of Integer. See ek9 -h E07830 for details.

Incorrect:

| tee in afterFilter
        | sort
        | collect as List of Integer

Correct:

| tee in afterFilter
        | sort
        | collect as List of String

E07830 — The tee and collect must match the pipeline type. After uniq the stream still contains Strings, so collecting into List of Integer fails. See ek9 -h E07830 for details.

Incorrect:

| tee in beforeUniq
        | uniq
        | collect as List of Integer

Correct:

| tee in beforeUniq
        | uniq
        | collect as List of String

E50060 — EK9 List has no toString() method. Use the $ prefix operator or string interpolation for string conversion. See ek9 -h E50060 for details.

Incorrect:

stdout.println(unique.toString())

Correct:

stdout.println(`Unique: ${unique}`)

Coming from another language?

Java: stream.peek() for side effects (like tee), stream.distinct() for unique elements. Python: no built-in tee (itertools.tee is different), set() for unique. JavaScript: no built-in pipeline tee, [...new Set(arr)] for unique. Rust: iter.inspect() for tee, iter.dedup() for consecutive dedup (like EK9 uniq). Go: manual implementation for both. EK9: | tee in variable for capture, | uniq for dedup by hashcode, | uniq by func for dedup by key.

Keywords: collection, tee, duplicate, tap, capture, copy, inspect, debug, stream, intercept, observe, effect, pipeline, unique, data-structure, uniq, intermediate, side