Remove duplicate integers from a list using sort and uniq.

← Streams and Pipelines · Ref: Q1137

uniq keeps the first occurrence of each value and drops every later one that is == to it, adjacent or not, so it alone deduplicates:

  cat items | uniq > stdout

Add sort to get the unique values in order:

  cat items | sort | uniq > stdout

The sort is for the ORDER of the result, not for uniq (which is not consecutive-only, unlike Unix uniq or Rust dedup). See Q989, Q891.

Example

defines module qa.streams.streamuniqgroup

  defines program

    StreamUniqGroupDemo()
      stdout <- Stdout()

      items <- [3, 1, 2, 1, 3, 2, 1]

      //uniq deduplicates on its own (global first-seen); the sort orders the result
      stdout.println("Sorted unique:")
      cat items | sort | uniq > stdout

      //Sort, group, then flatten
      stdout.println("Sort group flatten:")
      cat items | sort | group | flatten > stdout

Common mistakes

E50060 — A List has no distinct() method, so the call is not resolved (E50060). EK9 deduplicates in a stream with 'uniq' (cat items | uniq). It keeps the first of each equal item anywhere in the stream; the sort only orders the result. See ek9 -h E50060 for details.

Incorrect:

      items.distinct()

Correct:

      cat items | sort | uniq > stdout

Coming from another language?

Java: stream().distinct().sorted() - EK9 uniq is like distinct(), global first-seen. Python: sorted(set(items)). Rust: dedup() only drops consecutive repeats, so it needs a sort first; EK9 uniq does not. EK9: sort | uniq gives sorted unique values.

Keywords: uniq, group, unique, distinct, flatten, deduplicate, sort