How do I combine partial results from several tasks into the fullest Dict?

← Common Collection Tasks · Ref: Q1406

Merge each task's Dict into one with ':~:'. Merge never overwrites a set value; it fills whatever is unset or missing, key by key - so the first value recorded for a key wins and every gap is filled.

KEY BY KEY

- a key the target lacks is added
- a value the target holds unset is filled from a set source value
- a value the target holds set is kept

GATHERING

  gathered <- Dict() of (String, String)
  gathered :~: {"name": "Ada", "role": String()}
  gathered :~: {"name": "Grace", "role": "engineer"}
  stdout.println($gathered)

The first task recorded the name but not the role; the second supplies the role, and its name is ignored.

ORDER MATTERS

Because the first recorded value wins, merging A then B can differ from merging B then A when both set the same key. Merge the most trusted source first.

NOT THE SAME AS +

'+' builds a new Dict and, where keys overlap, the right-hand value is used. ':~:' changes the target and keeps what it already has.

TAKEN MEANS SHARED, NOT COPIED

What the target takes - a key it lacked, a value it held unset - is the same object the source holds (shallow, as a List's ':=:' is). Because merge recurses, a LATER merge into a structure the target took also changes the source's inner object:

  taskOne <- {"person": {"name": "Ada"}}
  taskTwo <- {"person": {"role": "engineer"}}
  combined <- Dict() of (String, Dict of (String, String))
  combined :~: taskOne   //takes taskOne's own inner Dict
  combined :~: taskTwo   //merges into that same Dict - taskOne now has the role too
  stdout.println($taskOne)
  stdout.println($combined)

To keep every source independent, give the target its own fresh inner structure first, so each merge fills the target's Dict rather than taking a source's:

  keptApart <- {"person": Dict() of (String, String)}

See Q1405 for the rule on single values. See Q1408 for Dicts nested inside Dicts. See Q186 for merging Lists and Dicts.

Example

defines module qa.collectiontasks.mergegatherpartialresults

  defines constant
    PERSON <- "person"
    NAME <- "name"
    ROLE <- "role"

  defines function

    personTask()
      ->
        field as String
        found as String
      <- rtn as Dict of (String, Dict of (String, String)): {PERSON: {field: found}}

    fieldOf()
      ->
        results as Dict of (String, Dict of (String, String))
        field as String
      <- rtn as String: results.getOrDefault(PERSON, Dict() of (String, String)).getOrDefault(field, "none")

    taskResults()
      <- rtn as List of Dict of (String, String): [
        {"name": "Ada", "role": String()},
        {"name": "Grace", "role": "engineer"},
        {"city": "London"}
        ]

  defines program

    MergeGatherPartialResultsDemo()
      stdout <- Stdout()

      //Each task found some fields; merge them all for the fullest set
      gathered <- Dict() of (String, String)
      for task in taskResults()
        gathered :~: task

      for entry in gathered
        stdout.println(`${entry.key()}: ${entry.value()}`)

      //The first value recorded for a key wins, so order matters
      first <- {"colour": "red"}
      second <- {"colour": "blue"}

      firstThenSecond <- Dict() of (String, String)
      firstThenSecond :~: first
      firstThenSecond :~: second
      stdout.println(`first then second: ${firstThenSecond.getOrDefault("colour", "none")}`)

      secondThenFirst <- Dict() of (String, String)
      secondThenFirst :~: second
      secondThenFirst :~: first
      stdout.println(`second then first: ${secondThenFirst.getOrDefault("colour", "unknown")}`)

      //A key the target lacked is TAKEN - the source's own inner Dict, shared not copied
      who <- "Ada"
      job <- "engineer"
      taskOne <- personTask(NAME, who)
      taskTwo <- personTask(ROLE, job)
      combined <- Dict() of (String, Dict of (String, String))
      combined :~: taskOne
      combined :~: taskTwo
      stdout.println(`task one shared, its role: ${fieldOf(taskOne, ROLE)}`)
      stdout.println(`combined: ${fieldOf(combined, NAME)} ${fieldOf(combined, ROLE)}`)

      //Give the target its own fresh inner Dict first: each merge fills it, the sources stay apart
      taskThree <- personTask(NAME, who)
      taskFour <- personTask(ROLE, job)
      keptApart <- {PERSON: Dict() of (String, String)}
      keptApart :~: taskThree
      keptApart :~: taskFour
      stdout.println(`task three kept apart, its role: ${fieldOf(taskThree, ROLE)}`)
      stdout.println(`kept apart gathered: ${fieldOf(keptApart, NAME)} ${fieldOf(keptApart, ROLE)}`)

Common mistakes

E50060 — EK9's Dict has no putAll() method; ':~:' is the merge, and unlike putAll() it never overwrites a value the target already holds. See ek9 -h E50060 for details.

Incorrect:

      gathered.putAll(task)

Correct:

      gathered :~: task

Other ways to ask this

Each task found some of the fields - how do I gather them into one Dict?
Which value wins when two Dicts I merge both have the key?
How do I fill the gaps in one Dict from another without losing what it has?
Does the order I merge Dicts in matter?
Does merge copy the nested Dicts it takes, or share them with the source?

Coming from another language?

Java: map.putAll(other) overwrites existing keys; gathering 'first found wins' needs putIfAbsent in a loop, and a null value is not filled. Python: dict.update overwrites; {**b, **a} keeps a's values but builds a new dict. Kotlin: map + other lets other win. EK9: target :~: source keeps every set value and fills missing keys and unset values in place, recursing into nested Dicts.

Keywords: partial results, merge, dict, shallow merge, putIfAbsent, shared not copied, :~:, gather, fill gaps, first recorded wins, merge order