How do I build up binary data or write a network protocol message in EK9?

BytesBuilder is the mutable working buffer - the write side of the family, and the counterpart to BytesReader.

WHY IT EXISTS (it is NOT just 'a faster Bytes')
Bytes is a value, so every + and | copies the whole accumulator; building byte-by-byte is quadratic. That matters for bulk work, but it is not the reason this type exists. Three things are inexpressible on Bytes at ANY speed:

  1. ENDIAN-AWARE WRITES. BytesReader has readUInt32BE; without a builder there is no endian-aware write anywhere in the language, so you hand-assemble bytes and reintroduce the endianness bugs the reader eliminated.
  2. IN-PLACE MUTATION - setByte, xorInto, copyInto. A value type cannot offer these; every operation on Bytes yields a new array.
  3. WIPE. A Bytes CANNOT be securely erased, because + and | already scattered copies of it. A builder is one buffer you can actually zero.

CREATING

  builder <- BytesBuilder()               // empty, and SET - empty is not unset
  sized <- BytesBuilder(1024)             // reserves capacity; length is still 0
  seeded <- BytesBuilder(someBytes)       // initialise from an immutable value

Capacity is backing storage, not content. reserve(n) grows it and never shrinks. Growth is geometric, so appending is amortised rather than quadratic.

APPENDING

  builder.append(aByte) / append(someBytes) / append(anotherBuilder)
  builder += aByte                        // and | through a stream terminal

The | and += spellings carry over from Bytes deliberately, so the familiar accumulation idiom reads the same - it is simply amortised here.

TYPED WRITES MIRROR THE READER EXACTLY

  appendUInt16BE/LE, appendUInt32BE/LE, appendUInt64BE/LE
  appendInt16BE/LE,  appendInt32BE/LE,  appendInt64BE/LE

Twelve writes against BytesReader's twelve reads. There is no appendUInt32 with a default, for the same reason there is no readUInt32: a silent endianness assumption is a bug category, so the defaulted spelling does not exist.

RANGE IS ENFORCED, NOT TRUNCATED

appendUInt16BE(0x10000) makes the builder UNSET rather than writing 0x0000. And appendUInt64BE(-1) is REFUSED rather than written as two's complement - the mirror image of readUInt64BE refusing to hand back a negative for a large positive. Use appendInt64BE when you mean a signed value.

IN-PLACE MUTATION

  builder.setByte(index, aByte)
  builder.xorInto(offset, source)
  builder.copyInto(offset, source)

All three write WITHIN the current length and never extend - use append to grow. A write that would run past the end is refused OUTRIGHT rather than part-applied, because half an applied round key is worse than none. xorInto is one primitive with three uses: AES AddRoundKey, the Argon2 block XOR and the ChaCha20 keystream application.

SECURE HYGIENE

  secret.wipe()

Zeroes the backing store and leaves the builder usable. Best-effort by nature - the JVM may have moved the buffer during GC and the OS may have paged it - but it is the only place erasure is even possible.

CROSSING BACK

  frozen <- builder.toBytes()

The single visible boundary between 'doing byte work' and handing a value to the rest of the program. It always COPIES, so a snapshot cannot change underneath its holder.

AND $ SHOWS SHAPE, NOT CONTENT

$builder gives BytesBuilder(4/16) - the length and capacity, never the bytes. A buffer holding a key must not leak into a log line through a debug print. For the same reason there is no #? - a hash that changes as you append makes a mutable key look usable, and that is a bug.

See Q1384 for BytesReader, Q1383 for Bytes and Q1382 for Byte. Use 'ek9 -h BytesBuilder' for the full API.

Example

defines module qa.bytesbuilder.usage

  defines constant

    tlsVersionTls12 <- 0x0303
    tlsBodyLength <- 8

  defines program

    BytesBuilderUsage()
      stdout <- Stdout()

      //The TLS record Q1384 parses - endianness is in the method name.
      record <- BytesBuilder()
      record.append(Byte("16"))
      record.appendUInt16BE(tlsVersionTls12)
      record.appendUInt16BE(tlsBodyLength)
      record.append(Bytes("034142430358595a"))

      framed <- record.toBytes()
      stdout.println(`framed: ${framed}`)

      //It reads back through the reader.
      cursor <- framed.reader()
      stdout.println(`type: ${cursor.readByte()}`)
      stdout.println(`version: ${cursor.readUInt16BE()}`)
      stdout.println(`declared length: ${cursor.readUInt16BE()}`)

      //BE and LE genuinely differ - no defaulted spelling exists.
      bigEndian <- BytesBuilder()
      bigEndian.appendUInt32BE(0x01020304)
      littleEndian <- BytesBuilder()
      littleEndian.appendUInt32LE(0x01020304)
      stdout.println(`big endian: ${bigEndian.toBytes()}`)
      stdout.println(`little endian: ${littleEndian.toBytes()}`)

      //Too wide for the field is REJECTED, never truncated.
      tooWide <- BytesBuilder()
      tooWide.appendUInt16BE(0x10000)
      stdout.println(`too wide accepted: ${tooWide?}`)

      //Negative unsigned is refused; the signed spelling accepts it.
      negativeUnsigned <- BytesBuilder()
      negativeUnsigned.appendUInt64BE(-1)
      stdout.println(`negative unsigned accepted: ${negativeUnsigned?}`)
      negativeSigned <- BytesBuilder()
      negativeSigned.appendInt64BE(-1)
      stdout.println(`negative signed: ${negativeSigned.toBytes()}`)

      //xorInto is AddRoundKey / Argon2 block XOR / ChaCha20 keystream, self-inverse.
      block <- BytesBuilder(Bytes("ffffffff"))
      block.xorInto(0, Bytes("0f0f0f0f"))
      stdout.println(`after xor: ${block.toBytes()}`)
      block.xorInto(0, Bytes("0f0f0f0f"))
      stdout.println(`xor undone: ${block.toBytes()}`)

      //A write past the end is refused outright.
      overrun <- BytesBuilder(Bytes("ffffffff"))
      overrun.xorInto(2, Bytes("0f0f0f"))
      stdout.println(`overrun accepted: ${overrun?}`)

      //wipe zeroes the buffer - an immutable Bytes cannot be erased.
      secret <- BytesBuilder(Bytes("cafebabedeadbeef"))
      secret.wipe()
      stdout.println(`wiped length: ${length secret}`)
      stdout.println(`still usable: ${secret?}`)

      //toBytes always COPIES - a snapshot cannot move under its holder.
      snapshotting <- BytesBuilder(Bytes("deadbeef"))
      snapshot <- snapshotting.toBytes()
      snapshotting.append(Byte("FF"))
      stdout.println(`snapshot: ${snapshot}`)
      stdout.println(`builder now: ${snapshotting.toBytes()}`)

      //$ reports SHAPE, never CONTENT - a key must not reach a log line.
      stdout.println(`shape only: ${$snapshotting}`)

Other ways to ask this

Is there a StringBuilder equivalent for bytes in EK9?
How do I write a big-endian integer into bytes?
How do I XOR one buffer into another in EK9?
How do I erase a key or password buffer in EK9?

Coming from another language?

Java: ByteBuffer.putInt writes in whatever order the buffer's order() setting happens to be - decided elsewhere in the program - and silently truncates a long into an int field; a password in a String cannot be erased at all, which is why the JDK's own APIs take char[]. C: a memcpy of a struct writes host byte order and host padding, correct on the machine it was written on, and memset on a dead buffer may be optimised away entirely - which is why memset_s and explicit_bzero exist. Python: struct.pack raises on an out-of-range value, which is right, but the endianness prefix is easy to omit and native order is then used; bytes are immutable so their copies cannot be cleared. Go: binary.Write takes an explicit binary.BigEndian, but a too-wide value is handled by the conversion at the call site, so the truncation just moves. EK9 puts the endianness in the method name, makes an out-of-range write visibly absent instead of quietly plausible, and gives key material one buffer that can actually be zeroed.

Keywords: wipe, buffer, xor, mutable, bytesbuilder, crypto, append, builder, littleendian, binary, endian, bigendian