How do I work with binary data (Bytes) in EK9?

← Advanced Type System · Ref: Q1383

Bytes is an immutable sequence of unsigned 8-bit values - the public face of binary data. It follows the sequence semantics of Bits rather than those of a number: + CONCATENATES, | ACCUMULATES, and there is no arithmetic at all.

CREATING

  payload <- 0yDEADBEEF                 // the LITERAL - 0y then hex digits IN PAIRS
  fromHex <- Bytes("DEADBEEF")        // hex, 0x prefix optional, case-insensitive
  single <- Bytes(Byte("FF"))          // one octet
  packed <- Bytes(someBits)            // whole bytes only
  zeroed <- Bytes().withLength(16)     // 16 zero bytes
  filled <- Bytes().withLength(16, Byte("FF"))

The withLength pair follows the standard-library 'with' convention - pure, returns a new value, and called on a default-constructed receiver.

THE LITERAL IS 0y, NOT 0x
0x stays Integer and 0b stays Bits - all three coexist and none is ambiguous. Hex digits come in PAIRS and the rule is enforced by the LEXER, so 0yF does not parse at all rather than becoming an unset Bytes to be diagnosed later. Case follows 0x/0b: 0Y works too.

EMPTY IS NOT UNSET

Bytes() is UNSET. A zero-length Bytes is SET and empty. Binary protocols depend on that difference - a field that was absent and a field that was present and empty are not the same fact.

HEX BOTH WAYS

$ emits upper-case hex and the String constructor parses hex, so Bytes($b) == b always holds. An ODD number of hex digits is UNSET - half a byte is not a byte. Decimal text is not accepted.

Base64 is NOT guessed at, because "DEADBEEF" is simultaneously valid hex and valid Base64. Name the encoding:

  fromB64 <- Bytes(text, "base64")
  asB64 <- payload.toBase64()

BITWISE NEEDS EQUAL LENGTHS

and, or and xor require BOTH operands to be the same length. A mismatch is UNSET. This is deliberate and it is the one place Bytes departs from Bits: zero-extending a short key leaves the tail of the message UNCHANGED, so the 'ciphertext' would carry plaintext while the operation reported success. Length is semantic for binary data - a 16-byte key IS 16 bytes.

SEQUENCE OPERATIONS

  joined <- first + second            // concatenation
  extended <- payload + Byte("FF")    // append one octet
  collector <- Bytes()
  cat payload.iterator() > collector   // | accumulates through a stream

Ordering (<, <=, >, >=, <=>) is lexicographic over UNSIGNED octets, so 0xFF sorts ABOVE 0x7F.

SLICING AND SEARCH

slice(from, count), first(n) and last(n) return VALUES, not views, so a slice can never mutate its source. Out of range is UNSET rather than clamped. indexOf returns UNSET when the needle is absent - there is no -1 to be mistaken for a real index.

COMPARING SECRETS

Use constantTimeEquals for MACs, tokens and digests. It examines every byte regardless of where the first difference is; == is free to short-circuit and must not be used for secrets.

See Q1382 for Byte, the scalar element type. Use 'ek9 -h Bytes' for the full API.

Example

defines module qa.bytes.usage

  defines program

    BytesUsage()
      stdout <- Stdout()

      //Hex in, hex out - the round trip always holds.
      //The LITERAL form - 0y then hex digits in pairs. 0x is Integer, 0b is Bits.
      payload <- 0yDEADBEEF
      stdout.println(`payload: ${payload}`)
      stdout.println(`length: ${length payload}`)
      stdout.println(`base64: ${payload.toBase64()}`)

      //Buffers via the standard 'with' factory convention.
      zeroed <- Bytes().withLength(4)
      stdout.println(`zeroed: ${zeroed}`)
      filled <- Bytes().withLength(4, Byte("FF"))
      stdout.println(`filled: ${filled}`)

      //'+' concatenates - Bytes has no arithmetic.
      joined <- Bytes("DEAD") + Bytes("BEEF")
      stdout.println(`joined: ${joined}`)

      //XOR is its own inverse, which is why low level code leans on it.
      masked <- payload xor filled
      stdout.println(`masked: ${masked}`)
      stdout.println(`recovered: ${(masked xor filled) == payload}`)

      //Unequal lengths are UNSET, never zero extended - zero extension would leave the
      //tail of the longer operand untouched, which for a cipher means leaking plaintext.
      mismatch <- payload xor Bytes("FF")
      stdout.println(`mismatch set: ${mismatch?}`)

      //Slices are values, and absence is unset rather than a -1 sentinel.
      stdout.println(`first two: ${payload.first(2)}`)
      stdout.println(`indexOf BEEF: ${payload.indexOf(Bytes("BEEF"))}`)
      stdout.println(`not found set: ${payload.indexOf(Bytes("CAFE"))?}`)

Coming from another language?

Java: byte[] is mutable, has reference equality, no unset state, and every helper is hand-rolled; null vs empty is a constant confusion. C: unsigned char* plus a separate size_t that can drift out of step - the split that makes overruns possible. Python: bytes is immutable but uses None for absent, reintroducing null. Go: []byte slices share backing storage, so a slice can mutate its source. EK9: Bytes is immutable with value equality, carries its own length, is tri-state (unset vs empty vs set), slices are values not views, and and/or/xor REFUSE unequal lengths rather than zero-extending - which in other languages silently leaks the tail of the longer operand.

Keywords: buffer, constanttime, xor, sequence, immutable, bytes, octet, base64, binary, slice, hex, mask