How do I parse binary data or a network protocol in EK9?

← Advanced Type System · Ref: Q1384

BytesReader is a forward-only cursor over a Bytes, obtained with aBytes.reader(). It is what you use for binary protocol and format parsing.

THE CENTRAL IDEA - MALFORMED INPUT IS A VALUE, NOT AN EXCEPTION
Every read is tri-state. A read that would run past the end returns UNSET and LEAVES THE CURSOR WHERE IT WAS. So parsing is a sequence of ordinary reads followed by a set-check, with no try/catch and no bounds test before every field. Because a failed read does not move the cursor, a smaller read can still succeed afterwards:

  cursor <- Bytes("010203").reader()
  complete <- cursor.readUInt32BE()?   // false - only three bytes
  start <- cursor.position()           // still 0
  smaller <- cursor.readUInt16BE()     // 0x0102 - works

CURSOR STATE

  .position()   bytes consumed so far
  .remaining()  bytes left
  .atEnd()      nothing left
  .peek()       next byte WITHOUT consuming it

ENDIANNESS IS ALWAYS EXPLICIT

There is no readUInt32() with a default. There is readUInt32BE and readUInt32LE, and nothing else. A silent endianness default is a known bug category - it is correct on the machine it was written on and wrong in production - so the defaulted spelling does not exist. The same applies at every width: readUInt16BE/LE, readUInt64BE/LE, and the signed readInt16BE/LE, readInt32BE/LE, readInt64BE/LE.

Signed reads sign-extend; unsigned ones do not:

  Bytes("fffe").reader().readInt16BE()   // -2
  Bytes("fffe").reader().readUInt16BE()  // 65534

WHY readUInt64BE CAN BE UNSET
EK9's Integer is 64-bit SIGNED, so an unsigned 64-bit value with the top bit set has no correct representation - returning it would hand back a NEGATIVE number for a large positive one. It is UNSET instead, and deliberately NOT consumed, so you can reinterpret the same bytes:

  wide <- Bytes("ffffffffffffffff").reader()
  fits <- wide.readUInt64BE()?   // false - will not fit
  signed <- wide.readInt64BE()   // -1 - same bytes, your interpretation

NAVIGATION

  .skip(n)    advances; answers false and does NOT move if it would run past the end - it never clamps
  .readBytes(n)  reads n bytes as a Bytes

SLICE - WHAT MAKES NESTED FORMATS SAFE

  body <- cursor.slice(declaredLength)

This is a bounded sub-reader over the next n bytes which CONSUMES them from the parent. The sub-reader cannot read past its own boundary however malformed its content is, and the parent carries on after it. That is exactly what a length-prefixed structure needs - a TLS extension inside a record, a DER SEQUENCE inside a SEQUENCE.

See Q1383 for Bytes and Q1382 for Byte. Use 'ek9 -h BytesReader' for the full API.

Example

defines module qa.bytesreader.usage

  defines program

    BytesReaderUsage()
      stdout <- Stdout()

      //A TLS-record shape: type, version, body length, then a length-prefixed body.
      record <- Bytes("1603030008034142430358595a")
      cursor <- record.reader()

      stdout.println(`type: ${cursor.readByte()}`)
      stdout.println(`version: ${cursor.readUInt16BE()}`)

      bodyLength <- cursor.readUInt16BE()
      stdout.println(`body length: ${bodyLength}`)

      //A bounded sub-reader - it cannot read past its own boundary however malformed the body is.
      body <- cursor.slice(bodyLength)
      fieldLength <- body.readByte().toInteger()
      stdout.println(`field: ${body.readBytes(fieldLength)}`)

      //slice consumed the body from the parent, so the parent is done.
      stdout.println(`parent at end: ${cursor.atEnd()}`)

      //A short read is UNSET and leaves the cursor alone, so a smaller read still works.
      short <- Bytes("010203").reader()
      stdout.println(`underflow set: ${short.readUInt32BE()?}`)
      stdout.println(`cursor unmoved: ${short.position()}`)
      stdout.println(`smaller read: ${short.readUInt16BE()}`)

      //An unsigned 64-bit value with the top bit set cannot fit a SIGNED Integer, so it is
      //unset rather than negative - and is not consumed, so the bytes can be reinterpreted.
      wide <- Bytes("ffffffffffffffff").reader()
      stdout.println(`uint64 fits: ${wide.readUInt64BE()?}`)
      stdout.println(`same bytes signed: ${wide.readInt64BE()}`)

Coming from another language?

Java: ByteBuffer throws BufferUnderflowException on a short read, so untrusted input needs a try/catch or a hasRemaining() check per field; endianness is a mutable order() setting carried on the buffer, often set far from the read. C: reading past the end is undefined behaviour - nothing to catch, and the single most productive source of remote vulnerabilities in binary parsers. Python: struct.unpack raises while slicing silently returns short - two failure modes in one parse. Go: binary.Read returns an error per field, easy to assign and ignore. EK9: a short read is UNSET and does not move the cursor, so the ordinary and malformed paths are the same code; endianness is in the method name and cannot be defaulted; slice gives a hard boundary for nested structures.

Keywords: buffer, tls, cursor, protocol, parse, underflow, bytesreader, littleendian, slice, binary, endian, bigendian