How long is a String with an emoji in it in EK9?

← Common String Operations · Ref: Q2285

A String is a sequence of Unicode code points, and a Character is one code point. So every length, index and width counts code points - never bytes, never UTF-16 units - on every back end:

  latin <- length "café"   //4
  mixed <- length "a😀b"   //3: the emoji is ONE characterr
  smile <- '😀'             //a Character literal can be any characterr

INDICES AND WIDTHS

indexOf, count, subString, #< and #>, first(), last(), the iterator ('for character in text') and the paddings (leftPadded, rightPadded, center, zeroPad) all work in code points, so subString can never cut a character in half and a padded column of names lines up by characters.

CODE POINTS AS NUMBERS

  point <- smile.codePoint()    //128512 (U+1F600)
  accented <- Character(233)    //'é'

Character(Integer) is unset outside 0 to 1114111 (U+10FFFF) and for the surrogates U+D800 to U+DFFF, which are halves of a character rather than characters. ++ and -- step by code point and give unset on reaching one.

ORDER

Strings and Characters order by code point, and <=> answers -1, 0 or 1.

ESCAPES

A unicode escape is four hex digits ('\u00e9'). A character above U+FFFF is written as itself, or in a String as its two surrogate escapes together ("\uD83D\uDE00"); a surrogate escape on its own is E07904.

WHAT IS NOT A CODE POINT

A user-perceived character can be several code points - a letter and a combining accent, a flag, an emoji with a skin tone - and each of those code points counts as one.

See Q170 for length and empty. See Q1192 for Character operations.

Example

#!ek9
defines module qa.stringops.codepoints

  defines function

    described() as pure
      -> text as String
      <- rtn as String: `[${text}] length ${length text} first ${#< text} last ${#> text}`

  defines program

    CodePoints()
      stdout <- Stdout()
      stdout.println(described("cafe"))
      stdout.println(described("café"))
      stdout.println(described("a😀b")))

      smile <- '😀'
      stdout.println(`smile is code point ${smile.codePoint()}`)
      stdout.println(`code point 233 is ${Character(233)}`)

      greeting <- "hi 😀!"
      stdout.println(`after the emoji: ${greeting.subString(4)}`)
      stdout.println(`emoji at ${greeting.indexOf(smile)}`)
      stdout.println(`[${greeting.leftPadded(8)}]`)

      count <- 0
      for character in greeting
        count++
      stdout.println(`characters iterated: ${count}`)

Common mistakes

E07904 — A surrogate escape is half of a character above U+FFFF, not a character. Write the character itself, or in a String give both halves together. See ek9 -h E07904 for details.

Incorrect:

      smile <- '\uD83D'

Correct:

      smile <- '😀'
Other ways to ask this
  • Does EK9 count characters or bytes in a String?
  • What is a Character in EK9 - a byte, a UTF-16 unit or a code point?
  • How do I get the Unicode code point of a Character in EK9?
  • How do I make a Character from a code point number in EK9?
  • Can subString cut an emoji in half in EK9?

Coming from another language?

Java and JavaScript: length() and charAt() count UTF-16 units, so an emoji is 2; codePointCount() and codePoints() count code points. Python 3: len() counts code points, ord()/chr() convert. Rust: len() counts UTF-8 bytes, chars().count() counts code points; char is a scalar value. Go: len() counts bytes, a rune is a code point. Swift: count counts grapheme clusters. C#: Length counts UTF-16 units. EK9: length, every index and Character are code points; codePoint() and Character(Integer) are ord() and chr().

Keywords: Character, UTF-16, UTF-8, Unicode, chr, code point, codePoint, emoji, length, ord, subString, surrogate