Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Unicode, a 💌

Unicode, a 💌

Florian Gilcher

July 02, 2015
Tweet

More Decks by Florian Gilcher

Other Decks in Programming

Transcript

  1. Let’s talk basic Strings Strings are: a memory abstraction saved

    as bytes, packed semantically mapped to characters
  2. What are Strings used for? Strings are used for: Human

    readable, semi-structured, binary data Words and Text
  3. string = "user/c9e4e625-b872-4e5b-9d6c-2d8dde25bbe1" string.bytes #=> [117, 115, 101, 114, 47,

    99, 57, 101, # 52, 101, 54, 50, 53, 45, 98, 56, 55, 50, 45, # 52, 101, 53, 98, 45, 57, 100, 54, 99, 45, 50, # 100, 56, 100, 100, 101, 50, 53, 98, 98, 101, 49]
  4. Information content An address of an object In a readable

    form A property of an object (the type) Structure is implicit, the reader has to know it
  5. Operations on strings Determining length Indexing into the string (e.g.

    get characters 0-3) Parsing for structured information Splitting Merging
  6. text = <<COC Podstawowym celem wszelkich konferencji i grup użytkowników

    powołujących się na ten Kodeks postępowania jest otwartość na jak największą liczbę osób o jak najbardziej urozmaiconych i różnorodnych korzeniach. COC
  7. Information content The language (Polish) Pronounciation information (“o” vs. “ó”)

    A statement (“We want to be inclusive”) Sentences, Words, syllables, etc.
  8. Strings have many uses Strings - the memory abstraction -

    have many uses. They can be data containers, hold semi-structured data or hold actual text. Each of those uses needs to be treated differently.
  9. What do character encodings do? “a character encoding is used

    to represent a repertoire of characters by some kind of an encoding system.”
  10. Morse .- / .–. .-. .. – .- .-. -.–

    / –. — .- .-.. / — ..-. / .- .-.. .-.. / - .... . / -.-. — -. ..-. . .-. . -. -.-. . ... / .- -. -.. / ..- ... . .-. / –. .-. — ..- .–. ...
  11. US-ASCII 7 Bits, up to 128 characters Encodes the basic

    latin alphabet (a-zA-Z) Punctuation Additional special chars
  12. Special chars? A notification to the user A notification to

    the program A notification for the hardware
  13. Characters are not always text! The most common mistake in

    text handling is to think that characters are all like “a” and “b”. They encode many things.
  14. Unicode is a Text encoding Defines a vast set of

    characters Assigns them to numeric codes Includes almost all characters used in writing systems... ...plus their control instructions. (e.g. right to left writing)
  15. About unicode Many special encodings were developed for multiple writing

    systems Unicode unifies (almost) all Maintained by the Unicode Consortium
  16. Full members of the Consortium Adobe Apple Google Huawei IBM

    Microsoft Ministry of Awqwaf and Religious Affairs of the Sultanate of Oman Oracle SAP Yahoo!
  17. Very brief history US-ASCII-compatible (UTF-8) Unicode unifies (almost) all A

    universal character list Defines numeric values (Codepoints) for every of those Maintained by the Unicode Consortium
  18. Unicode is not an encoding Unicode is a machine-readable description

    of many characters, not an encoding. Unicode has multiple ways to be encoded. Examples: UTF-8,16,32 UCS-2 (warning, this is what Postgres uses!) WTF-8 (for mapping between weird Unicode implementations)
  19. Consequences Characters have changing (memory) width Jumping in the String

    encoding the Text means having to read it left-to-right Compatibility with the most important legacy encoding.
  20. So why not fixed-width? Wouldn’t it be better to have

    all characters in a fixed width, which would allow us to skip forward and back without reading the string? UTF-32 is fixed-length and encodes all Unicode codepoints
  21. Composition What is an ¨ u even? A character on

    its own? A “u” with a trema on top?
  22. Composition Even with fixed-length encoding, reading our desired info out

    of a String is still left-to-right. Reading a too low number of bytes might change the meaning of a character Reading from some point in between may miss context and yield non-sensical data.
  23. Operations become non-trivial string1 = "¨ u" string2 = "\u0075\u0308"

    string1 == string2 #=> false Unicode says they are equal.
  24. Normalization All Unicode Strings need to be normalized before comparison.

    Normalization ensures ambiguities to be mapped to one. Normalization is costly. There are multiple Normalization strategies.
  25. ”Fully Unicode compatible Strings” Many languages boast themselves as fully

    compatible. This is deceiving. You don’t want many of the text operations on Strings.
  26. Unicode upcase/downcase Because it is the most practically useless feature

    ever. Upcase and downcase are operations for using Strings as program labels. Don’t use them on text.
  27. When handling text, learn about handling text Remember the following:

    folding: Like upcasing, but more general. Folding follows rules like “ß” to “SS” collation: Sorting values by the rules of the respective language locales: always work within the context of a locale.
  28. Unicode metadata Unicode ships with huge metadata collections about characters.

    Type of character (e.g. non-printable, word-separating, whitespace, etc.) Origin of character Variants of character Role (Punctuation, etc.)
  29. How much support to expect from a programming language? Validity

    checks (are all characters even Unicode?) Transcoding, conversion Knowledge of character boundaries Knowledge about character metadata in Regexes
  30. Social aspects The Unicode can be racist (Emoji skin colors)

    The Unicode can be sexist (Emoji figures assigning roles to gender) The Unicode can be normative (Decides about correct writing and display) The Unicode can have severe social impact (for people where the name is not encodable)
  31. Social aspects “I Can Text You A Pile of Poo,

    But I Can’t Write My Name” https://modelviewculture.com/pieces/ i-can-text-you-a-pile-of-poo-but-i-cant-write-my-name “Han Unification” https://en.wikipedia.org/wiki/Han_unification
  32. Why I love Unicode Because it codifies one of the

    oldest cultural techniques of humans.
  33. Why I love Unicode Because it is inseperable from its

    social impact, for better or worse.