Character Building - Fun with charsets and encodings

Transcript

1. Christoph Lühr @chluehr / @bephpug 2013 "Fun with charsets and encodings" Character Building ٩(͡๏̯͡๏)۶
2. basilicom
4. �
5. Image source: http://www.flickr.com/photos/stinajonsson/3932774410 CC BY-NC 2.0
6. Charset vs. Encoding
7. Set of Characters [ A B C ... 1 2 3 ... @#$ ] UNICODE / CODE PAGES
8. U+2278 NEITHER LESS-THAN NOR GREATER-THAN
9. U+2620 SKULL AND CROSSBONES
10. U+FDFA ARABIC LIGATURE SALLALLAHOU ALAYHE WASALLAM "May Allah bless him and grant him peace"
11. Encoding = Mapping A = 0x65 UTF-8, ISO-8859-1
12. Single Byte vs. Multi Byte
13. UTF-16 variable length!
14. UTF-16/32 Big- / vs Little-Endian
15. U+FEFF: BOM Byte-Order-Mark zero width non-breaking space
16. U+FFFE: BOM Byte-Order-Mark
17. BOM BOM BOM... UTF8 BOM 0xEF 0xBB 0xBF UTF32BE BOM 0x00 0x00 0xFE 0xFF UTF32LE BOM 0xFF 0xFE 0x00 0x00
18. How to debug? terminal.
19. hexdump -C foo.txt 00000000 48 61 6c 6c 6f 20 62 65 70 68 70 75 67 21 0a 48 |Hallo bephpug!.H| 00000010 69 65 72 20 65 69 6e 20 61 2d 55 6d 6c 61 75 74 |ier ein a-Umlaut| 00000020 3a c3 a4 21 0a 48 69 65 72 20 65 69 6e 20 61 2d |:..!.Hier ein a-| 00000030 6d 69 74 2d 4b 72 69 6e 67 65 6c 3a c3 a5 21 0a |mit-Kringel:..!.| 00000040 0a
20. How to re-encode? iconv -f FROM -t TO
21. PHP? strlen, substr, ...
22. � c3 a4 = ä c3 =
23. PHP! mbstring mb_* iconv_*
24. Transliteration ü => ue ü => u
25. Databases
26. DB - Connection SET NAMES 'UTF8'
27. DB - Storage Table vs. DB
28. DB - Collation ü..rstuvw ... rstuüvw
29. Problems Weird Stuff
30. PHP: Identifiers $Schüssel = new Müsli (T_FRÜCHTE); [a-zA-Z_\x7f-\xff][a-zA-Z0-9_\x7f-\xff]*
31. Different Line-Endings
32. <?php CR LF LF // lets say hello! LF echo "hello" LF
33. <?php CR LF LF // lets say hello! LF echo "hello" LF <?php // lets say hello! LF echo "hello"
34. Diacritics ü => u+" U+00FC ü c3 bc LATIN SMALL LETTER U WITH DIAERESIS U+0075 u 75 LATIN SMALL LETTER U U+0308 _̈ cc 88 COMBINING DIAERESIS
35. Advice
36. Use UTF-8
37. UTF-8
38. PHP/Server header( 'Content-type: text/html; charset=utf-8' ); HTML <meta http-equiv="Content-Type" content=" text/html; charset=UTF-8" >
39. Database SET NAMES UTF8 (or PDO) [mysql] default-character-set=utf8
40. Contact Christoph Lühr eMail: luehr@r-pentomino.de, christoph.luehr@basilicom.de Twitter: @chluehr Slides license Attribution-NonCommercial-ShareAlike 3.0 http://creativecommons.org/licenses/by-nc-sa/3.0/ Thanks! Questions? U+3020 POSTAL MARK FACE
41. Links • Kore Nordmann (FAQ!) http://kore-nordmann.de/blog/0082_charset_versus_encoding.html http://kore-nordmann.de/blog/php_charset_encoding_FAQ.html • Misc. Resources http://www.iana.org/assignments/character-sets/character-sets.xml http://www.joelonsoftware.com/articles/Unicode.html http://www.unicode.org/charts/ http://t-a-w.blogspot.de/2008/12/funny-characters-in-unicode.html http://www.utf8-zeichentabelle.de/unicode-utf8-table.pl?number=1024 http://stackoverflow.com/questions/3417180/exotic-names-for-methods- constants-variables-and-fields-bug-or-feature