続・mruby/cにUTF-8 を実装する

Transcript

1. ଓɾmruby/cʹUTF-8 Λ࣮૷͢Δ 2023-08-19 ima1zumi
2. Introduction • @ima1zumi (Mari Imaizumi) • Character encoding lover • IRB and Reline committer • ESM, inc. 2
3. ఏڙ
4. RubyKaigi 2023Ͱ࿩ͨ͜͠ͱ • mruby/cʹUTF-8Λ࣮૷ͯ͠·͢
5. mruby/cͱ͸ • ϝϞϦ࢖༻ྔ͕গͳ͍ • < 40KB • ϫϯνοϓϚΠίϯ͕λʔήοτ • CͱRubyͰ࣮૷͞Ε͍ͯΔ 5 [^3]: https://github.com/mrubyc/mrubyc
6. mruby/c !== mruby 6 Ref: mruby/cͰ࢝ΊΔΦϦδφϧIoTσόΠε࡞Γ https://magazine.rubyist.net/articles/0059/0059-original_mrubyc_iot_device.html
7. mruby/cͷString • RubyͰ͍͏ͱ͜ΖͷASCII-8BIT͔͠࢖͑ͳ͍ • UTF-8΋࢖͑ΔΑ͏ʹ࣮૷த
8. ରԠ͠ͳ͍͍ͯ͘ϝιου • new, +, *, to_i, to_f, to_s, b, clear, chomp, chomp!, dup, empty?, getbyte, lstrip, lstrip!, rstrip, rstrip!, strip!, to_sym, start_with?, end_with?, include?, bytes, ==, <=> • 25ݸ 8
9. ରԠ͕ඞཁ • ࣮૷ࡁ: 12 • index, size(length), slice([]), slice!([]=), insert, inspect, ord • <<, Integer#chr, each_char, encoding, valid_encoding? • Todo: 3 • tr, tr!, split 9
10. Integer#chr • selfΛίʔυϙΠϯτͱΈͳͯ͠จࣈΛฦ͢ • UTF-8ͷ৔߹͸Unicode scalar valueͱΈͳ͢ • Scalar value͸্Լαϩήʔτ(0xD800͔Β0xDFFF)͸ؚ·ͳ͍
11. https://www.unicode.org/versions/ Unicode15.0.0/ch03.pdf
13. 0b0011_0000_0100_0010 (U+3042) ↑scalar value
14. 0b0011_0000_0100_0010 (U+3042) 1110zzzz 10yyyyyy 10xxxxxx ↑scalar value
15. 1110zzzz 10yyyyyy 10xxxxxx ↑scalar value 0b0011_0000_0100_0010 (U+3042)
16. 11100011 10yyyyyy 10xxxxxx ↑scalar value 0b0011_0000_0100_0010 (U+3042)
17. 0b0011_0000_0100_0010 (U+3042) 11100011 10yyyyyy 10xxxxxx ↑scalar value
18. 0b0011_0000_0100_0010 (U+3042) 11100011 10000001 10xxxxxx ↑scalar value
19. 0b0011_0000_0100_0010 (U+3042) 11100011 10000001 10xxxxxx ↑scalar value
20. 0b0011_0000_0100_0010 (U+3042) 11100011 10000001 10000010 ↑scalar value
21. 0b00110000_01000010 (U+3042) 0b11100011_10000001_10000010 "あ".bytes.map { _1.to_s(2) } => ["11100011", "10000001", "10000010"]
23. String#<<
24. String#encoding • mruby/cͷจࣈίʔυ੾Γସ͑͸ϏϧυΦϓγϣϯͰ੍ޚ • ϏϧυΦϓγϣϯΛݟͯฦ͚ͩ͢
25. String#valid_encoding? • ASCII-8BIT͸ͳΜͰ΋OK • UTF-8͸well-formed͔Ͳ͏͔ͷ൑ఆ͕ඞཁ • ϏϧυΦϓγϣϯͰStringͷencodingΛ੾Γସ͍͑ͯΔͨΊɺUTF-8ͱͯ͠well-formed͔Ͳ͏͔ Λ͢΂ͯͷStringͰ֬ೝ͢ΔͱόΠφϦ͕StringʹೖΒͳ͍ͱ͍͏໰୊͕͋Δ • ͔ͱ͍ͬͯෆਖ਼UTF-8͸ڐͨ͘͠ͳ͍ • ηΩϡϦςΟϦεΫ • Stringͷߏ଄ମΛେ͖ͨ͘͘͠΋ͳ͍ • ં஭Ҋ: valid_encoding?ͰνΣοΫՄೳʹ͢Δ
26. Well-formed UTF-8 byte sequences • mruby/c͸ϏϧυΦϓγϣϯͰStringͷencodingΛ੾Γସ͑Δ • શStringͰਖ਼͍͔֬͠ೝ͢ΔͱόΠφϦ͕࡞੒ෆՄೳʹͳΔ • ͔ͱ͍ͬͯෆਖ਼UTF-8͸ڐͨ͘͠ͳ͍ • ηΩϡϦςΟϦεΫ • Stringͷߏ଄ମΛେ͖ͨ͘͘͠΋ͳ͍ • ં஭Ҋ: valid_encoding?ͰνΣοΫՄೳʹ͢Δ
29. valid_encoding? • UTF-8ͷόϦσʔγϣϯΞϧΰϦζϜ͸͍Ζ͍Ζ • [௒ߴ଎ͳUTF-8όϦσʔγϣϯɺrangeΞϧΰϦζϜͷ঺հ - ͖ͯͱ͏ ͳ͍͞ͱɻ΂͐ͨ͹Μ](https://tekitoh-memdhoi.info/views/872) • https://github.com/cyb70289/utf8 • [[2010.03090] Validating UTF-8 In Less Than One Instruction Per Byte](https://arxiv.org/abs/2010.03090) • Θ͔Γ͍ͨ
30. upcase, downcase • ରԠ͠ͳ͍ • Unicodeͷେจࣈɺখจࣈม׵͸Ϛοϐϯάςʔϒϧ͕ඞਢ • "LJ".downcase == "lj" • ϚΠίϯͰͦ͜·Ͱ͍ͨ͠Ϣʔεέʔε͸͋·Γͳͦ͞͏