GoでParserを書く

kamakura.go #6

https://pkg.go.dev/golang.org/x/tools/cmd/goyacc
https://github.com/alecthomas/participle

https://github.com/karupanerura/gqlparser
https://github.com/karupanerura/runetrie

Transcript

1. GoͰParserΛॻ͘ kamakura.go #6 @karupanerura
2. @karupanerura • Perl / Go / Java / TypeScript / etc.. • Software Engineer @ DeNA, Co,. LTD. • ୅දཧࣄ @ Japan Perl Association • GoCon͸ؾ෇͍ͨΒCfPऴΘͬͯͨ
3. Parserྺ • ͦͦ͜͜Parserॻ͍ͨ͜ͱ͋Δ • Perl (JSON5, TOML::Parser, Text::MustacheTemplate) • C (c-geohex3, MySQL::Dump::Parser::XS, etc..) • Go (gqlparser, google-cloud-work fl ow-emulator) • But ମܥతʹֶΜͩ͜ͱ͸ͳ͍ • ΋͠ͳΜ͔͓͔͠ͳ͜ͱݴͬͯͨΒποίϛ΄͍͠Ͱ͢
4. jpa.Promotion()
6. 2024/10/05 (Sat) ͸ͩͯ͜ະདྷେֶ
7. gopher΋׻ܴ
8. ຊ୊
9. ࿩͢͜ͱ • ςΩετΛର৅ͱͨ͠Parserͷجૅ • ط஌ͷΞϧΰϦζϜ΍πʔϧͷ؆୯ͳ঺հ • GoͰ࣮ࡍʹॻ͍ͨParserͷ࣮૷ํ๏ͷ঺հ
10. ࿩͞ͳ͍͜ͱ • ςΩετҎ֎ͷύʔεʢόΠφϦͷύʔεͳͲʣ • ط஌ͷΞϧΰϦζϜͷৄࡉ • طଘͷϥΠϒϥϦͷ࢖͍ํ • ChatGPT-4o
11. Parser
12. ಛఆͷߏจΛղऍ͢Δ΍ͭ
13. Parser is Կ • ߏจʹԊͬͨςΩετΛղऍͯ͠ϓϩάϥϜ͕ѻ͑Δߏ଄ʹ͢Δॲཧ • ߏจղੳɺParseͱݺͿ • GopherʹҰ൪ೃછΈ͕ਂ͍ͱ͜ΖͰ͸GoͷίʔυͷParser? • go/parser • go/ast
14. ߏจΛղੳ͢ΔҰൠతͳखॱ • (Tokenize) ςΩετΛҙຯͷ͋Δ୯Ґʹ·ͱΊΔ • ྫ: "123+456" -> "123", "+", "456" • ͜ΕΛ΍Δ΍ͭ͸LexerɺTokenizerɺScannerͳͲͱݺ͹ΕΔ • (Parse) (্هͷ·ͱ·ΓΛ΋ͱʹ)ͦͷҙຯΛදݱ͢Δߏ଄ʹม׵͢Δ • ྫ: "123", "+", "456" -> Plus{Left: 123, Right: 456} • ڱٛͷParserͱͯ͠Parserͱݴ͍ͭͭ͜Ε͚ͩΛࢦ͢͜ͱ΋͋Δ
15. ͦͷଞɺසग़୯ޠ • AST (Abstract Syntax Tree) ͋Δ͍͸ ந৅ߏจ໦ • ςΩετΛ͋Δߏจͱͯ͠ղऍͨ͠ͱ͖ͷҙຯΛߏ଄ͱͯ͠දݱ • લड़ͷParserͷ݁Ռͱͯ͠͸େ఍͜ΕΛ࡞Δ • BNF • ߏจͷߏ଄Λهड़͢ΔͨΊͷߏจ • ABNFͳͲ೿ੜܗ΋͋Δ
16. ਺ࣜͷ৔߹
17. 1+2
18. Integer(1) Symbol('+') Integer(2)
19. Binary{ OP: '+', L: Integer(1), R: Integer(2), }
20. 123+456
21. Integer(123) Symbol('+') Integer(456)
22. Binary{ OP: '+', L: Integer(123), R: Integer(456), }
23. 1+(2-1)*2+3
24. Integer(1) Symbol('+') Symbol('(') Integer(2) Symbol('-') Integer(1) Symbol(')') Symbol('*') Integer(2) Symbol('+') Integer(3)
25. Binary{ OP: '+', L: Binary{ OP: '+', L: Integer(1), R: Binary{ OP: '*', L: Binary{OP: '-', L: Integer(2), R: Integer(1)}, R: Integer(2), }, }, R: Integer(3), }
26. ͜͏͍͏ײ͡ͷ͜ͱΛ͢Δ
27. ༏ઌ౓͕བྷΉͱͪΐͬͱ໽հ Platt Parserͱ͔ʹ͢Δඞཁ͕͋Δ
28. ࣮૷͖͋Β͔ʹ໘౗ͦ͘͞͏
29. ࿕ใ
30. ࣗಈੜ੒Ͱ͖·͢
31. goyacc • yaccͷgo൛ • yaccͱ͍͏΍͕ͭੲ͔Β͋Δ • BNFΛ࢖ͬͯߏจͷߏ଄ͱͦΕͧΕʹର͢ΔॲཧΛهड़͢Δ • ͍ΘΏΔڱٛͷParser͚ͩΛੜ੒͢Δ • Lexer͸࡞ͬͯ͘Εͳ͍ • ͳΜͱgoʹҰॹʹ͍ͭͯ͘Δ
32. goyaccͷྫ • άάΔͱ͍ͬͺ͍Ͱ͖ͯ·͢
33. goyaccҎ֎ͷࣅͨΑ͏ͳ΍ͭ΋͋Δ • ͜͏͍͏΍ͭΛҰൠʹParser Generatorͱ͍͍·͢ • github.com/alecthomas/participle • Goߏ଄ମλάͳͲͰߏจΛදݱ͢Δ • ANTLR • ͍ΖΜͳݴޠͰੜ੒Ͱ͖Δ • ͳͲͳͲ
34. ͜͏͍͏ͷͰ͍͍͡ΌΜ
35. ऴ
36. ͦ͏͸ͳΒΜ͜ͱ΋͋ΔΜ΍ • goyaccͱ͔΋େ͚֓ͬ͜͏ͩΔ͍ʢ˞ݸਓͷײ૝Ͱ͢ʣ • BNFͷྨ͸ॻ͍͍ͯͯ͋Μ·Γ໘ന͘ͳ͍ʢ˞ݸਓͷײ૝Ͱ͢ʣ • πʔϧ΍ߏจࣗମΛͪΌΜͱֶश͢Δඞཁ͕͋Δ • ୭Ͱ΋΄͍ͦΕͱ࢖͑Δ΄Ͳʹ͸؆୯Ͱ͸ͳ͍ • ࡉ͔͍ͱ͜ΖͷখճΓ͕ޮ͔ͳ͍ • ύϑΥʔϚϯεɾνϡʔχϯά΋೉͍͠৔߹͕͋Δ
37. ※େ֓͸Parser Generator࢖͏΄͏͕ྑ͍ • ParserΛॻ͘ͷ͸େม • ߏจ͕ෳࡶͰ͋Ε͹ෳࡶͰ͋Δ΄ͲԸܙ͸Ͱ͔͍ • ࡉ͔͍ͱ͜ΖͷখճΓ͕ޮ͔ͳͯ͘΋ͳΜͱ͔ͳΔ͜ͱ΋ଟ͍
38. खॻ͖͢Δͧ
39. ୊ࡐ: GQL Parser • GraphQLͰ͸ͳ͘Google Cloudʹ͋ΔಠࣗͷSQLతͳݴޠ • Cloud Firestore datastore mode (Cloud Datastore) ͷΫΤϦݴޠ • https://cloud.google.com/datastore/docs/reference/gql_reference • goyaccʹΑΔ࣮૷͸طग़ • https://github.com/nshmura/dsio/blob/master/gql/parser.go.y
40. GQLͷྫ • SELECT * FROM foo • SELECT __key__ FROM foo • SELECT DISTINCT f1, f2 FROM foo • SELECT DISTINCT ON (f1, f2) f1, f2, f3 FROM foo • SELECT * FROM foo WHERE f1 = 1 AND (f2 = 2 OR f3 = 3)
41. github.com/karupanerura/gqlparser • github.com/karupanerura/datastore-cli Λ࡞͍ͬͯΔͱ͖ʹੜ·Εͨ • github.com/nshmura/dsio ͕ઌߦͯ͠ଘࡏ͚ͨ͠Ͳ஌Βͳ͔ͬͨ • Ͱ΋ͲͷΈͪࣗ෼ͷ࢖͍ํʹ߹Θͳ͍ΠϯλʔϑΣʔεͩͬͨ • GQLΛParseͯ͠ASTΛ࡞ͬͯ͘ΕΔ
42. ※͔͜͜Βgqlparserͷ࣮૷঺հ
43. Lexerͷख࡞Γ • ద౰Ͱ΋ͳΜͱ͔ͳΔ • ෦෼Ұக͢ΔΑ͏ͳΩʔϫʔυͷѻ͍ʹ஫ҙ • ࠷௕ҰகͰѻ͏ͱ͔ϧʔϧΛܾΊΔ • SliceΛฦ͢ΑΓio.Readerతͳ΍ͭΛฦ͢΄͏͕ศར • ઌಡΈΈ͍ͨͳ͜ͱΛͨ͘͠ͳΔ৔߹͕͋Δ • https://github.com/karupanerura/gqlparser/blob/main/lexer.go
44. github.com/karupanerura/runetrie • LexerͰ࠷௕Ұக͢Δͷ͕ͩΔͯ͘࡞ͬͨ • ೚ҙͷจࣈྻͷηοτΛTrie(τϥΠ໦)ʹ·ͱΊΔ • จࣈྻͷઌ಄ʹ࠷௕Ұகͨ͠ཁૉΛฦ͢͜ͱ͕Ͱ͖Δ • LongestMatchPre fi xOf
45. Parserͷख࡞Γ • ؾ߹ͰͳΜͱ͔͢Ε͹ͳΜͱ͔ͳΔ • ͨͩ͠Τϥʔॲཧ͕ύϥύϥͯ͠େม • ಡΈͮΒ͞Λղফ͢ΔͨΊʹ಺෦DSLͬΆ͍ײ͡Ͱॻ͚Δͱ͏Ε͍͠ • ࠓճ͸ࣗ࡞ͨ͠
46. tokenAcceptor type tokenAcceptor interface { accept(tokenReader) error } type tokenReader interface { Next() bool Read() (Token, error) } // ҎԼ͞·͟·ͳtokenAcceptorͷ࣮૷
47. tokenAcceptor࢖༻ྫ func acceptQuery(query *Query) tokenAcceptor { return tokenAcceptors{ skipWhitespaceToken, acceptKeyword("SELECT"), acceptWhitespaceToken, acceptSelectQueryBody(query), } } func acceptSelectQueryBody(query *Query) tokenAcceptor { return tokenAcceptors{ &conditionalTokenAcceptor{ ifAccept: acceptKeyword("DISTINCT"), andThen: acceptDistinctBody(query), orElse: nopAcceptor, }, ...
48. ࢖͍ํ • ߏจʹ͋ΘͤͯtokenAcceptorΛ߹੒͍ͯ͘͠ • ৚݅෼ذΈ͍ͨͳ͜ͱ΋Ͱ͖ΔΑ͏ʹͳ͍ͬͯΔ • ઌಡΈͷ୅ΘΓʹಡΜͩҐஔΛϩʔϧόοΫ͢Δ࢓૊Έ͕͋Δ • https://github.com/karupanerura/gqlparser/blob/main/ parser.go#L396 • ͜͏͍͏ͷҰൠతʹͳΜ͍ͯ͏ΜͰ͔͢ʁ಺෦DSLͷPEGʁ
49. tokenAcceptorͰߏ੒͍ͯ͠ͳ͍෦෼ • ͜ͷ෦෼͚ͩ୯ಠͷtokenAcceptorͱͯ͠ఆٛͯ͠ಠཱ • WHERE۟ͷ෦෼ͷParser • Platt Parserʹͳ͍ͬͯΔ • ANDͱORͱͦΕҎ֎ͷԋࢉࢠͷ༏ઌॱҐͱ͔͕͋Δ • https://github.com/karupanerura/gqlparser/blob/main/ condition_parser.go
50. ͏·͍͜ͱͰ͖ͨ
51. ·ͱΊ • ParserΛॻ͘ͷ͸೉͍͠ • Parser Generatorͱ͔࢖ͬͨ΄͏͕ݡ͍ • جຊతʹखॻ͖͸Ἒͷಓ • ParserΛॻ͘ͷ͸ָ͍͠ • ීஈWeb։ൃͰॻ͔ͳ͍Α͏ͳίʔυ͕ॻ͚Δ • ؾ෼స׵ʹͽͬͨΓͳͷͰ͓͢͢Ί