Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Parsing case study in Go / Go Conference mini 2...

Parsing case study in Go / Go Conference mini 2023 Winter IN KYOTO

Avatar for Ken’ichiro Oyama

Ken’ichiro Oyama

December 01, 2023
Tweet

More Decks by Ken’ichiro Oyama

Other Decks in Technology

Transcript

  1. Parsing case study in Go খࢁ݈Ұ࿠ / GMO PEPABO inc.

    2023.12.2 Go Conference mini 2023 Winter IN KYOTO 1
  2. 6 ຊηογϣϯʹ͓͚Δύʔε ύʔε • ຊηογϣϯʹ͓͚Δύʔεͱ͸ʮจࣈྻ΍όΠτྻ͔Βҙຯͷ͋Δߏ଄ԽσʔλΛநग़͠׆༻ ͢Δ͜ͱʯΛࢦ͢ɻҰൠతͳʮύʔεʯͷఆٛΑΓ΋গ͠޿͍ղऍɻ • จࣈྻ΍όΠτྻ … string,

    []byte, io.Reader • ߏ଄Խσʔλ … struct, map, slice, chan • ຊηογϣϯͰ͸σγϦΞϥΠζ΍σίʔυ΋ؚΉʢͳͷͰɺ͋·Γʮύʔεʯͷఆٛʹͩ͜ΘΒͳ͍Ͱ΋Β͑Δͱ خ͍͠Ͱ͢🙏 ʣɻ
  3. 10 README.mdͷҰ෦͚ͩࠩ͠ସ͑Δ github.com/k1LoW/repin • README.mdͳͲɺෳ਺ߦʹΘͨΔจࣈྻͷҰ෦͚ͩΛࠩ͠ସ͑ΔͨΊͷπʔϧ/ϥΠϒϥϦ • ։࢝ΩʔϫʔυͱऴྃΩʔϫʔυΛࢦఆ͢ΔͱͦͷΩʔϫʔυؒͷจࣈྻΛࠩ͠ସ͑Δ • ྫ: ։࢝Ωʔϫʔυ

    ```console ɺऴྃΩʔϫʔυ ``` ʹͯ͠ίϚϯυͷϔϧϓͷग़ ྗΛ࠷৽ʹࠩ͠ସ͑ • ॲཧͷಛ௃ • ͻͱ͔ͨ·Γͷෳ਺ߦͷจࣈྻΛೖྗͱͯͦ͠ͷ్தͷจࣈྻΛࠩ͠ସ͑Δ • ෳ਺ߦΛ·ͱΊͯॲཧ͢Δඞཁ͕͋Δ • ܾ·ͬͨߏจͱ͍ͬͨ΋ͷ͸ͳ͍
  4. 11 README.mdͷҰ෦͚ͩࠩ͠ସ͑Δ ύʔεͷ࣮૷ • ࣈ۟ղੳΛbufio.ScannerͰ࣮૷ • ಠࣗbufio.SplitFuncΛॻ͘͜ͱͰΩʔϫʔυ֎ͱΩʔϫʔυؒͰ෼ׂ • ࣈ۟ղੳ݁ՌΛ Scan()

    Ͱड͚औ͍ͬͯ͘ʢ []string ʣ • ॱ൪ʹΩʔϫʔυ֎ -> Ωʔϫʔυؒ -> Ωʔϫʔυ֎ -> … • ͋ͱ͸Ωʔϫʔυؒͷจࣈྻ͚ͩࠩ͠ସ͑ͭͭ૊Έཱͯͯग़ྗ͢Δ͚ͩ
  5. 14 ඪ४ग़ྗʹྲྀΕ͍ͯΔϩάΛύΠϓͰड͚औͬͯ৭෇͚͢Δ github.com/k1LoW/colr • ඪ४ೖྗʹྲྀΕΔϩάʹରͯ͠ࢦఆͨ͠จࣈྻʹϚονͨ͠Β͚ͦͩ͜৭෇͚Λ͢Δπʔϧ • ϩά ”Error” ΍ “status:500”

    ͳͲΛڧௐ͢Δ͜ͱͰϩάΛΈ΍͘͢͢ΔͨΊͷ΋ͷ • tail -F /var/log/nginx/access.log | colr POST GET 404 500 search • ॲཧͷಛ௃ • ೖྗ͸ऴ୺͕ෆ໌ͷετϦʔϜɻॱ࣍ॲཧ͍ͯ͘͠ඞཁ͕͋Δ • ୯ʹ৭෇͚ͳͷͰɺϩάͷߏจΛߟྀ͢Δඞཁ͸ͳ͍ɻ1ߦͮͭॲཧͯ͠ྑ͍ɻ
  6. 16 ඪ४ग़ྗʹྲྀΕ͍ͯΔϩάΛύΠϓͰड͚औͬͯ৭෇͚͢Δ ύʔεͷ࣮૷ʢHandle(context.Context, io.Reader) <-chan stringʣ • bufio.NewReader(os.Stdin) -> ReadString(‘\n’)Ͱߦ୯Ґʹ෼ׂ

    • ߦ͝ͱʹ৭෇͚ ( Paint() ) • ಺෦͸࣮࣭strings.ReplaceAll • ৭෇͚݁ՌΛ chan stringʹྲྀ͢ • ʢ࣮͸ʣp.Handle()͕ඪ४ग़ྗΛड͚औͬ ͯ΋ྑ͍ • ʢ࣮͸ʣchan []byte Ͱ΋ྑ͍
  7. 20 VIEWςʔϒϧͰ࢖༻͞Ε͍ͯΔςʔϒϧͷҰཡΛऔಘ͢Δ ςʔϒϧ໊ͷ൑ఆ • “FROM” ͔ “JOIN” ͷ࣍ͷτʔΫϯ͕ςʔϒϧ໊ ͱ͍͏൑ఆͰे෼ •

    ಺ɺ“WITH” Ͱఆٛ͞ΕͨҰ࣌తͳςʔϒ ϧ໊͚ͩআ͘ • VIEWςʔϒϧͷDDLͷಛ௃Λཧղ͓͚ͯ͠͹શ ͯΛτʔΫϯ൑ผ͢Δඞཁ͸ͳ͍
  8. 22 σʔλϕʔεʹྲྀΕΔύέοτΛղੳͯ͠ߏ଄ԽΫΤϦϩάΛ࡞Δ github.com/k1LoW/tcpdp • σʔλϕʔεʹྲྀΕΔύέοτΛղੳͯ͠σʔλϕʔεͷߏ଄ԽΫΤϦϩάΛ࡞੒͢Δπʔϧ • 3ύλʔϯʹରԠ 1. ΫϥΠΞϯτͱσʔλϕʔεͷؒʹTCPϓϩΩγαʔόͱͯ͠ىಈ 2.

    libpcapΛ࢖ͬͯtcpdumpͱಉ͡࢓૊ΈͰύέοτΛΩϟϓνϟ͢Δ 3. tcpdump͕μϯϓͨ͠pcapϑΝΠϧΛղੳ͢Δ • ॲཧͷಛ௃ • ෳ਺ͷΫϥΠΞϯτ͔Βͷ઀ଓ͕͋Δ • 1ͭͷTCPύέοτʹΫΤϦ͕ऩ·Δͱ͸ݶΒͳ͍ • σʔλϕʔεϓϩτίϧҙຯͷ͋Δ௨৴͸ΫϥΠΞϯτ-αʔόؒͰԿճ͔ԟ෮͢Δ • 3ύλʔϯͷύέοτऔಘʹରԠ͢Δඞཁ͕͋Δ͕ύʔεॲཧ͸ڞ௨Խ͍ͨ͠
  9. 27 ͜͜·Ͱͷֶͼ ࣈ۟ղੳ͸έʔεʹΑͬͯ͸ࡶʹͰ͖Δ • ೖྗΛઌ಄͔ΒॱʹಡΈࠐΜͰ͍͘ͷ͕ϕʔγοΫͳख๏ • τʔΫϯʹ෼ׂ͢ΔͨΊʹ͸ઌಡΈͯ͠͸͡ΊͯτʔΫϯछผΛ൑ఆͰ͖Δ͜ͱ͕ଟ͍ • ઌಡΈʹٻΊΒΕΔཁ͕݅ෳࡶͰ͋Ε͹͋Δ΄Ͳࣈ۟ղੳͷ࣮૷͕೉͘͠ͳΔʢײ૝ʣ •

    ҰํͰ • έʔεʹΑͬͯ͸εϖʔε۠੾ΓͰτʔΫϯʹ෼ׂͰ͖ΔͳΒॳख strings.Split ͰࡁΉ৔߹ ΋͋Δ • έʔεʹΑͬͯ͸શͯͷจࣈྻΛτʔΫϯʹ͢Δඞཁ͸ͳ͍ʢۭന΍վߦΛແࢹ͢Δͷͱ ಉ͡ʣ
  10. 29 ͜͜·Ͱͷֶͼ bufio.ScannerʢVS bufio.Reader VS []rune VS []byteʣ • Ұ౓ʹಡΈࠐΉόΠτ਺ͷ্ݶ͕Θ͔͍ͬͯΔ৔߹ʢ΋͘͠͸͋Δఔ౓খ͍͞ͱΘ͔͍ͬͯΔ৔

    ߹ʣbufio.Scannerͷ࢖͍উख͕ྑ͍ • ؆қͳτʔΫϯ෼ׂʹ͓͍ͯ͸bufio.SplitFuncͷ֦ு͕ޮ͘ • ಡΈࠐΉόΠτ਺ͷ্ݶ͕Θ͔͍ͬͯΔͳΒ͹ bufio.Scanner.Buffer() Ͱ্ݶΛઃఆͰ͖Δ • ಡΈࠐΉόΠτ਺͕Θ͔Βͳ͍৔߹ʢେ͖͘ͳΔՄೳੑ͕େ͖͍৔߹ʣ͸ bufio.Reader • จࣈͱͯ͠ύʔε͢Δ৔߹͸[]runeͰʢྫ: github.com/goccy/go-yaml/scannerʣ • ϓϩτίϧͷϔομͳͲαΠζ͕໌Β͔ͳ΋ͷͷ৔߹͸[]byteͰ
  11. 31 ֤छίʔυΧόϨοδϨϙʔτ͔Β৘ใΛऔಘ͢Δ github.com/k1LoW/octocov • ίʔυΧόϨοδΛऩूɾϨϙʔτ͢Δπʔϧ • πʔϧͷಛੑ্ɺαϙʔτ͢ΔΧόϨοδϑΥʔϚοτ͸ଟ͚Ε͹ଟ͍΄Ͳྑ͍ • v0.54.6Ͱͷαϙʔτ …

    Go coverageɺLCOVɺSimpleCovɺCloverɺCoberturaɺJaCoCo • ॲཧͷಛ௃ • େ఍1ͭͷϑΝΠϧ • ΋ͱ΋ͱΧόϨοδϨϙʔτϑΝΠϧ͸ϚγϯϦʔμϒϧͳߏ଄ʹͳ͍ͬͯΔ
  12. 35 YAMLϑΝΠϧͷҰ෦Ͱม਺ల։Λ͢Δ github.com/k1LoW/expand • YAMLϑΝΠϧͷҰ෦Ͱม਺ల։Λ͢ΔػೳΛ࡞Δ͜ͱ͕Ͱ͖ΔϥΠϒϥϦ • ؀ڥม਺ల։ͱ͔ɺGitHub ActionsͷWorkflow YAMLͷΑ͏ͳม਺ల։ͱ͔ •

    ॲཧͷಛ௃ • ม਺ల։ͨ͋͠ͱͷߏจղੳ͸YAMLύʔαʔʹ೚͍ͤͨ • env DEBUG=true ͷจࣈྻ true Λͦͷ··YAMLύʔαʔʹ౉ͯ͠bool஋ͱ͍ͨ͠ • JSONΛ౉ͨ͠ΒYAMLͷߏจͱͯ͠ల։͞Εͯཉ͍͠ • ͱ͸͍͑ɺల։ͨ͠஋ͷߏ଄Ͱల։લͷYAMLͷߏจ໦͕ݩͱେ͖ͣ͘ΕΔΑ͏ͳ࣮૷Ͱ ͸͋ͬͯ͸͍͚ͳ͍ʢ text/template ͳͲʹΑΔม਺ల։Ͱ͸೉͍͠ͱ൑அ ʣ
  13. 41 ੜ੒͞ΕͨGoͷίʔυΛ͞Βʹॻ͖׵͑Δ ύʔεͷ࣮૷ʢRewrite(ctx context.Context, p string, w io.Writer) errorʣ •

    astutil.Apply() Λ࢖ͬͯ௚઀ASTΛࠩ͠ସ͑ ͍ͯΔ • ੜ੒͞ΕͨGoͷίʔυͳͷͰɺ͋Δఔ౓ ϧʔϧԽ͞Ε͍ͯΔͨΊɺࠩ͠ସ͑ର৅Λ ಛఆ͢ΔͨΊͷ൑ఆ৚݅΋ͦ͜·Ͱ࡞Γࠐ Ήඞཁ͕ͳ͍
  14. 43 ຊηογϣϯʹ͓͚Δύʔε ߏจղੳ͕ඞཁͩͬͨΓৄࡉͳࣈ۟ղੳ͕ඞཁͳύʔε͸࣮૷ίετ͕ߴ͍ • ղੳ͍ͨ͠ߏจʹΑ࣮ͬͯ૷͕େ͖͘ҟͳΔʢͦΕ͸ͦ͏ʣ • ϓϩάϥϛϯάݴޠ΍ͦΕʹྨ͢Δ΋ͷ͸ந৅ߏจ໦ʢASTʣʹͳΔ͜ͱ͕ଟ͍ • ͦΕʹྨ͢Δ΋ͷ …

    JSON΍YAML͸ͦͷ࠷ͨΔྫ • ߏจղੳͷख๏͸ଟ͘ఏҊ͞Ε͍ͯΔ • ʢಛʹAST͕ඞཁͳϨϕϧͷʣύʔαΛಠࣗʹ։ൃ͢Δͷ͸͔ͳΓίετ͕ߴ͍ͨΊɺαʔυ ύʔςΟ΋ؚΊద੾ͳύοέʔδʹ೚ͤΔͷ͕ྑ͍
  15. 45 ͦͷଞ: ύοέʔδΛ׆༻͍ͯ͠Δέʔε έʔε͝ͱʹ׆༻͍ͯ͠Δศརύοέʔδ • .ssh/config Λ࢖༻ͨ͠SSH઀ଓʢgithub.com/k1LoW/sshcʣ • ssh_configͷύʔε …

    github.com/kevinburke/ssh_config • .protoϑΝΠϧͷύʔεͱͦΕΛར༻ͨ͠gRPC௨৴ʢgithub.com/k1LoW/grpcstubʣ • .protoϑΝΠϧͷύʔε … github.com/bufbuild/protocompile • OpenAPI SpecʹΑΔHTTPϦΫΤετ/ϨεϙϯεͷόϦσʔγϣϯʢgithub.com/k1LoW/runnʣ • OpenAPIυΩϡϝϯτͷύʔε … github.com/getkin/kin-openapi • άϧʔϐϯάͰ͖ΔProcess exporter for Prometheusʢgithub.com/k1LoW/grouped_process_exporterʣ • /procʢprocfsʣ ͷύʔε … github.com/prometheus/procfs
  16. • ຊηογϣϯͰ͸ʮύʔεʯͷ࣮૷ʹ͍༷ͭͯʑͳέʔεΛ঺հͨ͠ɻ • ࠓճ͸ΑΓଟ͘ͷέʔεΛ঺հ͢Δ͜ͱͰԿ͔͠Βͷؾ͖͕ͮಘΒΕΕ͹ͱ͍͏ظ଴Λࠐ Ίͯߏ੒ͨ͠ɻ • Ұݴʹʮύʔεʯͱ͍ͬͯ΋έʔε͝ͱʹಛ௃͕͋Γɺ࣮૷ํ๏͕͋Δɻ • ҰํͰέʔεΛ၆ᛌͯ͠ΈΔͱڞ௨͍ͯ͠Δύʔλϯ΋͋ΔΑ͏ʹݟ͑Δɻ •

    ύʔεύοέʔδΛఏڙͯ͘͠Ε͍ͯΔOSSϝϯςφͷํʹ͸ײँ͔͠ͳ͍ɻ • ֤έʔεͷ۩ମతͳ࣮૷Λ֬ೝ͍ͨ͠৔߹ɺ঺հͨ͠έʔε͸શͯOSSʹͳ͍ͬͯΔͷͰɺ֤Ϧ ϙδτϦͰࢀরՄೳʢPull Request଴ͬͯ·͢ʂʣɻ 47 ͓ΘΓʹ ͓ΘΓʹ