Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AWSの「隙間」を埋める隙間家具 OSS 開発 / AWS DevDay Tokyo 2019

AWSの「隙間」を埋める隙間家具 OSS 開発 / AWS DevDay Tokyo 2019

Avatar for FUJIWARA Shunichiro

FUJIWARA Shunichiro

October 03, 2019
Tweet

More Decks by FUJIWARA Shunichiro

Other Decks in Technology

Transcript

  1. ྫ: RDS for MySQL ͷྺ࢙ 2009-10 ϩʔϯν 2010-05 Multi-AZػೳ 2015-12

    ೚ҙλΠϜκʔϯઃఆ 2018-01 CloudWatch Logsʹϩάग़ྗػೳ αʔϏεͷࠜװʹؔΘΔίΞػೳ (ྫ:Multi-AZ) ͸ ͘͝ॳظʹ࣮૷͞ΕΔ͕… ޻෉ͰԿͱ͔ͳΔػೳ (ྫ:λΠϜκʔϯ) ଞαʔϏεͱͷ࿈ܞศརػೳ (ྫ:Logsग़ྗ) ͸ޙճ͠ʹ͞Ε͕ͪ ← 伱ؒ
  2. ޻෉ͰԿͱ͔ͳΔػೳ͸ޙճ͠ʹͳΓ͕ͪ ྫ͑͹λΠϜκʔϯΛ JST ʹઃఆ͍ͨ͠৔߹ ύϥϝʔλάϧʔϓͷ init_connect ͰҎԼͷ SQL Λࢦఆ1 SET

    SESSION time_zone = CASE WHEN POSITION('rds' IN CURRENT_USER()) = 1 THEN 'UTC' ELSE 'Asia/Tokyo' END; rds ͔Β࢝·ΔϢʔβʔ(RDS಺෦Ͱར༻͢Δ΋ͷ)Ͱ͸ UTC ͦΕҎ֎ͷϢʔβʔʹ͸ Asia/Tokyo (JST) ʹઃఆ͢Δ Bad Know-how... 1 https://qiita.com/j3tm0t0/items/089ef96ba131df079ca4
  3. 伱ؒͷ࣮ྫ - S3 ͱ Redshift ͷؒ Redshift ʹσʔλΛऔΓ͜Ήʹ͸ S3 ͔Βͷίϐʔ͕ඞਢ

    Redshift ʹ઀ଓ͠ɺऔΓࠐΉΦϒδΣΫτΛࢦఆͯ͠ COPY ΫΤϦΛൃߦ 2012-11 ʹ Redshift ͕ϩʔϯνͯ͠3೥ S3͔ΒܧଓతʹऔΓ͜ΉϚωʔδυͳํ๏͸ଘࡏ͠ͳ͔ͬͨ
  4. 伱ؒՈ۩ OSS ͷ࣮ྫ - Rin github.com/fujiwara/Rin S3 Πϕϯτ௨஌Ͱ SQS ʹૹ৴

    SQS ͷϝοηʔδΛݩʹ Redshiftʹ COPY Λൃߦ ͯ͠औΓࠐΈΛߦ͏ Go ੡πʔϧ 2015.05 ʹ։ൃ
  5. ݩʑผͷํ๏Λ࢖༻͍ͯͨ͠ fluent-plugin-redshift Fluentd ͷ output plugin ϝοηʔδͷoutputॲཧͱͯ͠ S3 ΁Ξοϓϩʔυ Redshift

    ΁ͷCOPYൃߦ Λಉظతʹ࣮ߦ Fluentd ʹϩάΛૹΔ͚ͩͰ Redshift ·ͰऔΓ͜ ·ΕΔͷͰศརͱࢥͬͯಋೖ
  6. Rin ͷઃܭํ਑ S3 ΁ͷΞοϓϩʔυ͸ଞʹ೚ͤΔ (fluentd) S3, SQS ͷՄ༻ੑ͸େมߴ͍ Redshift ͷμ΢ϯλΠϜ͸ൺֱతେ͖͍

    ϛεϚον෦෼ΛҰ౓ʹॲཧ͠ͳ͍͜ͱͰ ϦτϥΠΛ༰қʹ Redshift ΁ͷ COPY ൃߦʹಛԽ͢Δπʔϧ
  7. Rin ͷઃఆϑΝΠϧ queue_name: my_queue_name # SQS queue name targets: -

    s3: region: ap-northeast-1 bucket: my.test.bucket key_prefix: test/foo/ redshift: host: redshift.example.com port: 5439 dbname: test user: test password: xxxxxxxx schema: public table: foo sql_option: "JSON 'auto' GZIP"
  8. ॊೈͳऔΓࠐΈઃఆ targets: - s3: bucket: my.test.bucket key_regexp: test/schema-([a-z0-9]+)/table-([a-z0-9]+)/ redshift: schema:

    $1 # ^ͷਖ਼نදݱͰcaptureͨ͠஋Λల։ table: $2 େ఍ͷΞϓϦέʔγϣϯͰ͸औΓࠐΈςʔϒϧ͸ෳ਺ʹͳΔ ॊೈʹऔΓ͜ΊΔΑ͏ʹಈతઃఆΛՄೳʹ
  9. Rin ͷण໋͸௵͑ͨͷ͔? ࣮͸·ͩੜ͖͍ͯ·͢ Fluentd͔ΒͷऔΓࠐΈ͸Firehoseܦ༝ͰΑ͘ͳ͕ͬͨ ผͷϢʔεέʔεͰ࢖͑Δ ELB / ALB ͷϩά͕ S3

    ʹ഑ஔ͞ΕΔ΋ͷΛஞ࣍औΓࠐΈ͢Δͷʹ׆༂த ϢʔεέʔεΛߜͬͨద౓ͳ൚༻ੑ͕૗ޭ͍ͯ͠Δ
  10. 伱ؒՈ۩ OSS ͷྫ - s32cs Amazon CloudSearch 2012-04 ϦϦʔε 2019-10

    ݱࡏ ϚωʔδυͰσʔλΛܧଓతʹऔΓ͜Ήํ๏͸ͳ͍ (console/ aws-cli / API / HTTP POST ͷΈ)
  11. CloudSearch ΁ͷ౤ೖॲཧ CloudSearch ΁͸ 5MB ҎԼͷ JSON ഑ྻΛ౤ೖ ࡉ͔͍୯ҐͰ౤ೖ͢ΔͱύϑΥʔϚϯε͕ྼԽ͢Δ s32cs

    = ഑ྻܗࣜ΁ͷՃ޻ɺ෼ׂͱ HTTP POST Λ΍ͬͯ͘ΕΔ͚ͩͷπʔϧ {"id": "123", "type": "add", "fields": {"title": "hoge", "message": "Θ͍Θ͍"]}} {"id": "345", "type": "delete"} ↓ [ {"id": "123", "type": "add", "fields": {"title": "hoge", "message": "Θ͍Θ͍"]}}, {"id": "345", "type": "delete"} ]
  12. Firehose → S3 ͷΠϕϯττϦΨ͔Βͳʹ͔͢Δύλʔϯ n෼͝ͱʹσʔλॲཧ͢ΔόονΛΠϕϯτυϦϒϯʹม׵͍ͯ͠Δ s32cs Ҏલ cronͰաڈ5෼ͷσʔλΛDB͔ΒऔಘɺՃ޻ɺHTTP POST Ͱ౤ೖ͢Δόον

    σʔλऔಘɺՃ޻ɺ౤ೖ͕ҰମͰෆՄ෼ s32cs Ҏޙ ੜ੒͸ΞϓϦέʔγϣϯͰͷϩάૹ৴ 5෼͝ͱʹσʔλΛ·ͱΊΔͷ͸ Firehose ·ͱ·ͬͨσʔλΛՃ޻ͯ͠౤ೖ͢Δ͚͕ͩ࢓ࣄ
  13. Firehose ͰͷετϦʔϜ → S3ΠϕϯττϦΨॲཧ n෼͝ͱͷcronΛ୅ସ ετϦʔϜΛn෼͝ͱͷchunkͰS3ʹు͖ग़͢ॲཧ = Firehose S3 ʹ͸

    ࢦఆ࣌ؒ (60ʙ900ඵ) ΋͘͠͸ࢦఆαΠζ (1ʙ128MB)ͷ ઌʹୡͨ͠৚݅ͰΦϒδΣΫτ͕ੜ੒͞ΕΔ ෼ࢄॲཧ͕ՄೳʹͳΔ όονॲཧଆʹঢ়ଶΛ࣋ͭඞཁ͕ͳ͍ ҆৺ͯ͠ΦϯϝϞϦॲཧ͕Ͱ͖Δʂ
  14. ϚωʔδυαʔϏεʹ΋͍Ζ͍Ζ͋Δ Ϋϥ΢υωΠςΟϒ = ΠϯελϯεΛҙࣝ͠ͳͯ͘Α͍(Ͱ͖ͳ͍) SQS, S3, CloudWatch, Labmda, DynamoDB... Ϋϥ΢υωΠςΟϒʹ͍ۙ

    = Πϯελϯε͸ᐓؾʹݟ͑Δ͕εέʔϧ͕ࣗಈ ELB, CloudSearch... Ϋϥ΢υωΠςΟϒʹͳΓ͖Εͳ͍=Πϯελϯε͕ݟ͑εέʔϧ͕ࣗಈͰͳ͍ RDS, ElastiCache... Ϋϥ΢υωΠςΟϒͳ΋ͷΛ׆༻͍ͯ͘͠ͱεέʔϧ͠΍͍͢ Ϋϥ΢υωΠςΟϒͳࢥߟ๏͸େࣄ
  15. 伱ؒՈ۩ OSS ͷྫ - ssmwrap github.com/handlename/ssmwrap SSM Parameter Store ͷ஋Λ؀ڥม਺ʹઃఆͯ͠ίϚϯυΛ

    exec ͢Δ wrapper ੈͷதʹྨࣅ඼͸͍ͬͺ͍͋Δ͕͋͑ͯ… github.com/remind101/ssm-env github.com/okzk/env-injector github.com/jamietsao/aws-ssm-env
  16. ssmwrap $ ssmwrap -path /prod/ -- mycommand ྫ͑͹ SSM ύϥϝʔλετΞʹ

    /prod/DB_PASSWORD, /prod/API_TOKEN ͕ೖ͍ͬͯΔ৔߹ ؀ڥม਺ DB_PASSWORD, API_TOKEN ʹ஋Λઃఆͯ͠ mycommand Λ࣮ߦ 2018.07 ECS task ʹ SSM ͷ஋Λ౉ͨ͢Ίʹ։ൃ
  17. ssmwrap ͷ͍͍ͱ͜Ζ exec ͢Δ(ࢠϓϩηεͱͯ͠ىಈ͠ͳ͍) wrapper ͳͷͰ ίϯςφͷ entrypoint ʹ௚઀ࢦఆͰ͖Δ #

    ssmwrap -path /prod/ -- command ͱಉ౳ͷىಈॲཧ ENTRYPOINT ["/usr/bin/ssmwrap"] ENV SSMWRAP_PATHS=/prod/ CMD ["--", "mycommand"] ϦτϥΠ͕ࢦఆͰ͖Δ (ྫ -retries=3 or SSMWRAP_RETRIES=3) ύϥϝʔλετΞ͕ΤϥʔΛฦͨ͠৔߹ɺϦτϥΠͰ͖ͳ͍ͱ
  18. ssmwrap ͷར༻Ձ஋͸ͳ͘ͳͬͨͷ͔? ECS Ҏ֎Ͱ΋࢖͑Δ EC2 Ͱಈ࡞͢Δ shell script ͷ shebang

    ʹࢦఆͨ͠Γ #!/usr/bin/ssmwrap -path /prod/ -- /bin/sh echo $DB_PASS ... AWS֎ (ྫ͑͹ CircleCI) Ͱͷ࣮ߦʹ΋ಉ༷ʹ࢓ࠐΊΔ γεςϜ͢΂͕ͯ ECS Ͱͳ͍ঢ়ଶͰ͸·ͩ·ͩར༻Ձ஋͸͋Δ
  19. GoͷϥΠϒϥϦͱͯ͠΋࢖͑Δ ͜Ε͕໌ه͞Ε͍ͯΔྨࣅπʔϧ͸গͳ͍(͸ͣ) godoc.org/github.com/handlename/ssmwrap#Export ϥΠϒϥϦͱͯ͠࢖͑Δ → Lambda Ͱ࢖͑Δ import "github.com/handlename/ssmwrap" err

    := ssmwrap.Export(ssmwrap.ExportOptions{ Paths: []string{"/prod/"}, Retries: 3, }) // ී௨ʹ؀ڥม਺Λࢀর͢Δ͚ͩ dbpass := os.Getenv("DB_PASS")
  20. ssmwrap ͷࢥ૝ Ϛωʔδυʹͳͬͨͱ͖ʹऔΓ֎͠΍͍͢Α͏ͳઃܭ ؀ڥม਺/ϑΝΠϧʹSSMύϥϝʔλετΞͷ஋Λઃఆ͢Δ͜ͱʹूத͢Δ ECSɿsecrets ͰαϙʔτࡁΈ Lambda: ͦͷ͏ͪͰ͖Δ(͖ͬͱ) ϑΝΠϧʹॻ͖ग़͔ͯͯ͠͠Βίϯςφىಈ →

    k8sͰ͸Ͱ͖Δɺͭ·ΓECS΋… কདྷϚωʔδυʹͳΓͦ͏ͳ෦෼ΛΞϓϦέʔγϣϯͷ΂ͨॻ͖Ͱղܾ͠ͳ͍ খ͍͞πʔϧ/ϥΠϒϥϦʹ੾Γग़͓ͯ͘͠ औΓ֎͕͠༰қʹͳΔ
  21. OSS ͱͯ͠࡞Δཧ༝(3) ຐվ଄൛͕૿৩͢ΔͷΛ๷͙ ʮGitHubͰόΠφϦఏڙͯ͠ΔͷͰͦΕΛ࢖͍ͬͯͩ͘͞ʯ ʮRubyGems / CPAN / npm /

    etc. ͔ΒΠϯετʔϧ͍ͯͩ͘͠͞ʯ ಠཱͨ͠ύοέʔδʹͳͬͯͳ͍ͱίϐϖ͞ΕΔ ίϐϖޙʹվ଄͞ΕΔͱ৽ػೳ΍όάϑΟοΫεʹ௥ैͰ͖ͳ͍