Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SQS を使った非同期処理の導入

SQS を使った非同期処理の導入

AWS cafeteria #1 〜サイバーエージェント×ゆめみ×クラスメソッド 3社共催LT会〜 で発表した資料です。
https://cyberagent.connpass.com/event/303690/

#AWScafeteria

Keigo Ibaraki

January 25, 2024
Tweet

More Decks by Keigo Ibaraki

Other Decks in Programming

Transcript

  1. ຊ೔ͷ಺༰ 1. ීஈͲΜͳ͜ͱΛ͍ͯ͠Δ͔ʁ 2. ඇಉظॲཧͷಋೖͱ͸ʁ 3. SQS ͱ͸ʁ 4. ࣮૷લޙͷΞʔΩςΫνϟ

    5. SQS બఆ࣌ʹߟྀͨ͠఺ 6. SQS બఆޙʹߟྀͨ͠఺ 7. ෛՙࢼݧͷ࿩ 8. ࠓޙͷల๬ 3
  2. SQS ͱ͸ʁ 16 ࢀߟ : https://aws.amazon.com/jp/sqs/ https://d1.awsstatic.com/webinars/jp/pdf/services/20190717_AWS-BlackBelt_AmazonSQS.pdf ʮΩϡʔʯͷʮϝοηʔδʯΛ औಘ͢ΔΞϓϦέʔγϣϯ ίϯγϡʔϚ

    ϓϩσϡʔα͕ੜ੒͢Δσʔλ ϝοηʔδ ʮΩϡʔʯʹʮϝοηʔδʯΛ ૹ৴͢ΔΞϓϦέʔγϣϯ ϓϩσϡʔα ϝοηʔδΛΩϡʔΠϯά͢Δ ෼ࢄΩϡʔ Ωϡʔ
  3. • ECS ͕௚઀ DynamoDB ͱ Aurora ʹ ʮಡΈऔΓʯʮॻ͖ࠐΈʯΛߦ͍ͬͯͨ લޙͷΞʔΩςΫνϟ •

    ؒʹ SQS ͱ Lambda ΛڬΉ͜ͱͰɺ ʮ༧ࢉফԽʯ෦෼ͷΈΛ੾Γ཭ͨ͠ ࣮૷લ ࣮૷ޙ 18
  4. ඪ४Ωϡʔ vs FIFO Ωϡʔͷ࿩ ͋ • API ΞΫγϣϯ͝ͱʹ 1ඵ͋ͨΓ 300τϥϯβΫγϣϯ

    ͷΫΥʔλΛα ϙʔτ • ϝοηʔδ͸ ඞͣ1ճ ͚ͩ഑৴ ඪ४Ωϡʔ FIFO Ωϡʔ • API ΞΫγϣϯ͝ͱʹ 1ඵ͋ͨΓ ΄΅ແ੍ݶ ͷ API ίʔϧΛαϙʔτ • ϝοηʔδ͸ গͳ͘ͱ΋1ճ ഑৴ 21 ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/quotas-messages.html https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues-exactly-once-processing.html
  5. • FIFO ΩϡʔͷߴεϧʔϓοτϞʔυΛద༻͢Δͱɺ1ඵʹ͖ͭ 3000݅ ͷϦΫΤετʹ ରԠͰ͖Δ ◦ 2023.10.04 ࣌఺Ͱ౦ژϦʔδϣϯͰ͸ඵؒ 9000τϥϯβΫγϣϯ

    • ϝοηʔδάϧʔϓΛ෼ࢄͤ͞Δ͜ͱͰɺͦΕͧΕͷ ϝοηʔδάϧʔϓ͕1ඵ͋ͨΓ 300݅ͷϦΫΤετΛαϙʔτ ͢Δ High Throughput Queue ʹ͍ͭͯ 22 ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/high-throughput-fifo.html https://aws.amazon.com/jp/about-aws/whats-new/2023/10/amazon-sqs-increased-throughput-quota-fifo-high-throughput-mode/
  6. Dead Letter Queue ʹ͍ͭͯ 24 ϓϩσϡʔα͕ϝοηʔδΛ ૹ৴͢ΔͨΊͷओཁͳΩϡʔ ιʔεΩϡʔ ॲཧ͕ࣦഊͨ͠ࡍͷ ૹ৴ઌͱͳΔΩϡʔ

    σουϨλʔΩϡʔ ίϯγϡʔϚ͕ϝοηʔδΛ ड৴͠Α͏ͱ͢Δճ਺ maxReceiveCount ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
  7. ΠϕϯτιʔεϚοϐϯά 28 • ΠϕϯτιʔεʢSQS ͷΩϡʔ౳ʣ͔ΒΠϕϯτΛಡΈऔΓɺLambda ؔ਺Λݺͼग़ ͢ Lambda Ϧιʔεͷ͜ͱ •

    Πϕϯτιʔεͷ SQS ʹϝοηʔδ͕ૹ৴͞Εͨ͜ͱΛݕ஌ͯ͠ɺLambda ؔ਺͕࣮ ߦ͞ΕΔ ࢀߟ : https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/invocation-eventsourcemapping.html
  8. ΠϕϯτιʔεϚοϐϯάͷ batch sizeʢಉ࣌ݺͼग़্͠ݶʣ • ΠϕϯτιʔεʢSQS ͷΩϡʔ౳ʣ͔Β Lambda ͕ಉ࣌ʹಡΈऔΕΔΠϕϯτ਺ • FIFO

    Ωϡʔͷ৔߹͸࠷େ஋͸ 10 • Lambda ͷಉ࣮࣌ߦ਺੍ݶ͕͋ΔͷͰɺ1ճͷ Lambda ىಈͰΑΓଟ͘ͷϝοηʔδ ΛॲཧͰ͖ΔΑ͏ʹͨ͠ 29 ࢀߟ : https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-sqs.html
  9. ΠϕϯτιʔεϚοϐϯάͷ Lambda ࠷େಉ࣮࣌ߦ਺ 30 • ΠϕϯτιʔεʢSQS ͷΩϡʔ౳ʣ͕ಉ࣌ʹݺͼग़ͤΔ Lambda ؔ਺ͷ࠷େ਺ •

    FIFO Ωϡʔͷ৔߹ɺಉ࣌ݺͼग़͠ͷ্ݶ͸ɺʮϝοηʔδάϧʔϓ ID ͷ਺ʯ·ͨ͸ ʮ࠷େಉ࣮࣌ߦ਺ͷઃఆʯͷͲͪΒ͔ খ͍͞ํ ࢀߟ : https://docs.aws.amazon.com/ja_jp/lambda/latest/dg/with-sqs.html
  10. SQS ͷՄࢹੑλΠϜΞ΢τʹ͍ͭͯ 34 • ϝοηʔδ͕ड৴͞Εͨ௚ޙʹɺଞͷίϯγϡʔϚ͕ಉ͡ϝοηʔδͷड৴΍ॲཧΛ Ͱ͖ͳ͘ͳΔظؒͷ͜ͱ • ՄࢹੑλΠϜΞ΢τ͸ Lambda ͷ࣮ߦ࣌ؒͷগͳ͘ͱ΋

    6ഒ ʹઃఆ͞ΕΔ͜ͱ͕ਪ঑ ͞Ε͍ͯΔ ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html
  11. SQS ͷ maxReceiveCount ʹ͍ͭͯ 35 • ίϯγϡʔϚʔ͕ϝοηʔδΛड৴͠Α͏ͱͨ͠ࡍʹɺmaxReceiveCount ΛॳΊͯ ௒͑ͨ࣌ʹ DLQ

    ʹߦ͘࢓༷ʹͳ͍ͬͯΔ • ϝοηʔδ͕ॲཧ͞ΕΔ֬཰ΛߴΊΔʹ͸ maxReceiveCount Λ 5 Ҏ্ʹઃఆ͢Δ͜ ͱ͕ਪ঑͞Ε͍ͯΔ ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html
  12. • ࠷ॳ͸ϝοηʔδάϧʔϓ ID Λ୯Ұʹ͍ͯͨ͠ͷͰύϑΥʔϚϯε͕Ͱͳ͔ͬͨ • SQS FIFO ΩϡʔͷߴεϧʔϓοτϞʔυͰ͸ɺϝοηʔδάϧʔϓ ʹ͖ͭ 1ඵؒ

    300 ݅ͷϦΫΤετΛαϙʔτ͢ΔͷͰɺͦ͜Λద੾ʹ෼ࢄͤͨ͞ Queue ͷ API Limit ͷ্ݶʹର͢Δݟམͱ͠ 38 ࢀߟ : https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dead-letter-queues.html