Python × AWS × Serverless 初学者が次の一歩を踏み出すためのテクニック/Python & AWS & Serverless - Step to the next stage from a beginner

PyCon JP 2020 ( https://pycon.jp/2020/en/ ) の登壇資料

https://pycon.jp/2020/en/timetable/?id=203588

(2020/08/30 追記) Qiita 補足ブログ書きました https://qiita.com/hassaku_63/items/875ba4694538c6484526

Transcript

1. Python × AWS × Serverless ॳֶऀ͕࣍ͷҰาΛ౿Έग़ͨ͢ΊͷςΫχοΫ Python & AWS & Serverless - Step to the next stage from a beginner PyCon JP 2020 #pyconjp_2 @hassaku
2. ͸͡Ίʹ
3. Speaker hassaku (Twitter: @hassaku_63) Serverworks Co., Ltd.
4. Speaker ࣮ફ AWS CDK - TypeScript Ͱ
 Πϯϑϥ΋ΞϓϦ΋ʂ ڞஶ͠·ͨ͠ʢٕज़ॻయ8ʣ BOOTH: https://booth.pm/ja/items/1881928
5. Serverworks • AWS ಛԽͷ SIer • ʮΫϥ΢υͰɺੈքΛɺ΋ͬͱɺ͸ͨΒ͖΍͘͢ʯ • AWSύʔτφʔωοτϫʔΫ
 ʮϓϨϛΞίϯαϧςΟϯάύʔτφʔʯೝఆ
6. ൃදͷલʹ - ͓͜ͱΘΓ - • ࠂ஌ࡁΈͷΞδΣϯμ͔Βएׯͷมߋ͕͋Γ·͢ʢݟղͷมԽʹ൐͏΋ͷʣ • ૝ఆಡऀ • AWS ʹ͍ͭͯɺଟগͷར༻ܦݧ͕͋Δ • ʮαʔόʔϨεʯͷ֓೦΍ɺྑ͞͸ͳΜͱͳ͘஌͍ͬͯΔ • Lambda Function ΛσϓϩΠͨ͜͠ͱ͕͋Δʗ֓೦Λཧղ͍ͯ͠Δ • ʮαʔόʔϨεʯೖ໳هࣄͷ࣍ʹԿΛͨ͠Βྑ͍ͷ͔Θ͔Βͳ͍ • Զͨͪ͸งғؾͰαʔόʔϨεΛ΍͍ͬͯΔ
7. Agenda 1. AWSʹ͓͚ΔαʔόʔϨεΞϓϦέʔγϣϯߏஙͷجຊ 2. Serverless Framework جຊͷ͓͞Β͍ 3. ϓϩδΣΫτߏ੒ͷྫ 4. ʮεςʔδʯͷ؅ཧ 5. ϩΪϯά 6. ·ͱΊ
8. AWSʹ͓͚ΔαʔόʔϨε
 ΞϓϦέʔγϣϯߏஙͷجຊ
9. αʔόʔϨεͷԿ͕೉͍͠ʁ • ֶशίετͷߴ͞ • ֓೦ʹೃછΈ͕ͳ͍ • ʮೖ໳ॻͷ࣍ʯ໰୊
10. αʔόʔϨεͳʮࢥߟʯ • ඇಉظ • ΠϕϯτυϦϒϯ • εςʔτϨε
11. αʔόʔϨεͳʮࢥߟʯ • ඇಉظ • ΠϕϯτυϦϒϯ • εςʔτϨε
12. αʔόʔϨεͳొ৔ਓ෺ • ϒϩʔΧʔ • Πϕϯτϋϯυϥ • σʔλετΞ CSPLFS FWFOU
    IBOEMFS %BUB
    TUPSF CSPLFS e.g SQS, SNS, Kinesis e.g DynamoDB Lambda
13. See also • [PyCon JP 2017] The theory of Serverless development by Python (ཧ࿦ ͔ΒֶͿPythonʹΑΔαʔόϨε։ൃ) @Masashi Terui
14. Serverless Framework جຊͷ͓͞Β͍
15. Serverless Framework • OSSͷσϓϩΠπʔϧ • Infrastructure as Code (IaC) • ϚϧνΫϥ΢υରԠ • Serverless Frameworkͷ࢖͍ํ·ͱΊ https://www.serverless.com/
16. Serverless Framework 1. ϓϩδΣΫτ࡞੒ 2. ίʔυΛॻ͘ (yaml, python) 3. σϓϩΠύοέʔδΛ࡞੒ 4. σϓϩΠ … sls create … sls deploy (or sls package) … sls deploy
17. σϓϩΠͷಈ͖ > sls deploy —stage dev
18. σϓϩΠͷಈ͖ > sls deploy —stage dev $MPVE'PSNBUJPO
    ςϯϓϨ
    -BNCEB
    σϓϩΠύοέʔδ
    Λ࡞੒
19. σϓϩΠͷಈ͖ S3 όέοτΛੜ੒͠ɺ
 ҎԼΛΞοϓϩʔυ ɾLambda ͷσϓϩΠύοέʔδ
 ɾCloudFormation tempalte "84
    4
    ΁Ξοϓϩʔυ
20. σϓϩΠͷಈ͖ $MPVE'PSNBUJPO
    σϓϩΠ CloudFormation deploy Λ࣮ߦ
21. Serverless Framework ͷಠࣗ֓೦
 ˞σϓϩΠઌ؀ڥΛ۠ผ͢ΔͨΊͷ༻్ > sls deploy —stage dev > sls deploy —stage prod → “dev” ༻ͷ CloudFormation ελοΫ͕࡞੒͞ΕΔ → “prod” ༻ͷ CloudFormation ελοΫ͕࡞੒͞ΕΔ Note: εςʔδ
22. See also • [Qiita] Serverless Framework ͷ࢖͍ํ·ͱΊ @horike37
23. ϓϩδΣΫτߏ੒ͷྫ
24. ྫʣఆظδϣϒͷ࣮ߦ
25. ྫʣఆظδϣϒͷ࣮ߦ δϣϒύϥϝʔλੜ੒
26. ྫʣఆظδϣϒͷ࣮ߦ δϣϒ࣮ߦج൫
    δϣϒύϥϝʔλΛ
    %ZOBNP%#͔Βऔಘ
27. ྫʣఆظδϣϒͷ࣮ߦ service dir lambda functions scheduled task step functions task utility libs jinja2 tempalte files tests ࣮૷ྫ: Qiita هࣄࣗಈߋ৽ https://github.com/hassaku63/qiita-autogenaration
28. σΟϨΫτϦߏ੒ͷҙਤ ʮԿʹؔ͢Δίʔυͳͷ͔ʯͷݟ௨͠Λྑ͔ͨͬͨ͘͠ • αʔόʔϨε͸ߏ੒ਤ͕ϐλΰϥεΠονʹͳΓ͕ͪ • ී௨ʹϞδϡʔϧ෼ׂ͍ͯ͘͠ײ֮ͱ΄ͱΜͲಉ͡ͰOK • ϚΠΫϩαʔϏε͝ͱʹτοϓϨϕϧͷ֊૚Λ෼͚Δ • ϚΠΫϩαʔϏε಺͸ Lambda ͱͦΕҎ֎ͷϞδϡʔϧͰ෼཭ • lambda functions ʢτϦΨʔ͝ͱʹαϒϞδϡʔϧʹ෼͚Δʣ • ൚༻Ϟδϡʔϧʗઃఆ༻ͷม਺؅ཧ
29. ʮεςʔδʯͷ؅ཧ
30. ΍Γ͍ͨ͜ͱ • ʮεςʔδʯʹґଘ͠ͳ͍ίʔυϕʔεʹ͍ͨ͠ • ϩʔΧϧͰ΋ؾָʹಈ͔͍ͨ͠
31. ΍Γ͍ͨ͜ͱ • ʮεςʔδʯʹґଘ͠ͳ͍ίʔυϕʔεʹ͍ͨ͠ • ϩʔΧϧͰ΋ؾָʹಈ͔͍ͨ͠ → ʮεςʔδʯݻ༗ͳ஋Λ͍͍ײ͡ʹ؅ཧ͢Δ → ϩʔΧϧͰ΋ lambda ͷϩδοΫίʔυΛ࣮ߦ͍ͨ͠
 ˞ ίϯςφϕʔεͷϩʔΧϧϥϯλΠϜͷ࿩͸ࠓճׂѪ
32. 1. εςʔδʹґଘ͠ͳ͍ίʔυϕʔεʹ͍ͨ͠
33. εςʔδ͝ͱʹ஋͕มΘΔ΋ͷ • Ϧιʔε໊ͷࢀরɹ • AWS Ϧιʔεͷ Resource Name, ARN • Endpoint (API Gateway, SQS Queue URL) • ΞϓϦέʔγϣϯݻ༗ͷઃఆ৘ใɹ • ϩάϨϕϧ • ར༻͢ΔAWSϦιʔεͷAPIࢀর༻ͷ໊લʢARN ΍ Ϧιʔε໊ɺURL ͳͲʣ • ઀ଓ͢Δ֎෦αʔϏεͷઃఆ (e.g Slack ͷ௨஌ઌνϟϯωϧʣ
34. εςʔδ͝ͱʹ஋͕มΘΔ΋ͷ • Ϧιʔε໊ͷࢀর → ελοΫͷఆٛͰରॲ • AWS Ϧιʔεͷ Resource Name, ARN • Endpoint (API Gateway, SQS Queue URL) • ΞϓϦέʔγϣϯݻ༗ͷઃఆ৘ใ → ʢLambda ͷʣίʔυϕʔεͰରॲ • ϩάϨϕϧ • ར༻͢ΔAWSϦιʔεͷAPIࢀর༻ͷ໊લʢARN ΍ Ϧιʔε໊ɺURL ͳͲʣ • ઀ଓ͢Δ֎෦αʔϏεͷઃఆ (e.g Slack ͷ௨஌ઌνϟϯωϧʣ
35. ؀ڥม਺ʹΑΔεςʔδݻ༗ม਺ͷ؅ཧ େࡶ೺ͳ؅ཧํ਑ɿ • ੩తͳ஋͔ͭػີ৘ใͰͳ͍ͳΒ؀ڥม਺ • ੩తͳ஋͔ͭػີ৘ใͳΒ Secret Manager/ Parameter Store • ಈతͳ஋ͳΒσʔλετΞ (e.g. DynamoDB)
36. ؀ڥม਺ʹΑΔεςʔδݻ༗ม਺ͷ؅ཧ • python-dotenv • .env ϑΝΠϧΛѻ͏ύοέʔδ • serverless-dotenv-plugin • Serverless Framework ϓϥάΠϯ • .env.<stage> ϑΝΠϧͷએݴΛελοΫఆٛ಺ͰࢀরͰ͖Δ • ϦϦʔε໊ʢͷҰ෦͋Δ͍͸શ෦ʣͷએݴ • Lambda ؀ڥม਺ʹ౉͢ → ${provider.environment.XXX}
37. 2. ϩʔΧϧͰ΋ؾָʹಈ͔͍ͨ͠
38. جຊ • લड़ͷ͜ͱ͕Ͱ͖͍ͯΕ͹ɺ͋ͱ͸΄΅ʮςετ༰қੑʯͷ࿩ • Lambda handler ͷؔ਺ʹϕλॻ͖͗͢͠ͳ͍ • ϏδωεϩδοΫ΍֎෦αʔϏεݺͼग़͠͸ผϞδϡʔϧʹ෼཭ • ϋϯυϥ಺ͷ࢓ࣄ͸ event ͷύʔεͳͲɺ࠷௿ݶʹཹΊΔ • ϩʔΧϧίϯςφϥϯλΠϜͷ࢖༻΋ࣗવͱ΍Γ΍͘͢ͳΔ • e.g) SAM Local, DynamoDB Local, localstack
39. αʔόʔϨεͬΆ͍෦෼ • εςʔδґଘͷ஋͸؀ڥม਺ͱͯ͠ѻ͏ • ؅ཧ͢ΔϞδϡʔϧΛ࡞ͬͯू໿ • e.g) settings.py, constants.py • example: hassaku63/sls-env-example settings.py
40. Example • https://github.com/hassaku63/sls-env-example ૝ఆߏ੒ਤ
41. Example • https://github.com/hassaku63/sls-env-example ૝ఆߏ੒ਤ 424
    2VFVF
    63-
    
    Λ஌͍ͬͯΔඞཁ͕͋Δ
42. templates.yml
43. templates.yml MBNCEB
    ͷ؀ڥม਺ʹ
    Ϧιʔεಛఆʹ
    ඞཁͳ৘ใΛ౉͢
44. settings.py
45. settings.py εςʔδ༻ͷ
    EPUFOW
    Λϩʔυ 2VFVF
    63-
    ͷಛఆʹ
    ඞཁͳม਺Λϩʔυ
46. ϩΪϯά
47. ͠Ό΂Δ͜ͱʢ༧ఆʣ • JSON ϩΨʔΛ࢖ͬͯߏ଄Λ࣋ͨͤΔ • τϨʔε༻ͷIDΛ෇͚Δ • σίϨʔλͰڞ௨Խɹ
48. ͠Ό΂Δ͜ͱʢ༧ఆมߋʣ • JSON ϩΨʔΛ࢖ͬͯߏ଄Λ࣋ͨͤΔ • τϨʔε༻ͷIDΛ෇͚Δɹ→ɹߏ੒ਤ͕೐Θ͖ͬͯͨΒ΍Δ • σίϨʔλͰڞ௨Խɹ→ɹ࣮૷ऀͷ޷Έ࣍ୈ
49. JSONͰߏ଄Խ CloudWatch Logs Insights ͷݕࡧ͕ḿΔ
 (υοτ۠੾ΓͰωετͨ͠ϑΟʔϧυʹΞΫηεՄೳ) Amazon CloudWatch Logs - JSON ϩάͷϑΟʔϧυ https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html#CWL_AnalyzeLogData-discoverable-JSON-logs
50. JSONͰߏ଄Խ ࣮૷ͷબ୒ࢶ 1. ϩΪϯά࣌ͷ msg Ҿ਺ʹ͸ඞͣ json.dump() ͔ͯ͠Β஋Λ౉͢ 2. JSON ϩΨʔΛαϙʔτ͢ΔύοέʔδΛ࢖͏ 3. ࣗલͰॻ͘ 1. Logger, Formatter Λܧঝ 2. Logging Adapter ͷར༻ʢݕ౼தʣ
51. JSONͰߏ଄Խ Logger ʹٻΊ͍ͨཁ݅ʢ໛ࡧதʣ • JSONܗࣜͰग़ྗͰ͖Δ͜ͱ • Ұఆͷߏ଄ԽΛڧ੍Ͱ͖Δ͜ͱ • ʢগͳ͘ͱ΋ʣτοϓϨϕϧͷJSONϓϩύςΟ͕Ұ؏͍ͯ͠Δ • Lambda ͷ࣮ߦ؀ڥ৘ใΛউखʹ஫ೖͯ͘͠ΕΔ
52. JSONͰߏ଄Խ ࢀߟʣLambda Context ͷ৘ใ͸༗༻ͳͷͰϩά౳Ͱ׆༻Ͱ͖Δ (doc) • Lambda ؔ਺໊ʗARNʗόʔδϣϯ • LogGroup / LogStream • Invocation timeout ·Ͱͷ࢒Γ࣌ؒ • etc ..
53. τϨʔε༻IDͷ෇༩ • αʔόʔϨεߏ੒Ͱ͸ɺ1ͭͷϢʔεέʔε͕ෳ਺ͷϚΠΫϩ αʔϏεΛލ͍Ͱॲཧ͞ΕΔ • ӡ༻্ͷτϥϒϧγϡʔςΟϯάͰ͸Ϣʔεέʔε୯ҐͰϩά ΛߜΓࠐΈ͍ͨ • αʔϏεԣஅͰ࣮ߦ͝ͱʹҰҙͱͳΔID͕औΕΕ͹Good
54. τϨʔε༻IDͷ෇༩ • جຊ͸ AWS X-Ray Λ࢖͓͚ͬͯ͹OK • τϨʔε༻IDʹ x-amzn-trace-id Λར༻Մೳ • Serverless Framework ͳΒ༗ޮԽ͸؆୯ • Step Functions Λ࢖͏ߏ੒ͩͱҰ೧Γඞཁ • Step Functions ͸ X-Ray ʹඇରԠʢ2020೥8݄ݱࡏʣ
55. τϨʔε༻IDͷ෇༩ - X-Ray ͷ༗ޮԽ - template.yml ʹ2-3ߦ௥Ճ
 ʢ͓ΘΓʣ
56. σίϨʔλͰڞ௨Խ Lambda ϋϯυϥ༻ͷσίϨʔλΛ࡞ͬͯΞλον͢Δ • e.g) ϋϯυϥͷ࣮ߦલޙͰඞͣϩΪϯάΛڬΉΑ͏ʹ͢Δ • e.g) ෳ਺ϨίʔυΛόϥͯ͠σίʔυ͔ͯ͠Β౉͢ʢSQS/Kinesis τϦΨʔʣ
57. https://github.com/hassaku63/qiita-autogenaration/blob/master/qp/logs.py ࢖༻ྫ (Decorator)
58. ࢖༻ྫ (lambda handler) https://github.com/hassaku63/qiita-autogenaration/blob/master/qp/functions/sfn/publish.py#L17-L28
59. σίϨʔλͰڞ௨Խ Pros • ϋϯυϥ಺ͷهड़ྔ͕ݮΓɺຊ࣭తͳϩδοΫ෦෼͕ڧௐ͞ΕΔ • ൚༻తͳڞ௨ॲཧͷ࠶ར༻ • ։ൃϙϦγʔͷڧ੍ • ࣮ࡍʹ͜ͷख๏Λ࢖࣮ͬͨ૷͕͋Δ (e.g: awslabs/aws-lambda-powertools-python)
60. σίϨʔλͰڞ௨Խ Cons • ΍Γ͗͢͸٫ͬͯίʔυ͕ಡΈͮΒ͘ͳΔʢओ؍ʣ • ଟগ௕͘ͳͬͯ΋ʮԿͯ͠Δ͔ʯ͕௚ײతʹಡΈԼͤΔͳΒ໰୊ͳ͍ɺͱ͍͏ߟ͑΋͋Δ • logging ࣌ͷελοΫϑϨʔϜ͕σίϨʔλʹͳͬͯ͠·͏ • ※ͨͩ͠ɺ΍ΓΑ͏͸͋Δʢݺͼग़͠ݩͷελοΫϑϨʔϜΛḷΔʗσίϨʔλͷҾ਺ͰରԠʣ • ϋϯυϥͷ్தͰ΋ϩΪϯά͍ͨ͠έʔε͸ग़ͯ͘Δ
61. ·ͱΊ
62. ·ͱΊ • αʔόʔϨεʹͳͬͯ΋ɺΞϓϦέʔγϣϯͷ෦෼͸͘͝ී௨ͷ Python • ߏ੒೺Ѳ͕΍Γ΍͘͢ͳΔΑ͏ʹιʔεΛߏ੒͢Δ • εςʔδݻ༗ͷύϥϝʔλ͸؀ڥม਺ͳͲ࢖ͬͯ؅ཧ • ϩΪϯά • JSONϩΨʔΛ࢖͍ͭͭɺσʔλߏ଄ΛҰ؏ͤ͞Δ • ՄೳͳΒτϨʔεIDΛৼΔ • σίϨʔλʹΑΔڞ௨Խ͸͓޷ΈͰ
63. ʢࢀߟʣ৘ใݯ • AWS • AWS Ϋϥ΢υαʔϏε׆༻ࢿྉू: https://aws.amazon.com/jp/aws-jp-introduction/ • Webinar: https://aws.amazon.com/jp/about-aws/events/webinars/ • Connpass: https://aws-serverless.connpass.com/ • Serverless community • Connpass: https://serverless.connpass.com/ • Serverless Operations • Blog archives: https://serverless.co.jp/blog-archives • Qiita: https://qiita.com/organizations/serverless-operations • Twitter • Yan Cui (@theburningmonk / AWS Serverless Hero): https://twitter.com/theburningmonk • Kensuke Shimokawa (@kensh /AWS Japan Serverless Specialist SA): https://twitter.com/_kensh • Keisuke Nishitani (@Keisuke69 / AWS Japan): https://twitter.com/Keisuke69 • Takahiro Horike (@horike37 / Serverless Operations): https://twitter.com/horike37
64. We here hiring!