よく聞くけど使ったことないソフトウェアNo.1 KafkaとSnowflake

2024年4月22日に行われたPyData Okinawaで行なったLTです
https://pydataokinawa.connpass.com/event/314900/

言及してるQuickStartはこれ
Getting Started with Snowpipe Streaming and Amazon MSK
- https://quickstarts.snowflake.com/guide/getting_started_with_snowpipe_streaming_aws_msk/index.html#0

Getting Started with Snowflake and Amazon Data Firehose
- https://quickstarts.snowflake.com/guide/getting_started_with_snowflake_and_aws_kdf/index.html

Transcript

1. Α͘ฉ͚͘Ͳ࢖ͬͨ͜ͱແ͍ι ϑτ΢ΣΞ/P ,BGLB ͱ 4OPXGMBLF  ͪΎΒσʔλ $50
   ඛপ ༤ଠ ˜
   
   $IVSB
   %"5"
   JOD
   
2. ˜
   
   $IVSB
   %"5"
   JOD
   ࣗݾ঺հ ΦϑϥΠϯͷ࢟
3. ΦϯϥΠϯͷ࢟ ˜
   
   $IVSB
   %"5"
   JOD
   
4. ˜
   
   $IVSB
   %"5"
   JOD
   
5. ˜
   
   $IVSB
   %"5"
   JOD
   ࣗݾ঺հ ͪΎΒσʔλ
   $50 %"56.
   456%*0
   ࣥߦ໾һ 4OPXGMBLF
   ͷ
   %BUB
   )FSP
   PG
   5IF
   :FBS 
   4OPX7JMMBHF
   ͷଜ௕ ͜ͷࢠ͸ Ζ͋ͪΌΜ
6. ˜
   
   $IVSB
   %"5"
   JOD
   ͪΎΒσʔλ
   $50 %"56.
   456%*0
   ࣥߦ໾һ 4OPXGMBLF
   ͷ
   %BUB
   )FSP
   PG
   5IF
   :FBS 
   4OPX7JMMBHF
   ͷଜ௕ ͜ͷࢠ͸ Ζ͋ͪΌΜ ͘͘͞͞ ΍͍ͬͯͧ͘ʂ
7. ,BGLB
   ฉ͍ͨ͜ͱ͋Γ·͔͢ʁ ˜
   
   $IVSB
   %"5"
   JOD
   
8. ˜
   
   $IVSB
   %"5"
   JOD
   "QBDIF
   ,BGLB͸ɺ෼ࢄܕΠϕϯτε τΞ͓ΑͼετϦʔϜॲཧϓϥοτ ϑΥʔϜͰ͋Δɻ"QBDIFιϑτ ΢ΣΞࡒஂ͕؅ཧ͢ΔΦʔϓϯιʔ εͷγεςϜͰɺ+BWBͱ4DBMBΛ༻ ͍࣮ͯ૷͞Ε͍ͯΔɻ
9. ˜
   
   $IVSB
   %"5"
   JOD
   ,BGLBͷΞʔΩςΫνϟ #SPLFS
   DMVTUFS ;PP,FFQFS
   DMVTUFS ΞϓϦέʔγϣϯ 1VCMJTIFS ΞϓϦέʔγϣϯ 4VCTDSJCFS ,BGLB
   1SPEVDFS ˞ॻ͖ࠐΈϥΠϒϥϦ ,BGLB
   $POTVNFS ˞ಡΈࠐΈϥΠϒϥϦ #SPLFS #SPLFS #SPLFS 5PQJD 1BSUJUJPO 1BSUJUJPO 1BSUJUJPO 1BSUJUJPO 1BSUJUJPO 1BSUJUJPO ;PP,FFQFS ;PP,FFQFS ;PP,FFQFS • ,BGLB͸ෳ਺#SPLFSͰΫϥελߏ ੒ɺΫϥελ৘ใ͸;PP,FFQFS ͕࣋ͪɺ#SPLFSͷ୆͕Ϧʔμʔ ͱͳΓΫϥελ؅ཧ • ,BGLBͷϝοηʔδ͸ΩʔόϦϡʔ ܗࣜͷ3FDPSEͱݺ͹Εɺͭͷ 5PQJD͸ෳ਺ͷ#SPLFSʹ෼ࢄ͞Εͨ 1BSUJUJPOͰߏ੒ɺ5PQJD΁ͷ3FDPSE ͷಡΈॻ͖Λ1BSUJUJPO୯ҐͰ෼ࢄฒ ྻԽ͢Δ͜ͱͰߴ଎ͳॲཧΛ࣮ݱ
10. ετϦʔϛϯάͰσʔλΛ౤ೖ͢ΔͳΒ,BGLB Α͋͘Δ͋ΔͰ͢ ˜
    
    $IVSB
    %"5"
    JOD
    
11. େྔͷσʔλΛ౤ೖͯ͠෼ੳ͢ΔͳΒ 0-"1ܕͷ%#͕ඞཁͰ͢ΑͶʁ ʢ࠷ۙ͸)5"1΋͋Δ͚Ͳʣ ˜
    
    $IVSB
    %"5"
    JOD
    
12. 0-"1ͱ͍͑͹ࠓ͸4OPXGMBLFͱ#JH2VFSZ͕࠷ڧ ˜
    
    $IVSB
    %"5"
    JOD
    
13. ࠓ೔ͷຊ୊ͷ4OPXGMBLFͱ,BGLB ˜
    
    $IVSB
    %"5"
    JOD
    
14. ˜
    
    $IVSB
    %"5"
    JOD
    4OPXGMBLF
    ͷ
    ,BGLB
    $POOFDUFS
    ͸ํࣜ͋Δ ,BGLB
    5PQJD 1BSUJUJPO
     1BSUJUJPO
     1BSUJUJPO
     1BSUJUJPO
     ,BGLB
    $POOFDUFS QPMM ˞
    IUUQTEPDTTOPXGMBLFDPNFOVTFSHVJEFEBUBMPBETOPXQJQFTUSFBNJOHLBGLB
    هࡌͷਤΛݩʹ୯७Խͯ͠࡞ਤ͠·ͨ͠ 4OPXGMBLF *OUFSOBM
    4UBHF 4 +40/
    'JMF 5BCMF 4OPXQJQF 4OPXQJQF 4OPXQJQF 4USFBNJOH ,BGLB
    5PQJD 1BSUJUJPO
     1BSUJUJPO
     1BSUJUJPO
     1BSUJUJPO
     ,BGLB
    $POOFDUFS QPMM 4OPXGMBLF 5BCMF $IBOOOFM
     $IBOOOFM
     $IBOOOFM
     $IBOOOFM
     • 4OPXQJQF ্ ͸ɺ,BGDB
    $POOFDUFSͰσʔλΛड͚ͯɺ4΁ϑΝΠϧΛ165ɺͦͷޙ4OPXQJQFͱ͍͏ σʔλऔΓࠐΈϓϩάϥϜΛ3&45
    "1*Ͱىಈ͢ΔͷͰɺ෼஗Ε͘Β͍ͰऔΓࠐ·ΕΔɺ͢ͳΘͪϨΠς ϯγେ͖Ί • 4OPXQJQF
    4USFBNJOH Լ ͸ɺ,BGLB
    $POOFDUFSͰड͚ͯ1BSUJPOͱಉ͡਺ͷ$IBOOFMͰฒྻऔΓࠐΈΛߦ͏ ͷͰϨΠςϯγ͸ඵ୯ҐͰύϑΥʔϚϯε΋ߴ͍ʢ(#Tͱ͔ग़ΔΒ͍͠Αʣ • අ༻తʹ͸࣌ؒҐͰສߦྲྀ͠ࠐΜͰɺυϧ͘Β͍ͩΑ ˞1BSUJUJPO
    ͱ
    $IBOOOFM
    ͸ ͔͜͜Βࠨ͸ετϦʔϛϯά ͔͜͜Βӈ͸ϚΠΫϩόον
15. • (FUUJOH
    4UBSUFE
    XJUI
    4OPXQJQF
    4USFBNJOH
    BOE
    "NB[PO
    .4, • IUUQTRVJDLTUBSUTTOPXGMBLFDPNHVJEFHFUUJOH@TUBSUFE@XJUI@TOPXQJQF@TUSFBNJOH@BXT@ NTLJOEFYIUNM ˜
    
    $IVSB
    %"5"
    JOD
    2VJDL
    4UBSU΋͋Δ͔ΒνϟϨϯδͯ͠ΈͯͶ
16. ͪͳΈʹ࠷ۙ͸͞Βʹ ετϦʔϛϯάͷϋʔυϧ͕Լ͕ͬͨ ˜
    
    $IVSB
    %"5"
    JOD
    
17. ,JOFTJT
    %BUB
    'JSFIPTF ,BGLB͡Όͳ͍Α
    ͷ 4OPXGMBLF
    4OPXQJQF
    4USFBNJOHରԠ ˜
    
    $IVSB
    %"5"
    JOD
    
18. ˜
    
    $IVSB
    %"5"
    JOD
    %BUB
    'JSFIPTF
    ͷ
    4OPXQJQF
    4USFNJOHରԠͷൃද ౦ژʜͳ͍ͩͱʁ
19. ˜
    
    $IVSB
    %"5"
    JOD
    ͱ͜Ζ͕ɺ͖ͬ͞ݟͨΒ Ωςλʔʔ
20. ˜
    
    $IVSB
    %"5"
    JOD
    • (FUUJOH
    4UBSUFE
    XJUI
    4OPXGMBLF
    BOE
    "NB[PO
    %BUB
    'JSFIPTF • IUUQTRVJDLTUBSUTTOPXGMBLFDPNHVJEFHFUUJOH@TUBSUFE@XJUI@TOPXGMBLF@BOE@BXT@LEGJO EFYIUNM 2VJDL
    4UBSU΋Ͱ͍ͯΔ
21. ࠷ޙʹ ˜
    
    $IVSB
    %"5"
    JOD
    
22. ˜
    
    $IVSB
    %"5"
    JOD
    ͪΎΒσʔλͷએ఻ίʔφʔ