Ruby で作るデータ分析基盤

Rails Developers Meetup 2018 Day 3 Extreme https://techplay.jp/event/679666

Transcript

1. ©2018 Wantedly, Inc. 3VCZͰ࡞Δσʔλ෼ੳج൫
   
 
   3BJMT
   ΞϓϦέʔγϣϯʹ͓͚Δσʔλ෼ੳͷมભ
    Rails Developer Meetup 2018 Day 3 Extreme - Sohei Takeno
2. ©2018 Wantedly, Inc. 4PIFJ
   5BLFOP !"MUFDI@  ೥d
   8BOUFEMZ
    8BOUFEMZ
   7JTJU
   ͷ։ൃΛ೥ؒ୲౰
    άϩʔεɺݕࡧɾਪનɺσʔλج൫ɺ
 ։ൃج൫ͳͲ
    ݱࡏ͸
   8BOUFEMZ
   1FPQMF
   ͷόοΫΤϯυΛ։ൃ
    ޷͖ͳϓϩάϥϛϯάݴޠ͸
   3VCZ ࣗݾ঺հ
3. ©2018 Wantedly, Inc. ⾣ σʔλ෼ੳͷܦҢ
   ⾣ σʔλ෼ੳج൫ ໨࣍
4. ©2018 Wantedly, Inc. ⾣ σʔλ෼ੳͷܦҢ
   w ͳͥσʔλ෼ੳΛ͍͔ͨ͠
   w αʔϏε։࢝౰ॳͷ࣮ݱํ๏
   w ੒௕͖ͯͨ͜͠ͱͰى͖ͨ໰୊
   ⾣ σʔλ෼ੳج൫ ໨࣍
5. ©2018 Wantedly, Inc. ⾣ ೥͔ΒαʔϏεఏڙ
   ⾣ ϞϊϦγοΫͳ
   3BJMTΞϓϦέʔγϣϯ
   w σʔλ෼ੳ΋ϑΣʔζͷมԽʹ߹ΘͤͯมΘ͍ͬͯͬͨ ࣄྫɿ8BOUFEMZ
   7JTJU
6. ©2018 Wantedly, Inc. ⾣ ϢʔβʔͷߦಈΛݩʹΞϓϦέʔγϣϯ ͷಈ͖Λม͍͑ͨ
   w FH
   ॏཁͳίϯόʔδϣϯɾεςοϓΛ௨ա͍ͯ͠Δ͔൱͔
   w FH
   ίϯόʔδϣϯʹࢸΔ·Ͱͷࡉ͔͍εςοϓ͝ͱͷέΞ
   w ʮ☓☓Λ/ճݟ͍ͯͨΒʯ
   w ػցֶशʹΑΔ࠷దԽ
   ⾣ Ϣʔβʔʹ෼ੳػೳΛఏڙ͍ͨ͠
   w FH
   ʮࡢ೔ͷ
   17
   ͱɺ͜Ε·Ͱͷ
   17
   ͷ߹ܭʯ
   w FH
   ʮଞͱൺֱ͢ΔͨΊͷ
   17
   ϥϯΩϯάʯ ͳͥσʔλ෼ੳΛ͍͔ͨ͠
   
7. ©2018 Wantedly, Inc. ⾣ σʔλΛݩʹҙࢥܾఆ͍ͨ͠
   w FH
   ,1*
   ͷܭࢉ
   w FH
   ,1*
   ʹؔ࿈͢Δॏཁࢦඪͷܭࢉ
   w FH
   "#ςετ ͳͥσʔλ෼ੳΛ͍͔ͨ͠
   
8. ©2018 Wantedly, Inc. αʔϏε։࢝౰ॳͷ࣮ݱํ๏ ೥d
9. ©2018 Wantedly, Inc. ⾣ ॏཁͳίϯόʔδϣϯɾεςοϓΛ௨ա͍ͯ͠Δ͔൱͔
 ˠ
   ౰વϦϨʔγϣφϧσʔλϕʔεʹอଘ͞Ε͍ͯΔ
   ⾣ ʮϢʔβʔ:͕ΞΠςϜΛ/ճݟ͍ͯͨΒʯ
 ˠ
   ࣄ৅͕ى͜Δ౓ʹΧ΢ϯτ͓ͯ͘͠
   PS
   ϩά͔Βܭࢉ͢Δ
   ⾣ ʮϢʔβʔ:͕ΞΠςϜ9Λݟ͍ͯͨΒʯ
 ˠ
   ϦϨʔγϣφϧσʔλϕʔεʹϩάΛอଘͯ͠ࢀর͢Δ ϢʔβʔͷߦಈΛݩʹΞϓϦέʔγϣϯͷಈ͖Λม͍͑ͨ
10. ©2018 Wantedly, Inc. ⾣ ʮࡢ೔ͷ
    17
    ͱɺ͜Ε·Ͱͷ
    17
    ͷ߹ܭʯ
 ˠ
    ΞΫηεϩά͔Β೔͝ͱͷ
    17
    Λதؒσʔλͱͯ͠࡞੒
    ⾣ ʮൺֱͰ͖ΔΑ͏ʹ
    17
    ϥϯΩϯάΛग़͢ʯ
 ˠ
    தؒσʔλશମΛूܭ͢ΔΫΤϦΛॻ͖ɺ3BJMT
    Ωϟο γϡͰܰ͘͢Δ Ϣʔβʔʹ෼ੳػೳΛఏڙ͍ͨ͠
11. ©2018 Wantedly, Inc. ੒௕͖ͯͨ͜͠ͱͰى͖ͨ໰୊ ೥d
12. ©2018 Wantedly, Inc. ⾣ lϦϨʔγϣφϧσʔλϕʔεʹϩάΛอଘz
    w %#༰ྔΛѹഭɺू໿ܭࢉ΋͠ʹ͘͘த్൒୺ͳଘࡏʹ
    ⾣ lதؒσʔλશମΛूܭ͢ΔΫΤϦΛॻ͖ɺ3BJMT
    Ωϟογϡz
    w ࣌ؒͷܦա
    ʜ
    தؒσʔλͩͱࢥ͍ͬͯͨ΋ͷࣗମ͕ॏ͘ͳΔ
    w Ϣʔβʔ਺ͷ૿Ճ
    ʜ
    ಉ࣌ΞΫηε਺͕૿Ճ͠ɺ3BJMT
    Ωϟογϡ ͕ඈΜͩλΠϛϯάͰͷ࠶ܭࢉ͕ಉ࣌ฒߦͰ૸Δ
    w ্هͷֻ͚߹ΘͤͰ
    %#
    ͕ϝϞϦΛ࢖͍Ռͨ͢ࣄ݅΋ ੒௕͖ͯͨ͜͠ͱͰى͖ͨ໰୊
    
13. ©2018 Wantedly, Inc. ⾣ σʔλ΢ΣΞϋ΢εར༻ͷଅਐ
    w 5SFBTVSF%BUB
    ೥d
    #JH2VFSZ
    ೥d
    ⾣ ૊৫͕େ͖͘ͳΓɺαʔϏε͕੒௕͢Δʹ࿈Εͯҙࢥܾఆͷ ͨΊͷσʔλ෼ੳ͕૿͑Δ
    w ଟ਺ͷ෼ੳΫΤϦΛ
    8FC
    6*
    Ͱ؅ཧ
    ୭͕Ͳ͏͍͏ҙਤͰ࡞ΓͲ͏มߋ͞Ε͔ͨ෼͔Βͳ͍
    ΫΤϦ͕ؒҧ͍ͬͯͯ൑அΛޡΔ ੒௕͖ͯͨ͜͠ͱͰى͖ͨ໰୊
    
14. ©2018 Wantedly, Inc. ⾣ ෼ੳϩδοΫͷมߋͷ౓ʹେ͖͘ͳͬͨ
    3BJMT
    ΞϓϦέʔ γϣϯશମΛ
    EFQMPZ
    ͢Δඞཁ͕͋Δ
    w ੜ࢈ੑ্͕͕Βͳ͍
    ⾣ δϣϒͷ඼࣭͕όϥόϥ
    w ႈ౳ੑͷ͋Δδϣϒɺͳ͍δϣϒ
    w ࠶࣮ߦՄೳͳδϣϒɺෆՄೳͳδϣϒ ੒௕͖ͯͨ͜͠ͱͰى͖ͨ໰୊
    
15. ©2018 Wantedly, Inc. ⾣ σʔλ෼ੳͷܦҢ
    ⾣ σʔλ෼ੳج൫ ໨࣍
16. ©2018 Wantedly, Inc. ⾣ σʔλ෼ੳͷܦҢ
    ⾣ σʔλ෼ੳج൫ ໨࣍
17. ©2018 Wantedly, Inc. ⾣ σʔλ෼ੳͷܦҢ
    ⾣ σʔλ෼ੳج൫ ໨࣍
18. ©2018 Wantedly, Inc. ⾣ #JH2VFSZ
    ͷΫΤϦͷ݁ՌΛ೔࣌Ͱςʔϒϧʹॻ͖ग़͢ ࢖͍ํ
     export do table :daily_page_views columns [:day, :pv] mode :update, [:day] end schedule do frequency :daily end run :bq, <<SQL SELECT DATE(_WT_SCHEDULED_TIME, '+09:00') day, COUNT(*) AS pv FROM `log.accesses*` WHERE _TABLE_SUFFIX = FROMAT_TIMESTAMP(”%Y%m%d”, TIMESTAMP_SUB(_WT_SCHEDULED_TIME, INTERVAL 1 DAY) ) SQLn @85@4$)&%6-&%@5*.&
    
    δϣϒͷ࣮ߦ࣌ؒΛදٙ͢ࣅม਺
19. ©2018 Wantedly, Inc. ⾣ δϣϒϑΝΠϧΛ௥Ճ͠ɺ
    QVMMSFRVFTU
    ʹ͢Δ
    ⾣ Ϛʔδ͢Δͱɺδϣϒͱͯ͠ొ࿥͞Εఆظ࣮ߦ͞ΕΔ
    ⾣ $-*
    ͔Β೚ҙͷ೔ͷδϣϒΛ࣮ߦՄೳ
    ⾣ ೚ҙͷ
    3VCZ
    ίʔυΛ࣮ߦ͢Δ
    SVOOFS
    ࢖͍ํ
     $ ./bin/job run daily_page_view -s '2018-07-14' run :proc, -> (scheduled_time) { # … return [[:foo, 1], [:bar, 2]] }
20. ©2018 Wantedly, Inc. ⾣ ΠγϡʔΛཱͯͯɺ1VMM3FRVFTU
    Λग़͠ɺϨϏϡʔ͢Δ
    ⾣ CMBNF
    ͔ΒḷΕ͹എܠɾҙਤ͕෼͔Δ
    w ΫΤϦͷΦʔφʔΛ୳͠ճΔ࡞ۀ΋ඞཁͳ͍
    ⾣ ϫʔΫϑϩʔ͕͋Δ
 ͔Βෳࡶͳ΋ͷΛ
 ੵΈ্͛ΒΕΔ (JU
    
    (JU)VC
    ͷϫʔΫϑϩʔʹ৐Δ
21. ©2018 Wantedly, Inc. ⾣ DG
    ΦϯϥΠϯͰॏ͍ूܭΛͯ͠
    %#
    ࢮΜͩࣄ݅
    ΞʔΩςΫνϟΛݻఆ͢Δ 3BJMT
    "QQ XSJUF RVFSZ 3BJMT
    "QQ XSJUFPOMZ BOBMZUJDT SFBEPOMZ MPHHJOH RVFSZ αΠΫϧճ͢୯ํ޲ͷΞʔΩςΫνϟ
22. ©2018 Wantedly, Inc. ⾣ ʮࡢ೔ΠϯϑϥϨϕϧͰෆ۩߹͕͋ͬͨͷͰʓʓؔ܎ͷδϣϒ Λػցతʹ࠶࣮ߦ͢Δʯ͕Մೳ͔ʁ
    ⾣ ΠϯλʔϑΣΠεɾϨϕϧͰ࠶࣮ߦՄೳੑΛଅਐ
    w FYQPSU
    ͷࡍ͸
    VQEBUF
    ͕σϑΥϧτ
    w SVO
    ͸֎෦͔Β࣮ߦ࣌ؒΛҾ਺Λ༩͑Δ
    w TDIFEVMJOH
    ͸
    EBJMZ
    ͷΑ͏ͳେ·͔ͳࢦఆ͕σϑΥϧτ
    ⾣ δϣϒϑΝΠϧ͸
    3VCZ
    ΦϒδΣΫτͱͯ͠ϩʔυ͞ΕΔͷͰɺ ϓϩάϥϚϒϧʹ΋ॲཧͰ͖Δ δϣϒͷ඼࣭ jobs = Analytics::Backend::JobLoader.load_all; daily_jobs = jobs.select {|job| job.schedule&.frequency == :daily } daily_jobs.each do |job| job.execute(scheduled_time: Time.new(2018,1,10), export: true) end
23. ©2018 Wantedly, Inc. ⾣ ΫϥελϦϯά͞Εͨෳ਺ͷϚγϯ্Ͱ࣮ߦ͞ΕΔ
 ʢޮ཰Խɾ৑௕Խʣ
    ⾣ ,VCFSOFUFT
    ͷίϚϯυͰొ࿥δϣϒͷҰཡ΍࣮ߦεςʔλ ε͕͋Δఔ౓෼͔Δ δϣϒ࣮ߦج൫ͱͯ͠ͷ
    ,VCFSOFUFT
    ͷ׆༻ export do table :daily_page_views columns [:day, :pv] mode :update, [:day] end schedule do frequency :daily end run :bq, <<SQL SELECT DATE(_WT_SCHEDULED_TIME, '+09:00') day, COUNT(*) AS pv FROM `log.accesses*` WHERE _TABLE_SUFFIX = Ruby ϑΝΠϧ apiVersion: batch/v1beta1 kind: CronJob metadata: name: visit--user-impressed-companies labels: namespace: visit basename: user_impressed_companies role: job namespace: analytics spec: schedule: "20 6,21 * * *" concurrencyPolicy: "Replace" suspend: false successfulJobsHistoryLimit: 10 failedJobsHistoryLimit: 3 jobTemplate: metadata: name: visit--user-impressed-companies labels: namespace: visit basename: user_impressed_companies role: job spec: backoffLimit: 5 template: metadata: name: visit--user-impressed-companies labels: YAML ϑΝΠϧ Kubernetes HFOFSBUF BQQMZ SVO
    BGUFS
    NFSHFE
24. ©2018 Wantedly, Inc. ⾣ σʔλ෼ੳͷܦҢ
    ⾣ σʔλ෼ੳج൫ ໨࣍
25. ©2018 Wantedly, Inc. ⾣ ܧଓతʹ࢖ΘΕଓ͚͍ͯΔ
    w ೥݄ݱࡏɺδϣϒ
    ⾣ ఆظతͳػೳ֦ு
    w #JH2VFSZ
    
    #JH2VFSZ
    ΁ͷॻ͖ग़͠
    w JNQPSU
    KPC
    
    3%#
    
    #JH2VFSZ
    ͷ
    &5- ಋೖޙ
26. ©2018 Wantedly, Inc. ⾣ σʔλ෼ੳͷͨΊͷίʔυϕʔεΛ෼཭ͯ͠࢓૊ΈΛ࡞ͬͨ
    w ࢖͍΍͍͢
    *'
    ͷఏڙ
    w σϓϩΠ࣌ؒͷ୹ॖ
    w ϫʔΫϑϩʔͷ౷Ұ
    w ΑΓ҆શͳΞʔΩςΫνϟ΁ͷҠߦ
    w δϣϒͷ඼࣭޲্
    w ֦ுՄೳͳ࢓૊Έ
    ⾣ ݁Ռͱͯ͠ɺ෼ੳܥΛ
    3BJMT
    ΞϓϦ͔Β੾Γ཭͢͜ͱʹ੒ޭ
    ⾣ ґଘؔ܎ͷ؅ཧͳͲࠓޙཉ͘͠ͳΔՄೳੑͷ͋Δػೳ͸͋Δ ·ͱΊ