Share
Embed
Ruby で作るデータ分析基盤
Rails Developers Meetup 2018 Day 3 Extreme https://techplay.jp/event/679666
Altech
July 14, 2018
Transcript
26 slides
-
©2018 Wantedly, Inc. 3VCZͰ࡞Δσʔλੳج൫ 3BJMTΞϓϦέʔγϣϯʹ͓͚Δσʔλੳͷมભ Rails Developer Meetup 2018
Day 3 Extreme - Sohei Takeno -
©2018 Wantedly, Inc. 4PIFJ5BLFOP !"MUFDI@ d8BOUFEMZ 8BOUFEMZ7JTJUͷ։ൃΛؒ୲
άϩʔεɺݕࡧɾਪનɺσʔλج൫ɺ ։ൃج൫ͳͲ ݱࡏ8BOUFEMZ1FPQMFͷόοΫΤϯυΛ։ൃ ͖ͳϓϩάϥϛϯάݴޠ3VCZ ࣗݾհ -
©2018 Wantedly, Inc. ⾣ σʔλੳͷܦҢ w ͳͥσʔλੳΛ͍͔ͨ͠ w αʔϏε։࢝ॳͷ࣮ݱํ๏ w
͖ͯͨ͜͠ͱͰى͖ͨ ⾣ σʔλੳج൫ ࣍ -
©2018 Wantedly, Inc. ⾣ ϢʔβʔͷߦಈΛݩʹΞϓϦέʔγϣϯ ͷಈ͖Λม͍͑ͨ w FHॏཁͳίϯόʔδϣϯɾεςοϓΛ௨ա͍ͯ͠Δ͔൱͔ w FHίϯόʔδϣϯʹࢸΔ·Ͱͷࡉ͔͍εςοϓ͝ͱͷέΞ
w ʮ☓☓Λ/ճݟ͍ͯͨΒʯ w ػցֶशʹΑΔ࠷దԽ ⾣ ϢʔβʔʹੳػೳΛఏڙ͍ͨ͠ w FHʮࡢͷ17ͱɺ͜Ε·Ͱͷ17ͷ߹ܭʯ w FHʮଞͱൺֱ͢ΔͨΊͷ17ϥϯΩϯάʯ ͳͥσʔλੳΛ͍͔ͨ͠ -
©2018 Wantedly, Inc. ⾣ σʔλΛݩʹҙࢥܾఆ͍ͨ͠ w FH,1*ͷܭࢉ w FH,1*ʹؔ࿈͢Δॏཁࢦඪͷܭࢉ w
FH"#ςετ ͳͥσʔλੳΛ͍͔ͨ͠ -
©2018 Wantedly, Inc. ⾣ ॏཁͳίϯόʔδϣϯɾεςοϓΛ௨ա͍ͯ͠Δ͔൱͔ ˠવϦϨʔγϣφϧσʔλϕʔεʹอଘ͞Ε͍ͯΔ ⾣ ʮϢʔβʔ:͕ΞΠςϜΛ/ճݟ͍ͯͨΒʯ ˠࣄ͕ى͜ΔʹΧϯτ͓ͯ͘͠PSϩά͔Βܭࢉ͢Δ ⾣
ʮϢʔβʔ:͕ΞΠςϜ9Λݟ͍ͯͨΒʯ ˠϦϨʔγϣφϧσʔλϕʔεʹϩάΛอଘͯ͠ࢀর͢Δ ϢʔβʔͷߦಈΛݩʹΞϓϦέʔγϣϯͷಈ͖Λม͍͑ͨ -
©2018 Wantedly, Inc. ⾣ lϦϨʔγϣφϧσʔλϕʔεʹϩάΛอଘz w %#༰ྔΛѹഭɺूܭࢉ͠ʹ͘͘த్ͳଘࡏʹ ⾣ lதؒσʔλશମΛूܭ͢ΔΫΤϦΛॻ͖ɺ3BJMTΩϟογϡz w
࣌ؒͷܦաʜதؒσʔλͩͱࢥ͍ͬͯͨͷࣗମ͕ॏ͘ͳΔ w Ϣʔβʔͷ૿Ճʜಉ࣌ΞΫηε͕૿Ճ͠ɺ3BJMTΩϟογϡ ͕ඈΜͩλΠϛϯάͰͷ࠶ܭࢉ͕ಉ࣌ฒߦͰΔ w ্هͷֻ͚߹ΘͤͰ%#͕ϝϞϦΛ͍Ռͨ͢ࣄ݅ ͖ͯͨ͜͠ͱͰى͖ͨ -
©2018 Wantedly, Inc. ⾣ σʔλΣΞϋεར༻ͷଅਐ w 5SFBTVSF%BUBd #JH2VFSZd ⾣ ৫͕େ͖͘ͳΓɺαʔϏε͕͢Δʹ࿈Εͯҙࢥܾఆͷ
ͨΊͷσʔλੳ͕૿͑Δ w ଟͷੳΫΤϦΛ8FC6*Ͱཧ ୭͕Ͳ͏͍͏ҙਤͰ࡞ΓͲ͏มߋ͞Ε͔͔ͨΒͳ͍ ΫΤϦ͕ؒҧ͍ͬͯͯஅΛޡΔ ͖ͯͨ͜͠ͱͰى͖ͨ -
©2018 Wantedly, Inc. ⾣ ੳϩδοΫͷมߋͷʹେ͖͘ͳͬͨ3BJMTΞϓϦέʔ γϣϯશମΛEFQMPZ͢Δඞཁ͕͋Δ w ੜ࢈ੑ্͕͕Βͳ͍ ⾣ δϣϒͷ࣭͕όϥόϥ
w ႈੑͷ͋Δδϣϒɺͳ͍δϣϒ w ࠶࣮ߦՄೳͳδϣϒɺෆՄೳͳδϣϒ ͖ͯͨ͜͠ͱͰى͖ͨ -
©2018 Wantedly, Inc. ⾣ #JH2VFSZͷΫΤϦͷ݁ՌΛ࣌Ͱςʔϒϧʹॻ͖ग़͢ ͍ํ export do table
:daily_page_views columns [:day, :pv] mode :update, [:day] end schedule do frequency :daily end run :bq, <<SQL SELECT DATE(_WT_SCHEDULED_TIME, '+09:00') day, COUNT(*) AS pv FROM `log.accesses*` WHERE _TABLE_SUFFIX = FROMAT_TIMESTAMP(”%Y%m%d”, TIMESTAMP_SUB(_WT_SCHEDULED_TIME, INTERVAL 1 DAY) ) SQLn @85@4$)&%6-&%@5*.&δϣϒͷ࣮ߦ࣌ؒΛදٙ͢ࣅม -
©2018 Wantedly, Inc. ⾣ δϣϒϑΝΠϧΛՃ͠ɺQVMMSFRVFTUʹ͢Δ ⾣ Ϛʔδ͢Δͱɺδϣϒͱͯ͠ొ͞Εఆظ࣮ߦ͞ΕΔ ⾣ $-*͔ΒҙͷͷδϣϒΛ࣮ߦՄೳ ⾣
ҙͷ3VCZίʔυΛ࣮ߦ͢ΔSVOOFS ͍ํ $ ./bin/job run daily_page_view -s '2018-07-14' run :proc, -> (scheduled_time) { # … return [[:foo, 1], [:bar, 2]] } -
©2018 Wantedly, Inc. ⾣ ΠγϡʔΛཱͯͯɺ1VMM3FRVFTUΛग़͠ɺϨϏϡʔ͢Δ ⾣ CMBNF͔ΒḷΕഎܠɾҙਤ͕͔Δ w ΫΤϦͷΦʔφʔΛ୳͠ճΔ࡞ۀඞཁͳ͍ ⾣
ϫʔΫϑϩʔ͕͋Δ ͔ΒෳࡶͳͷΛ ੵΈ্͛ΒΕΔ (JU(JU)VCͷϫʔΫϑϩʔʹΔ -
©2018 Wantedly, Inc. ⾣ DGΦϯϥΠϯͰॏ͍ूܭΛͯ͠%#ࢮΜͩࣄ݅ ΞʔΩςΫνϟΛݻఆ͢Δ 3BJMT"QQ XSJUF RVFSZ 3BJMT"QQ
XSJUFPOMZ BOBMZUJDT SFBEPOMZ MPHHJOH RVFSZ αΠΫϧճ͢୯ํͷΞʔΩςΫνϟ -
©2018 Wantedly, Inc. ⾣ ʮࡢΠϯϑϥϨϕϧͰෆ۩߹͕͋ͬͨͷͰʓʓؔͷδϣϒ Λػցతʹ࠶࣮ߦ͢Δʯ͕Մೳ͔ʁ ⾣ ΠϯλʔϑΣΠεɾϨϕϧͰ࠶࣮ߦՄೳੑΛଅਐ w FYQPSUͷࡍVQEBUF͕σϑΥϧτ
w SVO֎෦͔Β࣮ߦ࣌ؒΛҾΛ༩͑Δ w TDIFEVMJOHEBJMZͷΑ͏ͳେ·͔ͳࢦఆ͕σϑΥϧτ ⾣ δϣϒϑΝΠϧ3VCZΦϒδΣΫτͱͯ͠ϩʔυ͞ΕΔͷͰɺ ϓϩάϥϚϒϧʹॲཧͰ͖Δ δϣϒͷ࣭ jobs = Analytics::Backend::JobLoader.load_all; daily_jobs = jobs.select {|job| job.schedule&.frequency == :daily } daily_jobs.each do |job| job.execute(scheduled_time: Time.new(2018,1,10), export: true) end -
©2018 Wantedly, Inc. ⾣ ΫϥελϦϯά͞ΕͨෳͷϚγϯ্Ͱ࣮ߦ͞ΕΔ ʢޮԽɾԽʣ ⾣ ,VCFSOFUFTͷίϚϯυͰొδϣϒͷҰཡ࣮ߦεςʔλ ε͕͋Δఔ͔Δ δϣϒ࣮ߦج൫ͱͯ͠ͷ,VCFSOFUFTͷ׆༻
export do table :daily_page_views columns [:day, :pv] mode :update, [:day] end schedule do frequency :daily end run :bq, <<SQL SELECT DATE(_WT_SCHEDULED_TIME, '+09:00') day, COUNT(*) AS pv FROM `log.accesses*` WHERE _TABLE_SUFFIX = Ruby ϑΝΠϧ apiVersion: batch/v1beta1 kind: CronJob metadata: name: visit--user-impressed-companies labels: namespace: visit basename: user_impressed_companies role: job namespace: analytics spec: schedule: "20 6,21 * * *" concurrencyPolicy: "Replace" suspend: false successfulJobsHistoryLimit: 10 failedJobsHistoryLimit: 3 jobTemplate: metadata: name: visit--user-impressed-companies labels: namespace: visit basename: user_impressed_companies role: job spec: backoffLimit: 5 template: metadata: name: visit--user-impressed-companies labels: YAML ϑΝΠϧ Kubernetes HFOFSBUF BQQMZ SVOBGUFSNFSHFE -
©2018 Wantedly, Inc. ⾣ ܧଓతʹΘΕଓ͚͍ͯΔ w ݄ݱࡏɺδϣϒ ⾣ ఆظతͳػೳ֦ு w
#JH2VFSZ#JH2VFSZͷॻ͖ग़͠ w JNQPSUKPC3%##JH2VFSZͷ&5- ಋೖޙ -
©2018 Wantedly, Inc. ⾣ σʔλੳͷͨΊͷίʔυϕʔεΛͯ͠ΈΛ࡞ͬͨ w ͍͍͢*'ͷఏڙ w σϓϩΠ࣌ؒͷॖ w
ϫʔΫϑϩʔͷ౷Ұ w ΑΓ҆શͳΞʔΩςΫνϟͷҠߦ w δϣϒͷ্࣭ w ֦ுՄೳͳΈ ⾣ ݁Ռͱͯ͠ɺੳܥΛ3BJMTΞϓϦ͔ΒΓ͢͜ͱʹޭ ⾣ ґଘؔͷཧͳͲࠓޙཉ͘͠ͳΔՄೳੑͷ͋Δػೳ͋Δ ·ͱΊ