Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
GCPではじめるスモールスタートなデータ活用
Search
Takashi Nishibayashi
September 06, 2016
Technology
3
2.9k
GCPではじめるスモールスタートなデータ活用
2016-09-06
bq_sushi #4での発表資料です
Takashi Nishibayashi
September 06, 2016
Tweet
Share
More Decks by Takashi Nishibayashi
See All by Takashi Nishibayashi
診断前の病歴テキストを対象としたLLMによるエンティティリンキング精度検証
hagino3000
1
110
論文紹介 Improving Medical Reasoning through Retrieval and Self-Reflection with Retrieval-Augmented Large Language Models
hagino3000
0
850
論文紹介 Audience Size Forecasting Fast and Smart Budget Planning for Media Buyers
hagino3000
0
240
論文紹介 Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems
hagino3000
1
630
論文紹介 Budget Management Strategies in Repeated Auctions (公開版)
hagino3000
2
290
論文紹介 A Request-level Guaranteed Delivery Advertising Planning: Forecasting and Allocation
hagino3000
1
120
論文紹介 Online Experimentation with Surrogate Metrics Guidelines and a Case Study
hagino3000
0
330
論文紹介 Bidding Machine: Learning to Bid for Directly Optimizing Profits in Display Advertising
hagino3000
1
200
論文紹介 Balancing Relevance and Discovery to Inspire Customers in the IKEA App
hagino3000
0
730
Other Decks in Technology
See All in Technology
20250719_JAWS_kobe
takuyay0ne
1
170
MCPに潜むセキュリティリスクを考えてみる
milix_m
1
840
地図と生成AI
nakasho
0
810
激動の時代、新卒エンジニアはAIツールにどう向き合うか。 [LayerX Bet AI Day Countdown LT Day1 ツールの選択]
tak848
0
600
AIを使っていい感じにE2Eテストを書けるようになるまで / Trying to Write Good E2E Tests with AI
katawara
3
1.8k
Ktor + Google Cloud Tasks/PubSub におけるOTel Messaging計装の実践
sansantech
PRO
1
320
大規模イベントを支える ABEMA の アーキテクチャ 変遷 2025
nagapad
3
360
AIエージェントを支える設計
tkikuchi1002
11
2k
Wasmで社内ツールを作って配布しよう
askua
0
150
AIに全任せしないコーディングとマネジメント思考
kikuchikakeru
0
220
東京海上日動におけるセキュアな開発プロセスの取り組み
miyabit
0
180
(HackFes)米国国防総省のDevSecOpsライフサイクルをAWSのセキュリティサービスとOSSで実現
syoshie
5
670
Featured
See All Featured
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
760
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Raft: Consensus for Rubyists
vanstee
140
7k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.9k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Testing 201, or: Great Expectations
jmmastey
43
7.6k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Making Projects Easy
brettharned
116
6.3k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
Transcript
1 GCPͰ͡ΊΔ εϞʔϧελʔτͳσʔλ׆༻ #bq_sushi ver. bq_sushi #4 2016-09-06 Takashi Nishibayashi
2 Takashi Nishibayashi Software Engineer Zucks AdNetwork, Zucks Inc. Data
analysis team ݱࡏ৴ޮͷ࠷దԽ ೖࡳՁ֨ࣗಈௐϩδοΫɺ৴αʔ όʔͷࠂબϩδοΫͷ։ൃʹैࣄ @hagino3000
3 3 ͜ΕԿ͔ ಉͷGCP NEXT TOKYOͷࣄྫհηογϣ ϯͰൃදͨ͠༰ͷॖখ൛Ͱ͢
4 4 Zucks AdNetwork ͷσʔλ׆༻ͷมભ
5 5 5 ϓϩδΣΫτ։࢝࣌ͷཧͱݱ࣮
6 6 6 ࢦ͢ॴ(Ծ) ࠂ৴αʔόʔͰΠϯϓϨογϣϯຖʹػցֶशϞσϧʹΑΔίϯ όʔδϣϯ༧ଌɺΫϦοΫ༧ଌΛߦͳ͍৴ޮΛΞοϓ ݱ࣮ େྔͷϩάϑΝΠϧ͕༷ʑͳϑΥʔϚοτͰAWS S3ʹஔ͔Ε͍ͯΔ ϚελσʔλMySQLʹ֨ೲ͞Ε͍ͯΔ
Elastic SearchʹೖͬͯΔͷۙ2िؒ
7 7 7
8 8 8 ͍͖ͳΓ౸ୡͰ͖ͳ͍
9 1ظ: ·ͣσʔλαΠΤϯςΟετ͕ར༻Ͱ͖ΔΑ͏ʹ ü ωοτࠂۀքͰػցֶश͕ྲྀߦ͍ͬͯΔͱ͍͑ɺࣗαʔϏεͷ σʔλͰͦΕ͕Մೳͳͷ͔ݕূ͍ͨ͠ ü ࣮ݧԾઆݕূͷͨΊʹਓ͕ؒσʔλΛखܰʹར༻͍ͨ͠ ü ݶΒΕͨਓ͕ؒΫΤϦूܭΛ࣮ߦͰ͖Εྑ͍
ü ඦϛϦඵͷԠੑೳٻΊͳ͍ ü σʔλετΞͷཧʹख͕͔͔ؒΒͳ͍ࣄ͕ॏཁ ü σʔλྔ 600GByte/day ఔ͕ͩɺ·ͩ·ͩ૿͑ͦ͏
10 1ظ: ·ͣσʔλαΠΤϯςΟετ͕ར༻Ͱ͖ΔΑ͏ʹ ² ࠂͷ৴ϩάΛBigQueryʹྲྀ͠ࠐΜͩ ² MySQLͷϚελσʔλBigQueryʹಉظ ² WebUIPandasɺBigQuery Pythonܦ༝Ͱར༻
² BigQueryͰαϒαϯϓϦϯάͯ͠ϩʔΧϧϚγϯͰֶश ² AWS EMRୀ ² Elastic Searchୀ ² Cloud Datalab betaʹඈͼ͍ͭͯരࢮ (20161݄)
11 2ظ: όονॲཧ͔Βར༻Ͱ͖ΔΑ͏ʹ ü ܧଓతʹճ͍࣮ͨ͠ݧɺ༧ଌॲཧͷόονΛcronͰΒ͍ͤͨ ü ੳλεΫʹݶΒͣɺ৴γεςϜଆͷόονॲཧ͍͍ͨ ü ػೳຖͷ༻ঢ়گ(ΫΤϦίετ)Ѳ͍ͨ͠
12 2ظ: όονॲཧ͔Βར༻Ͱ͖ΔΑ͏ʹ ² CloudLoggingͷઃఆͰBigQueryͷࠪϩάΛBigQueryʹΤΫεϙʔτ ² ػೳຖʹαʔϏεΞΧϯτΛ͍ग़ͯ͠ɺ༻ঢ়گΛѲ ² ίετ͕ͶͨΒ௨ ²
ೖࡳ୯Ձࣗಈௐόονɺෆਖ਼ΫϦοΫఆόον͕Քಈ ² ϧʔϧϕʔεɺҟৗݕϕʔεͷࣝผλεΫSQLͰॻ͚Δ ² ࣮ݧ݁ՌCloud Storage/BigQueryʹอଘ
13
14
15 Audit Logͷ༻్ ² ػೳຖͷΫΤϦίετ ² ຖͷΫΤϦίετ ² ςετ༻ͷςʔϒϧ࡞ऀௐࠪ ²
ΘΕ͍ͯͳ͍ςʔϒϧௐࠪ
16 3ظ: ͯ͢ͷ৬छͷϝϯόʔ͕σʔλΛར༻Ͱ͖ΔΑ͏ʹ ü ఆܕͷௐࠪλεΫΤϯδχΞ๊͕͑ͨ͘ͳ͍ ü ίετ͕രൃ͠ͳ͍Α͏ʹར༻ऀΛ૿͍ͨ͠ ü SQLॻ͚Δਓ͕૿͑Δͱྑ͍ײ͡ʹͳΔͷͰ
17 3ظ: ͯ͢ͷ৬छͷϝϯόʔ͕σʔλΛར༻Ͱ͖ΔΑ͏ʹ ² re:dashͰΫΤϦͰ͖ΔΑ͏ʹͨ͠ ² ΤϯδχΞ͕ཁΛݩʹςϯϓϨʔτͷΫΤϦΛ࡞ ² Ϩϙʔτը໘ͷϓϩτλΠϓʹ ²
ΫΤϦ୯ҐͷίετϦϛοτઃఆ(re:dashͷػೳ)ͰߴֹΫΤϦ࣮ߦ Λࢭ
18 ཁٻ͞ΕΔσʔλ࣭ϨϕϧมΘΔ ü Ϣʔεέʔε͕૿͑Δͱσʔλ࣭͕՝ʹ ü 23࣌ͷϩάऔΓࠐΈ͕ऴͬͨޙʹॲཧΛΒ͍ͤͨΜ͚ͩͲ? ² Stream Insert, Batch
Insert, ΫΤϦશͯϦτϥΠػߏඞਢ ² ݄ʹ1BigQueryͷௐࢠͷѱ͍͕͋Δ ² σʔλͷऔΓࠐΈ࿙ΕɺॏෳऔΓࠐΈνΣοΫͷόονΛՔಇ ² σʔλͷऔΓࠐΈঢ়گ͕֎෦͔Β֬ೝͰ͖ΔΈ
19 ෭࣍తՌ • ΤϯδχΞ͕͍ͭͰ৴ϩάͷௐ͕ࠪՄೳʹ • MySQLͰѻ͑ͳ͔ͬͨαΠζͷσʔλΛݩʹͨ͠ҙࢥܾఆ͕Մೳʹ • ༷ʑͳόονॲཧ͕σʔλΛར༻Մೳʹ • SQLΛॻ͚ͩ͘ͰϨϙʔτ͕ࣗ༝ʹ࡞Մೳʹ
• ϓϩδΣΫτͷϝϯόʔશһ͕σʔλʹΞΫηεՄೳʹ
20 ͦͷଞ • ΦϯϥΠϯͰσʔλΛࢀর͢ΔΑ͏ͳॲཧʹBigQuery͔ͳ͍ • Key-ValueͰҾ͚ΔΑ͏ʹͯ͠BigtableΛͬͨํ͕͍͍ • BigQueryͷલʹΩϟογϡϨΠϠΛ༻ҙ͢Δࣄྫ • Cloud
Dataproc or Cloud Dataflow…… • SpotifySparkෳࡶ͗ͯ͑͢ͳ͍ͱͷࣄͰDataflowΛscala͔Βར༻ • https://github.com/spotify/scio • Cloud Datalab͕৽͘͠ͳͬͨͦ͏ͳͷͰظ • Jupyter NotebookͷΫϥυ൛
21 ·ͱΊ • ͍͖ͳΓ͍͠ॴΛૂ͏ͱՌ͕ग़Δ·Ͱ͕͔͔࣌ؒΔͨΊɺͳΒ͠Λ͠ ͳ͕Βσʔλ׆༻ΛਐΊ͍ͯΔ • SQLͰهड़Ͱ͖Δϧʔϧϕʔεҟৗݕϕʔεͷॲཧػցֶशͱൺֱ͢ Δͱૣ͘Ռ͕ग़ͤΔ • Cloud
Storage, Cloud Logging, Cloud Dataprocͱͷ࿈ܞ͕ڧԽ͞Εɺ BigQueryͷϢʔεέʔε͕૿͑ͨ • ඦmsecͷԠੑೳɺಉ࣌ΫΤϦ࣮ߦɺ҆ఆੑΛٻΊͳ͚ΕBigQuery Ϧʔζφϒϧʹ͑Δ
22 ิ BigQueryͰ౷ܭྔΛग़࣌͢ʹ͏ΫΤϦϝϞ http://qiita.com/hagino3000/items/e9ed62638ebe54391188
23 23 Thank You