Lock in $30 Savings on PRO—Offer Ends Soon! ⏳
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Hive 集計テクニック
Search
Yuki Ishikawa
April 19, 2019
Technology
0
440
Hive 集計テクニック
2019.04.19 中国地方DB勉強会 in 沖縄
Yuki Ishikawa
April 19, 2019
Tweet
Share
More Decks by Yuki Ishikawa
See All by Yuki Ishikawa
ORM と向き合う
hoto17296
8
7.1k
明日業務で役立たない Web 開発 TIPS
hoto17296
0
100
Python をフル活用した工場への AI 導入 & データ活用基盤構築事例
hoto17296
0
2.5k
クソ bot 実装ライブコーディング
hoto17296
0
180
DeepGBM 論文の紹介
hoto17296
0
500
試行錯誤のための Docker 活用術
hoto17296
4
2.7k
データ分析と Docker / Data Analysis with Docker
hoto17296
0
310
DeepCluster 論文の紹介
hoto17296
7
2.4k
最新論文を追う技術 / Technology to follow the latest paper
hoto17296
2
240
Other Decks in Technology
See All in Technology
Reliability Engineering at Studist
katsuhisa91
PRO
0
120
Nihonbashi Test Talk #3_WebDriver BiDiと最新の実装状況 / WebDriver BiDi latest status
takeyaqa
1
150
How is Cilium Tested?
yutarohayakawa
5
300
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
5
52k
Will Positron accelerate us?
lycorptech_jp
PRO
1
120
Advancing the 3D Geospatial Ecosystem in Japan via Global Collaborations
osgeojp
0
170
pmconf2024_UPSIDER
upsider_tech
0
7.3k
イベントをどう管理するか
mikanichinose
1
110
12/2(月)のBedrockアプデ速報(re:Invent 2024 Daily re:Cap #1 with AWS Heroes)
minorun365
PRO
2
310
深層学習のリペア技術の最新動向と実際 / DNN Repair Techniques for AI Performance Alignment for Safety Requirements
ishikawafyu
0
480
Classmethod_regrowth_2024_tokyo_security_identity_governance_summary
hiashisan
0
380
PR TIMESにおけるNext.jsとcacheの付き合い方
apple_yagi
2
170
Featured
See All Featured
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
1
94
Producing Creativity
orderedlist
PRO
341
39k
A Modern Web Designer's Workflow
chriscoyier
693
190k
No one is an island. Learnings from fostering a developers community.
thoeni
19
3k
Building an army of robots
kneath
302
43k
Faster Mobile Websites
deanohume
305
30k
BBQ
matthewcrist
85
9.3k
4 Signs Your Business is Dying
shpigford
181
21k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
126
18k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Visualization
eitanlees
145
15k
Mobile First: as difficult as doing things right
swwweet
222
8.9k
Transcript
Hive ूܭςΫχοΫ 2019.04.19 தࠃํDBษڧձ in ԭೄ @hoto17296
RDB ͚͕ͩ DB Ͱͳ͍ ʂʂʂʂʂ
@hoto17296 • ͪΎΒσʔλגࣜձࣾ σʔλΞφϦετ • ԭೄͷडୗσʔλੳձࣾ • ࠓͷΠϕϯτʹԿਓ͔ࣾһ͍Δ
খωλͰ͢
Apache Hive • Hadoop ͷࢄετϨʔδ (HDFS) ্ͷ σʔλΛ SQL ϥΠΫʹૢ࡞Ͱ͖Δݴޠ
• େنσʔλੳج൫ͳͲʹΑ͘ΘΕΔ • PostgreSQL ͱ͔ͱಉ͡ϊϦͰॻ͘ͱࢮ͵ • Map Reduce ͷ͓ؾ࣋ͪΛͯ͠
Γ͍ͨ͜ͱ time ࠂID ੑผ طࠗ ऩ 1555664019 253678c9 உ
20-24 ະࠗ 500ສ 1555664020 2baf1f40 ঁ 35-38 طࠗ 400ສ 1555664022 105d9501 உ 25-29 ະࠗ 500ສ 1555664022 fe319a8e உ 30-34 طࠗ 600ສ 1555664024 c463fdf4 ঁ 20-24 طࠗ 400ສ 1555664027 325a2a5d உ 40-44 طࠗ 800ສ 1555664029 f445adf3 ঁ 25-29 ະࠗ 500ສ 1555664034 fe319a8e உ 35-38 طࠗ 600ສ 1555664034 f572f626 ঁ 25-29 ະࠗ 400ສ Α͋͘ΔΦʔσΟΤϯεσʔλ
Γ͍ͨ͜ͱ time ࠂID ੑผ طࠗ ऩ 1555664019 253678c9 உ
20-24 ະࠗ 500ສ 1555664020 2baf1f40 ঁ 35-38 طࠗ 400ສ 1555664022 105d9501 உ 25-29 ະࠗ 500ສ 1555664022 fe319a8e உ 30-34 طࠗ 600ສ 1555664024 c463fdf4 ঁ 20-24 طࠗ 400ສ 1555664027 325a2a5d உ 40-44 طࠗ 800ສ 1555664029 f445adf3 ঁ 25-29 ະࠗ 500ສ 1555664034 fe319a8e உ 35-38 طࠗ 600ສ 1555664034 f572f626 ঁ 25-29 ະࠗ 400ສ ࠂ ID ϢχʔΫ͔ͱࢥ͍͖ ΊͪΌͪ͘Όॏෳ͍ͯ͠Δ ಉ͡ ID Ͱଐੑσʔλ (ਪఆ) ͕ ͦΕͧΕҧ͏
Γ͍ͨ͜ͱ time ࠂID ੑผ طࠗ ऩ 1555664019 253678c9 உ
20-24 ະࠗ 500ສ 1555664020 2baf1f40 ঁ 35-38 طࠗ 400ສ 1555664022 105d9501 உ 25-29 ະࠗ 500ສ 1555664022 fe319a8e உ 30-34 طࠗ 600ສ 1555664024 c463fdf4 ঁ 20-24 طࠗ 400ສ 1555664027 325a2a5d உ 40-44 طࠗ 800ສ 1555664029 f445adf3 ঁ 25-29 ະࠗ 500ສ 1555664034 fe319a8e உ 35-38 طࠗ 600ສ 1555664034 f572f626 ঁ 25-29 ະࠗ 400ສ time ͕࠷৽͍͠1ߦ͚ͩͯ͠ଞશͯআ֎͍ͨ͠
1. ·ͣࢥ͍ͭͭ͘
Ϛονϣա͗Δ • શΧϥϜॻ͔ͳ͍ͱ͍͚ͳ͍ • ༻్͕ຊདྷͷ MAX Ͱͳ͍ • จࣈྻΧϥϜʹ MAX
͢Δͷؾ͕Ҿ͚Δ • ࠷৽ͷσʔλ͕औΕΔΘ͚Ͱͳ͍
2. ΟϯυؔΛ͏ͭ
ROW_NUMBER Πϝʔδ time ࠂID ੑผ طࠗ ऩ rank 1555664019
253678c9 உ 20-24 ະࠗ 500ສ 1 1555664020 2baf1f40 ঁ 35-38 طࠗ 400ສ 1 1555664022 105d9501 உ 25-29 ະࠗ 500ສ 1 1555664022 fe319a8e உ 30-34 طࠗ 600ສ 2 1555664024 c463fdf4 ঁ 20-24 طࠗ 400ສ 1 1555664027 325a2a5d உ 40-44 طࠗ 800ສ 1 1555664029 f445adf3 ঁ 25-29 ະࠗ 500ສ 1 1555664034 fe319a8e உ 35-38 طࠗ 600ສ 1 1555664034 f572f626 ঁ 25-29 ະࠗ 400ສ 1 PARTITION BY ORDER BY
ROW_NUMBER ศར • ͱͯΘ͔Γ͍͢ • ͔֬ PostgreSQL Ͱ͑Δ • ൚༻ੑ
(ʁ) ͕͋ͬͯྑ͍
3. Hivemall Λ͏ͭ
Hivemall • Hive ্Ͱػցֶश͢ΔͨΊͷϥΠϒϥϦ • SQL ͰػցֶशͰ͖Δ • Apache Incubation
Project ʹબΕͨ • ͍͢͝ (খฒײ)
EACH_TOP_K ؔ • Hivemall ʹؚ·Ε͍ͯΔؔ • Ϋϥελʔ͝ͱʹ K ݸͷσʔλΛऔಘͰ͖Δ •
(ϢʔεέʔεʹΑͬͯ) ROW_NUMBER ΑΓ͍
͞ͷݕূ • Treasure Data ͷαϯϓϧσʔληοτͰ͋Δ NASDAQ ͷגՁσʔλ (880ສߦ) Λର •
֤ฑ͝ͱͷ࠷৽ͷגՁΛऔಘ͢ΔΫΤϦΛ ROW_NUMBER ͱ EACH_TOP_K Ͱॻ͖ɺ ࣮ߦ࣌ؒΛܭଌ͢Δ
ݕূ݁Ռ 1.46 ഒ͘Β͍ EACH_TOP_K ͷํ͕ ͔ͬͨ ROW_NUMBER EACH_TOP_K 85 ඵ
124 ඵ
ͳ͍͔ͥʁ time symbol volume rank 1555102800 APPL 198.87 1 1555016400
APPL 198.95 2 1554930000 APPL 200.62 3 1554843600 APPL 199.50 4 1555102800 MSFT 120.95 1 1555016400 MSFT 120.33 2 1554930000 MSFT 120.19 3 1554843600 MSFT 119.28 4 ROW_NUMBER શͯͷߦʹ൪߸Λ ৼ͔ͬͯΒߜΓࠐΉ
ͳ͍͔ͥʁ time symbol volume 1555102800 APPL 198.87 1555016400 APPL 198.95
1554930000 APPL 200.62 1554843600 APPL 199.50 1555102800 MSFT 120.95 1555016400 MSFT 120.33 1554930000 MSFT 120.19 1554843600 MSFT 119.28 EACH_TOP_K ඞཁͳ͚ͩऔಘͨ͠Β ͋ͱॲཧ͠ͳ͍
Φν͕ͳ͍ (·ͱΊ) • Hive Ͱʮάϧʔϓ͝ͱʹτοϓ N ݅Λऔಘʯ ͍ͨ͠߹ ROW_NUMBER ͕͑Δ
• Hivemall ͕͑Δ߹ EACH_TOP_K Λ ͏ͱΑΓ͘ͳΔ͔͠Εͳ͍
ʲPRʳ