Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
大容量SSDとOpenStreetMap
Search
Taro Matsuzawa aka. btm
October 22, 2021
Technology
0
100
大容量SSDとOpenStreetMap
OpenSourceConference 2021 Online/Fallの資料です。
Taro Matsuzawa aka. btm
October 22, 2021
Tweet
Share
More Decks by Taro Matsuzawa aka. btm
See All by Taro Matsuzawa aka. btm
OpenLayers ext TypeScript declarationの開発
smellman
0
480
平成生まれのためのUNIX&IT歴 史講座 ~番外編~
smellman
2
450
掛川城の点群データをiTownsで表示しよう
smellman
0
240
そのJavascript、全部TypeScriptにしちゃえ
smellman
1
190
MapLibreとtile.openstretmap.jpで始めるベクトル地図プログラミング
smellman
1
520
国内向けタイルサーバの構築と運用について
smellman
0
1.1k
Python/Javascriptで読む点群
smellman
1
4.8k
日本のCommunity向け タイルサーバの現状
smellman
0
2.6k
Pythonで点群を読んでみよう
smellman
1
2.3k
Other Decks in Technology
See All in Technology
アジャイルでの品質の進化 Agile in Motion vol.1/20241118 Hiroyuki Sato
shift_evolve
0
170
オープンソースAIとは何か? --「オープンソースAIの定義 v1.0」詳細解説
shujisado
10
1.1k
プロダクト活用度で見えた真実 ホリゾンタルSaaSでの顧客解像度の高め方
tadaken3
0
180
Lambda10周年!Lambdaは何をもたらしたか
smt7174
2
110
誰も全体を知らない ~ ロールの垣根を超えて引き上げる開発生産性 / Boosting Development Productivity Across Roles
kakehashi
1
230
開発生産性を上げながらビジネスも30倍成長させてきたチームの姿
kamina_zzz
2
1.7k
【令和最新版】AWS Direct Connectと愉快なGWたちのおさらい
minorun365
PRO
5
760
Oracle Cloud Infrastructureデータベース・クラウド:各バージョンのサポート期間
oracle4engineer
PRO
28
13k
OCI 運用監視サービス 概要
oracle4engineer
PRO
0
4.8k
BLADE: An Attempt to Automate Penetration Testing Using Autonomous AI Agents
bbrbbq
0
320
Application Development WG Intro at AppDeveloperCon
salaboy
0
190
EventHub Startup CTO of the year 2024 ピッチ資料
eventhub
0
120
Featured
See All Featured
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
25
1.8k
Product Roadmaps are Hard
iamctodd
PRO
49
11k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
42
9.2k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
20k
5 minutes of I Can Smell Your CMS
philhawksworth
202
19k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
900
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Unsuck your backbone
ammeep
668
57k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.3k
It's Worth the Effort
3n
183
27k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Transcript
େ༰ྔSSDͱOpenStreetMap Taro Matsuzawa(@smellman) OpenStreetMap Foundation Japan 1
ࣗݾհ ཧۭؒܥΤϯδχΞ ຊUNIXϢʔβձձ/Ұൠࣾஂ๏ਓOSGeoຊࢧ෦ཧࣄ/Ұൠࣾஂ๏ਓΦʔ ϓϯετϦʔτϚοϓϑΝϯσʔγϣϯδϟύϯϝϯόʔ ౦ژిػେֶOB/౦ژిػେֶCySecݱੜ breakcoreΫϥελ ͱݺΕͯ·͢
ʢୀࡁΈʣ
OSSͷ׆ಈ React NativeܥϥΠϒϥϦͷվम react-native-static-server (iOSରԠ), redux-persist (ϥΠϒϥϦΛ·Δ͝ͱ TypeScriptʹॻ͖͑Δ) ͳͲ OpenMapTilesϓϩδΣΫτͷߩݙɺཧؔ։ൃ༁
ckanext-spatial (Python3ରԠ) ͳͲ ͦͷଞɺؾ͕͘··ίʔυυΩϡϝϯτΛॻ͍ͯ·͢
ࣥචྺ ͦͷଞC MagazineͷಛूɺSoftware Designͷ࿈ࡌͳͲ ݩʑFirefox(Mozilla)Λத৺ʹ׆ಈɺݱࡏཧεϚϗΞϓϦ͕ϝΠϯ
એ
FOSS4G 2021 Japan Online 12/4-5 OSSͳཧͷΠϕϯτͰ͢ ίΞσΠͱνϡʔτϦΞϧσΠͷೋؒ ։࠵ ίΞσΠ͕ൃදத৺ Youtube
LiveͰͲͳͨͰݟΔࣄ͕Ͱ͖· ͢ https://www.osgeo.jp/events/foss4g-2021/foss4g-2021-japan-online
GEOSPATIAL Hackers Program ૯লओ࠵ͷཧۭؒܥΤϯδχΞͷҭϓϩάϥϜ ࠓ݄͙Β͍͔Βืू͕։࢝͞ΕΔͱࢥ͏ͷͰཁνΣοΫ ͳ͓ɺνϡʔλʔΛΓ·͢ https://ghp.connpass.com/
ຊ
େ༰ྔSSDͱOpenStreetMap Powered by Kioxia
͜ͱͷ͡·Γ ʮɺSSDି͢ͷͰϕϯνϚʔΫهࣄΛಉਓࢽʹॻ͍ͯΈͳ͍ʁʯ
ಉਓࢽ Kioxia(چ౦ࣳϝϞϦ)͕Maker Fair Tokyo ͚ʹ࡞ͨ͠ಉਓࢽ ΠϕϯτͷΦϯϥΠϯԽʹ͍PDFͰ ແঈͰެ։ SSDͷೱ͍͕ͨ͘͞Μ ͳ͔ͥͦ͜ʹࠞͬͨ͟ https://about.kioxia.com/ja-jp/news/2021/20210930-1.html
ࠓճಉਓࢽͷݩωλΛ͢ͱ͍͏ײ͡Ͱ͢ɻ ৽࡞ϕϯνϚʔΫͪΐͬͱ͚ͩ͋Δɻ
ςʔϚ OpenStreetMapʹ͍ͭͯ ି͠ग़͠ػࡐͱ࣮ݧڥʹ͍ͭͯ ࣮ࢪͨ͠ϕϯνϚʔΫͷհ ࠓޙͷలʹ͍ͭͯ
OpenStreetMapʹ͍ͭͯ
OpenStreetMapͱ(1) ୭Ͱࣗ༝ʹਤΛॻ͘͜ͱ͕Ͱ͖Δϓ ϥοτϑΥʔϜ ཁϢʔβొ ੈքதͷਤσʔλΛ୯Ұͷσʔλϕʔ εͰಈ͔͍ͯ͠Δ
OpenStreetMapͱ(2) σʔλ node, way, relation ͷΈ߹ΘͤͰΓཱͭσʔλϕʔε σʔλͷදݱܗࣜͱͯ͠XMLΛ࠾༻͍ͯ͠Δ σʔλ ODbL ͱ͍͏ϥΠηϯεͰެ։͞Ε͍ͯΔ
༻ར༻OK ͪͳΈʹosm.orgͰϗεςΟϯά͞ΕͯΔʮը૾ʯCC BY-SA
OpenStreetMapͱ(3) ͼ͗Ͷͬͱ͞ΜΛྫʹ͢Δͱ node͕ձࣾͷPoI(Point of Interest) ดͨ͡wayͰձࣾͷܗ(Polygon)Λදݱ ඒ௨Γ(Polyline)wayͰදݱ
OpenStreetMapͱ(4) ੈքதͷσʔλ(Planet)ΛXMLͰදݱ͢Δ ͱ1.5TB ѹॖͯ͢͠Δͷ͕ී௨ Bzip2: 109GB Protocol Buffer: 60GB
Protocol Bufferʹ͍ͭͯ XMLʹ࠷దԽ͞Εͨσʔλަܗࣜ ͔ͳΓσʔλѹॖ͕Մೳ 1.5TB XML -> 60GB Protocol Buffer
࠷ۙͷVector Tile෦Protocol Bufferܗ͕ࣜத৺ ϥΠϒϥϦͳͲॆ࣮͍ͯ͠ΔͷͰOSMͷσʔλΛ͏ͳΒσʔλ༰ྔʹ ߹ΘͤͯProtocol BufferͱXMLΛ͍͚Δͱྑ͍
ି͠ग़͠ػࡐͱ࣮ݧڥʹ͍ͭͯ
ࠓճػࡐ(PC)Λ·Δ͝ͱିͯ͠Β͍·ͨ͠ɻ ͳ͓ɺࠓିͯ͠Β͍ͬͯ·͢ɻ
εϖοΫ ROG STRIX Z390-F GAMING Intel Core i7-9700F 3.00GHz (8core/8thread)
DDR4 PC4-19200 CL15 8GBx2 SSD KXD5YLN13T84 (3,840GB NVMe)
εϖοΫ(͓·͚) AMD/ATI Vega 20 Ͳ͏ߟ͑ͯήʔϛϯάϚγϯͰ͢
None
SSD KXD5YLN13T84 Kioxiaͷσʔληϯλʔ͚SSD Ұൠ༻ʹʮചΒΕ͍ͯͳ͍ʯ ͍͓ͪ͏յౖͯ͠ΒΕͳ͍ͱ͍͏ʹͳͬͯΔ .oO(͍͘Β͢ΜͩΖɺාͯ͘ฉ͚ͳ͍) https://business.kioxia.com/ja-jp/ssd/data-center-ssd/xd5.html
None
None
جຊతͳڥ Debian/Bullseyes IPΞυϨεΛݻఆԽ ͋ͱSSHͰΑ͠ͳʹ جຊίϯιʔϧͰ࡞ۀ
جຊߏ
࣮ݧڥ ࣮ՈͷRaspberry Pi͔ΒࣗʹVPNͰৗ࣌ଓ ͔ࣗΒ࣮ՈͷRaspberry PiʹϩάΠϯ Raspberry PiΛ౿Έʹ࣮ͯ͠ݧػʹϩάΠϯͯ͠ͻͨ͢Β࡞ۀ ࣮ՈͷRaspberry PiʹPrometheusΛಈ͔ͯ͠I/OΛܭଌ
None
࣮ࢪͨ͠ϕϯνϚʔΫʹ͍ͭͯ
Overpass APIͷߏங OpenStreetMapͰҰൠతʹΘΕ͍ͯΔAPI XMLͰΫΤϦΛॻ͍ͯɺXMLͰฦΛड͚Δ ؆ུԽͨ͠ܗࣜͰΫΤϦΛॻ͘Overpass Turboͱ͍͏ͷ͋Δ ͋͘·Ͱଟ͘Πϯελϯε͕͋Δͷ͕Overpass API
Πϯετʔϧ https://wiki.openstreetmap.org/wiki/Overpass_API/Installation جຊతʹ͜͜ʹ͋Δํ๏ͰOK init_osm3s.sh ͷ࣮ߦ࣌ؒΛܭଌ
݁Ռ bin/init_osm3s.sh ../data/planet-210816.osm.bz2 $DB_DIR $EXEC_DIR 107634.58s user 12608.34s system 91%
cpu 36:32:35.51 total 1.5΄ͲͰΠϯϙʔτྃ
ߟ ೖྗܗ͕ࣜbzip2ܗࣜ ѹॖΛղ͘ͷʹίετ͕͔͔ͬͯͳ͍͔ʁ εΫϦϓτΛݺΜͰbunzip2Λ࣮ߦͯ͠Δͱ͜ΖΛಛఆͯ͠ɺ1.5TBͷXML ΛಡΈࠐ·ͤͯΈͨ
݁Ռ มԽͳ͠orz ୯ҰϓϩηεͰಈ࡞͢ΔͨΊɺΠϯϙʔτϓϩηεࣗମ͕ͦͦϘτ ϧωοΫʹͳ͍ͬͯͨ ಉ͘͡500MB/sec͙Β͍ͷεϐʔυ͕ग़͍ͯͨ ·ͩ·ͩߦ͚Δͣ
imposm3ʹΑΔPostgreSQLͷσʔλೖ Imposm3ͱ͍͏ෳͷϓϩηεͰΠϯϙʔτΛߦ͏πʔϧ͕͋Δ ΠϯϙʔτઌPostgreSQL+PostGIS+hstore PostgreSQL PGTune Λར༻ͯ͠νϡʔχϯά ೖྗϑΝΠϧProtocol Bufferܗࣜ https://imposm.org/docs/imposm3/latest/
݁Ռ ./imposm import -mapping mapping.json -connection -read -write 118021.70s user
28366.90s system 387% cpu 10:29:22.71 total 10࣌ؒ30΄ͲͰΠϯϙʔτྃ
ߟ ຊͷΠϯϙʔτͱੈքͷΠϯϙʔτΛൺֱ ੈքͷํ͕ύϑΥʔϚϯεʹҧ͍͕ग़ͨʂ
ੈքͷΠϯϙʔτ ຊͷΠϯϙʔτ࠷େͰread͕400MB/ sec͕ͩͬͨɺੈքͷΠϯϙʔτͰ࠷ େ1600MB/sec·ͰύϑΥʔϚϯε͕ग़ͨ ཧͷ3/4͙Β͍·Ͱग़͍ͯΔʂ
osm2pgroutingͷϕϯνϚʔΫ pgRoutingͱ͍͏ܦ࿏୳ࡧ༻ͷϓϩάϥϜΛௐࠪ ͳ͓ɺओͳpgRoutingओʹฐ͕ࣾϝϯςφϯε͍ͯ͠·͢ ͦͦIPAͷະ౿ϓϩδΣΫτൃ https://pgrouting.org/
τϥϒϧ osm2pgrouting͕ϝϞϦϦʔΫ͕ܹ͘͠ɺσϑΥϧτͷઃఆͰ͙͢ʹམ ͪͯ͠·͏ SwapϑΝΠϧΛ512GB༻ҙͯ͠ରԠ େ༰ྔSSDͳΒͰͷ͍ํ(·ͯ
݁Ռ ຊશҬΛΠϯϙʔτͨ͠ͱ͜Ζɺܦ࿏͕ਖ਼͘͠ΠϯϙʔτͰ͖ͳ͔ͬ ͨorz ΘΓʹؔ౦ͷΤϦΞΛΠϯϙʔτ ./osm2pgrouting -d pgrouting_kanto -U osm -W
osm -c -f 603.93s user 216.84s system 22% cpu 1:00:40.31 total 1࣌ؒ΄ͲͰΠϯϙʔτྃ
ܦ࿏୳ࡧٴͼ݁Ռ ࣗసं͚ϧʔτΛܭࢉ ΠϯϙʔτγϯάϧϓϩηεͳͷͰɺ readੑೳͰ200MB/secఔ ϚϧνϓϩηεԽͨ͠Βͬͱྑ͘ ͳΓͦ͏͕ͩɺ݁ߏͦ͠͏ʼʻ
ύϑΥʔϚϯε·ͱΊ ϚϧνϓϩηεͰ͔ͭೖྗ͕େ͖͍ϑΝΠϧͷಡΈࠐΈͷ߹ʹύ ϑΥʔϚϯε͕ߴ͘ग़Δͱ͍͏݁Ռʹͳͬͨ େ༰ྔSSDͳΒੈքதͷσʔλΛͿΜճͤΔʂ
ࠓޙͷలʹ͍ͭͯ
openmaptilesͷςετ ຊͷλΠϧαʔόߏஙʹ͍ͬͯΔϓϩάϥϜΛςετ͍ͯ͠Δ ෦తʹimposm3Λར༻ ݩʑDesk mini a300ͱ͍͏ϚγϯͰ͍ͬͯͨ࡞ۀ Desk mini a300ʹ٧ΜͰΔͷRyzen 3400G
(4core/8thread)ͳͷͰൺֱର ͱͯͪ͠ΐͬͱऑ͍ https://openmaptiles.org/
ੑೳൺֱ Asia ͷൣғͷΠϯϙʔτͷൺֱ Desk mini a300: 20ఔ Kioxiaͷݕূظ: 16ఔ ୯७ʹCPUͷੑೳࠩͱࢥΘΕΔ
όʔδϣϯΞοϓ OpenmaptilesϓϩδΣΫτͷDockerϑΝΠϧ͕࠷ۙόʔδϣϯΞοϓ ૣൺֱͯ͠Έͨ
݁Ռ 2021-10-4 (kioxia) Japan: 131345 seconds (36.5࣌ؒ) 2021-10-19 (kioxia) Japan:
100535 seconds (28࣌ؒ) ࠩ: 30810ඵ=8.5࣌ؒͷੑೳ্ ϨϯλϦϯάͷੑೳ͕େ෯ʹ্͕ͬͨͱࢥΘΕΔ
࠷ۙͷٞ JsonbܗࣜͰࠃࡍԽΛ͍ͯ͠ΔͷΛΒͨ͠ΒύϑΥʔϚϯε্͕͕Δ ͷͰʁ SQLϕʔεͰ͍ͬͯΔ࡞ۀΛPythonʹஔ͖͑ͨΒͲ͏ͳΔ͔ʁ νϡʔχϯάͷͨΊͷݕূ͕ٞ͞Ε͍ͯΔ
ͦͷଞͷϓϩάϥϜ https://tilemaker.org/ OpenStreetMap Foundation (ຊՈ)Ͱݕ౼͞Ε͍ͯΔϓϩάϥϜ OpenMapTilesͷՌΛ͍ͭͭɺ৽͍͠ΠϯϙʔτͷΈΛ࡞Ζ͏ ͱ͍ͯ͠Δ ͕࣌ؒऔΕͨΒͬͪ͜ͷϕϯνϚʔΫΛऔΖ͏͔ͱࢥ͍·͢
·ͱΊ
େ༰ྔSSDͱڊେͳσʔλϕʔε૬ੑ͕ྑ͍ ͨͩ͠ɺϚϧνϓϩηεͳͷͰͳ͍ͱSSDͷੑೳ͕ੜ͔ͤͳ͍ େ༰ྔSSDເ͕͋Δ 1TBͰΓ͘Γ͢ΔͷͦΖͦΖෆໟ
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠