Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ScaleShift-jp-2019-summer

Avatar for ryo nakamaru ryo nakamaru
July 17, 2019
Research
230
1

 ScaleShift-jp-2019-summer

ScaleShift の機能概要、Kubernetes クラスタとの連携について

Avatar for ryo nakamaru

ryo nakamaru

July 17, 2019

More Decks by ryo nakamaru

See All by ryo nakamaru
AWSで楽をするサービスメッシュ入門/appmesh-trial
Avatar for ryo nakamaru pottava
1
1.6k
reinforce-2019-recap-lt
Avatar for ryo nakamaru pottava
2
4.2k
Firecracker とは何か/what is Firecracker
Avatar for ryo nakamaru pottava
12
5.6k
ハイブリッド並列 on Kubernetes/hybrid-parallel-program-on-kubernetes
Avatar for ryo nakamaru pottava
1
460
AWS Fargate + Code 兄弟で始める継続的デリバリー / Continuous Delivery with AWS Fargate and Code brothers
Avatar for ryo nakamaru pottava
12
3.3k
Singularity と NVIDIA GPU Cloud で作る ハイブリッド機械学習環境の構築 / Building a hybrid environment for Machine Learning with Singularity and NGC
Avatar for ryo nakamaru pottava
3
1.5k
明日から始めるちょい足し λ / get-started-with-aws-lambda
Avatar for ryo nakamaru pottava
4
2.5k
NGC と Singularity によるハイブリッド機械学習環境 / A hybrid environment for Machine Learning with NGC and Singularity
Avatar for ryo nakamaru pottava
0
530
NGC × Singularity での機械学習環境/MachineLearning environment with NGC and Singularity
Avatar for ryo nakamaru pottava
2
880

Other Decks in Research

See All in Research
多様なデータを許容し学習し続ける模倣学習 / Advanced Imitation Learning for VLA
Avatar for PRIN Lab prinlab
0
190
typst の使い方：言語学を研究する学生のために
Avatar for Tomoya gitomochang
0
430
COFFEE-Japan PROJECT Impact Report（海ノ向こうコーヒー）
Avatar for 坂ノ途中 ontheslope
0
1.7k
RS-Agent: Automating Remote Sensing Tasks through Intelligent Agent
Avatar for SatAI.challenge satai
0
220
2026-01-30-MandSL-textbook-jp-cos-lod
Avatar for Yuka Egusa yegusa
1
1.2k
[BlackHatAsia2026] Hidden Telemetry: Uncovering TraceLogging ETW Providers You're Not Using (Yet)
Avatar for Asuka Nakajima asuna_jp
1
460
Research Engineerという仕事 / Research Engineering: Bridging Research and Business
Avatar for chck chck
1
150
正規分布と最適化について
Avatar for koide3 koide3
0
210
コーディングエージェントとABNを再考
Avatar for Hironobu Fujiyoshi hf149
2
670
討議：RACDA設立30周年記念都市交通フォーラム2026
Avatar for Traffic Brain trafficbrain
0
900
Unified Audio Source Separation (Defense Slides)
Avatar for Kohei Saijo kohei_1979
1
600
第66回コンピュータビジョン勉強会＠関東 Epona: Autoregressive Diffusion World Model for Autonomous Driving
Avatar for Kento Sasaki kentosasaki
0
610

Featured

See All Featured
Designing Dashboards & Data Visualisations in Web Apps
Avatar for Des Traynor destraynor
231
55k
It's Worth the Effort
Avatar for 3n 3n
188
29k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
66
8.5k
Groundhog Day: Seeking Process in Gaming for Health
Avatar for Sebastian Deterding codingconduct
0
190
Building a Scalable Design System with Sketch
Avatar for Laura Van Doore lauravandoore
463
34k
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
333
25k
Design of three-dimensional binary manipulators for pick-and-place task avoiding obstacles (IECON2024)
Avatar for konakalab konakalab
0
430
SEO Brein meetup: CTRL+C is not how to scale international SEO
Avatar for Linda Hogenes lindahogenes
1
2.7k
Jess Joyce - The Pitfalls of Following Frameworks
Avatar for Tech SEO Connect techseoconnect
PRO
1
150
Lightning Talk: Beautiful Slides for Beginners
Avatar for Ines Montani inesmontani
PRO
1
560
The Illustrated Guide to Node.js - THAT Conference 2024
Avatar for David Neal reverentgeek
1
360
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
47
8.1k

Transcript

  1. ScaleShift ΦϯϓϨϛε / Ϋϥ΢υ Ͱ࣮ݱ͢Δػցֶश؀ڥ June, 2019

  2. !2 ScaleShift Docker ϕʔεɺΦʔϓϯιʔεͷ Web ΫϥΠΞϯτ ΞϓϦέʔγϣϯͰ͢ • ϞσϧߏஙϑΣʔζ -

    NGC / ࣗࣾϦϙδτϦ͔Βػցֶश Docker ΠϝʔδΛϫϯΫϦοΫͰऔಘ - ͦͷ೚ҙͷ Docker ΠϝʔδΛ Jupyter notebook ίϯςφͱͯ͠ىಈ • ϞσϧֶशϑΣʔζ - ߏஙʹར༻ͨ͠ϥΠϒϥϦ͝ͱ Docker ΠϝʔδʹݻΊϦϙδτϦ΁อଘ - ΫϦοΫ͚ͩͰ Kubernetes Ϋϥελ / Rescale ΁େن໛ܭࢉλεΫΛૹ৴

  3. جຊతͳಈ͖ !3 How does it work?

  4. !4 ScaleShift ͷىಈ ϩʔΧϧʹ Web αʔόʔ্ཱ͕͕ͪΓ·͢

  5. !5 ػցֶशιϑτ΢ΣΞͷΠϯετʔϧ NGC / ϓϥΠϕʔτϨδετϦ ͔ΒϫϯΫϦοΫͰμ΢ϯϩʔυ

  6. !6 Jupyter notebook ͰͷϞσϧߏங Jupyter Ͱϥοϓͨ͠ίϯςφ͕͔ΜͨΜʹىಈ ϙʔτ΋࡞ۀྖҬ΋ ίϯςφ͝ͱʹ ෼཭͞Εͨ ΫϦʔϯͳ؀ڥ

    ɹ.

  7. !7 େن໛ܭࢉͷͨΊͷϥοϐϯά ґଘϥΠϒϥϦ΍ιʔείʔυ܈Λ·ͱΊɺͻͱͭͷΠϝʔδʹݻΊ·͢

  8. !8 ࣾ಺Ϋϥελ / Ϋϥ΢υ΁ܭࢉλεΫ౤ೖ ౤ೖઌʹԠͯ͡ඞཁͳ API ͕࣮ߦ͞Ε·͢ ར༻ϦιʔεྔΛܾΊ Ϋϥελ΁λεΫ౤ೖ

  9. Kubernetes ࿈ܞ !9 Integration with a kubernetes cluster

  10. !10 ػցֶश ͱ Kubernetes Web ք۾Λத৺ʹίϯςφΦʔέετϨʔγϣϯͷσϑΝΫτʹͳͬͨ k8sɻ ػցֶशͷจ຺Ͱ΋ίϯςφར༻͕੝ΜʹͳΓɺԠ༻ࣄྫ͕૿͍͑ͯ·͢ɻ - NVIDIA

    ͕ެࣜʹαϙʔτΛද໌ [ GTC 2018 Keynote, March 27 ] - Mercari ML Ops Night Vol.1 [ גࣜձࣾ ϝϧΧϦ / May 23, 2018 ] ɹhttps://mercari.connpass.com/event/85931/presentation/ - Jupyter ͚ͩͰػցֶशΛ࣮αʔϏεల։Ͱ͖Δج൫ [ גࣜձࣾϦΫϧʔτϥΠϑελΠϧ ] ɹhttps://engineer.recruit-lifestyle.co.jp/techblog/2018-10-04-ml-platform/ - KubernetesʹΑΔػցֶशج൫΁ͷ௅ઓ [ גࣜձࣾ Preferred Networks / Dec 4, 2018 ] ɹhttps://www.slideshare.net/pfi/kubernetes-125013757

  11. !11 ScaleShift + Kubernetes ߏ੒ྫ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC

    DockerHub ϓϥΠϕʔτ ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ ScaleShift ೖΓ ϩʔΧϧ୺຤

  12. !12 1. ػցֶशιϑτ΢ΣΞͷબ୒ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ GUI ͔ΒબͿ͚ͩͰ μ΢ϯϩʔυ͕࢝·Γ·͢

  13. !13 2. Ϟσϧߏங ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ ScaleShift ͕ ϊʔτϒοΫΛىಈ͠·͢

  14. !14 3. ࣮ߦ؀ڥɾೖྗσʔλͷసૹ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ScaleShift ͕಺෦తʹ ඞཁͳసૹΛߦ͍·͢ ݚڀ / ։ൃνʔϜ

  15. !15 4. େن໛ܭࢉͷ࣮ߦΛࢦࣔ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ Kubernetes ͷ Job ͱͯ͠ ܭࢉ৚݅Λૹ৴͠·͢

  16. !16 5. େن໛ܭࢉͷ࣮ߦ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub Kubernetes

    ϓϥΠϕʔτ ϨδετϦ ݚڀ / ։ൃνʔϜ

  17. !17 6. ܭࢉ݁Ռͷ֬ೝ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ

  18. !18 Kubernetes ઃఆ / λεΫ࣮ߦը໘

  19. ScaleShift ͷઃఆ !19 Configurations

  20. !20 ֎෦࿈ܞ ࿈ܞػೳ ઃఆ஋ NVIDIA GPU CLOUD • NVIDIA ࣾͷ؅ཧ͢Δػցֶश

    Docker Πϝʔδͷ Ұཡ / ৄࡉ৘ใऔಘɺΠϝʔδͷμ΢ϯϩʔυ ɹAPI Ωʔ & Ϣʔβઃఆ ϓϥΠϕʔτϨδετϦ • ࣗࣾͰ؅ཧ͢Δػցֶश Docker Πϝʔδͷ Ұཡ৘ใऔಘɺΠϝʔδͷμ΢ϯϩʔυ ɹ઀ଓઌ & Ϣʔβઃఆ AWS • ػցֶश Docker Πϝʔδͷμ΢ϯϩʔυ • ϩʔΧϧϑΝΠϧγεςϜͱ S3 ؒͷσʔλ࿈ܞ ʢ࣮૷༧ఆʣ Kubernetes • ࣾ಺Ϋϥελ / Ϋϥ΢υͰͷେن໛ܭࢉ࣮ߦ ɹkubecfg Rescale • Rescale ϓϥοτϑΥʔϜͰͷେن໛ܭࢉ࣮ߦ ɹ஍Ҭࢦఆ & API Ωʔ

  21. !21 ىಈΦϓγϣϯʢൈਮʣ ઃఆ֓ཁ ॳظ஋ SS_JUPYTER_MINIMUM_PORT ίϯςφ΁ͷ઀ଓϙʔτಈతׂ౰։࢝൪߸ ɾ30000 SS_LOG_LEVEL ΞϓϦέʔγϣϯͷϩάग़ྗϨϕϧ ɹwarn

    SS_WORKSPACE_HOST_DIR ϗετଆͷ࡞ۀσʔλอଘྖҬ ɹͳ͠ʢࢦఆඞਢʣ SS_NGC_REGISTRY_ENDPOINT NGC ઀ଓઌ ɹhttps://registry.nvidia.com SS_NGC_REGISTRY_USER_NAME NGC Ϣʔβʔ໊ ɹ$oauthtoken SS_RESCALE_SINGULARITY_VERSION Rescale Ͱͷ Singularity ϥϯλΠϜόʔδϣϯ ɹ3.2.0 SS_RESCALE_JOB_WALLTIME Rescale ͰͷλεΫ࣮ߦ࠷େ࣌ؒ ɾ3600 docker-compose.yml ʹઃఆΛهࡌɺىಈͰ͖·͢