Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ScaleShift-jp-2019-summer

Avatar for ryo nakamaru ryo nakamaru
July 17, 2019
Research
1
200

 ScaleShift-jp-2019-summer

ScaleShift の機能概要、Kubernetes クラスタとの連携について
Avatar for ryo nakamaru

ryo nakamaru

July 17, 2019
Tweet

More Decks by ryo nakamaru

See All by ryo nakamaru
AWSで楽をするサービスメッシュ入門/appmesh-trial
Avatar for ryo nakamaru pottava
1
1.4k
reinforce-2019-recap-lt
Avatar for ryo nakamaru pottava
2
4.1k
Firecracker とは何か/what is Firecracker
Avatar for ryo nakamaru pottava
12
5.4k
ハイブリッド並列 on Kubernetes/hybrid-parallel-program-on-kubernetes
Avatar for ryo nakamaru pottava
1
420
AWS Fargate + Code 兄弟で始める継続的デリバリー / Continuous Delivery with AWS Fargate and Code brothers
Avatar for ryo nakamaru pottava
12
3.2k
Singularity と NVIDIA GPU Cloud で作る ハイブリッド機械学習環境の構築 / Building a hybrid environment for Machine Learning with Singularity and NGC
Avatar for ryo nakamaru pottava
3
1.3k
明日から始めるちょい足し λ / get-started-with-aws-lambda
Avatar for ryo nakamaru pottava
4
2.4k
NGC と Singularity によるハイブリッド機械学習環境 / A hybrid environment for Machine Learning with NGC and Singularity
Avatar for ryo nakamaru pottava
0
480
NGC × Singularity での機械学習環境/MachineLearning environment with NGC and Singularity
Avatar for ryo nakamaru pottava
2
760

Other Decks in Research

See All in Research
ASSADS：ASMR動画に合わせて撫でられる感覚を提示するシステムの開発と評価 / ec75-shimizu
Avatar for yumulab yumulab
1
190
請求書仕分け自動化での物体検知モデル活用 / Utilization of Object Detection Models in Automated Invoice Sorting
Avatar for Sansan R&D sansan_randd
0
180
Pix2Poly: A Sequence Prediction Method for End-to-end Polygonal Building Footprint Extraction from Remote Sensing Imagery
Avatar for SatAI.challenge satai
3
290
90 分で学ぶ P 対 NP 問題
Avatar for E869120 e869120
15
6.9k
さくらインターネット研究所 アップデート2025年
Avatar for MATSUMOTO Ryosuke matsumoto_r
PRO
0
530
言語モデルによるAI創薬の進展 / Advancements in AI-Driven Drug Discovery Using Language Models
Avatar for tsurubee tsurubee
2
310
Self-supervised audiovisual representation learning for remote sensing data
Avatar for SatAI.challenge satai
3
130
言語モデルLUKEを経済の知識に特化させたモデル「UBKE-LUKE」について
Avatar for 田村光太郎 petter0201
0
360
チャッドローン：LLMによる画像認識を用いた自律型ドローンシステムの開発と実験 / ec75-morisaki
Avatar for yumulab yumulab
1
200
資産間の相関関係を頑健に評価する指標を用いたファクターアローケーション戦略の構築
Avatar for s-noma nomamist
0
190
A Segment Anything Model based weakly supervised learning method for crop mapping using Sentinel-2 time series images
Avatar for SatAI.challenge satai
3
300
Scale-Aware Recognition in Satellite images Under Resource Constraints
Avatar for SatAI.challenge satai
3
180

Featured

See All Featured
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
280
13k
Building Better People: How to give real-time feedback that sticks.
Avatar for Will Jessup wjessup
367
19k
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
Typedesign – Prime Four
Avatar for Hannes Fritz hannesfritz
41
2.6k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
63
7.7k
Optimizing for Happiness
Avatar for Tom Preston-Werner mojombo
378
70k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
Avatar for Tammy Everts tammyeverts
52
2.4k
RailsConf 2023
Avatar for Aaron Patterson tenderlove
30
1.1k
KATA
Avatar for Megan Lloyd mclloyd
29
14k
Bash Introduction
Avatar for André Augusto Costa Santos 62gerente
612
210k
The Success of Rails: Ensuring Growth for the Next 100 Years
Avatar for Eileen M. Uchitelle eileencodes
45
7.2k
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
48
49k

Transcript

  1. ScaleShift ΦϯϓϨϛε / Ϋϥ΢υ Ͱ࣮ݱ͢Δػցֶश؀ڥ June, 2019

  2. !2 ScaleShift Docker ϕʔεɺΦʔϓϯιʔεͷ Web ΫϥΠΞϯτ ΞϓϦέʔγϣϯͰ͢ • ϞσϧߏஙϑΣʔζ -

    NGC / ࣗࣾϦϙδτϦ͔Βػցֶश Docker ΠϝʔδΛϫϯΫϦοΫͰऔಘ - ͦͷ೚ҙͷ Docker ΠϝʔδΛ Jupyter notebook ίϯςφͱͯ͠ىಈ • ϞσϧֶशϑΣʔζ - ߏஙʹར༻ͨ͠ϥΠϒϥϦ͝ͱ Docker ΠϝʔδʹݻΊϦϙδτϦ΁อଘ - ΫϦοΫ͚ͩͰ Kubernetes Ϋϥελ / Rescale ΁େن໛ܭࢉλεΫΛૹ৴

  3. جຊతͳಈ͖ !3 How does it work?

  4. !4 ScaleShift ͷىಈ ϩʔΧϧʹ Web αʔόʔ্ཱ͕͕ͪΓ·͢

  5. !5 ػցֶशιϑτ΢ΣΞͷΠϯετʔϧ NGC / ϓϥΠϕʔτϨδετϦ ͔ΒϫϯΫϦοΫͰμ΢ϯϩʔυ

  6. !6 Jupyter notebook ͰͷϞσϧߏங Jupyter Ͱϥοϓͨ͠ίϯςφ͕͔ΜͨΜʹىಈ ϙʔτ΋࡞ۀྖҬ΋ ίϯςφ͝ͱʹ ෼཭͞Εͨ ΫϦʔϯͳ؀ڥ

    ɹ.

  7. !7 େن໛ܭࢉͷͨΊͷϥοϐϯά ґଘϥΠϒϥϦ΍ιʔείʔυ܈Λ·ͱΊɺͻͱͭͷΠϝʔδʹݻΊ·͢

  8. !8 ࣾ಺Ϋϥελ / Ϋϥ΢υ΁ܭࢉλεΫ౤ೖ ౤ೖઌʹԠͯ͡ඞཁͳ API ͕࣮ߦ͞Ε·͢ ར༻ϦιʔεྔΛܾΊ Ϋϥελ΁λεΫ౤ೖ

  9. Kubernetes ࿈ܞ !9 Integration with a kubernetes cluster

  10. !10 ػցֶश ͱ Kubernetes Web ք۾Λத৺ʹίϯςφΦʔέετϨʔγϣϯͷσϑΝΫτʹͳͬͨ k8sɻ ػցֶशͷจ຺Ͱ΋ίϯςφར༻͕੝ΜʹͳΓɺԠ༻ࣄྫ͕૿͍͑ͯ·͢ɻ - NVIDIA

    ͕ެࣜʹαϙʔτΛද໌ [ GTC 2018 Keynote, March 27 ] - Mercari ML Ops Night Vol.1 [ גࣜձࣾ ϝϧΧϦ / May 23, 2018 ] ɹhttps://mercari.connpass.com/event/85931/presentation/ - Jupyter ͚ͩͰػցֶशΛ࣮αʔϏεల։Ͱ͖Δج൫ [ גࣜձࣾϦΫϧʔτϥΠϑελΠϧ ] ɹhttps://engineer.recruit-lifestyle.co.jp/techblog/2018-10-04-ml-platform/ - KubernetesʹΑΔػցֶशج൫΁ͷ௅ઓ [ גࣜձࣾ Preferred Networks / Dec 4, 2018 ] ɹhttps://www.slideshare.net/pfi/kubernetes-125013757

  11. !11 ScaleShift + Kubernetes ߏ੒ྫ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC

    DockerHub ϓϥΠϕʔτ ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ ScaleShift ೖΓ ϩʔΧϧ୺຤

  12. !12 1. ػցֶशιϑτ΢ΣΞͷબ୒ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ GUI ͔ΒબͿ͚ͩͰ μ΢ϯϩʔυ͕࢝·Γ·͢

  13. !13 2. Ϟσϧߏங ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ ScaleShift ͕ ϊʔτϒοΫΛىಈ͠·͢

  14. !14 3. ࣮ߦ؀ڥɾೖྗσʔλͷసૹ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ScaleShift ͕಺෦తʹ ඞཁͳసૹΛߦ͍·͢ ݚڀ / ։ൃνʔϜ

  15. !15 4. େن໛ܭࢉͷ࣮ߦΛࢦࣔ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ Kubernetes ͷ Job ͱͯ͠ ܭࢉ৚݅Λૹ৴͠·͢

  16. !16 5. େن໛ܭࢉͷ࣮ߦ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub Kubernetes

    ϓϥΠϕʔτ ϨδετϦ ݚڀ / ։ൃνʔϜ

  17. !17 6. ܭࢉ݁Ռͷ֬ೝ ετϨʔδ ؅ཧϊʔυ ܭࢉϊʔυ ࣾ಺ωοτϫʔΫ NGC DockerHub ϓϥΠϕʔτ

    ϨδετϦ Kubernetes ݚڀ / ։ൃνʔϜ

  18. !18 Kubernetes ઃఆ / λεΫ࣮ߦը໘

  19. ScaleShift ͷઃఆ !19 Configurations

  20. !20 ֎෦࿈ܞ ࿈ܞػೳ ઃఆ஋ NVIDIA GPU CLOUD • NVIDIA ࣾͷ؅ཧ͢Δػցֶश

    Docker Πϝʔδͷ Ұཡ / ৄࡉ৘ใऔಘɺΠϝʔδͷμ΢ϯϩʔυ ɹAPI Ωʔ & Ϣʔβઃఆ ϓϥΠϕʔτϨδετϦ • ࣗࣾͰ؅ཧ͢Δػցֶश Docker Πϝʔδͷ Ұཡ৘ใऔಘɺΠϝʔδͷμ΢ϯϩʔυ ɹ઀ଓઌ & Ϣʔβઃఆ AWS • ػցֶश Docker Πϝʔδͷμ΢ϯϩʔυ • ϩʔΧϧϑΝΠϧγεςϜͱ S3 ؒͷσʔλ࿈ܞ ʢ࣮૷༧ఆʣ Kubernetes • ࣾ಺Ϋϥελ / Ϋϥ΢υͰͷେن໛ܭࢉ࣮ߦ ɹkubecfg Rescale • Rescale ϓϥοτϑΥʔϜͰͷେن໛ܭࢉ࣮ߦ ɹ஍Ҭࢦఆ & API Ωʔ

  21. !21 ىಈΦϓγϣϯʢൈਮʣ ઃఆ֓ཁ ॳظ஋ SS_JUPYTER_MINIMUM_PORT ίϯςφ΁ͷ઀ଓϙʔτಈతׂ౰։࢝൪߸ ɾ30000 SS_LOG_LEVEL ΞϓϦέʔγϣϯͷϩάग़ྗϨϕϧ ɹwarn

    SS_WORKSPACE_HOST_DIR ϗετଆͷ࡞ۀσʔλอଘྖҬ ɹͳ͠ʢࢦఆඞਢʣ SS_NGC_REGISTRY_ENDPOINT NGC ઀ଓઌ ɹhttps://registry.nvidia.com SS_NGC_REGISTRY_USER_NAME NGC Ϣʔβʔ໊ ɹ$oauthtoken SS_RESCALE_SINGULARITY_VERSION Rescale Ͱͷ Singularity ϥϯλΠϜόʔδϣϯ ɹ3.2.0 SS_RESCALE_JOB_WALLTIME Rescale ͰͷλεΫ࣮ߦ࠷େ࣌ؒ ɾ3600 docker-compose.yml ʹઃఆΛهࡌɺىಈͰ͖·͢