goodby-waiting-status-forever

Mesos Meetup Tokyo #3
https://mesos.connpass.com/event/67877/

ref1. Apache Mesos 
with Amazon EC2 SpotFleet
https://speakerdeck.com/kotatsu360/apache-mesos-with-amazon-ec2-spotfleet

Transcript

1. © 2017 VASILY,Inc. શͯͷ“Waiting”Λ ੜ·ΕΔલʹফ͠ڈΓ͍ͨɻ Mesos Meetup Tokyo #3
 2017/11/01 Wed. Tatsuro MITSUNO
2. © 2017 VASILY,Inc. ࣗݾ঺հ ▸ Tatsuro Mitsuno / ޫ໺ ୡ࿕ ▸ 2012/04 Yahoo Japan Corporation ▸ 2016/04 VASILY, Inc. ▸ Infrastructure Engineer ▸ Twitter GitHub Qiita: @kotatsu360 Icon illustrated by YOSHI 
 https://ja-jp.facebook.com/yoshi.yone.7 Yukihiro "Matz" Matsumoto
 (VASILY technical advisor ) 
3. © 2017 VASILY,Inc. Mesos Meetup Tokyo #2 ʹͯ https://speakerdeck.com/kotatsu360/apache-mesos-with-amazon-ec2-spotfleet Apache Mesos ͱ Amazon EC2 SpotFleetͰߏங͞Εͨ
 ฐࣾͷΫϩʔϥʔج൫ʹ͍ͭͯ͝঺հ͠·ͨ͠ɻ
4. © 2017 VASILY,Inc. VASILYͷΫϩʔϥʔج൫  scaling request control SpotFleet Instance scaling request Lambda CloudWatch / CloudWatch Event update request
5. © 2017 VASILY,Inc. ຊ೔ͷςʔϚ Marathonͷ“Waiting”Λղܾ͢Δ ͜Ε
6. © 2017 VASILY,Inc. Waitingͱ͸  ▸ Marathon͕λεΫʹඞཁͳϦιʔεΛ֬อͰ͖ͳ͍ঢ়ଶ ▸ ެࣜυΩϡϝϯτͷτϥϒϧγϡʔςΟϯάͷୈҰ߲໨͸
 An app stays in “Waiting” forever ▸ ୭΋͕ૺ۰͢Δ ▸ ࣌ؒܦաͰղܾ͠ͳ͍ ▸ ղܾํ๏ɿෆ଍͍ͯ͠ΔϦιʔεΛಛఆͯ͠ɺ௥Ճɾ։์
7. © 2017 VASILY,Inc. Waitingͷ೉఺ ▸ ͲͷϦιʔε͕ෆ଍͍ͯ͠Δ͔Θ͔Βͳ͍ɾɾɾ ▸ ϝϞϦɾCPUɾϙʔτɾetc,,, ▸ Mesos͕֬อՄೳͳϦιʔεશ͕ͯݪҼʹͳΓ͏Δ ▸ ͔͠͠ɺز͔ͭͷ͋Γ͕ͪͳύλʔϯʹ෼ྨՄೳ ▸ ຊ೔͝঺հ͢Δͷ͸࣮ࡍʹࢲ͕ૺ۰ͨ͠Waitingͱͦͷରࡦ 
8. © 2017 VASILY,Inc. Waitingͦͷ1ɿΫϥελϦιʔε଍Γͯ·͔͢ʁ
9. © 2017 VASILY,Inc.  Waitingͦͷ1ɿΫϥελϦιʔε଍Γͯ·͔͢ʁ ݪҼɿλεΫ͕࢖͏Ϧιʔε > MesosΫϥελͷۭ͖Ϧιʔε ֬ೝɿλεΫ͕ඞཁͱ͢ΔϝϞϦ΍CPUΛܭࢉͯ͠ɺMesos UIͰۭ͖Λ֬ೝ ରࡦɿMesosΫϥελΛ૿΍͢ɾλεΫ΁ͷׂ౰ྔΛݮΒ͢ λεΫA ϝϞϦ512M ϝϞϦ 2048M ʮ2560M֬อʯ ʮϝϞϦ଍Βͳ͍ʯ λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫAΛ5ݸ ϝϞϦ512M Waiting
10. © 2017 VASILY,Inc. Waitingͦͷ2ɿλεΫͷ੍໿ຬͨͯ͠·͔͢ʁ
11. © 2017 VASILY,Inc.  Waitingͦͷ2ɿλεΫͷ੍໿ຬͨͯ͠·͔͢ʁ ݪҼɿ੍໿͖ͭλεΫͷ੍໿͕ຬͨͤͳ͍ ֬ೝɿλεΫͷ੍໿͕ຬͨ͞Ε͍ͯΔ͔ݟ௚͢ʢಛʹϢχʔΫ੍໿ʣ ରࡦɿϊʔυΛ૿΍͢ λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫBΛ5ݸ Constraints = hostname, unique Waiting ΤʔδΣϯτ ʮ1λεΫ/1ϗετʯ ʮϗετ଍Γͳ͍ʯ ΤʔδΣϯτ ΤʔδΣϯτ ΤʔδΣϯτ
12. © 2017 VASILY,Inc.  Waitingͦͷ2ѥछɿϦιʔεยدͬͯແ͍Ͱ͔͢ʁ ݪҼɿ੍໿͖ͭλεΫ͕࢖͏Ϧιʔε>ಛఆϗετͷۭ͖Ϧιʔε ֬ೝɿλεΫ͕ಛఆͷϗετʹยدͬͯͳ͍͔ʢكͳঢ়گʣ
 ରࡦɿఆظతʹλεΫΛϦελʔτ͢Δɾ੍໿͖ͭλεΫΛे෼ʹখ͘͢͞Δ λεΫA ϝϞϦ512M λεΫA ϝϞϦ512M λεΫBΛ2ݸ Constraints = hostname, unique Waiting ʮ1λεΫ/1ϗετʯ ʮ1୆ͰϝϞϦ଍Γͳ͍ʯ ΤʔδΣϯτ ϝϞϦ࢖༻཰ 90% ΤʔδΣϯτ ϝϞϦ࢖༻཰ 20%
13. © 2017 VASILY,Inc. Waitingͦͷ3ɿϦιʔεͷ࢖༻ڐՄ͋Γ·͔͢ʁ
14. © 2017 VASILY,Inc.  Waitingͦͷ3ɿϦιʔεͷ࢖༻ڐՄ͋Γ·͔͢ʁ ݪҼɿλεΫ͕࢖͏ϙʔτͷ࢖༻ڐՄ͕ͳ͍ ֬ೝɿλεΫ͕ಛผͳϦιʔεΛ࢖Θͳ͍͔֬ೝ ରࡦɿλεΫʹrequirePortsɺagentϓϩηεʹresourcesΦϓγϣϯΛ௥Ճ
 https://mesosphere.github.io/marathon/docs/troubleshooting.html#requiring-a-specific-host-port Waiting λεΫCΛ1ͭ ϗετϙʔτ80Λ࢖༻ ʮ80ϙʔτʯ ʮڐՄແ͠ʯ σϑΥϧτઃఆͷ
 mesos-agentϓϩηε
15. © 2017 VASILY,Inc.  Waitingͦͷ3ѥछɿطଘλεΫ͕ར༻͍ͯ͠·ͤΜ͔ʁ ݪҼɿ৽نλεΫ͕࢖͍͍ͨϙʔτΛطଘλεΫ͕όΠϯυࡁΈ ֬ೝɿൃੜλΠϛϯάɻσϓϩΠ࣌ͳΒϙʔτ͕͔ͪ߹͍ͬͯΔՄೳੑ
 ରࡦɿupgradeStrategyͰɺminimumHealthCapacityͱmaximumOverCapacityʹ0Λࢦఆ
 ʢ৽نλεΫͷσϓϩΠલʹطଘλεΫ͕kill͞ΕΔʣ Waiting λεΫCΛ1ͭ ϗετϙʔτ80Λ࢖༻ ʮ80ϙʔτʯ ʮ࢖༻தʯ λεΫC ʢrunningʣ
16. © 2017 VASILY,Inc. ·ͱΊ ▸ λεΫ͕ඞཁͱ͢Δ֤छϦιʔε͕֬อͰ͖ͳ͍৔߹ɺWaitingʹͳΔ ▸ ୅දతͳWaitingύλʔϯ ▸ ϝϞϦɾCPU͕଍Γͳ͍ ▸ ੍໿͕ຬͨͤͳ͍ ▸ ಛఆͷϦιʔεΛ࢖͏ݖݶ͕ͳ͍ ▸ ద੾ͳϦιʔε෼഑ͱಉ࣌ʹɺλεΫࣗମͷݟ௚͠΋ ▸ ੍໿΍ϗετϙʔτ͕ඞཁͳ͍λεΫͰ͋Ε͹ࣗવʹ࢖͑Δ 
17. © 2017 VASILY,Inc. We are Hiring https://www.wantedly.com/companies/vasily