Kubernetes で解決したこと新たに出てきた課題 /#jtf2017

1年を通して Kubernetes で解決したこと新たに出てきた課題をどう立ち向かったを発表しました

at July Tech Festa 2017

Transcript

1. ,PEBJ
   4BLBCF
   !LPVEBJJJ
   8BOUFEMZ
   *OD Kubernetes Ͱղܾͨ͜͠ͱ ৽ͨʹग़͖ͯͨ՝୊ July Tech Festa 2017
2. ABOUT ME — Software Engineer — 2010/4- TIS ࡔ෦ ޿େ (KODAI SAKABE) 2015/8- Wantedly @koudaiii Site: https://koudaiii.com
4. Kubernetes ෳ਺ͷϗετؒͰίϯςφԽ͞ΕͨΞϓϦέʔγϣϯΛ؅ཧ͢ΔͨΊͷOSS ΞϓϦέʔγϣϯͷdeployment, maintenance, scalingͷͨΊͷجຊతͳϝΧχζϜΛఏڙ https://github.com/kubernetes/kubernetes
5. Kubernetes Λར༻ͨ݁͠Ռ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͠΍͢͞ • εέʔϧͷ͠΍͢͞ • ՝୊ʹͳͬͨ͜ͱ • खಈͰ΍͖ͬͯͨ͜ͱͷݶք • Kubernetes ͷ Master ͕ SPOF • ϞχλϦϯάͷෳࡶ͞
6. Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
7. ΞδΣϯμ ڈ೥·ͰͷৼΓฦΓ ࣮ફ Kubernetes ղܾͨ͜͠ͱ / ՝୊΁ͷऔΓ૊Έ τΠϧ๾໓ӡಈ The UNIX Philosophy ϚϧνϚελʔ ࣗಈϞχλϦϯά
8. ڈ೥·ͰͷৼΓฦΓ
9. ͜Ε·ͰͷऔΓ૊Έ • Docker / Chef / Packer • Blue-Green Deployment • Terraform • CoreOS • cell
10. Docker / Chef / Packer • Docker image ͷதʹ Chef Λ࢖ͬͯΠϯϑϥΛߏங • Packer Ͱ AWS ͷ instance ͷ AMI Λ࡞Δ • Base ͱͳΔ template Λ༻ҙͯ͠ɺ৽͍͠αʔϏε͕ग़͖ͯͯ΋ྲྀ༻Ͱ ͖ΔΑ͏ʹͨ͠ • ΞϓϦέʔγϣϯͷߏங෦෼Λ Code Խ
11. Blue-Green Deployment • /healthcheck ͰϨεϙϯεΛฦͨ͠Β੾Γସ͑Δ UBHT
    
    OBNF
    BQQ
    
    BQQ
    SBJMT
    
    DPMPS
    HSFFO UBHT
    
    OBNF
    BQQ
    
    BQQ
    SBJMT
    
    DPMPS
    CMVF DBQ
    EFQMPZTXJUDI@DPMPS
    CMVF QPSU QPSU
12. Terraform • AWS(S3/RDS/ELB etc) / dnsimple Λૢ࡞ IUUQTTQFBLFSEFDLDPNEUBOUFSSBGPSNBUXBOUFEMZUFDIDJSDMFOVNCFS
13. TAG ͱ AutoScaleGroup(ASG) • instance ͷ໾ׂΛɺ EC2 ͷ TAG Λ࢖ͬͯ؅ཧ • Կ୆ಈ͔͍͔ͨ͠ʁ => ASGͰઃఆ • ΠϯϑϥπʔϧΛ LaunchConfig ʹఆٛ • instance ্ཱ͕͕ͪΔͱ౰࣌ʹࣗ෼ʹ෇͚ΒΕͨ TAG Λ؍ͯɺඞཁͳ service Λ systemd ʹ൓ө͠ɺαʔϏεΛىಈ(cell)
14. cell • AWS TAG Λݩʹ systemd ʹαʔϏεΛొ࿥ • αʔϏεͷ status ͕ ok ʹͳͬͨΒ DNS ࣗ਎Λ௥Ճ
15. ٕज़ελοΫ • ৽نαʔϏεΛ্ཱͪ͛Δࡍ͸͜ͷٕज़ ελοΫΛશ෦༻ҙ͢Δ • Πϯϑϥͷ҆ఆԽͷͨΊͷ༷ʑπʔϧͱ ࢓૊ΈΛ༻ҙ • αʔόʔΛߏங͢Δͱ͜Ζ͔ΒσϓϩΠɺ cell ·ͰΛίʔυԽ Web Application Monitoring Logging Auto Scale Load Balancer Internal DNS
16. ೉ղͳखॱ
17. ࣮ફ Kubernetes
18. Kubernetes ಋೖ΁ αʔϏε͕Ͱ͖Δ౓ʹҰݸͣͭαʔόʔΛ༻ҙͯ͠ߏங͢Δํ๏ ෳ਺ͷαʔόʔΛҰͭͷେ͖ͳαʔόʔʹݟཱͯͯར༻͢Δ ಉ͡Α͏ͳΠϯϑϥ࡞ۀΛݮΒ͠ɺϦιʔεΛޮ཰తʹར༻͢Δ ࢀߟ: CodeZineʮKubernetesΛ࢖ͬͨมԽʹڧ͍Πϯϑϥ――WantedlyͷΠϯϑϥνʔϜ͕େ੾ʹ͍ͯ͠Δ͜ ͱʯhttps://codezine.jp/article/detail/10357
19. The Twelve Factor App •12 ݸͷΞϓϦέʔγϣϯͷํ๏࿦ https://12factor.net/ja/ •Herokuࣄଶ΋ίϯςφٕज़ɻDocker Λӡ༻͍ͯ͘͠࠷దͳํ๏࿦ •σʔλετΞΛίϯςφʹؚΊͳ͍ •։ൃ؀ڥͱຊ൪؀ڥͷ ࠩҟΛ؀ڥม਺Ͱઃఆ •ܧଓతσϓϩΠ
20. ͦͷଞ௥Ճϧʔϧ Docker ͷ image ͸ܰ͘ CI Λ௨ͯ͠ɺimage ࡞੒ɺdeploy Λߦͳ͏ ϦϙδτϦͱ namespace ͕ 1ର1 ʹͳΔΑ͏ʹ
21. docker image ͸ܰ͘ https://github.com/koudaiii/sltd/blob/master/Dockerfile Golang Ͱ͋Ε͹όΠφϦΛΆΜى͖ͩͱখ͍͞ 1 FROM alpine:3.6 2 3 RUN apk add --no-cache --update ca-certificates 4 5 COPY bin/sltd /sltd 6 7 ENTRYPOINT ["/sltd"] 8 CMD ["help"]
22. CI/CD .travis.yml ΑΓൈਮ 58 script: 59 - ./script/ci-test # Test 60 - ./script/ci-build # docker build 61 - ./script/ci-push # docker push 62 ɾɾɾɾɾɾɾɾɾɾɾ 69 deploy: 70 skip_cleanup: true 71 provider: script 72 script: ./script/ci-deploy # Release 73 on: 74 all_branches: true
24. ϦϙδτϦ = namespace ֤ϦϙδτϦʹ /kubernetes Λ഑ஔ $ tree kubernetes/ kubernetes/ ᵓᴷᴷ jobs ᴹ ᵓᴷᴷ create-blog-job.yaml ᴹ ᵓᴷᴷ destroy-blog-job.yaml ᴹ ᵋᴷᴷ update-monthly-ranking-job.yaml ᵓᴷᴷ namespace.yaml ᵓᴷᴷ rails-docker-sample-run.yaml ᵓᴷᴷ rails-docker-sample-hpa.yaml ᵋᴷᴷ rails-docker-sample-svc.yaml
25. ϚχϑΣετϑΝΠϧ Namespace ઐ༻ͷ໊લۭؒ Deployment ΞϓϦέʔγϣϯͷ࣮ߦ HorizontalPodAutoscaler cpu/memory ͔Β pod Φʔτεέʔϧ Pod one-off ίϯςφ༻(rails c ౳Λߦͳ͏༻) Service SSL෇͖ELB Job oneshot Ͱ࣮ߦ༻(rake db:migrate ౳) CronJob Cron ܗࣜͰλεΫΛ࣮ߦ(rake task ౳)
26. Namespace ઐ༻ͷ໊લۭؒΛ࡞੒ 1 apiVersion: v1 2 kind: Namespace 3 metadata: 4 name: hoge
27. Deployment 1 apiVersion: extensions/v1beta1 2 kind: Deployment ɾɾɾɾɾɾɾɾɾɾɾ 9 spec: 11 strategy: 12 type: RollingUpdate 13 rollingUpdate: 14 maxSurge: 50% # Ұ౓൒෼ೖΕସ͑ 15 maxUnavailable: 0 # replicas Λҡ࣋ ɾɾɾɾɾɾɾɾɾɾɾ 23 spec: 24 containers: 25 - image: wantedly/python:latest 26 name: python 27 ports: 28 - containerPort: 8000 29 readinessProbe: # ready ͷఆٛ 30 httpGet: 31 path: /ping 32 port: 8000 33 initialDelaySeconds: 10 34 timeoutSeconds: 1 35 envFrom: 36 - secretRef: 37 name: dotenv 38 command: ["script/server"]
28. HorizontalPodAutoscaler Pod ͷ autoscale 1 apiVersion: autoscaling/v1 2 kind: HorizontalPodAutoscaler 3 metadata: 4 name: hoge 5 namespace: hoge 6 spec: 7 maxReplicas: 30 8 minReplicas: 1 9 scaleTargetRef: 10 apiVersion: extensions/v1beta1 11 kind: Deployment 12 name: python 13 targetCPUUtilizationPercentage: 50
29. Pod one-off Container(ྫ: rails c) 1 apiVersion: v1 2 kind: Pod 3 metadata: 4 name: {USER} 5 namespace: hoge 6 labels: 7 role: console 8 spec: ɾɾɾɾɾɾɾɾɾɾɾ 11 containers: 12 - image: wantedly/bash:latest 13 imagePullPolicy: Always 14 name: {USER} 15 command: 16 - bash 17 stdin: true 18 stdinOnce: true 19 terminationMessagePath: /dev/termination-log 20 tty: true 21 envFrom: 22 - secretRef: 23 name: dotenv
30. Service SSLূ໌ॻ෇͖ ELB 1 apiVersion: v1 2 kind: Service 3 metadata: 4 name: hoge 6 annotations: 7 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:reagion:x:certificate/x 8 service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http 9 labels: 10 name: hoge 11 role: web 12 spec: 13 ports: 14 - port: 443 15 protocol: TCP 16 targetPort: 8000 17 selector: 18 name: hoge 19 role: web 20 type: LoadBalancer
31. Job (ྫ rake db:migrate) 1 apiVersion: batch/v1 2 kind: Job ɾɾɾɾɾɾɾɾɾɾɾ 16 spec: 17 restartPolicy: Never 18 containers: 19 - name: db-migrate-[REPLACE_WITH_DATETIME] 20 image: wantedly/rails:[REPLACE_WITH_TAG] 21 command: ["bundle", "exec", "rake", "db:migrate"] 22 envFrom: 23 - secretRef: 24 name: dotenv
32. CronJob (ྫ rake task) concurrencyPolicy Ͱલ࣮ߦ͕࢒ͬͨ৔߹ͷϋϯυϦϯά͕ग़དྷΔ 1 apiVersion: batch/v2alpha1 2 kind: CronJob 3 metadata: 4 namespace: hoge 5 name: create-post 6 labels: 7 job: create-post 9 spec: 10 schedule: "*/30 * * * *" 11 concurrencyPolicy: “Replace" # “Allow” or “Forbid” 12 successfulJobsHistoryLimit: 5 13 failedJobsHistoryLimit: 5 ɾɾɾɾɾɾɾɾɾɾɾ
33. ղܾͨ͜͠ͱ / ՝୊΁ͷऔΓ૊Έ
34. ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͠΍͢͞ • εέʔϧͷ͠΍͢͞
35. ৽نαʔϏεͷग़͠΍͢͞ ͜Ε·Ͱ ݱࡏ
36. εέʔϧͷ͠΍͢͞ •͜Ε·Ͱ͸ instance ϨϕϧͰɺεέʔϧΛߦͳ͏࢓૊ΈΛ࡞Δඞཁ͕͋ͬͨ •ϔϧενΣοΫ •αʔϏεσΟεΧόϦ •DNSొ࿥ / Load Balancer •ࣗݾम෮ •Kubernetes ͔ΒɺԼهͷ఺͔Β༰қʹͳͬͨ •αʔϏεσΟεΧόϦͱDNSɺLoad Balancer ྑ͠ͳʹ͞Ε͍ͯΔ఺ •ίϯςφϨϕϧͷεέʔϧग़དྷΔ఺ $ kubectl autoscale deployment foo --min=2 —-max=10 —cpu-percent=50 # શମͷCPUUtilization 50% ᮢ஋Ͱ૿ݮ $ kubectl scale --replicas=3 deployment foo # ͦͷ৔Ͱ3ͭʹ͢Δ
37. ՝୊ʹͳͬͨ͜ͱ 1. खಈͰ΍͖ͬͯͨ͜ͱͷݶք 2. Kubernetes ͷ Master ͕ SPOF 3. ϞχλϦϯάͷෳࡶ͞
38. ՝୊΁ͷऔΓ૊Έ 1. τΠϧ๾໓ӡಈ 2. ϚϧνϚελʔ 3. ࣗಈϞχλϦϯά
39. “τΠϧͱ͸ɺϓϩμΫγϣϯαʔϏεΛಈ࡞ͤ͞Δ͜ͱʹ ؔ܎͢Δ࡞ۀͰɺख࡞ۀͰ܁Γฦ͠ߦΘΕɺࣗಈԽ͢Δ͜ͱ ͕ՄೳͰ͋Γɺઓज़తͰ௕ظతͳՁ஋Λ࣋ͨͣɺ࡞ۀྔ͕αʔ Ϗεͷ੒௕ʹൺྫ͢Δͱ͍ͬͨ܏޲Λ࣋ͭ΋ͷͰ͢ɻ” ൈਮɿ: Betsy Beyer “SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά”
40. ҰͭҎ্౰ͯ͸·ΔͱτΠϧͷՄೳੑ༗ • ख࡞ۀͰ͋Δ͜ͱ • ܁Γฦ͞ΕΔ͜ͱ • ࣗಈԽग़དྷΔ͜ͱ • ઓज़తͰ͋Δ͜ͱ(ઓུతͰ͋ͬͨΓ༧ଌʹجͮ͘΋ͷͰ͸ͳ͘ɺׂΓࠐΈͰ࢝·Γɺ໰୊ͳͲ ͕ੜͨ͜͡ͱ΁ͷରԠ) • ௕ظతͳՁ஋Λ࣋ͨͳ͍͜ͱ • αʔϏεͷ੒௕ʹରͯ͠O(n)Ͱ͋Δ͜ͱ
41. τΠϧ๾໓ӡಈ •test-build-push-release => CI ͷ template Խ •autoscale => HorizontalPodAutoscaler ϚχϑΣετΛॻ͘ •migrate => Job ϚχϑΣετΛॻ͘ •one-off => ઐ༻ͷ Pod ϚχϑΣετΛॻ͘ •SSL ͷखܰ͞ => Service ϚχϑΣετʹ annotation Λ௥ه͢Δͱઃఆ͞ΕΔ •deploy & rollback => The UNIX Philosophy ଇͬͨπʔϧ࡞੒ ࣮͸΄΅
    ,VCFSOFUFT
    Ͱ࣮ݱ
42. The UNIX Philosophy
43. The UNIX Philosophy ൈਮ • Small is beautiful. খ͍͞΋ͷ͸ඒ͍͠ • Make each program do one thing well. 1ͭͷϓϩάϥϜʹ͸1ͭͷ͜ͱΛ͏·͘΍ ΒͤΔ • Choose portability over efficiency. ޮ཰ΑΓҠ২ੑΛ༏ઌ͢Δ • Use software leverage to your advantage. ιϑτ΢ΣΞΛᑏࢠ(ͯ͜)ͱͯ͠࢖͏ • Avoid captive user interfaces. ա౉ͷର࿩తΠϯλʔϑΣʔεΛආ͚Δ
44. Server Tools ͱ Client Tool • Ұͭͷ͜ͱΛ্ख͘΍Δπʔϧ࡞Γɺᑏࢠͱͯ͠ར༻ => ͍ͭͰ΋ަ׵Մೳ • ΫϥΠΞϯτଆ͸׳Ε਌͠Μͩύοέʔδ؅ཧ => homebrew Ͱ഑৴ • ΫϥΠΞϯτଆ͸πʔϧΛݺͼग़ͨ͢Ίͷϥούʔʹઐ೦ • ΫϥΠΞϯτଆʹग़དྷΔݶΓϩδοΫΛೖΕͳ͍ => ॲཧ͸ server ্ • ωοτ੾அΛ૝ఆ͢Δ => server ͱ container ͸ҡ࣋
45. homebrew Golang πʔϧΛ homebrew Ͱ binary Λ഑৴ ΫϩείϯύΠϥͰOSґଘ͕ۃྗͳ͘πʔϧ͕࢖͑ΔΑ͏ʹ
46. kube ࣾ಺πʔϧ
47. खಈσϓϩΠ ϚχϑΣετϑΝΠϧʹ annotations Λ௥ه͢Δ͜ͱͰखಈσϓϩΠ͕Ͱ͖Δ k8ship deploy [BRANCH|COMMIT_SHA1] 1 apiVersion: extensions/v1beta1 2 kind: Deployment 3 metadata: 4 name: rails 5 namespace: rails 6 labels: 7 name: rails 8 role: web 9 annotations: 10 wantedly.com/deploy-target: “true" # ௥ه 11 wantedly.com/deploy-target-container: rails # ௥ه 12 wantedly.com/github: rails=wantedly/rails # ௥ه https://github.com/dtan4/k8ship
48. kube deploy ίϚϯυ -dry-run ΦϓγϣϯͰ diff Λग़͢ merge ޙ೚ҙͷλΠϛϯάͰ deploy ͍ͨ͠৔߹ʹར༻ $ kube qa deploy master --dry-run [dry-run] deploy to (deployment: "rails", container: "rails") [dry-run] before: wantedly/rails:d1cb608ee61cb18f8c397c2e27576573879b2fcf [dry-run] after: wantedly/rails:cb19269526f946222110fbdfb3d107c4b5a18fe7
49. ϚϧνϚελʔ
50. ϚϧνϚελʔ •Kubernetes ͷ version up ͸Ͳ͏͢Δͷʁ •Ϛελʔ͕ࢭ·ΔͱͲ͏ͳΔͷʁ •Ϋϥελͷఀࢭͳ͘ kops Λ࢖ͬͯ Rolling upgrade Ͱ͖ΔΑ͏ʹมߋ •ϚελʔΛࡾ୆༻ҙ͠ɺϚϧνϚελʔߏ੒ʹ͔͑ͯ SPOF վળ
51. ࣗಈϞχλϦϯά
52. ࣗಈϞχλϦϯά • 1ͭ1ͭઃఆ͢ΔͷͰ͸ͳ͘ɺ൚༻తʹద༻͞ΕΔΑ͏ʹ͢Δ • ྫ: High CPU {{.pod_name}} on {{.kube_namespace}} • request ͱ status code औಘ • μογϡϘʔυʹ൓ө • Deployment Ͱఆٛͨ͠਺
53. ࣗಈͰಉ͡඼࣭ͷϞχλϦϯάΛ໨ࢦ͢ https://www.datadoghq.com/blog/monitoring-101-collecting-data/ http://qiita.com/koudaiii/items/bc89368e1279649f2498
54. Dashboard
55. ϞχλϦϯάͰ࢖༻͍ͯ͠Δ΋ͷ • kelseyhightower/konfd => PostgresqlͳͲͷ৘ใΛϦϙδτϦʹؚΊͳ͍࣌ʹར༻ • kubernetes/kube-state-metrics => deployment ౳ͷ available Λऔಘ • datadog/docker-dd-agent => docker ༻ɺ web ༻ɺ db ༻ͱͯ͠໾ׂผͰ഑උ • koudaiii/sltd => Service ͷ metadata Λݩʹ ELB ͷ tag ʹ௥Ճ • logentries => ΫϥελશͯͷϩάΛૹΔ
56. ੒௕ਪҠ
57. αʔϏεਪҠ (2017/04Ἤ) Service 9 => 60 Container 700+ => 1,500 - 2,500+
58. ·ͱΊ • kubernetes ͸ɺΠϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬʹͳͬͨ • ղܾͨ͜͠ͱ • ৽نαʔϏεͷग़͠΍͢͞ • εέʔϧͷ͠΍͢͞ • ՝୊͔ΒऔΓ૊Μͩ͜ͱ • खಈͰ΍͖ͬͯͨ͜ͱͷݶք => τΠϧ๾໓ӡಈ • Ϛελʔ͕ SPOF => ϚϧνϚελʔԽ • ϞχλϦϯάͷෳࡶ͞ => ࣗಈϞχλϦϯά
59. Kubernetes Λར༻ͨ݁͠Ռ Πϯϑϥશମͷ࠷దԽΛߦͳ͏͖͔͚ͬ
60. IUUQTXXXXBOUFEMZDPNQSPKFDUT