Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How to Dev&Ops Internal PaaS

taichi nakashima
June 29, 2015
22
4.2k

How to Dev&Ops Internal PaaS

Talked at http://www.zusaar.com/event/9057007

taichi nakashima

June 29, 2015
Tweet

More Decks by taichi nakashima

See All by taichi nakashima
Platform Engineering at Mercari (Platform Engineering Kaigi 2024)
tcnksm
5
3.6k
Platform Engineering at Mercari
tcnksm
8
4.8k
Embedded SRE at Mercari
tcnksm
0
1.4k
How We Harden Platform Security at Mercari
tcnksm
2
1.7k
SRE Practices in Mercari Microservices
tcnksm
11
12k
開発者向けの基盤をつくる
tcnksm
38
12k
How We Structure Our Work At Mercari Microservices Platform Team
tcnksm
11
23k
Microservices Platform on Kubernetes at Mercari
tcnksm
16
16k
Introduction to Mercari Micorservices Platform Team
tcnksm
5
3.5k

Featured

See All Featured
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.2k
The Art of Programming - Codeland 2020
erikaheidi
53
13k
Put a Button on it: Removing Barriers to Going Fast.
kastner
59
3.6k
KATA
mclloyd
29
14k
Building Your Own Lightsaber
phodgson
103
6.1k
StorybookのUI Testing Handbookを読んだ
zakiyama
27
5.3k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
45
2.2k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
159
15k
Fantastic passwords and where to find them - at NoRuKo
philnash
50
2.9k
Practical Orchestrator
shlominoach
186
10k
Being A Developer After 40
akosma
87
590k
The Invisible Side of Design
smashingmag
298
50k

Transcript

  1. HOW TO DEV&OPS INTERNAL PAAS

  2. TAICHI NAKASHIMA @deeeet @tcnksm

  3. INTERNAL PAAS? = PaaS for Rakuten engineers

  4. ONLY FOR TEST? = No. It receives production requests

  5. WHY PAAS? = Fast app experimentation and iteration with PROD-grade

  6. WHY PAAS? = You don’t need to prepare servers by

    yourself

  7. WHY PAAS? = You don’t need to provision servers by

    yourself

  8. WHY PAAS? = You don’t need to prepare DBs by

    yourself

  9. WHY PAAS? = You can scale your app by *one

    command*

  10. WHY PAAS? = You can focus on development, not deployment

  11. WHY INTERNAL PAAS? = Easy to connect with other internal

    service

  12. WHY INTERNAL PAAS? = Instant support when something happen

  13. WHY INTERNAL PAAS? (From organizational point of view) = You

    can reduce duplicated tooling by different teams

  14. HOW LARGE? How many request? servers? language?

  15. 16000 req/sec. All application requests

  16. 2500 instances 1400 (PROD) + 700 (STG) + 400 (DEV)

  17. 4300 VMs 2800 (PROD) + 1200 (STG) + 300 (DEV)

  18. +300 VMs/mon. Growth forecasting

  19. 4 languages support Ruby, Node.js, Java, PHP

  20. 3 DB services Redis, MongoDB, Clustrix

  21. 100 Redis clusters 230 Instances

  22. 40 components Components (Roles) to run PaaS

  23. 320 chef recipes `ls cookbooks/*/recipes | wc -l`

  24. 8 Engineers Dev & Ops, From 7 Countries

  25. HOW TO DEV&OPS INTERNAL PAAS

  26. HOW TO DEV&OPS INTERNAL PAAS

  27. None

  28. Router API Health Check Messaging DBs Apps

  29. DEV FLOW RELEASE FLOW

  30. DEV FLOW RELEASE FLOW

  31. Create Ticket on JIRA Write code Write Chef cookbook Test

    on LAB Create PR (Git-Flow) Review

  32. DEV FLOW RELEASE FLOW

  33. Assign release manager Collect all JIRA tickets Write internal blog

    CanaryRelease Release

  34. 1 release for 1 week DEV (2day) , STG (2day)

    , PROD(3day)

  35. HOW TO RELEASE? = Chef + Capistrano

  36. RELEASE 1 SERVER

  37. Service-out Run Chef solo Run Serverspec Service-in

  38. Stop Load-Balancing Disable Health Check Stop monit Service-out Run Chef

    solo Run Serverspec Service-in Start monit Enable Health Check Start Load-Balancing

  39. /etc/service-out /etc/service-in Service-out Run Chef solo Run Serverspec Service-in

  40. Every server has same startup/stop scripts = workflow is same

    = automation is easy

  41. RELEASE X SERVERS

  42. cap service-in cap service-out cap setup-role Service-out X servers Run

    Chef solo X servers Run Serverspec X servers Service-in X servers

  43. Role A Role B Role C Operation 170.20.20.21.RoleA 170.20.20.22.RoleA 170.20.20.23.RoleA

    170.20.20.24.RoleA 170.20.20.25.RoleA 170.20.20.26.RoleA 170.20.20.27.RoleA VMLIST

  44. cap service-out 170.20.20.21.RoleA 170.20.20.22.RoleA 170.20.20.23.RoleA 170.20.20.24.RoleA 170.20.20.25.RoleA 170.20.20.26.RoleA 170.20.20.27.RoleA VMLIST

    Operation Role A Role B Role C Parallel execution

  45. cap setup-role Operation Parallel execution 170.20.20.21.RoleA 170.20.20.22.RoleA 170.20.20.23.RoleA 170.20.20.24.RoleA 170.20.20.25.RoleA

    170.20.20.26.RoleA 170.20.20.27.RoleA VMLIST Role A Role B Role C

  46. cap service-in Role A Role B Role C Operation 170.20.20.21.RoleA

    170.20.20.22.RoleA 170.20.20.23.RoleA 170.20.20.24.RoleA 170.20.20.25.RoleA 170.20.20.26.RoleA 170.20.20.27.RoleA VMLIST Parallel execution

  47. cap service-out Operation Parallel execution 170.20.20.31.RoleB 170.20.20.32.RoleB 170.20.20.33.RoleB 170.20.20.34.RoleB 170.20.20.35.RoleB

    170.20.20.36.RoleB 170.20.20.37.RoleB VMLIST Role A Role B Role C

  48. cap service-out 170.20.20.21.RoleA VMLIST Operation Role A Role B Role

    C Start from Canary

  49. HOW TO DEV&OPS INTERNAL PAAS

  50. LOGGING MONITORING ALERT HANDLING SUPPORT IAAS

  51. LOGGING MONITORING ALERT HANDLING SUPPORT IAAS

  52. 700GB/day logs All logs produced in PaaS

  53. LOGGING IN PAAS? = Application logs + Component logs

  54. APPLICATION LOG ? = PaaS should provide user the way

    to debug

  55. Instant logs Midterm logs Longterm logs Real time 1-2 weeks

    - 6 month

  56. Router API Health Check Messaging DBs Apps Instant log

  57. Log Server Apps Object Storage Instant log Midterm log Longterm

    log

  58. Log Server Apps Instant log Midterm log Hadoop (BigData team)

    Analytics

  59. Log Server Apps Instant log Midterm log Splunk  Dashboard

  60. COMPONENT LOG ? = Log which we use for debug

    PaaS itself

  61. Log Server Object Storage

  62. Log Server Object Storage We can debug CF here

  63. Log Server Object Storage GlusterFS LeoFS

  64. Log Server Object Storage GlusterFS

  65. LOGGING METRICS ALERT HANDLING SUPPORT IAAS

  66. OpenTSDB, Pandra FMS

  67. LOGGING METRICS ALERT HANDLING SUPPORT IAAS

  68. 1 week, 24H charge Primary & Sub admin

  69. ✉

  70. 2500 ✉/day MAX. Need to fix…

  71. LOGGING METRICS ALERT HANDLING SUPPORT IAAS

  72. JIRA, HipChat Instant support is one of *good* point of

    Internal PaaS

  73. LOGGING METRICS ALERT HANDLING SUPPORT IAAS

  74. IAAS Operating PaaS also means operating IaaS

  75. vSphere

  76. HOW TO BOOT SERVERS? = Internal tool like terraform

  77. Role A vSphere Operation rvc create -c rvc.yml 170.20.21.RoleA RoleA:

    cpu: 2 mem: 8192 rvc.yml 170.20.20.21.RoleA 170.20.20.22.RoleA 170.20.20.23.RoleA VMLIST

  78. Role A vSphere Operation VMLIST rvc create -c rvc.yml 170.20.21.RoleA

    RoleA: cpu: 2 mem: 8192 rvc.yml 170.20.20.21.RoleA 170.20.20.22.RoleA 170.20.20.23.RoleA VMLIST

  79. Role A vSphere Operation rvc create -c rvc.yml 170.20.21.RoleA RoleA:

    cpu: 2 mem: 8192 rvc.yml 170.20.20.21.RoleA 170.20.20.22.RoleA 170.20.20.23.RoleA VMLIST

  80. Role A vSphere Operation rvc create -c rvc.yml 170.20.22.RoleA RoleA:

    cpu: 2 mem: 8192 rvc.yml 170.20.20.21.RoleA 170.20.20.22.RoleA 170.20.20.23.RoleA VMLIST

  81. Role A vSphere Operation rvc create -c rvc.yml 170.20.23.RoleA RoleA:

    cpu: 2 mem: 8192 rvc.yml 170.20.20.21.RoleA 170.20.20.22.RoleA 170.20.20.23.RoleA VMLIST

  82. cap setup-role Role A Operation vSphere 170.20.20.21.RoleA 170.20.20.22.RoleA 170.20.20.23.RoleA VMLIST

  83. cap setup-role Role A Operation vSphere 170.20.20.21.RoleA 170.20.20.22.RoleA 170.20.20.23.RoleA VMLIST

  84. Easy to boot & setup servers = If there is

    *physical resource*

  85. FUTURE? = We are moving to *version 2*

  86. BE GOPHER CloudFoundry moves from Ruby to Golang

  87. NO FORK Everything goes to upstream

  88. BE OPEN Building tool as OSS

  89. ✉

  90. NO MORE TOO MUCH ✉ Planing to use Pagerduty +

    Riemann

  91. Log Server Object Storage GlusterFS LeoFS

  92. Object Storage LeoFS Kafka

  93. MORE FLEXIBLE LOG STACK Planning to use Apache Kafka

  94. NEW METRICS STACK Planning to use InfluxDB + Grafana

  95. CONTAINER Planning to support Docker

  96. MORE HA Planning to have a ChaosMonkey

  97. NEW IAAS Migrating to OpenStack

  98. NEW IAAS Planning to Hybrid Cloud

  99. WE HAVE MANY CHALLENGES

  100. WE ARE HIRING http://corp.rakuten.co.jp/careers/experienced/

  101. @deeeet