Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Service Operation Centered Development

Service Operation Centered Development

DevOpsDaysTokyo 2019

Mitsuyuki Shiiba

April 05, 2019

More Decks by Mitsuyuki Shiiba

Other Decks in Technology


  1. Service Operation Centered Development 2019/04/09 DevOpsDays Tokyo 2019 Mitsuyuki Shiiba

    (@bufferings) EC Incubation Development Dept. Rakuten, Inc.
  2. @bufferings #devops_b2 30 countries & regions with local operational presence

    70 + services Almost 1.3 B global members ¥ 15.4 T global gross transaction value *FY2018 Figures are from https://global.rakuten.com/corp/ accessed on April 3rd, 2019
  3. @bufferings #devops_b2 Too many alerts Non-automated tests Long methods Meaningless

    names Manual deployment etc… Not updated documents Big ball of mud Dirty servers What makes Service Operation tough
  4. @bufferings #devops_b2 We should connect dev & ops in our

    heads putting Service Operation at the center
  5. @bufferings #devops_b2 Good culture, feature flags, chatops, and more. Rakuma

    team wants you! -> (Wantedly) https://shiiba.page.link/rakuma if current_time <= 7m
  6. @bufferings #devops_b2 It's US who get woken up at midnight,

    not anyone else. Photo by Andre Mouton on Unsplash
  7. @bufferings #devops_b2 Is this really necessary? • Can we check

    it next morning? • Can we retry it? • Can we develop automatic recovery? What's actually happened? • What's the impact on the users? • What should we do for it?
  8. @bufferings #devops_b2 so that everyone can think about the impact

    & the solution LogMessageBuilder message("What's happened?") .cause("What's the cause?") .impact("What's the user impact?") .solution("What do we have to do?") .build()
  9. @bufferings #devops_b2 to know the actual impact on the users

    (Learned) Handle them at the controller layer Data Access Application Service Controller ← We can know the user impact ← We can't know the user impact
  10. @bufferings #devops_b2 Fool Proof to use the tools at ease

    Fail Safe to keep consistency for every single line Idempotence so that we can send a same message multiple times Eventual Consistency for automatic recovery Decoupled Architecture to minimize the user impact
  11. @bufferings #devops_b2 Codes/Tests/Docs Readable • having reason for every single

    line • meaningful names • small methods • overview → detail Maintainable • keep them updated • just enough quality & quantity
  12. @bufferings #devops_b2 EC Start-up Group is mobbing everyday! -> (Rakuten

    Careers) https://shiiba.page.link/ecsu if current_time <= 15m
  13. @bufferings #devops_b2 Feel the forces in the org. Respect &

    trust other people. Doing the right thing in your place doesn't work
  14. @bufferings #devops_b2 Trust everyone feels joy to make the service

    better. Trust people, doubt Ba(場: environment)
  15. @bufferings #devops_b2 See what would come next for the service,

    and prepare for them • business • technology • design • organization Everything is changing rapidly
  16. @bufferings #devops_b2 Service Operation Development Interaction 1. Reply to all

    the alerts 2. Safe by Design 3. Services over Projects 1. Feel the forces 2. See around the corner Summary
  17. @bufferings #devops_b2 10+ Deploys Per Day: Dev and Ops Cooperation

    at Flickr https://conferences.oreilly.com/velocity/velocity2009/public/sc hedule/detail/7641 Appendix
  18. @bufferings #devops_b2 "Yutapon coding" copyright © 2007 plx system co.

    http://net2.system.to/pc/font.html About Font