2019-10-17 17Media SRE Journey

2019-10-17 DevOpsDays Taipei

Transcript

1. 1 17Media SRE Journey Sammy Lin 2019-10-18 DevOpsDays Taipei
2. 43& Site Reliability Engineering
3. SRE is what you get when you treat operations as if it’s a software problem. Our mission is to protect, provide for, and progress the software and systems behind all of Google’s public services — Google Search, Ads, Gmail, Android, YouTube, and App Engine, to name just a few — with an ever-watchful eye on their availability, latency, performance, and capacity. What is SRE? 3FGFSFODF
   
   IUUQTMBOEJOHHPPHMFDPNTSF
5. SRE is not just a position. It's a culture.
6. 6 Job Description System Administrator • Architecture planning, setup, backup, update, security protection and management of LINUX servers and operating systems. DevOps Engineer • Build and improve our CI/CD process and tools
 • Manage AWS or GCP environment SRE • Scale our applications and infrastructure. • Develop monitoring systems. • Participate in our on-call rotation.
8. There are no best practices. It has to be customized toward your organization.
9. 17Media SRE Milestone
10. Why called SRE? ை
11. 11 17Media SRE 2015 • 17Media founded • Builded on AWS 2016/8 • First DevOps Engineer joined 2018/5 • Migrated to GCP 2017/1 • Migrated from beanstalk to ECS 2018/6 • Migrated from node.js to golang 2019/8 • Building GRE
13. 3FGFSFODF
    
    IUUQTDMPVEHPPHMFDPNCMPHQSPEVDUTHDQCSJOHJOHQPLFNPOHPUPMJGFPOHPPHMFDMPVE
15. (3& Group Reliability Engineering
16. 16 舉⼿手電商
 有限公司 Paktor Group
 拍拖集團 17 Media BVI ⿇麻吉⼀一七股份有限公司 台灣分公司
17. 17 Our Vision • The SREs in different companies can support each other. • Training Jr. SRE to Sr. • Stronger negotiation power toward 3rd party vendors. • Copy successful experience.
18. SRE Culture
19. 19 Broken into 5 key areas: Accept Failure as Normal Reduce Organization Silos Implement Gradual Change Leverage Tooling & Automation Measure Everything
22. Accept Failure as Normal
23. Psychological
 Safety Blamelessness Accept Failure as Normal
24. Psychological Danger Psychological Safety Fear of admitting mistakes Blaming others Less likely to share different views Common knowledge effect Comfort admitting mistakes Learning from failure Everyone openly shares ideas Better innovation & decision- making
25. Story 三塊巧克⼒：法國媽媽如何教育孩⼦ 故事內容：https://kknews.cc/education/3jm6nay.html Blamelessness TPVSDF
    
    IUUQTLLOFXTDDFEVDBUJPOKNOBZIUNM
26. 26 Learning in this story Blameless Action Items Lessons Learned
27. 27 Failure is the key to success each mistake teaches us something. — Morihei Ueshiba
28. How does 17Media Accept Failure as Normal?
29. 29 One of Our Postmortem • Summary • Full disk caused SQL outage • Root Cause • mysql-tailer didn't set log rotation • mysql-4 boot disk was full • Timeline • 2019/04/26 10:25: Alert triggered. • 2019/04/26 10:28: Discussion thread began. • 2019/04/26 10:35: Alert closed. • 2019/04/26 10:42: Fully recovered. • How to prevent this happening again? • Add log rotate configuration
30. Reduce Organizational Silos
31. 31 Reduce Organizational Silos
32. How does 17Media Reduce Organizational Silos?
33. 33 Sharing on a regular basis
34. 34 1 on 1 You Colleague Colleague Manager/ Supervisors Subordinates
35. 35 1 on 1 • Have regular 1:1’s. • Optimize settings. • Have a doc.
36. Implement Gradual Change
38. In early January 2016, Netflix Cloud Migration Complete. Netflix recently announced, through their company blog, their journey to the cloud is finally complete after more than 7 years.
39. How does 17Media Implement Gradual Change?
40. 40 17Media SRE 2015 • 17Media founded • Build on AWS 2016/8 • First DevOps Engineer join 2018/5 • Migrate to GCP 2017/1 • Migrate from beanstalk to ECS 2018/6 • Migrate from node.js to golang 2019/8 • Building GRE
41. 41 Migrate from node.js to golang • Step 1: Create a proxy layer to redirect traffic • Step 2: Develop new features in golang • Step 3: Migrate node.js to golang one by one Node.js Proxy LB Node.js LB Proxy Node.js LB Golang
42. Leverage Tooling & Automation
43. 43 Leverage Tooling & Automation • Tooling to automate repetitive work • release • create an account for new users • automatic repair interrupt? • Less manual work. More R&D • Something to show off on resume
44. How does 17Media Leverage Tooling & Automation?
45. 45 Leverage Tooling & Automation - 1 • Situation • The sensitive data like token and password are in the repository so that anyone can access this. • Solution • #1: Manually encrypt passwords. Each new password takes 2 developer days. • #2: Develop a tool called: http://github.com/17media/ macgyver
 3FGFSFODF
    
    IUUQTHJUIVCDPNNFEJBNBDHZWFS
46. 46 Leverage Tooling & Automation - 1
47. 47 Leverage Tooling & Automation - 1
48. 48 Leverage Tooling & Automation - 2 • Situation • Our application release on daily basis, and this process need to take 30 min. Git Tag Deploy to Staging Unit test/ Image Build E2E test Deploy to Production
49. 49 Leverage Tooling & Automation - 2 Git Tag Deploy to Staging Unit test/ Image Build E2E test Deploy to Production
50. 50 Leverage Tooling & Automation - 2 • Solution • Auto deploy to staging at 6am everyday
51. Measure Everything
52. 52 Measure Everything • A dashboard to see all states. • Can go deep to trace issues. • Help to make important design decisions.
53. How does 17Media Measure Everything?
55. 55 Measure Everything • Cluster Level • Node count • Pod count • ….
56. 56 Measure Everything • System Level • CPU • Memory • Disk • IOPS • …
57. 57 Measure Everything • Application Level • Latency • QPS • Error rate • ….
58. 58 Measure Everything • Code level • Cache hit rate • Concurrent users • How many times does your code go into “else” statement
59. It has to be customized toward your organization.
60. SRE Resource in Taiwan 60 IUUQTXXXGBDFCPPLDPNHSPVQTTSFUBJXBO IUUQTXXXGBDFCPPLDPNHSPVQT%FW0QT5BJXBO
61. Try to Change your organization
62. Try to Change your organization Job
63. 63 We're hiring Site Reliability Engineer - Infrastructure Site Reliability Engineer - Automation & Tools
 Site Reliability Engineer - DBA
 Backend Engineer Check our job page! 17 Media