Community Driven DevOps

DevOps Evangelist 經驗分享
軟體公司的數字化轉型?
快速交付是種「能力」！
Trunk-based Development & Feature Toggle
基本定義 (Definitions) 的重要性
Community Driven DevOps
DevOps Elite Culture

Transcript

1. 
 Community Driven DevOps Get Together, Go Faster
2. Bryan Liu • 現任 LINE TW DevOps TF Lead & DevGov Facilitator • 經歷 RD, PM, AE 等職位 • ⾃ 2013 開始研究持續整合到嘗試導入 DevOps 等更有效率的開發⽅式 演講經歷 • LINE TechPulse 2023，以 DevOps 思維實施開發治理 • LINE TechPulse 2020，通過測試隔離改進⾃動化驗收測試 • DevOpsDays Taipei 2017，百倍數交付談主幹開發 • DevOps Summit Taipei 2016，Test Automation
3. Agenda
4. About LINE TW • TW 在地的服務 ➡ (Very biz Driven) • RD established ≈ 2015 (Good Eng. culture) • Back to 2018 when I joined: • All run Scrum w/ similar work fl ow • Di ff erent languages, stacks, release process • It’s hard to retrieve SW Px metrics
5. Results Demo
6. You Got to Believe in Something 
 Faith
7. 解決⾝邊沒效率的問題 • ⼀個 hot fi x 到底要測幾次？ • Bug 重複出現 • Merge hell • Long integration & code freeze • QA 沒時間執⾏探索測試
8. • 質量反饋滯後性 (⾃動化測試與⽣產程式碼不同步) • 每天早上花 1 ~ 2 hr 檢查為何失敗 • New feature code checked in • Code bugs • Test automation fl aky • 數百、千隻的 test automations • Duplicated test coverage • RD 從來也不會來看測了什麼 • Pipeline con fi dence 完全沒幫助 解決⾝邊不合理的流程
9. DevOps Evangelist & Faith (Goals) (Realities)
10. DevOps Evangelist & Faith (Goals) (Realities)
11. Digital Transformation? Why Change is So Hard …
13. 快速交付是種「能⼒」！ 現代企業⽣存的關鍵
14. Source: The Art of Corporate Endurance, Harvard Business Review、DevOps Enterprise Journal、cloud.google.com S P 
 E 
 E 
 D
15. 不進則退...
16. 實際情況 … • Biz 說「不需要這麼快啦」 • 我們團隊做 platform 的，重要的是要穩定！ • PV、廣告營收不要掉，穩穩地比較重要 ... Source: Not Wrong Long - is perfect software necessary in the world of continuous delivery 但別⼈可不是這麼想！
17. Source: Insights of Agile at Tesla with Joe Justice; Is continuous testing the key to SpaceX software success 具備「快」的「能⼒」 “Reduce cycle time to enable FAST experiments and learning! ”
18. Trunk-based 
 With 
 Feature Toggling vs Git fl ow with Alpha, Beta, Staging, Production Envs. “抓⼤、放⼩、 縮短 cycle time”
19. De fi nition Matters Essential Rules and Practices for CI: • Don’t check in on a broken build • Always run all commit tests locally before committing • Wait for commit tests to pass before moving on • Never go home on a broken build • Always be prepared to revert to the previous revision • Time-boxed fixing before reverting • Don’t comment out failed tests • Test-Driven Development • Take responsibility for all breakages that result from your changes
20. Controversial Still? “The data from State of DevOps Report is clear: trunk based development predicts higher throughput and better stability, and even higher job satisfaction and lower rate of burn-out”
21. 快速反饋中的測試驅動的開發 Source: SbE in DevOpsNotes (Blog)
22. Test Automation Traditional Automation Pipeline Driven Automation collaborative model RD: UT &| AE: IT & E2E Discuss “Acceptance Criteria” TOGETHER KPI Automation coverage rate KEY examples (scenario) coverage Pipeline # Multiple pipelines One pipeline to production De fi nitive Verdict Blue / Yellow / Red, human verdict Success / Fail Automation Strategy What ever works in a 2-week cycle Modularized, stub, test isolation, testContext, … Traits Alpha, beta, staging, production CUJ coverage, Release strategies, O11y Branching Strategy Feature Branches Trunk-based development + feature toggle Continuous Improvement Get “Green” before scheduled release Continuously improve pipeline cycle-time
23. Automation Strategies Source: 什麼是⾃動化測試的標的？ • Three levels of test isolation • TestContext,
 TestContainers &
 ConsumerDrivenContr actTest (CDCT) • Other test practices for stabilities
24. 「沒有嚴謹的準則、沒有固定⼯ 法，任何問題都會是值得討論，很 多⽅式看起來都是對的，但幾年後 你就會發現這些作法對 DevOps Elite 的⽬標毫無幫助，反⽽⼜會 變成要處理的 legacy。」 「別把 Weekly Release 當成團隊 ⽬標，它頂多是個 Milestone！因 為要練習的技能完全不⼀樣。」 「邁向 Elite 之路，終點與⼀路上 的風景，都跟別⼈很不⼀樣。」
25. Community Driven
26. • Unit test and CI Pipelines • Cloud native & Microservices • CI/CD & GitOps • Observability • Dev Governance History of DevOps TF • Platform Engineering • Pipeline optimization TF • Engineering operations TF • RUM & frontend tracing • Golden paths & Backstage 2018.06 2019.03 2020.08 2021.08 2022.08 2023.03 2023.06 • Task-Force: 為長期持續在某個領域的學習與導入 • Side-Project: 為中短期專注於所需的平台或⼯具的開發 • 團員皆為⾃願或團隊指派參與，約使⽤ 10 ~ 20% 時間
27. DevOps TF Illustration
28. Drive Change w/ Metrics
29. • 不設定覆蓋率的統⼀標準，但要⼀次比⼀次好（童軍守則） • 分階段：先求有、再求好、再來要求每個 PR 的 new code coverage • Workshops and more workshops • 公開的儀表板（Dashboard） • 找出表現好的團隊及發現好的作法 • 同儕比較的壓⼒ • 展現決⼼（commitment） • 可能是以季、年來規劃 • 固定週期拿出來檢視 • 隱惡揚善 Observe, Learn & Adapt
30. Team Autonomy (Test Coverage) : A circle means an average of code coverages of all repos in a project / service (Core components alone have much higher code coverage rate)
31. Together & Faster • Cloud native & GitOps • Observability • RUM & full trace (WIP) • A lot more …
32. Observe, Learn & Adapt • Infra team mindsets vs DevOps + SRE • Communication, communication & communication • Get together, Go fast • Fragmentation (down) • Cost of Maintenance (down) • Support E ff i ciency • Automatically upgrade • Dx satisfaction (up) “Only 10% of elite respondents indicated that their teams have fully implemented every SRE practice we investigated.” Source: state of DevOps 2021
33. Development Governance › Regulations: LINE開發者必須遵守的 8 條基本規則 › Guidelines: 描述實施法規的資訊和程序的指南，由 15 項組成 Development Operation
34. Observe, Learn & Adapt • 如何減少做表⾯功夫（常⾒於 Top-down） • 很多是需要持續改善（10,000 hrs） • 該做好的事就及早開始（Do it early） • 成果顯著（36% of best practice submission） • 開發治理有助於推⾏ DevOps practices
35. Platform Engineering TaskForce When platform, tools, SDKs are ready Life Finds a Way!
36. • 跨團隊（Cross-cutting） 都有的問題，似乎很適合以 “Guild” 社群⽅式來解決 • 平台開發者（RD）就是未來的使⽤者，很多問題都能及早發現並解決 • 持續深化 community driven - 以 Backstage 平台為載體 • 提⾼透明度，展現社群⽬標與進展 • Golden path 以 SW Template ⾃動化 • Tech Radar 活化架構選型彈性 • 社群適合組織特定⽬標、流程、⼯法、PoC 等等討論與驗證 • 運維（operational tasks）與後續開發還是需要固定⼈⼒ Observe, Learn & Adapt
37. Takeaway
38. “看症狀來開藥單... 其實很多問題都是⾃⼰創造出來的。”
39. • 更重要的是培養「快」的能⼒ • De-couple code deployment and release • Staged rollout, progressive release • Scalable A/B testing and reduce each experiment costs • Don’t forget the core abilities (UT, code review, …) • Develop “abilities” probably more important than “methodologies” • But “methodologies” provide “de fi nitions” which do matters! Trunk-based or NOT
40. • It’s all about “Solving problems” over “Building solutions” • Aim high, start low and make changes • (Leadership) Increase visibility & in fl uence outside of ur project • (Leadership) Authority comes from an ability to in fl uence others • Consensus and endorsement from mgmt (bonus, OKR also) • One man’s opinion v.s. team consensus • Top-down：短時間要有⼀定成果，對「持續改善」可能有負⾯效果 • Community：進度緩慢、團隊向⼼⼒（team moral ）不易維持 Community in a Digital World
41. “沒有⼀家公司是完美的，抱怨永遠不會結束， 但很多公司卻會結束。” Dragon boat festival, this is culture! （⽂化沒有對與錯，可能也不是 DevOps 能否導入的關鍵。） For Evangelists “如何幫助公司、團隊，學習到更有效 率的開發⽅式。” “想想如何說服⼤家，相信⼯程師⼤部分都是理性的。” “⾝為⼀位佈道者，你還是得相信點什麼， 然後學會不要臉！”
42. 《從 A 到 A ＋》書的結尾說到：『做到卓越不⾒得比做到優秀困難， 反⽽可以少吃點苦頭，累積的動能會注入更多的能量，持續維持在平庸 將不斷耗掉能量，新注入的能量卻很少。』 Good v.s. Excellent • Ex: Trunk-based development Engineering Culture Very very few use trunk- based development Elite companies scale well with it New unicorns advertise it Best-selling books promote it Statistic data prove it
43. Source: https://cudl.lib.cam.ac.uk/