Upgrade to Pro — share decks privately, control downloads, hide ads and more …

那些年我們做過的 DevOps Pipeline @ DevOpsDays Taipei 2023

Cheng-Wei Chen
September 28, 2023

那些年我們做過的 DevOps Pipeline @ DevOpsDays Taipei 2023

那些年我們做過的 DevOps Pipeline @ DevOpsDays Taipei 2023

CI / CD Pipeline 是 DevOps 工程實踐中非常重要的一項工具,這麼多年過去 CI / CD 技術與工具的發展,已經從地端到雲端、VM 到 Container、老爺爺到太空章魚⋯⋯等經歷多個年代。到底為何我們仍需要 CI / CD Pipeline?

就讓我們來回顧一下 CI / CD Pipeline 的歷史軌跡,放眼過去、展望未來,認識這項在實踐 DevOps 時,你絕對少不了的重要工具!

Cheng-Wei Chen

September 28, 2023
Tweet

More Decks by Cheng-Wei Chen

Other Decks in Technology

Transcript

  1. Photo by Simon Stratford on Unsplash Cheng Wei Chen @

    DevOpsDays Taipei 2023 那些年我們做過的 DevOps Pipeline
  2. Cheng Wei Chen 陳正瑋 Athemaster 炬識科技 / Technology Consultant 《Effective

    DevOps 中⽂版》譯者、GitLab Hero、DevOps Taiwan Community 志⼯ 著作《和艦長⼀起30天玩轉GitLab》 https://chengweichen.com
  3. Testing Code Deploy Build Debug / Feature Local 環境 composer

    source code (*.php) code/ -> /var/www/ 開啟瀏覽器,點⼀點⋯
  4. Testing Local Code Deploy Build Local 環境 Testing Main Code

    Deploy Build 其他環境 Debug / Feature
  5. 開發⼈員應遵守的 CI 七⼤要領: • Commit code frequently • Don’t commit

    broken code • Fix broken builds immediately • Write automated developers tests • All tests and inspections must pass • Run private builds • Avoid getting broken code 網路上查得到
  6. Code Delivery Release source code (*.php) artifacts.zip User Install /

    Config / Provision / Setup … Photo by Fernando Lavin on Unsplash Photo by Roberto Sorin on Unsplash Photo by Campaign Creators on Unsplash Photo by William Hook on Unsplash Application / Services Install / Config / Provision / Setup … Deploy
  7. Code Delivery Release Deploy environment dev staging prod local 開發

    測試 老闆 顧客 Install / Config / Provision / Setup … source code (*.php) artifacts.zip con fi g fi les env fi les + VM VM VM (Cloud)
  8. Code Delivery Release Deploy environment dev staging prod local 開發

    測試 老闆 顧客 持續交付 Install / Config / Provision / Setup …
  9. 修復問題的耗費的資源 尚未整合的「異動」數量 修復問題 之 惡性循環 發生問題 + + + 持續(頻繁)的整合與交付⼩範圍的異動(包含程式、環境⋯⋯)

    每次異動都靠 CI / CD 守護最低限度的品質 及早發現問題 及早修復問題 + 潛在問題的數量 / 複雜度 
 發生問題的機率
  10. Testing Local Code Deploy Build Local 環境 Testing Main Code

    Deploy Build 其他環境 Debug / Feature CI / CD 

  11. 1999 – VMware 2007 - Bamboo 2007 - Cruise 1999

    – Continuous Integration 1999 – eXtreme Programming 1994 - Origin of Continuous integration by Grady Booch 2007 - CI (Book) 資料來源: https://en.wikipedia.org/wiki/Continuous_integration 2010 - CD (Book) 2011 - Jenkins 2006 - TeamCity 2010 - GoCD 2005 - Huden 2009 - 1st DevOpsDays 2007 – VirtualBox 我們在這裡 2000 - Apache Subversion (SVN) 2005 - Git 1990 2000 2015 1995 2010 2005 2020 2025
  12. 第 1 代 CI / CD Pipeline CI Server VM

    SSH VM Agent Developer Jenkins Server Target Server / VM User / Password / SSH Key Actions / Shell scripts Pipeline / Step / Job Run backup / git pull / composer / con fi gure DB / runtime / nginx site con fi g / HA (LB) GUI
  13. 1999 – VMware 2007 - Bamboo 2007 - Cruise 1999

    – Continuous Integration 1999 – eXtreme Programming 1994 - Origin of Continuous integration by Grady Booch 2007 - CI (Book) 2010 - CD (Book) 2011 - Jenkins 2006 - TeamCity 2000 - Apache Subversion (SVN) 2010 - GoCD 2005 - Huden 2009 - 1st DevOpsDays 2007 – VirtualBox 資料來源: https://en.wikipedia.org/wiki/Continuous_integration 2008 - Bitbucket 2016 - Gitea 2005 - Git 2011 - Travis CI 2011 - CircleCI 2012 - GitLab CI 2018 - GitHub Action 1990 2000 2015 1995 2010 2005 2020 2025 2011 - GitLab 2008 - GitHub
  14. 第 2 代 CI / CD Pipeline Pipeline VCS YAML

    CI Service Developer Server Target Server / VM Actions / Shell scripts Pipeline / Step / Job Run backup / git pull / composer / con fi gure DB / runtime / nginx site con fi g / HA (LB) GitLab CI User / Password / SSH Key VM SSH CI Job
  15. 第 2 代 CI / CD Pipeline Pipeline VCS YAML

    CI Service Pipeline as Code GitLab & GitLab CI VM SSH CI Job Developer VCS CI / CD
  16. 1999 – VMware 2007 - Bamboo 2007 - Cruise 1999

    – Continuous Integration 1999 – eXtreme Programming 1994 - Origin of Continuous integration by Grady Booch 2007 - CI (Book) 2010 - CD (Book) 2011 - Jenkins 2006 - TeamCity 2000 - Apache Subversion (SVN) 2010 - GoCD 2005 - Huden 2009 - 1st DevOpsDays 2007 – VirtualBox 2016 - Gitea 2012 - GitLab CI 2008 - Bitbucket 2005 - Git 2011 - Travis CI 2011 - CircleCI 2018 - GitHub Action 2011 - GitLab 2008 - GitHub 資料來源: https://en.wikipedia.org/wiki/Continuous_integration 2014 - Docker (v1.0) 2015 - K8s 1990 2000 2015 1995 2010 2005 2020 2025
  17. 1999 – VMware 2007 - Bamboo 2007 - Cruise 1999

    – Continuous Integration 1999 – eXtreme Programming 1994 - Origin of Continuous integration by Grady Booch 2007 - CI (Book) 2010 - CD (Book) 2011 - Jenkins 2006 - TeamCity 2000 - Apache Subversion (SVN) 2010 - GoCD 2005 - Huden 2009 - 1st DevOpsDays 2007 – VirtualBox 2016 - Gitea 2012 - GitLab CI 2008 - Bitbucket 2005 - Git 2011 - Travis CI 2011 - CircleCI 2018 - GitHub Action 2011 - GitLab 2008 - GitHub 資料來源: https://en.wikipedia.org/wiki/Continuous_integration 2014 - Docker (v1.0) 2015 - K8s 1990 2000 2015 1995 2010 2005 2020 2025 第 2.5 代 CI / CD Pipeline VM VM Container
  18. build test release deploy dev / stg test dev /

    stg deploy prod code Local 第 1~2 代 CI / CD Pipeline build VM build VM zip dev VM stg VM prod VM test VM prod VM Local 容器 build 容器 test 容器 image dev 容器 stg 容器 第 2.5 代 CI / CD Pipeline prod VM prod VM
  19. 1999 – VMware 2007 - Bamboo 2007 - Cruise 1999

    – Continuous Integration 1999 – eXtreme Programming 1994 - Origin of Continuous integration by Grady Booch 2007 - CI (Book) 2010 - CD (Book) 2011 - Jenkins 2006 - TeamCity 2000 - Apache Subversion (SVN) 2010 - GoCD 2005 - Huden 2009 - 1st DevOpsDays 2007 – VirtualBox 2016 - Gitea 2012 - GitLab CI 2008 - Bitbucket 2005 - Git 2011 - Travis CI 2011 - CircleCI 2018 - GitHub Action 2011 - GitLab 2008 - GitHub 資料來源: https://en.wikipedia.org/wiki/Continuous_integration 2014 - Docker (v1.0) 2016 - IaC (Book, 1st) 2020 - IaC (Book, 2nd) 2015 - Spinnaker 2015 - Buddy 2019 - Argo CD 2018 - Jenkins X 1990 2000 2015 1995 2010 2005 2020 2025 2017 - GitOps 2021 - Terraform v1.0.0 第 3 代 CI / CD Pipeline Container VCS YAML Container Automation 2021 - Pulumi v2.0.0 2015 - K8s
  20. 第 3 代 CI / CD Pipeline Container VCS YAML

    Container Automation Everything as Code Dev / Ops / Infra 圖片來源:https://landscape.cncf.io
  21. build test release deploy dev / stg test dev /

    stg deploy prod code 第 3 代 CI / CD Pipeline Local 容器 build 容器 test 容器 image dev k8s stg k8s prod k8s Infrastructure Code / Manifest Code Application Code plan test code apply test
  22. 想像與思考 3 Golang AWS GitLab CI Microservice Dev Team: 30

    Go Micro 這些條件會產生何種 Pipeline? 自架 GitLab GCP Azure Argo CD K8s Platform Team: 6
  23. 想像與思考 4 JAVA AWS Jenkins Monolith Members: 10 Spring 如果加上更多的條件呢?

    Bitbucket VM 軟硬整合? 金融業? 多產品彼此相依? APP + Web?
  24. 程式語言 部署環境 可用 IT 資源 組織架構 工具決策 產品交付方式 資安規範 團隊文化

    軟體架構 系統架構 如何 build? 如何 test? 產業別 產品類型 團隊人數 產品的品質要求 產品迭代速度 產品交付速度
  25. The Question Behind the Question Photo by Fancycrave on Unsplash

    • 退後⼀步,了解全貌 • ⼈、流程、⼯具、資源限制, 
 權衡多項變因的多重相依關係 • 找出關鍵瓶頸點, 
 安排超前 2~3 步的規劃 • 交付「價值」,⽽不是無謂的⾃動化
  26. First success, Small Wins Photo by Fancycrave on Unsplash •

    搶下灘頭堡 • 借⼒使⼒ • 越過死亡⼭⾕ 死亡山谷 灘頭堡 Small Win Small Win
  27. Knowledge management Photo by Fancycrave on Unsplash • 關鍵知識、決策、技能的 •

    保存 • 交流 • 傳承 CNCF 也說了要好好的寫⽂件
  28. 建立管理相依關係的原則 Photo by Fancycrave on Unsplash • Dependency • Package

    • Release • Artifact • Container • Secret • Infrastructure • Environment management
  29. ⼩處著⼿,持續迭代 Photo by Fancycrave on Unsplash • 不要妄想⼀條龍,CI、CD 是可以拆開的 •

    Everything as Code • 良好的軟⼯技巧、架構設計 • 模板、模組、繼承 • 標準化與客製化的愛恨情仇 (80/20) • 測試再測試 • VSM、三步⼯作法、系統思考、全貌⋯⋯
  30. 修復問題的時間長度 尚未整合的「異動」數量 修復問題 之 惡性循環 潛在問題的數量 
 與發生問題的機率 發生問題 +

    + + 持續(頻繁)的整合與交付⼩範圍的異動(包含程式、環境⋯⋯) 每次異動都靠 CI / CD 守護最低限度的品質 及早發現問題 及早修復問題 +
  31. CI / CD 的原則: • 建立可重複且可靠的流程 • 盡可能的⾃動化 • 以版本控制管理並保護⼀切

    • 越是棘⼿的事,越要儘早且頻繁處理 • 重視品質 • 交付價值是每位成員的責任 •持續改善
  32. 第 1 代 CI / CD Pipeline CI Server GUI

    VM SSH VM Agent 第 2 代 CI / CD Pipeline Pipeline VCS YAML CI Service CI Job 第 3 代 CI / CD Pipeline Container VCS YAML Container Automation VM Container 2.5 VM
  33. 程式語言 部署環境 可用 IT 資源 組織架構 工具決策 產品交付方式 資安規範 團隊文化

    軟體架構 系統架構 如何 build? 如何 test? 產業別 產品類型 團隊人數 產品的品質要求 產品迭代速度 產品交付速度
  34. CI / CD / DevOps Process People Technology 涉及:流程、人、技術 Pipeline

    / Work fl ow Top Bottom Tools / Solution Team Department
  35. • Mindset • The Question Behind the Question • First

    success, Small Wins • Communicate, collaborate, align goals • Knowledge management Photo by Fancycrave on Unsplash 個⼈⼼得