Yet another Continuous Delivery talk - Incontro DevOps 2021

Continuous Delivery practices applied to a multi-cloud product born in 2013, through constant evolution of the pipelines to deliver software reliably and manage operations without fear.

Transcript

1. © Engineering Yet another Continuous Delivery talk Giovanni Toraldo / Solution Developer Engineering D.HUB Incontro Devops 2021 #idi2021 CONTINUOUS DELIVERY
2. © Engineering About me Open source Remote-first software developer / devops Solution developer in Engineering D.HUB Often teleported into the 14th century 🛡⚔🎯 CONTINUOUS DELIVERY / BIO Giovanni Toraldo
3. © Engineering About Cloudesire Whitelabel marketplace SaaS billing Channel management IaaS, PaaS, SaaS catalog 2012 - R&D project 2013 - Cloudesire.com 2016 - Tim Venture seed 600K 2020 - Engineering acquisition CONTINUOUS DELIVERY / BIO
4. © Engineering Antipatterns are everywhere – Manual deployments are a waste of time – Infrequent deployments cause late feedback – Manual configuration of environments leads to configuration drift – Manual testing leads to regressions – Can we do better? CONTINUOUS DELIVERY / THEORY
5. © Engineering What Continuous Delivery can do – Short cycles – Reliability – Early feedback – Market pressure – Customer satisfaction CONTINUOUS DELIVERY / THEORY
6. © Engineering Continuous Delivery phases CONTINUOUS DELIVERY / THEORY
7. © Engineering How we do achieve our goal? CONTINUOUS DELIVERY / PRACTICES
8. © Engineering Backlog ‒ One place for high-level issues (epic, stories) ‒ Focus on describing the problem you want to solve, not solutions • Developers knows what is good for you ‒ Prioritization as a contract with stakeholders ‒ Commit for a plan and try to stick with it (#noestimates) ‒ Current status should be transparent • Feedback to non-developers when releases happens • Meetings are ok but everyone happy and no meetings is better CONTINUOUS DELIVERY / PRACTICES
9. © Engineering Automated testing / TDD – Probably the most underrated practice for achieving velocity ‒ Writing tests is HARD for newcomers ‒ Maintaining test suites is HARD for everyone (e.g.: flappers) ‒ Still, it’s worth the time ‒ Optimize build times to avoid slow feedback ‒ Tests give hints on how a function should work CONTINUOUS DELIVERY / PRACTICES
10. © Engineering Continuous integration ‒ Treat it like your best friend, you can rely on it ‒ Ensure that you are not going to break everything in a single move ‒ Repetitive tasks belongs to here • Compile code • Execute regression tests • Generate documentation • Publish new artifacts CONTINUOUS DELIVERY / PRACTICES
11. © Engineering Code reviews ‒ Spot bugs before reaching users ‒ Share ownership and abandon blame culture ‒ Meet specifications ‒ Keep the bar high ‒ Written trail of design decisions ‒ Asynchronous feedback ‒ Mentor juniors without doing anything special CONTINUOUS DELIVERY / PRACTICES
12. © Engineering Packaging / Distribution ‒ A strict requirement for automated releases ‒ Every commit is a release candidate ‒ Standard packaging (e.g. Docker) ‒ Central artifacts repository (Nexus, Artifactory) • Accessible from all the execution environments CONTINUOUS DELIVERY / PRACTICES
13. © Engineering Automated releases ‒ Every environment should be a prod-like environment • Have at least a test environment for every prod environment ‒ Repeatable procedures reusable in every environment ‒ Rollback should be easy as a standard release ‒ Releases should be tracked somewhere (GIT, CI) ‒ Releases must not affects customers CONTINUOUS DELIVERY / PRACTICES
14. © Engineering Feedback loop ‒ Automatic deployment to one prod-like environment ‒ Release daily on staging environments ‒ Ask for explicit feedback after changes from tester/QA/stakeholders ‒ Meaningful and searchable application logs ‒ Proactively monitor exceptions and errors CONTINUOUS DELIVERY / PRACTICES
15. © Engineering Infrastructure as code ‒ Forgot SSH commands or clicking buttons ‒ Repeatable procedures ‒ Standardization ‒ Objects Inventory ‒ Document without actually writing documentation CONTINUOUS DELIVERY / PRACTICES
16. © Engineering Microservices architectures ‒ Maintaining a unique codebase has exponential difficulty • Divide et impera ‒ Requires a trained release pipeline ‒ Avoid repeating yourself via: • Boilerplate projects • Shared libraries • Automated updates • DOCUMENTATION CONTINUOUS DELIVERY / PRACTICES
17. © Engineering What’s in our toolbox CONTINUOUS DELIVERY / TOOLING
18. © Engineering Github ‒ GIT repository hosting ‒ Lightweight issue management • There is a label for everything • Type: Bug, new feature, change feature, operations, devops • Priority: P1, P2, P3, P4 • Effort: light, medium, high, epic • Workflow: in-progress, ready-for-staging, test, ready, awaiting-feedback, blocked • Topic: UI/UX, billing, ERP... • Customer • Stalebot that warns and close inactive issue for more than 3 months ‒ Documentation as markdown files ‒ Planning via weekly milestones • Internal dashboard to track milestones progress CONTINUOUS DELIVERY / TOOLING
19. © Engineering Internal dashboard for milestones tracking CONTINUOUS DELIVERY / TOOLING
20. © Engineering Dependabot ‒ Automatic dependencies updates for your repositories on Github ‒ PR get opened automatically with changelog between versions ‒ Triggered weekly by a schedule on early monday • PR are already built and ready to be merged ‒ Security updates with links to CVEs ‒ Supports most dependency management tools and Dockerfile ‒ Enabled on all repositories via Terraform CONTINUOUS DELIVERY / TOOLING
21. © Engineering .github/dependabot.yml version: 2 registries: maven-internal-releases: type: maven-repository url: https://repo.cloudesire.com/repository/internal username: ${{secrets.MAVEN_USER}} password: ${{secrets.MAVEN_PASS}} updates: - package-ecosystem: maven directory: /cloudesire-root schedule: interval: weekly day: sunday registries: - maven-internal-releases assignees: - gionn - malteo - package-ecosystem: docker directory: /cloudesire-root/cmw schedule: interval: weekly day: sunday assignees: - gionn - malteo CONTINUOUS DELIVERY / TOOLING
22. © Engineering Dependabot automated pull request CONTINUOUS DELIVERY / TOOLING
23. © Engineering CircleCI ‒ Cloud CI services better than in-house: • Scalability • Predictable pricing ‒ Batteries included compared to Jenkins ‒ A bit of downsides: • Not so configurable (e.g. tests parallelization) • Can’t add new projects via API • Docker integration slightly cumbersome since beginning ‒ Reuse pipeline configuration via ORB CONTINUOUS DELIVERY / TOOLING
24. © Engineering CircleCI ORB example CONTINUOUS DELIVERY / TOOLING
25. © Engineering CircleCI workflow diagram CONTINUOUS DELIVERY / TOOLING
26. © Engineering CircleCI job detail CONTINUOUS DELIVERY / TOOLING
27. © Engineering Sonar (static analysis) ‒ Catch bugs automatically ‒ Discovery and learn (why is this an issue?) ‒ Enforce standards (quality gates) ‒ Applied systematically for every project ‒ Reports integrated in PR (developer edition) ‒ Security audits (Hotspots) CONTINUOUS DELIVERY / TOOLING
28. © Engineering Terraform (Infrastructure as code) ‒ Supports many providers: • Iaas: AWS, Azure, GCP, ... • Saas: Cloudflare, Github, ... • API: Vault, ... ‒ Follow common development workflow • PR • Code review • CI pipelines CONTINUOUS DELIVERY / TOOLING
29. © Engineering Automated terraform plan on pull requests CONTINUOUS DELIVERY / TOOLING
30. © Engineering CircleCI workflow for terraform repository CONTINUOUS DELIVERY / TOOLING
31. © Engineering Chef (configuration management) ‒ Still the configuration management tool after 8+ yrs ‒ Provision new production environment in 30 minutes • script to generate new chef environments ‒ Nightly integration tests on cloud infrastructure ‒ Provision can run from CI or developers box CONTINUOUS DELIVERY / TOOLING
32. © Engineering Graylog ‒ Centralized logs collection • Applications via GELF • Syslog • Docker logs ‒ Elasticsearch monthly indexes • configurable retention ‒ Full-text search on every field, cross-environment CONTINUOUS DELIVERY / TOOLING
33. © Engineering Sentry ‒ Push applications exceptions via API ‒ Aggregate similar exceptions ‒ Mark as Resolved • Flagged as regression if found again ‒ Mark as Ignored for n time ‒ Searchable ‒ Slack integration • Each component in a different channel • Only for new errors, regressions or repeated errors in a timeframe CONTINUOUS DELIVERY / TOOLING
34. © Engineering Sentry issue example CONTINUOUS DELIVERY / TOOLING
35. © Engineering How to release CONTINUOUS DELIVERY / RELEASE
36. © Engineering Step 1: open a release PR on ops repo via Merger script ‒ Select which chef environment you want to update ‒ Iterate on all the platform components installed on the environment to: • Retrieve the latest successful build of master branch on CircleCI • Compare build number with currently installed docker image version • CI deploy jobs tag docker images with build number • If a new version is available, compare current and previous commit-id on Github • Parse commits to search for PR and referenced github issues ‒ Open a pull request with version changes on chef environment • Branch name is always `release-envName` • PR body contains the changelog for every components and links to issue • Issues are linked via a redirector domain to avoid references spam CONTINUOUS DELIVERY / RELEASE
37. © Engineering Example Release PR CONTINUOUS DELIVERY / TOOLING
38. © Engineering Step 2: PR review, approve, merge, provision ‒ Follows the standard development workflow (review, approve, merge) ‒ When building master, CI pipeline detect if the built commit is: • A release PR generated by Merger • A generic PR with a label `deploy/envName` ‒ Run Chef sequentially on all the nodes of the environment CONTINUOUS DELIVERY / RELEASE
39. © Engineering Step 3: profit ‒ If the release fails, the CI send an email to release owners and exceptions are pushed to sentry/slack ‒ If the release is successful: • The release PR get labeled as `deployed` and a comment is left with the link of the build • Every referenced issue in the release PR body receive a comment stating that a release is just happened • If the referenced issue had a label: • `workflow/ready-for-staging` is replaced by `workflow/test` • `workflow/ready` the issue is closed CONTINUOUS DELIVERY / RELEASE
40. © Engineering Release schedule via CircleCI ‒ Release PR are automatically opened following a weekly schedule: • Main staging environment every weekday until friday • Main demo environment on friday, copying main staging versions • Main production environment on monday, copying main demo versions • Secondary (clients) staging environment on different day of the week, copying main production versions • Secondary (clients) production environment on different days of the week, copying from their staging environment CONTINUOUS DELIVERY / RELEASE
41. © Engineering Wrap-up ‒ Continuous delivery is a game changer when developing products ‒ Try to continuously improve, one step at a time ‒ Automate all the things CONTINUOUS DELIVERY / SUMMARY
42. © Engineering www.eng.it Engineering Ingegneria Informatica SpA @EngineeringSpa gruppo.engineering LifeAtEngineering Thanks! Giovanni Toraldo @gionn