Real World App Deployment Processes @ Bright People

CI and CD of both massive legacy and greenfield applications in a Windows world. Automated build and deployment process using TeamCity and Octopus Deploy.

Transcript

1. Continuous Integration & Deployment Presented by: Josh Michielsen DevOps Systems Administrator Bright People Technologies
2. Product Overview 2 Legacy Applications › Workforce logistics › Workforce mobilisation › Planning and rostering › Reporting and forecasting › Workforce planning › Workforce validation › Verification and compliance › Self-service model Overview › Web based; SaaS › Focus on the resources industry › VB.NET/C#.NET › .NET Framework › ASP.NET Webforms, MVC, & Web API › MSTest; NUnit › SQL Server › 2+ Million LoC combined
3. Product Overview (2) 3 Cited Tech › Greenfield SaaS – https://cited.com.au › Targeted to all individuals and organisations › Workforce verification and compliance › E.g. National police checks; Visa checks; etc. › C#.NET › .NET Framework › ASP.NET MVC & Web API. Can be deployed independently. › NUnit › EventStore & MySQL
4. Hotfix Deployment Pipeline 4 Legacy Applications Git - Assembla › Automatically generate merge requests › Mandatory code reviews › Ticket/card tracking CI - TeamCity › Git hook, triggered after merge › Build via MSBuild › Test via NUnit › Generate .nupkg with OctoPack › Generate release notes (PoSH) › Push to NuGet feed › Build server config/environment specified in DSC as much as possible. CD - Octopus Deploy QA › Deploys and “installs” .nupkg files › Run scripts, configure IIS, import certificates, etc. › Repeatable single click deployments through environments › Enforces environment lifecycles, version requirements, etc. CI UAT Prod Demo › Alternating 2 week release cycle › All commits are made to master › Card-based testing › Branched for release at sprint milestone › Full regression testing › Pet-style infrastructure › Validate: Over 115 deployment steps, 12-18 mins › Logistics: 25 steps, 4-6 mins Master Channel Release Channel Details
5. Deployment Pipeline 5 Cited Git - Assembla › Automatically generate merge requests › Mandatory code reviews › Ticket/card tracking CI - TeamCity › Git hook, triggered before and after merge. › Build via MSBuild › Test via NUnit › Generate .nupkg with OctoPack › Generate release notes (PoSH) › Push to NuGet feed CD - Octopus Deploy QA › Deploys and “installs” .nupkg files › Run scripts, configure IIS, import certificates, etc. › Repeatable single click deployments through environments › Enforces environment lifecycles, version requirements, etc. CI (auto) UAT Prod Demo › Almost identical workflow as legacy teams › Multiple (no limit) releases per day to production (release early, release often). › All commits trigger CI build › No branching. Merged builds deployed automatically to CI › Some manual testing, much shorter than legacy › Infrastructure deployment and configuration automated (CloudFormation, cloud-init, and PowerShell DSC (deployed via Octopus) › Deployment time: 1-2 minutes Details
6. General Information 6 Who “owns” the deployment process? › DevOps (mostly me) invests the most time into management › Collaboration with the lead developers › We actively seek ways to improve the process, especially in the areas of speed and reliability Improvements in the last 12 months? › Migrated build and deployment infrastructure to cloud (AWS) › More proactive in updating CI/CD platforms › Dedicated resource (me), ‘DevOps’ process and implementation was started by a dev. When he left the business saw the need for a more permanent ‘DevOps’ engineer (with larger ops background). Further improvements in the next 12 months? › Heavier investment in PowerShell DSC › Octopus Deploy (via API) and TeamCity configuration (via their Kotlin DSL) as code and in VCS. › Really – just as much configuration as code as possible. › Provides visibility when changes are made › Allows for PRs/merge request and code reviews. › Provides repeatability and consistency › Devs have even greater control of environments and config, while having less access to the machines and admin consoles– massive improvement for security › Terraform, packer, and vault are all under consideration at the moment › Legacy: Better base images; Sub 10 min deployments for Validate (physics permitting) › Cited: Lambda (API), Elastic Beanstalk or Docker (MVC) Best/worst decisions made? › Best: Octopus Deploy › Worst: Devs with local admin to prod machines
7. Thanks! jmichielsen@brighttechnologies.com.au
8. Prod deployments (Cited) and downtime? › “Rolling” deployments - only 1 web server updated a time › Connections are drained from IIS before deployment All commits to master – dealing with “half baked features” › All developers are required to use feature toggles › Toggles are often controlled via Octopus variables › E.g. Large features are generally tested in UAT, therefore a featureX.IsEnabled variable would exist twice, once as a catch-all across environments (value: false), and again with a scope of only UAT (value: true) › In rare cases devs will branch and can manually trigger builds (+ tests) against that branch. Only master branches can be deployed though Questions Received How are the devs handling the move to ‘DevOps’? › Process started by a dev, definitely helped to sell it › Devs want to commit code and not have to worry about the how it gets delivered Removing developer access to machines & consoles – priority and feasibility? › DevOps and config management story has greatly improved in the Windows world, so feasibility › Greater investment into PowerShell DSC (think Puppet DSL, or Chef’s Ruby-based DSL) is a high priority, making the change much more feasible › Removing admin access to prod is very high priority