Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Intro to CI/CD in GitLab and Anatomy of a Pipeline

Intro to CI/CD in GitLab and Anatomy of a Pipeline

GitLab began as just another Git repository system, but CI pipelines were introduced after about a year and in the 6 years since GitLab CI has continuously improved to include CD and Security and is now the highest-rated CI/CD tool by Forrester.

Joining us will be Steve Grossman to describe the architecture of the GitLab CI/CD system and the basic parts of a pipeline definition including a sample pipeline. The majority of the functionality discussed is available to everyone in the open source self-hosted GitLab product or the free tier of gitlab.com.=

Keith Resar

August 22, 2019
Tweet

More Decks by Keith Resar

Other Decks in Technology

Transcript

  1. 2 Agenda • Introduction ◦ Who are we and who

    is GitLab? ◦ Where is your team today? ◦ Concepts defined - what is CI/CD? ◦ Why? ◦ How? • GitLab CI/CD Architecture • GitLab CI/CD Runners • CI/CD Pipeline Definition • Hands-On Workshop • Q&A
  2. 3 GitLab is the most popular solution for the Enterprise

    COMPANY - Incorporated in 2014 - 700+ employees across 56 countries - GitLab Federal entity est. in 2018 BROAD ADOPTION - 100,000+ organizations - Millions of users - 70% share of self-managed DevOps repository market STRONG COMMUNITY - Open source model - 2,200+ code contributors - 10,000+ total contributors
  3. 4 Values Collaboration ◦ Work asynchronously with fully remote workforce.

    ◦ Use GitLab to build GitLab. There’s an Issue and/or Merge Request for everything. Results ◦ Track outcomes, not hours. Diversity ◦ Remote-only tends toward global diversity, but we still have a ways to go. ◦ Hire those who add to culture, not those who fit with culture. We want cultural diversity instead of cultural conformity. Efficiency ◦ Boring solutions win. Complexity slows cycle time. Iteration ◦ Minimum Viable Change (MVC); if the change is better than the existing solution, ship it. Transparency ◦ Everything at GitLab is public by default: Strategy, Roadmap, Quarterly goals, Handbook, and Issue Trackers.
  4. 5 It’s Release Day! • We release the 22nd of

    EVERY month • We are close to 100 consecutive months of releases
  5. 6 Why Are We Here You Own GitLab Using it

    for code repo only Lots more to GitLab than just code repo, we’re going to show you another major piece
  6. 7 GitLab is the first single application for the entire

    DevOps lifecycle Manage Plan Create Verify Package Secure Release Configure Monitor Defend Since 2016 Since 2011 Since 2011 Since 2012 Since 2016 Since 2017 Since 2016 Since 2018 Since 2016 Coming soon: Cycle Analytics DevOps Score Audit Management Authentication and Authorization Coming soon: Code Analytics Workflow Policies Kanban Boards Project Management Agile Portfolio Management Service Desk Coming soon: Value Stream Management Requirements Management Quality Management Source Code Management Code Review Wiki Snippets Web IDE Coming soon: Design Management Live Coding Continuous Integration (CI) Code Quality Performance Testing Coming soon: System Testing Usability Testing Accessibility Testing Compatibility Testing Container Registry Maven Repository NPM Registry Coming soon: Rubygem Registry Linux Package Registry Helm Chart Registry Dependency Proxy SAST DAST Dependency Scanning Container Scanning License Management Coming soon: Secret Detection IAST RASP Continuous Delivery (CD) Release Orchestration Pages Review Apps Incremental Rollout Feature Flags Coming soon: Release Governance Auto DevOps Kubernetes Configuration ChatOps Serverless Coming soon: PaaS Chaos Engineering Runbook Configuration Cluster Cost Optimization Metrics Logging Cluster Monitoring Tracing Error Tracking Coming soon: Synthetic Monitoring Incident Management Status Page Runtime Application Self Protection Web Application Firewall Threat Detection Behavior Analytics Vulnerability Management Data Loss Prevention Container Network Security
  7. 9 What is CI/CD? Continuous Integration (CI) Automated testing and

    artifact creation (build) Continuous Delivery (CD) Automated deployment to test and staging environments Manual deployment to Production Continuous Deployment (CD) Automated deployment to Production
  8. 10 Why…? Getting it right matters High-performing teams deliver more,

    faster and cheaper 22% Less time on rework 2,555X Shorter lead times 200X More frequent deployments 29% More time on new work 2016 State of DevOps Report
  9. 11 GitLab - Leader in Forrester CI Tools WaveTM ▸

    Highest score overall for Current Offering ▸ Highest possible score for Strategy ▸ Top scores in Ease of Installation/Configuration, Configuring Builds and Build Reuse, Platform Support, Security Features, Container Build Support, Container Runtime Support, GUI, Analytics https://get.gitlab.com/forrester-wave-ci/ “GitLab’s vision is to serve enterprise-scale, integrated software development teams that want to spend more time writing code and less time maintaining their tool chain.” - The Forrester WaveTM: Continuous Integration Tools, Q3 2017 report The Forrester Wave™ is copyrighted by Forrester Research, Inc. Forrester and Forrester Wave™ are trademarks of Forrester Research, Inc. The Forrester Wave™ is a graphical representation of Forrester's call on a market and is plotted using a detailed spreadsheet with exposed scores, weightings, and comments. Forrester does not endorse any vendor, product, or service depicted in the Forrester Wave. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change.
  10. 14 GitLab Runners • Multi-platform • Multiple environments • Parallel

    builds • Built for docker • One install ➡ many runners • Pooled model for job execution, with exceptions • Easier to set up and manage than slave machines
  11. 16 GitLab CI/CD Architecture Detail GitLab Server Pipeline Coordinator Job

    Queue Git Repo Runner Server Runner Tag: Java Runner Tag: Windows MacOS Machine Runner Tag: MacOS iOS Poll Run .gitlab-ci.yml
  12. 17 Shared vs. Specific Runners Shared • Can be used

    by any project • Included in pool for ALL projects • Managed by GitLab Admin • Typically auto scaling or otherwise scaled Specific • Tied to one or more specific projects • In pool for ONLY specific projects • Managed by Runner Owner(s) • Typically for specialized builds, or if an org needs to do so for billing Tagged • Only used to run jobs tagged with same tag Untagged • Used to run jobs with no tags Protected • ONLY runs jobs from ◦ Protected Branches ◦ Protected Tags • Typically used for runners containing deploy keys or other sensitive capabilities Not Protected • Runs jobs from ANY branch • Used for ANY build
  13. 19 Platforms • Linux ◦ Debian/Ubuntu/CentOS/RedHat ◦ ANY • macOS

    • Windows • Docker service • Docker machine • FreeBSD • Kubernetes
  14. 20 Executors Executors support different platforms and methodologies for building

    your code • Shell • SSH • Docker • Docker Machine (auto-scaling) • Kubernetes • Parallels • VirtualBox
  15. 21 AWS Docker+Machine Kubernetes • Many active examples • EC2

    Spot Instances • Works with most cloud providers and many private cloud solutions • Docker gives Digital Ocean and AWS examples • Drivers also listed for: AWS, Azure, GCP, DO, Exoscale, Hyper-V, OpenStack, Rackspace, IBM Softlayer, VirtalBox, VMWare vCloud Air, VMWare Fusion, VMWare vSphere • Executor and a scaling method • Allows you to spin up a pod-per-job • K8s ConfigMap Runner Auto Scaling methods
  16. 23 Anatomy of a GitLab CI/CD build Pipeline ◦ Set

    of one or more jobs. Optionally organized into stages Stages ◦ Collection of jobs to be run in parallel ◦ e.g. Build, Test, Deploy Jobs ◦ Scripts that perform tasks ◦ e.g. npm test; mvn install; etc. Environments ◦ Where we deploy (Test, Review, Staging, Canary, Prod) All in one file - .gitlab-ci.yml
  17. 24 What does it look like? • Pipeline graph •

    Shows how jobs are executed in stages • Stages are run in serial to each other • Jobs in each stage executed in parallel • If one job in a stage fails, the next stage is not (usually) executed
  18. 25 GitLab CI/CD Basics - The .gitlab-ci.yml file • YAML

    syntax • Stored in root of project repo • Version controlled along with the rest of your code ◦ Enables (and enforces) different configurations for different branches ◦ Allows entire DevOps team to participate in pipeline definition • Contains ◦ Jobs ◦ Stages ◦ Environments • Other contents ◦ Include ◦ Image ◦ Services ◦ Before & After Scripts ◦ Caching ◦ Artifacts & On Success ◦ Only & Except ◦ When
  19. 27 Simple bash .gitlab-ci.yml example before_script: - echo "Before script

    section" - hostname - uname -a after_script: - echo "After script section" - echo "For example you might do some cleanup here" build1: tags: - steevo - macos stage: build script: - echo "Do your build here" test1: tags: - steevo - macos stage: test script: - echo "Do a test here" - echo "For example run a test suite" test2: tags: - steevo - macos stage: test script: - echo "Do another parallel test here" - echo "For example run a lint test"
  20. 31 Auto DevOps - The Easiest Way to Set Up

    A Pipeline Simply commit code and GitLab does the rest Auto DevOps . . . • Detects the language of the code ◦ Builds with a dockerfile if there is one ◦ Uses Heroku and herokuish build packs if there isn’t ▪ Build packs analyze the code in the project and figure out the best way to build based on convention not configuration. • Automatically builds, tests, and measures code quality • Scans for security and licensing issues • Packages • Instruments (for monitoring) • And deploys the application
  21. 32 Getting Started with CI/CD Yourself • All of the

    capabilities discussed tonight are available in the no-cost versions of GitLab ◦ GitLab CE - community edition, open source, good for personal projects ◦ GitLab EE Core - a good place to start on projects for work at no charge ◦ Gitlab.com - the free tier here also has CI/CD but with limited runner minutes ▪ Create your own local runners to avoid the runner minutes limitation • https://about.gitlab.com/product/continuous-integration/
  22. 33 GitLab is the single tool for the entire DevOps

    Lifecycle • We’ve only talked about one part of GitLab tonight • We cover everything from ◦ Agile Planning and Portfolio Management ◦ Git Repo, Branching, Merge Request Details ◦ CI/CD ◦ Security Scanning ◦ Security Defense (emerging) • https://about.gitlab.com •