Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Optimizing Your CI Pipelines

Optimizing Your CI Pipelines

Take your Continuous Integration to the next level! Learn how to optimize your pipelines for faster and more efficient builds through parallelization, caching, failing early, and more.

switowski

June 29, 2023
Tweet

More Decks by switowski

Other Decks in Programming

Transcript

  1. Example project ... build: stage: build script: - docker compose

    build test: stage: test script: - docker compose run --rm web python manage.py migrate - docker compose run --rm web pytest deploy: stage: deploy script: echo "Here goes deployment script" environment: production
  2. Improve your Docker config FROM python:3.10-slim-buster Bigger image Shorter build

    time FROM python:3.10-alpine Smaller image Longer build time
  3. Push (pull) image to (from) registry ... build: stage: build

    script: - docker compose build test: stage: test script: - docker compose run --rm web python manage.py migrate - docker compose run --rm web pytest
  4. Push (pull) image to (from) registry ... before_script: - docker

    login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY build: stage: build script: - docker compose build - docker push registry.gitlab.com/switowski/optimizing-ci-pipelines/web:dev test: stage: test script: - docker pull registry.gitlab.com/switowski/optimizing-ci-pipelines/web:dev - docker compose run --rm web python manage.py migrate - docker compose run --rm web pytest
  5. Push (pull) image to (from) registry ... before_script: - docker

    login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY build: stage: build script: - docker compose build - docker push registry.gitlab.com/switowski/optimizing-ci-pipelines/web:dev test: stage: test script: - docker pull registry.gitlab.com/switowski/optimizing-ci-pipelines/web:dev - docker compose run --rm web python manage.py migrate - docker compose run --rm web pytest
  6. With DAG build_3.8: stage: build script: - docker build -f

    Dockerfile_3.8 - docker push $CI_REGISTRY_IMAGE:3.8 test_3.8: stage: test needs: ["build_3.8"] script: - docker pull $CI_REGISTRY_IMAGE:3.8 - docker run $CI_REGISTRY_IMAGE:3.8 pytest release_3.8: stage: release needs: ["test_3.8"] script: echo "Release script"
  7. Downstream (child) pipelines frontend: trigger: include: frontend/.gitlab-ci.yml strategy: depend rules:

    - changes: [frontend/*] backend: trigger: include: backend/.gitlab-ci.yml strategy: depend rules: - changes: [backend/*]
  8. Run tests in parallel pytest-xdist Run tests across multiple CPUs

    $ pip install pytest-xdist $ pytest -n auto
  9. Run tests in parallel pytest-xdist Run tests across multiple CPUs

    pytest-test-groups Run tests across multiple runners
  10. Run tests in parallel pytest-xdist Run tests across multiple CPUs

    pytest-test-groups Run tests across multiple runners # requirements.in pytest-test-groups # .gitlab-ci.yml test: stage: test parallel: 5 script: - pytest \ --test-group-count $CI_NODE_TOTAL \ --test-group=$CI_NODE_INDEX
  11. Run tests in parallel pytest-xdist Run tests across multiple CPUs

    pytest-test-groups Run tests across multiple runners
  12. Run less and stop fast • Interruptible jobs build: stage:

    build interruptible: true script: - docker compose build
  13. Run less and stop fast • Interruptible jobs • Stop

    fast (pytest -x) build: stage: build interruptible: true script: - docker compose build
  14. Not every check is mandatory in the CI Some can

    run only on the main branches. Others can be triggered manually.
  15. Caching and cache policies You can use caching to, well,

    cache stuff between jobs. But you can also specify if you want to push or pull stuff to cache using policy key. default: cache: &global_cache key: $CI_COMMIT_REF_SLUG paths: - .cache/pip - some/other/path/ policy: pull-push job: cache: # inherit all global cache settings <<: *global_cache # override the policy policy: pull
  16. Fast zip For caching/artifacts, you can choose different level of

    compression (low level of compression runs faster, but results in a larger zip file). variables: FF_USE_FASTZIP: "true" # Available options are: # fastest, fast, default, slow, # or slowest ARTIFACT_COMPRESSION_LEVEL: "fastest" CACHE_COMPRESSION_LEVEL: "fastest"
  17. Takeaways • Learn concepts, not tools • There are no

    silver bullets 
 python-alpine or python-debian? Pull an image or build it?
  18. Takeaways • Learn concepts, not tools • There are no

    silver bullets 
 python-alpine or python-debian? Pull an image or build it? • Not every check has to run in every pipeline 
 make MR pipelines fast and main branch pipelines thorough
  19. Takeaways • Learn concepts, not tools • There are no

    silver bullets 
 python-alpine or python-debian? Pull an image or build it? • Not every check has to run in every pipeline 
 make MR pipelines fast and main branch pipelines thorough • Outdated CI setup is also a technical debt
  20. • Slide 1: https:/ /www.midjourney.com/app/search/?jobId=6f707c18-bf19-4fac-b87f-622ff3e1561b • Roadmap - containers: https:/

    /www.midjourney.com/app/search/?jobId=28e92637-bfd6-4d07-ae36-a279dd35c2d4 • Roadmap - rocket: https:/ /www.midjourney.com/app/search/?jobId=eaeeb022-17af-4b71-8438-a8eb05afc298 • Roadmap - traffic lights: https:/ /www.midjourney.com/app/search/?jobId=89159672-debf-4845-a822-be159983f172 • Roadmap - chest: https:/ /www.midjourney.com/app/search/?jobId=1bb66ddc-84a4-4ac1-a36a-7c7f8dbd1c04 • Gitlab logo: https:/ /www.midjourney.com/app/search/?jobId=a5491c2d-199e-43dd-b65f-9ebefc031f65 • Ribbon badge: https:/ /www.clipartmax.com/download/m2i8H7d3Z5i8G6K9_certificate-ribbons-ribbon-badge-vector-png/ • Containers: https:/ /www.midjourney.com/app/search/?jobId=f14ee5f1-b3a7-4bca-9907-e67607beff90 • DAG image: https:/ /www.midjourney.com/app/search/?jobId=4609a9aa-7154-49f2-9b10-38d600aa04f2 • Child pipelines: https:/ /www.midjourney.com/app/search/?jobId=39e19a04-1196-4c2e-99e9-f242464b5d7c • Rube Goldberg machine: https:/ /www.midjourney.com/app/search/?jobId=60ab5692-a07a-4992-8641-13dfbe6193f2 Attributions Most images come from midjourney.com Drawings were done with excalidraw.com