Concourse. CI с кубиками и на чистом YAML

Митап на тему "CI/CD", 21-11-2018
Максим Залысин (Devino Telecom)

CI систем сегодня не сосчитать — пальцев точно не хватит. Есть очень популярные, всегда на слуху, UI которых способен запутать в беготне по вкладкам. По ним написано не мало cookbook’ов, но сложности все равно возникают. Есть совсем простые — конфиг в проекте с несколькими строчками, и готово. И среди всех этих систем выделяется Concourse CI, с необычным UI и pipeline’ами на YAML.
В докладе я познакомлю с Concourse CI, расскажу, как готовятся pipeline’ы в Devino Telecom и какие планы по развитию CI, а также про недостатки которые не получилось устранить, и с которыми приходится мириться.

Transcript

1. Concourse CI с кубиками и на чистом YAML Максим Залысин
4. 4 О чем доклад? • Как устроен Concourse CI • Для чего CLI-утилита fly? • Что делает Concourse CI в Devino Telecom? • Что такое pipeline в Concourse? • Для чего resource? • Что происходит в job`е?
7. 7 TSA transportation security administration Secure Worker Registration ATC air traffic control Web UI, API, Pipeline Scheduler
8. 8 TSA transportation security administration Secure Worker Registration ATC air traffic control Web UI, API, Pipeline Scheduler
9. 9 TSA transportation security administration Secure Worker Registration ATC air traffic control Web UI, API, Pipeline Scheduler
10. 10 TSA transportation security administration Secure Worker Registration fly ATC air traffic control Web UI, API, Pipeline Scheduler
11. 11 TSA transportation security administration Secure Worker Registration fly ATC air traffic control Web UI, API, Pipeline Scheduler
12. 12 Для чего CLI-утилита fly? Управление team`ами $ fly -t main set-team --team-name=develop --ldap-group=… Управление pipeline`ами $ fly -t develop --pipeline=HELLO-WORLD --config=pipelines/hello-world.yml Тестирование task`ов $ fly -t develop execute --config=ci/build.yml --input=source=./ Отладка работающего task`а $ fly -t develop hijack --job=HELLO-WORLD/hello-world_build …
13. 13 Что делает Concourse CI в Devino Telecom?
14. 14 • Unit-test • CheckStyle • SpotBugs • PMD • Build • Autotests • … TEST Pull Request Status Comment
15. 15 • Unit-test • CheckStyle • SpotBugs • PMD • Build • Autotests • … TEST Pull Request Status Comment Review, Merge, Tag TEST • Unit-test • CheckStyle • SpotBugs • PMD • Build • Autotests • …
16. 16 • Unit-test • CheckStyle • SpotBugs • PMD • Build • Autotests • … • Get Dependencies • Compile • Pack TEST Pull Request Status Comment Review, Merge, Tag BUILD TEST N otify Project Channel Build & Push • Unit-test • CheckStyle • SpotBugs • PMD • Build • Autotests • …
17. 17 • Unit-test • CheckStyle • SpotBugs • PMD • Build • Autotests • … • Get Dependencies • Compile • Pack TEST Pull Request Status Comment Review, Merge, Tag BUILD Build & Push TEST • Prepare DEPLOY IN TEST N otify Edit Manifest Project Channel Deploy Notify TEST • Unit-test • CheckStyle • SpotBugs • PMD • Build • Autotests • …
18. 18 • Unit-test • CheckStyle • SpotBugs • PMD • Build • Autotests • … • Get Dependencies • Compile • Pack TEST Pull Request Status Comment Review, Merge, Tag BUILD TEST • Prepare DEPLOY IN TEST N otify Edit Manifest Project Channel Deploy Notify TEST PRODUCTION • Prepare DEPLOY IN PROD Deploy Deploy Channel N otify Edit Manifest Build & Push • Unit-test • CheckStyle • SpotBugs • PMD • Build • Autotests • …
19. 19 • Unit-test • CheckStyle • SpotBugs • PMD • Build • Autotests • … • Get Dependencies • Compile • Pack TEST Pull Request Status Comment Review, Merge, Tag BUILD • Unit-test • CheckStyle • SpotBugs • PMD • Build • Autotests • … TEST • Prepare DEPLOY IN TEST N otify Edit Manifest Project Channel Deploy Notify TEST PRODUCTION • Prepare DEPLOY IN PROD Deploy Deploy Channel N otify Edit Manifest Get Task Close Task Get Task Close Task Build & Push
20. 20 Что такое pipeline в Concourse?
21. Concourse CI v4.x.x - Pipeline
22. Concourse CI v4.x.x - Pipeline resource
23. Concourse CI v4.x.x - Pipeline job
24. 24 Concourse CI v4.x.x - Main Dashboard
25. 25 Для чего resource?
26. Concourse CI v4.x.x - Resource
27. 27 Concourse CI v4.x.x - Resource resources: - name: hello-world type: git source: uri: ssh://git@git.example.com/ex/hello-world.git private_key: {{bitbucket-private-key}} branch: master tag_filter: v*
28. 28 Concourse CI v4.x.x - Resource resources: - name: hello-world type: git source: uri: ssh://git@git.example.com/ex/hello-world.git private_key: {{bitbucket-private-key}} branch: master tag_filter: v*
29. 29 Concourse CI v4.x.x - Resource resources: - name: hello-world type: git source: uri: ssh://git@git.example.com/ex/hello-world.git private_key: {{bitbucket-private-key}} branch: master tag_filter: v*
30. 30 Concourse CI v4.x.x - Resource resources: - name: hello-world type: git source: uri: ssh://git@git.example.com/ex/hello-world.git private_key: | -----BEGIN EC PRIVATE KEY----- ... -----END EC PRIVATE KEY----- branch: master tag_filter: v*
31. Concourse CI v4.x.x - Resource (community)
32. 32 Concourse CI v4.x.x - Resource (community) resource_types: - name: bitbucket-pullrequest type: docker-image source: {repository: laurentverbruggen/concourse-bitbucket-pullrequest-resource} resources: - name: hello-world-pullrequest type: bitbucket-pullrequest source: uri: https://git.example.com/scm/ex/hello-world.git username: {{bitbucket-username}} password: {{bitbucket-password}} only_for_branch: master only_without_conflicts: true rebuild_when_target_changed: true
33. 33 Concourse CI v4.x.x - Resource (community) resource_types: - name: bitbucket-pullrequest type: docker-image source: {repository: laurentverbruggen/concourse-bitbucket-pullrequest-resource} resources: - name: hello-world-pullrequest type: bitbucket-pullrequest source: uri: https://git.example.com/scm/ex/hello-world.git username: {{bitbucket-username}} password: {{bitbucket-password}} only_for_branch: master only_without_conflicts: true rebuild_when_target_changed: true
34. 34 Concourse CI v4.x.x - Resource (community) resource_types: - name: bitbucket-pullrequest type: docker-image source: {repository: laurentverbruggen/concourse-bitbucket-pullrequest-resource} resources: - name: hello-world-pullrequest type: bitbucket-pullrequest source: uri: https://git.example.com/scm/ex/hello-world.git username: {{bitbucket-username}} password: {{bitbucket-password}} only_for_branch: master only_without_conflicts: true rebuild_when_target_changed: true
35. 35 Concourse CI v4.x.x - Resource (community) resource_types: - name: bitbucket-pullrequest type: docker-image source: {repository: laurentverbruggen/concourse-bitbucket-pullrequest-resource} resources: - name: hello-world-pullrequest type: bitbucket-pullrequest source: uri: https://git.example.com/scm/ex/hello-world.git username: {{bitbucket-username}} password: {{bitbucket-password}} only_for_branch: master only_without_conflicts: true rebuild_when_target_changed: true
36. 36 Concourse CI v4.x.x - Resource (community) resource_types: - name: bitbucket-pullrequest type: docker-image source: {repository: laurentverbruggen/concourse-bitbucket-pullrequest-resource} resources: - name: hello-world-pullrequest type: bitbucket-pullrequest source: uri: https://git.example.com/scm/ex/hello-world.git username: {{bitbucket-username}} password: {{bitbucket-password}} only_for_branch: master only_without_conflicts: true rebuild_when_target_changed: true
37. 37 Concourse CI v4.x.x - Resource (community) resource_types: - name: bitbucket-pullrequest type: docker-image source: {repository: laurentverbruggen/concourse-bitbucket-pullrequest-resource} resources: - name: hello-world-pullrequest type: bitbucket-pullrequest source: uri: https://git.example.com/scm/ex/hello-world.git username: concourse password: #sEcrEt_pAss-&1 only_for_branch: master only_without_conflicts: true rebuild_when_target_changed: true
38. 38 Concourse CI v4.x.x - Resource Types Provided With Concourse
39. 39 Concourse CI v4.x.x - Resource Types Provided With Concourse Community
40. 40 Concourse CI v4.x.x - Resource Types Provided With Concourse Community https://concourse-ci.org/community-resources.html
41. 41 Concourse CI v4.x.x - Resource (community) $ docker run --rm --entrypoint ls devinotelecom/concourse-slack-alert-resource -l /opt/resource total 19340 -rwxr-xr-x 1 root root 6451874 Nov 1 09:57 check -rwxr-xr-x 1 root root 6452080 Nov 1 09:57 in -rwxr-xr-x 1 root root 6879238 Nov 1 09:57 out https://concourse-ci.org/implementing-resources.html
42. 42 Что происходит в job`е?
43. 43 Concourse CI v4.x.x - Job "*_build"
44. 44 Concourse CI v4.x.x - Job "*_build" jobs: - name: hello-world_build plan: - get: hello-world passed: ["hello-world_test"] trigger: true - task: build file: hello-world/ci/build.yml input_mapping: {source: hello-world} - put: hello-world-docker params: {build: docker, tag: docker/version, tag_as_latest: true} on_failure: put: slack params: {alert_type: failed, message: "Build FAILED"} on_success: put: slack params: {alert_type: success, message: "Build SUCCESS"}
45. 45 Concourse CI v4.x.x - Job "*_build" jobs: - name: hello-world_build plan: - get: hello-world passed: ["hello-world_test"] trigger: true - task: build file: hello-world/ci/build.yml input_mapping: {source: hello-world} - put: hello-world-docker params: {build: docker, tag: docker/version, tag_as_latest: true} on_failure: put: slack params: {alert_type: failed, message: "Build FAILED"} on_success: put: slack params: {alert_type: success, message: "Build SUCCESS"}
46. 46 Concourse CI v4.x.x - Job "*_build" jobs: - name: hello-world_build plan: - get: hello-world passed: ["hello-world_test"] trigger: true - task: build file: hello-world/ci/build.yml input_mapping: {source: hello-world} - put: hello-world-docker params: {build: docker, tag: docker/version, tag_as_latest: true} on_failure: put: slack params: {alert_type: failed, message: "Build FAILED"} on_success: put: slack params: {alert_type: success, message: "Build SUCCESS"}
47. 47 Concourse CI v4.x.x - Job "*_build" jobs: - name: hello-world_build plan: - get: hello-world passed: ["hello-world_test"] trigger: true - task: build file: hello-world/ci/build.yml input_mapping: {source: hello-world} - put: hello-world-docker params: {build: docker, tag: docker/version, tag_as_latest: true} on_failure: put: slack params: {alert_type: failed, message: "Build FAILED"} on_success: put: slack params: {alert_type: success, message: "Build SUCCESS"}
48. 48 Concourse CI v4.x.x - Task ci/build.yml platform: linux image_resource: type: docker-image source: {repository: gradle, tag: 4.10.2-jdk8-alpine} inputs: - name: source outputs: - name: docker caches: - path: source/.gradle/caches run: dir: source user: root path: sh args: - -exc - | gradle --gradle-user-home=.gradle build cp build/libs/source.jar ../docker/application.jar cp ci/Dockerfile > ../docker/Dockerfile apk add --no-cache --no-progress git git describe --tags | sed 's/^v//' > ../docker/version
49. 49 Concourse CI v4.x.x - Job "*-pullrequest_test"
50. 50 Concourse CI v4.x.x - Job "*-pullrequest_test" jobs: - name: hello-world-pullrequest_test plan: - get: hello-world-pullrequest trigger: true - aggregate: - task: unit file: hello-world-pullrequest/ci/unit.yml input_mapping: {source: hello-world-pullrequest} - task: checkstyle file: hello-world-pullrequest/ci/checkstyle.yml input_mapping: {source: hello-world-pullrequest} on_failure: put: hello-world-pullrequest params: {path: hello-world-pullrequestt, status: failure} on_success: put: hello-world-pullrequest params: {path: hello-world-pullrequest, status: success}
51. 51 Concourse CI v4.x.x - Job "*-pullrequest_test" jobs: - name: hello-world-pullrequest_test plan: - get: hello-world-pullrequest trigger: true - aggregate: - task: unit file: hello-world-pullrequest/ci/unit.yml input_mapping: {source: hello-world-pullrequest} - task: checkstyle file: hello-world-pullrequest/ci/checkstyle.yml input_mapping: {source: hello-world-pullrequest} on_failure: put: hello-world-pullrequest params: {path: hello-world-pullrequestt, status: failure} on_success: put: hello-world-pullrequest params: {path: hello-world-pullrequest, status: success}
52. 52 Concourse CI v4.x.x - Job "*-pullrequest_test" jobs: - name: hello-world-pullrequest_test plan: - get: hello-world-pullrequest trigger: true - aggregate: - task: unit file: hello-world-pullrequest/ci/unit.yml input_mapping: {source: hello-world-pullrequest} - task: checkstyle file: hello-world-pullrequest/ci/checkstyle.yml input_mapping: {source: hello-world-pullrequest} on_failure: put: hello-world-pullrequest params: {path: hello-world-pullrequestt, status: failure} on_success: put: hello-world-pullrequest params: {path: hello-world-pullrequest, status: success}
53. 53 Concourse CI v4.x.x - Job "*-pullrequest_test" jobs: - name: hello-world-pullrequest_test plan: - get: hello-world-pullrequest trigger: true - aggregate: - task: unit file: hello-world-pullrequest/ci/unit.yml input_mapping: {source: hello-world-pullrequest} - task: checkstyle file: hello-world-pullrequest/ci/checkstyle.yml input_mapping: {source: hello-world-pullrequest} on_failure: put: hello-world-pullrequest params: {path: hello-world-pullrequestt, status: failure} on_success: put: hello-world-pullrequest params: {path: hello-world-pullrequest, status: success}
54. 54 Преимущества! • YAML - удобно, структурировано • CLI для управления CI`кой • Интеграция с чем угодно через resource • Просто написать свой resource • Job одновременно выполняет множество task`ов в связке с разными resource`ами • Переиспользуемые файлы task`ов
55. 55 Недостатки? • Требовательность к ресурсам • Бывает, worker залипает • Сложно повторно запустить job на устаревшем ресурсе • Нет разделения доступа по ролям в одной team`е
56. https://devinotele.com 56 Вопросы? Ссылки: • https://concourse-ci.org • https://github.com/devinotelecom • https://hub.docker.com/u/devinotelecom • https://t.me/magnaz