Share
Embed
CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.
CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにした話.
CircleCIでtiming dataに基づいたテスト分割をDartで利用できるようにするサンプルプロジェクト
https://github.com/operando/dart-test-CircleCI-timing-data
開発×テスト LT会 - vol.2 #devtestlt
https://rakus.connpass.com/event/237602/
operandoOS
March 15, 2022
Transcript
53 slides
-
ࣗݾհ • Shinobu Okano • @operandoOS / shinobu.dart • 10X,
Inc. / Software Engineer • Flutter / Dart / Android / Automation -
CircleCIͷςετϝλσʔλͷϑΥʔϚοτ • CircleCIͰςετ݁Ռ͕ςετϝλσʔλͱೝࣝ͞ΕΔϑΥʔϚοτҎԼ • JUnit XML • Cucumber JSON •
ଟ͘ͷݴޠϥΠϒϥϦͰαϙʔτ͍ͯ͠ΔͷJUnit XML • ຊࢿྉͰJUnit XMLΛѻ͍·͢ -
CircleCIͷςετׂ • ҎԼͷׂํ๏Λαϙʔτ͍ͯ͠Δ • timing data • ϑΝΠϧ໊ɾΫϥε໊ • ϑΝΠϧαΠζ
• CircleCItiming dataʹΑΔׂ͕࠷ྑͷํ๏ͱͯ͠Δ -
CircleCIͷtiming data • ֤ςετέʔεͷ݁ՌʹɺΫϥε໊ͱϑΝΠϧ໊ͳͲ͕ هࡌ͞Ε͍ͯΔ͜ͱΛCircleCIظ͍ͯ͠Δ • ֤ςετϑΝΠϧͷςετ࣮ߦʹ͔͔Δ͕࣌ؒΘ͔Δσʔλߏ • σϑΥϧτͷׂํ๏͕ɺϑΝΠϧ໊ʹجׂ͍ͮͨͳͷͰ্ه͕ඞཁ
• ςετϝλσʔλʹͦΕΒͷใ͕هࡌ͞Ε͍ͯͳ͍ͱtiming dataΛ ར༻ͯ͠ࢥͬͨͱ͓Γʹςετׂ͕࣮ߦ͞Εͳ͍ -
timing dataΛར༻ͯ͠ ςετׂ࣮ͯ͠ߦ͢Δྫʢrspecʣ circleci tests glob "test/**/*.rb" | circleci
tests split --split-by=timings > /tmp/tests-to-run bundle exec rspec $(cat /tmp/tests-to-run) -
• circleci tests splitͰςετϑΝΠϧͷҰཡऔಘ • --split-by=timingsͰtiming dataΛར༻࣮͠ߦڥ͝ͱʹ ςετׂΛߦ͏Α͏ʹࢦఆ •
bundle exec rspec࣮ߦڥ͝ͱʹׂ͞ΕͨςετΛ ࣮ߦ͢ΔΑ͏ʹࢦఆ timing dataΛར༻ͯ͠ ςετׂ࣮ͯ͠ߦ͢Δྫʢrspecʣ -
Why Dart test? • ࢲ͕ॴଐ͢Δձࣾ 10XͰ࡞͍ͬͯΔαʔϏε StailerͰ API࣮ʹαʔόʔαΠυDartΛ༻ •
αʔόʔαΠυDartͰCIʹCircleCIΛར༻͍ͯ͠Δ • DartͰςετΛॻ͘߹ʮtestʯύοέʔδ͕ϝδϟʔ • https://pub.dev/packages/test -
Dart sharding tests • testύοέʔδʹςετׂ͢Δػೳ͕͋Δʂʂ • ݱࡏ 10XͰtestͷػೳΛͬͯCircleCI্ͰςετΛׂɾฒྻ࣮ߦͯ͠ ͍Δ •
͔͠͠ɺ͜ͷׂํ๏֤ςετέʔεͷ࣮ߦ࣌ؒͳͲΛݩʹ ςετׂ͞ΕΔΘ͚Ͱͳ͍ • ͦͷͨΊExecutor͝ͱʹ࣮ߦ࣌ؒͷΒ͖͕ͭେ͖͍ͷ͕՝ -
Dart sharding tests dart test --total-shards 3 --shard-index 0 path/to/test.dart
dart test --total-shards 3 --shard-index 1 path/to/test.dart dart test --total-shards 3 --shard-index 2 path/to/test.dart -
Dart testͷ݁ՌΛJunit XMLͰग़ྗ͢Δ • Dart testςετ݁ՌΛJSONग़ྗ͢Δػೳ͕͋Δ • ͦͷJSONग़ྗ͞Εͨ݁ՌΛJunit XMLʹม͢Δπʔϧ͕͋Δ •
https://github.com/TOPdesk/dart-junitreport • ͜ΕΛͬͯDart testͷςετ݁ՌΛJunit XMLͰग़ྗ͢Δ -
ग़ྗͨ͠Junit XMLΛςετϝλσʔλͱͯ͠ CircleCIʹೝࣝͤ͞Δ • store_test_resultsεςοϓʹJunit XMLΛग़ྗͨ͠σΟϨΫτϦΛ ࢦఆ͢Δ • ͜ΕͰςετ݁ՌͷJunit XML͕CircleCIʹςετϝλσʔλ
ͱͯ͠ೝࣝ͞ΕΔ • ೝࣝͨ͠ςετϝλσʔλΛݩʹCircleCI͕timing dataΛੜ͢Δ -
ςετ࣮ߦͷίϚϯυΛॻ͖͑Δ • timing dataΛར༻ͯ͠ςετׂΛߦ͍ɺׂͨ͠ϑΝΠϧ ͷΈͰςετ࣮ߦ͢ΔΑ͏ʹίϚϯυΛॻ͖͑Δ TESTFILES=$(circleci tests glob "test/**/*_test.dart" |
circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded -
Կ͕ى͖͔ͨ...😨 • "No timing found for <ϑΝΠϧ໊>"ͱ͍͏ϝοηʔδ͕ग़ͨ • ςετϑΝΠϧʹର͢Δtiming data͕ଘࡏ͠ͳ͍ͥʂͬͯݴͬͯΔ
• timing dataͷϑΝΠϧੜ͞Ε͍͕ͯͨɺԿ͔͠ΒͷݪҼͰ ͏·͘ར༻͞Ε͍ͯͳ͍ͷͰɺ࠷దͳςετׂ͕ߦ͍͑ͯͳ͍ -
<testsuites> <testsuite errors="0" failures="0" tests="2" skipped="0" name=“test.sample_1" > <properties> <property
name="platform" value="vm" /> </properties> <testcase classname="test.sample_1" name=“String…” time="0.027" /> <testcase classname="test.sample_1" name=“String.trim…” time="0.005" /> </testsuite> <testsuite errors="0" failures="0" tests="4" skipped="0" name="test.sample_2" > <properties> <property name="platform" value="vm" /> </properties> <testcase classname="test.sample_2" name="String .s…” time="0.027" /> </testsuite> </testsuites> ग़ྗ͞ΕͨJunit XMLͷ༰ -
fi le keyͷ͕nullͳݪҼΛ୳Δ • testcase elementʹ֤ςετέʔεͷ݁Ռ͕هࡌ͞Ε͍ͯΔ • ͦ͜ʹ֤ςετέʔε͕هࡌ͞Ε͍ͯΔςετϑΝΠϧ໊͕ ॻ͔Ε͍ͯͳ͍͕ͳ͍͜ͱ͕ݪҼͬΆ͍͜ͱ͕Θ͔ͬͨ
• ݪҼ͕Θ͔ͬͨͷͰtestcase elementʹ fi le attributeͷ ՃΛࢼΈͨ -
ࢼͯ͠Έͨमਖ਼ • dart-junitreportͷ෦Ͱར༻͞Ε͍ͯΔdart-testreportϥΠϒϥϦͷ मਖ਼ • dart-junitreportͰJunit XMLग़ྗ͢Δࡍɺtestcase elementʹ fi
le attributeͷՃ • fi le attributeʹهࡌ͞Εͨ fi le pathΛCI্ͰsedίϚϯυΛ ར༻ͯ͠ॻ͖͑ -
version: 2.1 jobs: test: docker: - image: dart:2.16.1 parallelism: 2
steps: - checkout - run: name: Download dependencies command: pub get - run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart" | circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ workflows: version: 2 build: jobs: - test .circleci/con fi g.ymlͷ༰ ͍….จࣈখ͍͞…😇 -
- run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart"
| circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ େࣄͳͷ͜ͷ෦ʂ -
- run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart"
| circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ forkͨ͠ϦϙδτϦʹमਖ਼ΛೖΕͨͷͰ ࣗͷϦϙδτϦ͔Βऔಘ -
- run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart"
| circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ Junit XMLͰग़ྗ͠ɺsedͰ fi le attributeͷ༰Λॻ͖͑ -
- run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart"
| circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ ग़ྗͨ͠Junit XMLΛอଘʂ -
- run: name: Test command: | TESTFILES=$(circleci tests glob "test/**/*_test.dart"
| circleci tests split --split-by=timings) dart run test ${TESTFILES} --file-reporter json:test_results/tests_report.json --reporter expanded - run: name: Install JUnitReport when: always command: dart pub global activate --source git https://github.com/operando/dart-junitreport.git - run: name: Convert tests to JUnit when: always command: | $HOME/.pub-cache/bin/tojunit --input test_results/tests_report.json --output test_results/tests_report_junit.xml sed -i -e "s|file:///root/project/||g" test_results/tests_report_junit.xml - store_test_results: path: test_results/ - store_artifacts: path: test_results/ ͜ΕͰ͜ͷ෦ͷςετׂ͕͏·͘ಈ͍ͨʂ