Share
Embed
Transcript
82 slides
-
#railsdm2019 Analyze Rails CI Analyze Rails CI @mtsmfm Fumiaki Matsushima
RailsDM 2019 #railsdm2019 -
#railsdm2019 Analyze Rails CI ➔ Web Dev at Quipper ➔
Ruby, Mahjong, Dead by Daylight ➔ Nishi-nippori.rb organizer ➔ GraphQL Tokyo organizer @mtsmfm.inspect -
#railsdm2019 Analyze Rails CI I gave a talk about flaky
tests before https://speakerdeck.com/mtsmfm/how-do-e2e-tests-fail-randomly -
#railsdm2019 Analyze Rails CI “100% is the wrong reliability target
for basically everything” Site Reliability Engineering Edited by Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy https://landing.google.com/sre/sre-book/chapters/introduction/ -
#railsdm2019 Analyze Rails CI The percentage of passed jobs on
master branch in Feb 2019 in rails/rails repo -
#railsdm2019 Analyze Rails CI What I’d like to talk -
I’ve published Rails CI result as a public dataset on BigQuery https://console.cloud.google.com/bigquery?p=rails-travis -result&d=rails_travis_result&page=dataset - You can find the test was failed before or not on Rails CI via http://has-it-failed.herokuapp.com - There’s a room for improvement if we have CI result database (e.g. Make CI result passed if failed one is a flaky test) -
#railsdm2019 Analyze Rails CI Browser rails-ci-result-importer BigQuery PostgreSQL has-it-failed TravisCI
API Overview of systems I created for this talk -
#railsdm2019 Analyze Rails CI Heroku Scheduler Embulk BigQuery Redis diff.yml
TravisCI API bin/import embulk-input-travis embulk-output-bigquery -
#railsdm2019 Analyze Rails CI Why embulk? - Just curious -
Embulk plugin can be written in (J)Ruby -
#railsdm2019 Analyze Rails CI Heroku Scheduler Embulk BigQuery Redis diff.yml
TravisCI API bin/import embulk-input-travis embulk-output-bigquery -
#railsdm2019 Analyze Rails CI Resources on TravisCI Commit d8d6bd5 Build
59740 Job 59740.1 GEM=railties Ruby: 2.5.3 $ git push xxx Repository rails/rails Job 59740.2 GEM=railties Ruby: 2.6.0 Job 59740.3 GEM=ap,ac Ruby: 2.6.0 ... -
#railsdm2019 Analyze Rails CI Table schema on BigQuery Field name
Type Description id INTEGER Job id on TravisCI data STRING Job attributes in JSON log STRING Job log started_at TIMESTAMP When this job started build_number INTEGER Build number job belongs to build_data STRING Build attributes job belongs to commit_data STRING Commit attributes job belongs to -
#railsdm2019 Analyze Rails CI Important notice - This dataset doesn’t
have all results (for now) - Some rows don’t have commit_data (for now) - I paid to store this dataset but you need to pay money to run query -
#railsdm2019 Analyze Rails CI Table schema on BigQuery Field name
Type Description id INTEGER Job id on TravisCI data STRING Job attributes in JSON log STRING Job log started_at TIMESTAMP When this job started build_number INTEGER Build number job belongs to build_data STRING Build attributes job belongs to commit_data STRING Commit attributes job belongs to Partitioned by started_at -
#railsdm2019 Analyze Rails CI 40 / 1024 * 5 ≒
0.2 (USD) https://cloud.google.com/bigquery/pricing -
#railsdm2019 Analyze Rails CI Important notice - This dataset doesn’t
have all results (for now) - Some rows don’t have commit_data (for now) - I paid to store this dataset but you need to pay money to run query - Be sure to query with “WHERE started_at” - Full scan costs only $0.2 for now but amount of data keeps increasing -
#railsdm2019 Analyze Rails CI Browser BigQuery PostgreSQL Find all failed
tests via SQL ImportJob Store PG (cache) Web server -
#railsdm2019 Analyze Rails CI Browser BigQuery PostgreSQL Find all failed
tests via SQL ImportJob Store PG (cache) Web server -
#railsdm2019 Analyze Rails CI It’s just raw log... /home/travis/.rvm/rubies/ruby-head/bin/ruby -w
-Itest -Ilib -I../activesupport/lib -I../actionpack/lib -I../actionview/lib -I../activemodel/lib test/commands/server_test.rb Run options: --seed 10052 # Running: ....F Failure: Rails::Command::ServerCommandTest#test_using_server_mistype_without_suggestion [test/commands/server_test.rb:38]: Expected /Maybe you meant/ to not match "Could not find server \"t\". Maybe you meant \"cgi\"?\nRun `rails server --help` for more options.\n". rails test test/commands/server_test.rb:35 ..................... Finished in 0.760100s, 34.2060 runs/s, 71.0433 assertions/s. 26 runs, 54 assertions, 1 failures, 0 errors, 0 skips ^^^ +++ --- test/configuration/middleware_stack_proxy_test.rb -
#railsdm2019 Analyze Rails CI It’s just raw log... /home/travis/.rvm/rubies/ruby-head/bin/ruby -w
-Itest -Ilib -I../activesupport/lib -I../actionpack/lib -I../actionview/lib -I../activemodel/lib test/commands/server_test.rb Run options: --seed 10052 # Running: ....F Failure: Rails::Command::ServerCommandTest#test_using_server_mistype_without_suggestion [test/commands/server_test.rb:38]: Expected /Maybe you meant/ to not match "Could not find server \"t\". Maybe you meant \"cgi\"?\nRun `rails server --help` for more options.\n". rails test test/commands/server_test.rb:35 ..................... Finished in 0.760100s, 34.2060 runs/s, 71.0433 assertions/s. 26 runs, 54 assertions, 1 failures, 0 errors, 0 skips ^^^ +++ --- test/configuration/middleware_stack_proxy_test.rb Test class, test method Failed info -
#railsdm2019 Analyze Rails CI Browser BigQuery PostgreSQL Find all failed
tests via SQL ImportJob Store PG (cache) Web server -
#railsdm2019 Analyze Rails CI SELECT * FROM( SELECT DISTINCT id,
data, build_number, build_data, extractCiResult(log) AS parse_result FROM `rails-travis-result.rails_travis_result.jobs` WHERE "#{from.iso8601}" < started_at AND started_at < "#{to.iso8601}" ) WHERE parse_result <> "error" AND JSON_EXTRACT_SCALAR(data, "$.state") = "failed" AND JSON_ARRAY_LENGTH( JSON_EXTRACT(parse_result, "$.failedTests") ) > 0 -
#railsdm2019 Analyze Rails CI SELECT * FROM( SELECT DISTINCT id,
data, build_number, build_data, extractCiResult(log) AS parse_result FROM `rails-travis-result.rails_travis_result.jobs` WHERE "#{from.iso8601}" < started_at AND started_at < "#{to.iso8601}" ) WHERE parse_result <> "error" AND JSON_EXTRACT_SCALAR(data, "$.state") = "failed" AND JSON_ARRAY_LENGTH( JSON_EXTRACT(parse_result, "$.failedTests") ) > 0 -
#railsdm2019 Analyze Rails CI CREATE TEMP FUNCTION extractCiResult (log STRING)
RETURNS STRING LANGUAGE js AS """ try { const railsCiResult = (TravisResultParser.parse(log)).find( command => command.includes('[Travis CI]') ); return JSON.stringify(RailsCiParser.parse(railsCiResult)); } catch { return 'error'; } """ OPTIONS ( library=[ "gs://rails-travis-result/parser/v0.1.0/rails_ci.js", "gs://rails-travis-result/parser/v0.1.0/travis_result.js" ] ); -
#railsdm2019 Analyze Rails CI BigQuery supports JavaScript UDF - Let’s
write a parser in JavaScript and run on BigQuery directly! -
#railsdm2019 Analyze Rails CI PEG and PEG.js - Parsing Expression
Grammar - PEG.js is one of the most popular parser generator - Export parser from PEG - Online editor -
#railsdm2019 Analyze Rails CI Result = (Noise command:Command Noise {
return command; })+ Noise = (!Command .)* Command = TimeStart commandBody:CommandBody TimeEnd { return commandBody; } CommandBody = chars:(!TimeEnd char:. {return char})* { return chars.join('').replace(/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]/g, '').replace(/\r\n/g, "\n") } TimeStart = "travis_time:start:" (!"\n" .)+ "\n" TimeEnd = "travis_time:end:" (!"\n" .)+ "\n" -
#railsdm2019 Analyze Rails CI How to parse Rails CI log
1. Find output from entrypoint (ci/travis.rb) 2. Find test command e.g. $ ruby foo_test.rb 3. Find finished messages -
#railsdm2019 Analyze Rails CI CREATE TEMP FUNCTION extractCiResult (log STRING)
RETURNS STRING LANGUAGE js AS """ try { const railsCiResult = (TravisResultParser.parse(log)).find( command => command.includes('[Travis CI]') ); return JSON.stringify(RailsCiParser.parse(railsCiResult)); } catch { return 'error'; } """ OPTIONS ( library=[ "gs://rails-travis-result/parser/v0.1.0/rails_ci.js", "gs://rails-travis-result/parser/v0.1.0/travis_result.js" ] ); -
#railsdm2019 Analyze Rails CI Result = (Noise command:Command Noise {
return command; })+ Noise = (!Command .)* Command = TimeStart commandBody:CommandBody TimeEnd { return commandBody; } CommandBody = chars:(!TimeEnd char:. {return char})* { return chars.join('').replace(/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]/g, '').replace(/\r\n/g, "\n") } TimeStart = "travis_time:start:" (!"\n" .)+ "\n" TimeEnd = "travis_time:end:" (!"\n" .)+ "\n" -
#railsdm2019 Analyze Rails CI travis_fold:start:worker_info … travis_fold:end:worker_info … travis_time:start:08cc8690 $
bundle install --jobs 3 --retry 3 travis_time:end:08cc8690 … ... travis_time:start:262254e0 $ ci/travis.rb [Travis CI] railties Running command: bundle exec rake test travis_time:end:262254e0 … Done. Your build exited with 0. Result = (Noise command:Command Noise { return command; })+ -
#railsdm2019 Analyze Rails CI travis_fold:start:worker_info … travis_fold:end:worker_info … travis_time:start:08cc8690 $
bundle install --jobs 3 --retry 3 travis_time:end:08cc8690 … ... travis_time:start:262254e0 $ ci/travis.rb [Travis CI] railties Running command: bundle exec rake test travis_time:end:262254e0 … Done. Your build exited with 0. Noise Result = (Noise command:Command Noise { return command; })+ Noise Command Noise Noise Command -
#railsdm2019 Analyze Rails CI CREATE TEMP FUNCTION extractCiResult (log STRING)
RETURNS STRING LANGUAGE js AS """ try { const railsCiResult = (TravisResultParser.parse(log)).find( command => command.includes('[Travis CI]') ); return JSON.stringify(RailsCiParser.parse(railsCiResult)); } catch { return 'error'; } """ OPTIONS ( library=[ "gs://rails-travis-result/parser/v0.1.0/rails_ci.js", "gs://rails-travis-result/parser/v0.1.0/travis_result.js" ] ); -
#railsdm2019 Analyze Rails CI CREATE TEMP FUNCTION extractCiResult (log STRING)
RETURNS STRING LANGUAGE js AS """ try { const railsCiResult = (TravisResultParser.parse(log)).find( command => command.includes('[Travis CI]') ); return JSON.stringify(RailsCiParser.parse(railsCiResult)); } catch { return 'error'; } """ OPTIONS ( library=[ "gs://rails-travis-result/parser/v0.1.0/rails_ci.js", "gs://rails-travis-result/parser/v0.1.0/travis_result.js" ] ); -
#railsdm2019 Analyze Rails CI RailsCiResult = Noise tests:Test+ { return
{ failedTests: tests.filter(t => t.summary).filter(({summary: {failuresCount, errorsCount}}) => failuresCount + errorsCount > 0), coreDumpedTests: tests.filter(t => t.dump), noises: tests.filter(t => t.noise) } } Noise = (!Test .)* Test = command:TestCommand result:( results:TestFailureOrErrorResult* summary:TestFinished TestOutputNoise { return { summary, results } } / results:TestSuccessResult summary:TestFinished TestOutputNoise { return { summary, results } } / dump:RubyCoreDump { return { dump } } / noise:TestOutputNoise { return { noise } } ) { return { command, ...result }; } TestCommand = command:$("/home/travis/.rvm" $(!("/bin/" "j"? "ruby -w" / "\n") .)* "/bin/" "j"? "ruby -w" $(!"\n" .)*) "\n" { return command } TestOutputNoise = (!TestCommand .)* TestSuccessResult = message:$(!TestFinished .)* { return { message } } TestFailureOrErrorResult = (!(TestFailureOrErrorResultKeyword / TestFinished) .)* meta:TestFailureOrErrorResultKeyword message:TestFailureMessage { return { ...meta, message } } TestFailureMessage = $(!(TestFailureOrErrorResultKeyword / TestFinished) .)* TestFailureOrErrorResultKeyword = "\n" type:("Failure" / "Error") ":\n" testClass:TestClass "#" method:TestMethod (" [" file:TestFile ":" line:Int "]")? ":\n" { return { type, testClass, method } } TestClass = $[a-zA-Z:]+ TestMethod = $[a-zA-Z_]+ TestFile = $[a-z/\._]+ TestFinished = "\n" runsCount:Int " runs, " assertionsCount:Int " assertions, " failuresCount:Int " failures, " errorsCount:Int " errors, " skipsCount:Int " skips" "\n" { return { runsCount, assertionsCount, failuresCount, errorsCount, skipsCount } } RubyCoreDump = $((!RubyCoreDumpNote .)* RubyCoreDumpNote) RubyCoreDumpNote = "\n" "[NOTE]\n" "You may have encountered a bug in the Ruby interpreter or extension libraries.\n" "Bug reports are welcome.\n" "For details: https://www.ruby-lang.org/bugreport.html\n" "\n" "Aborted (core dumped)\n" Int = chars:$[0-9]+ { return parseInt(chars) } -
#railsdm2019 Analyze Rails CI RailsCiResult = Noise tests:Test+ { return
{ failedTests: tests.filter(t => t.summary).filter(({summary: {failuresCount, errorsCount}}) => failuresCount + errorsCount > 0), coreDumpedTests: tests.filter(t => t.dump), noises: tests.filter(t => t.noise) } } -
#railsdm2019 Analyze Rails CI Test = command:TestCommand result:( TestFailureOrErrorResult* TestFinished
TestOutputNoise / TestSuccessResult TestFinished TestOutputNoise / RubyCoreDump / TestOutputNoise ) -
#railsdm2019 Analyze Rails CI TestCommand = command:$("/home/travis/.rvm" $(!("/bin/" "j"? "ruby
-w" / "\n") .)* "/bin/" "j"? "ruby -w" $(!"\n" .)*) "\n" TestFailureOrErrorResult = (!(TestFailureOrErrorResultKeyword / TestFinished) .)* TestFailureOrErrorResultKeyword TestFailureMessage TestFailureOrErrorResultKeyword = "\n" type:("Failure" / "Error") ":\n" testClass:TestClass "#" method:TestMethod (" [" file:TestFile ":" line:Int "]")? ":\n" TestClass = $[a-zA-Z:]+ TestMethod = $[a-zA-Z_]+ TestFile = $[a-z/\._]+ TestFinished = "\n" runsCount:Int " runs, " assertionsCount:Int " assertions, " failuresCount:Int " failures, " errorsCount:Int " errors, " skipsCount:Int " skips" "\n" -
#railsdm2019 Analyze Rails CI /home/travis/.rvm/rubies/ruby-head/bin/ruby -w -Itest -Ilib -I../activesupport/lib -I../actionpack/lib
-I../actionview/lib -I../activemodel/lib test/commands/server_test.rb Run options: --seed 10052 # Running: ....F Failure: Rails::Command::ServerCommandTest#test_using_server_mistype_without_suggestion [test/commands/server_test.rb:38]: Expected /Maybe you meant/ to not match "Could not find server \"t\". Maybe you meant \"cgi\"?\nRun `rails server --help` for more options.\n". rails test test/commands/server_test.rb:35 ..................... Finished in 0.760100s, 34.2060 runs/s, 71.0433 assertions/s. 26 runs, 54 assertions, 1 failures, 0 errors, 0 skips ^^^ +++ --- test/configuration/middleware_stack_proxy_test.rb /home/travis/.rvm/rubies/ruby-head/bin/ruby -w -Itest -Ilib -I../activesupport/lib -I../actionpack/lib -I../actionview/lib -I../activemodel/lib test/configuration/middleware_stack_proxy_test.rb -
#railsdm2019 Analyze Rails CI /home/travis/.rvm/rubies/ruby-head/bin/ruby -w -Itest -Ilib -I../activesupport/lib -I../actionpack/lib
-I../actionview/lib -I../activemodel/lib test/commands/server_test.rb Run options: --seed 10052 # Running: ....F Failure: Rails::Command::ServerCommandTest#test_using_server_mistype_without_suggestion [test/commands/server_test.rb:38]: Expected /Maybe you meant/ to not match "Could not find server \"t\". Maybe you meant \"cgi\"?\nRun `rails server --help` for more options.\n". rails test test/commands/server_test.rb:35 ..................... Finished in 0.760100s, 34.2060 runs/s, 71.0433 assertions/s. 26 runs, 54 assertions, 1 failures, 0 errors, 0 skips ^^^ +++ --- test/configuration/middleware_stack_proxy_test.rb /home/travis/.rvm/rubies/ruby-head/bin/ruby -w -Itest -Ilib -I../activesupport/lib -I../actionpack/lib -I../actionview/lib -I../activemodel/lib test/configuration/middleware_stack_proxy_test.rb TestCommand Noise TestFailureOrErrorResultKeyword FailureMessage TestFinished -
#railsdm2019 Analyze Rails CI { "failedTests": [ { "command": "/home/travis/.rvm/rubies/ruby-head/bin/ruby
-w -Itest -Ilib -I../activesupport/lib -I../actionpack/lib -I../actionview/lib -I../activemodel/lib test/commands/server_test.rb", "summary": { "runsCount": 26, "assertionsCount": 54, "failuresCount": 1, "errorsCount": 0, "skipsCount": 0 }, "results": [ { "type": "Failure", "testClass": "Rails::Command::ServerCommandTest", "method": "test_using_server_mistype_without_suggestion", "message": "Expected /Maybe you meant/ to not match \"Could not find server \\\"t\\\". Maybe you meant \\\"cgi\\\"?\\nRun `rails server --help` for more options.\\n\".\n\nrails test test/commands/server_test.rb:35\n\n.....................\n\nFinished in 0.760100s, 34.2060 runs/s, 71.0433 assertions/s." } ... -
#railsdm2019 Analyze Rails CI Browser BigQuery PostgreSQL Find all failed
tests via SQL ImportJob Store PG (cache) Web server -
#railsdm2019 Analyze Rails CI has-it-failed web - Rails 6.0.0.beta3 -
Heroku - Nothing special -
#railsdm2019 Analyze Rails CI CREATE TEMP FUNCTION extractCiResult (log STRING)
RETURNS STRING LANGUAGE js AS """ try { const railsCiResult = (TravisResultParser.parse(log)).find( command => command.includes('[Travis CI]') ); return JSON.stringify(RailsCiParser.parse(railsCiResult)); } catch { return 'error'; } """ OPTIONS ( library=[ "gs://rails-travis-result/parser/v0.1.0/rails_ci.js", "gs://rails-travis-result/parser/v0.1.0/travis_result.js" ] ); -
#railsdm2019 Analyze Rails CI It’s much better to collect test
report in future -
#railsdm2019 Analyze Rails CI [Idea] Make CI passed if failed
one is flaky 1. Generate JUnit format XML test report 2. Send test report to the service 3. The service collects flaky tests data and returns 0 or 1 4. Use the value in step 3 as exit code -
#railsdm2019 Analyze Rails CI $ bin/ruby -I test || true
Flaky tests Data Web server $ test $(curl --form "[email protected]" https://server) = ‘0’ result.xml -
#railsdm2019 Analyze Rails CI $ bin/ruby -I test || true
Flaky tests Data 1. Generate XML and always returns 0 Web server $ test $(curl --form "[email protected]" https://server) = ‘0’ result.xml -
#railsdm2019 Analyze Rails CI $ bin/ruby -I test || true
Flaky tests Data 1. Generate XML and always returns 0 Web server $ test $(curl --form "[email protected]" https://server) = ‘0’ result.xml 2. Send test report to the service -
#railsdm2019 Analyze Rails CI $ bin/ruby -I test || true
Flaky tests Data 1. Generate XML and always returns 0 Web server $ test $(curl --form "[email protected]" https://server) = ‘0’ result.xml 3. Service returns 0 or 1 2. Send test report to the service -
#railsdm2019 Analyze Rails CI $ bin/ruby -I test || true
Flaky tests Data 1. Generate XML and always returns 0 Web server $ test $(curl --form "[email protected]" https://server) = ‘0’ result.xml 3. Service returns 0 or 1 4. Verify returned value 2. Send test report to the service -
#railsdm2019 Analyze Rails CI Conclusion - I’ve published Rails CI
result as a public dataset on BigQuery https://console.cloud.google.com/bigquery?p=rails-travis -result&d=rails_travis_result&page=dataset - You can find the test was failed before or not on Rails CI via http://has-it-failed.herokuapp.com - There’s a room for improvement if we have CI result database (e.g. Make CI result passed if failed one is a flaky test)