Share
Embed
Magento Test Automation - My Journey
Meet Magento Germany 2017
Slides as HTML: https://www.schmengler-se.de/slides/magento-test-automation-my-journey/
Fabian Schmengler
May 23, 2017
Transcript
68 slides
-
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 . Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 . Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 .. Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ... Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 .... SimpleTest? PHPUnit? Ibuildings MageTest? EcomDev PHPUnit? Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ..... Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ...... foo_bar? Foo_Bar? fooBar? foobar? bar? foo/bar? Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ....... TEST ALL THE THINGS! public function testObserver() { $this->assertEventObserverDefined( 'adminhtml', 'core_block_abstract_prepare_layout_before', SGH_OrderPrintPopups_Model_Observer_Block::MODEL, 'onCoreBlockAbstractPrepareLayoutBefore', 'orderprintpopups' ); } Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ........ TEST ALL THE THINGS! /** * @param Mage_Core_Block_Abstract $block * @return Varien_Event_Observer */ protected function _mockEventObserver($block, $otherArgs = array()) { $args = array_merge(array('block' => $block), $otherArgs); $observer = new Varien_Event_Observer(); $observer->setEvent(new Varien_Event($args)); return $observer; } Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ......... TEST ALL THE THINGS! $this->assertLayoutHandleLoaded( 'orderprintpopups_warensendung_index', 'module layout handle loaded' ); $this->assertLayoutBlockRendered( SGH_OrderPrintPopups_Block_Popup_Warensendung::ALIAS, 'popup block rendered' ); Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 .......... TEST ALL THE THINGS! $route = 'orderprintpopups/warensendung/barcodestatic'; $this->assertResponseBodyQueryRegex( '#sgh_warensendung', '#<img [^>]*src="[^"]+' . $route . '#' ); Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 .......... TEST ALL THE THINGS! /** * getter tests */ test("getter", function() { equal(this.block.getName(), "foo", "getName()"); equal(this.block.getContent(), "lorem ipsum falleri fallera", "getContent()"); equal(this.block.getPlaceholder(), "{foo}", "getPlaceholder()"); }); Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............ What lies behind the code Hours of debugging Hidden depenencies to global state Bugs of the testing frameworks Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............S A story of skipped tests $this->markTestSkipped('fatal error???'); Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............S A story of skipped tests $this->markTestSkipped('fatal error???'); $this->markTestSkipped('conflict with fixture and existing db'); Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............S A story of skipped tests $this->markTestSkipped('fatal error???'); $this->markTestSkipped('conflict with fixture and existing db'); $this->markTestSkipped('"out of stock", although fixture seems ok'); //FIXME Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............S A story of skipped tests $this->markTestSkipped('fatal error???'); $this->markTestSkipped('conflict with fixture and existing db'); $this->markTestSkipped('"out of stock", although fixture seems ok'); //FIXME $this->markTestSkipped( 'The new comparator classes in PHPUnit 3.6 cannot handle recursive comparision with $maxDepth parameter YET.' ); Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............S A story of skipped tests $this->markTestSkipped('fatal error???'); $this->markTestSkipped('conflict with fixture and existing db'); $this->markTestSkipped('"out of stock", although fixture seems ok'); //FIXME $this->markTestSkipped( 'The new comparator classes in PHPUnit 3.6 cannot handle recursive comparision with $maxDepth parameter YET.' ); $this->markTestSkipped( 'expectation + fixture changed. reason for missing prefix unknown '. 'but only occurs in test environment!' ); Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SS A story of skipped tests $this->markTestSkipped( 'Das klappt schon wieder alles nicht mit Adminlogin+Dispatch' ); Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SS A story of skipped tests $this->markTestSkipped( 'Das klappt schon wieder alles nicht mit Adminlogin+Dispatch' ); $this->markTestSkipped('Untestbar in der Form. Mglw. ein Bug in EcomDev_PHPUnit der zu leerem orig_data führt. Leider funktioniert @runInSeparateProcess nicht mit Magento.') Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSS Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSF Magento 1.6 Update Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF Magento Test Automation: My Journey @fschmengler -
06/2011 12/2011 06/2012 12/2012 06/2013 12/2013 06/2014 0 50 100
150 200 250 300 350 400 450 Product Description Blocks one of the first modules LLOC LLOC (Tests) 2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF. Magento Test Automation: My Journey @fschmengler -
01/2012 07/2012 12/2012 0 50 100 150 200 250 300
350 Payment one year into the project... LLOC LLOC (Tests) 2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S Magento Test Automation: My Journey @fschmengler -
01/2012 07/2012 12/2012 0 500 1000 1500 2000 2500 3000
3500 Payment Symfony Service LLOC LLOC (Tests) 2011 > 2012 > 2013 > 2014 > 2015 > 2016 > 2017 ............SSSFF...S. Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.. Diploma Thesis on Magento Development Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S... Research & Experiments Built-in tests Magento TAF for Smoke Test Suite Abstraction of automated acceptance tests + ZiBreve Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.... Selenium IDE > FitNesse Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S..... Agency project with Selenium Automated Acceptance Tests Selenium IDE Tests for every feature Partially written by QA Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S. Agency project with EcomDev_PHPUnit Automated Unit/Integration T ests Tests mandatory, coverage monitored Sophisticated setup Still typical problems Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.F DEADLINE IS COMING Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF M1 Integration Tests: Typical Problems Tests altering global state (singletons, registry) Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF M1 Integration Tests: Typical Problems Tests altering global state (singletons, registry) Tests depending on existing data, others delete it and use their own Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF M1 Integration Tests: Typical Problems Tests altering global state (singletons, registry) Tests depending on existing data, others delete it and use their own Tests not well isolated, not running in DB transaction Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF M1 Integration Tests: Typical Problems Tests altering global state (singletons, registry) Tests depending on existing data, others delete it and use their own Tests not well isolated, not running in DB transaction Complicated fixture setup Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF. THIS WEIRD UNIT TESTING TRICK BLEW MY MIND! Decouple business logic from Magento Use plain PHPUnit Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF.. Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF... Bad news we still need higher level tests Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF... Bad news we still need higher level tests Good news less of them Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF.... Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF..... Experiments Acceptance and integration test framework Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF..... Experiments Acceptance and integration test framework Acceptance test framework "THE SELENIUM-BASED TESTING FRAMEWORK FOR PEOPLE WHO HATE TESTING" Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF..... Experiments Acceptance and integration test framework Acceptance test framework "THE SELENIUM-BASED TESTING FRAMEWORK FOR PEOPLE WHO HATE TESTING" Test framework Codeception Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF...... A new era: Magento 2 Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF....... Magento 2 Extension Reused refactored lib from Magento 1 extension Integration tests for Magento 2 specific code Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........ Magento 2 Projects Great integration test framework Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........ Magento 2 Projects Great integration test framework Easier to start with Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........ Magento 2 Projects Great integration test framework Easier to start with More reliable Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S Magento 2 Projects Still frustrating sometimes Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S Magento 2 Projects Still frustrating sometimes Failures after minor version update Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S Magento 2 Projects Still frustrating sometimes Failures after minor version update Test runs not completely isolated Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S Magento 2 Projects Still frustrating sometimes Failures after minor version update Test runs not completely isolated Decoupled approach still works best Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S. TDD Katas: Back to the roots Regular practice Small tasks, no real project Goal: Make TDD my default habit https://www.schmengler-se.de/katas Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S.. How I would start today / Recommendations 1. For a quick win, create smoke test suite with Codeception Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S.. How I would start today / Recommendations 1. For a quick win, create smoke test suite with Codeception 2. Katas / small self contained projects to learn TDD Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S.. How I would start today / Recommendations 1. For a quick win, create smoke test suite with Codeception 2. Katas / small self contained projects to learn TDD 3. Start to build (small) parts of your code independent from Magento - "TDD" those parts Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S.. How I would start today / Recommendations 1. For a quick win, create smoke test suite with Codeception 2. Katas / small self contained projects to learn TDD 3. Start to build (small) parts of your code independent from Magento - "TDD" those parts 4. Try to add simple integration tests (in M1 with Xtest) but skip if it's getting complicated Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S... Mistakes I won't make again Try to sell tests as separate part of project quote or estimation Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S... Mistakes I won't make again Try to sell tests as separate part of project quote or estimation Try to teach "unit testing" and "testing Magento" at once Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S... Time: 40 minutes OK (42 slides, 0 cat pictures) Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S... Time: 40 minutes OK (42 slides, 0 cat pictures) Thank you! Magento Test Automation: My Journey @fschmengler -
2011 > 2012 > 2013 > 2014 > 2015 >
2016 > 2017 ............SSSFF...S.....S.FF........S... Time: 40 minutes OK (42 slides, 0 cat pictures) Thank you! Questions? Magento Test Automation: My Journey @fschmengler -
Announcement: MageTestFest NOVEMBER 17TH 2017 MAGENTO. SOFTWARE TESTING. PARTY. https://magetestfest.nl/
by Yireo. Magento Test Automation: My Journey @fschmengler -
Links http://xtest-mage.com/ https://magiumlib.com/ http://codeception.com/ Image sources Marvin: The Hitchhiker's Guide
to the Galaxy (2005 movie) Angry German: https://www.youtube.com/watch?v=-_xUIDRxdmc Black metal cat: https://twitter.com/evilbmcats Agile testing pyramid: http://www.agilecoachjournal.com Magento Test Automation: My Journey @fschmengler