of G¨ oteborg, Gothenburg, Sweden and Quviq AB
[email protected] John Hughes Chalmers University, Gothenburg, Sweden and Quviq AB
[email protected] Joakim Johansson Ulf Wiger Ericsson AB, ¨ Alvsj¨ o, Sweden
[email protected] [email protected] Abstract We present a case study in which a novel testing tool, Quviq QuickCheck, is used to test an industrial implementation of the Megaco protocol. We considered positive and negative testing and we used our developed specification to test an old version in order to estimate how useful QuickCheck could potentially be when used early in development. The results of the case study indicate that, by using Quviq QuickCheck, we would have been able to detect faults early in the development. We detected faults that had not been detected by other testing techniques. We found unclarities in the specifications and potential faults when the software is used in a different setting. The results are considered promising enough to Ericsson that they are investing in an even larger case study, this time from the beginning of the development of a new product. Categories and Subject Descriptors D.2.5 [Software Engineer- ing]: Testing and Debugging—Testing tools; D.2.4 [Software En- gineering]: Software/Program Verification—Formal methods General Terms Verification Keywords Test Automation, Property Based Testing 1. Introduction it potentially reduce testing time? Would it find obscure bugs, and help to improve final product quality? Our study is small and qual- itative, but it suggests that the answer to all of these questions is a resounding “Yes”. The rest of the paper is structured as follows. In section 2 we give an introduction to Quviq QuickCheck, and in section 3 we explain our case study, giving some background on the software testing methods already used at Ericsson. In section 4 we explain our approach in detail, with samples of the testing code and a description of the extensions we made in parallel to QuickCheck, to better support this kind of testing. In 5 we present the results we obtained, and in 8 we draw conclusions. 2. Quviq QuickCheck Quviq QuickCheck is a property-based testing tool, developed from Claessen and Hughes’ earlier QuickCheck tool for Haskell [3] and a re-design for Erlang [2]. Apart from adaption to an Erlang setting, Quviq QuickCheck includes a number of extensions, of which the most significant is an ability to simplify failing test cases automatically. Quviq QuickCheck is a product of Quviq AB. A user of QuickCheck writes properties that are expected to hold, as Erlang source code making use of the QuickCheck API. For example, one property of the standard list reversal function is