Software Quality Metrics

Transcript

1. ழ᜗ᐶࣖ ࡡᓃ߅Ҧ О͑೙ yho@bluetangstudio.com
2. Ң݊ም • ࡡᓃ߅Ҧ௴፬ɛ
3. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ
4. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ • Server Side Developer for 8 years.
5. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ • Server Side Developer for 8 years. • QA Engineer @ Glu Mobile
6. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ • Server Side Developer. • QA Engineer @ Glu Mobile • SDET @ Real Networks
7. Ң݊ም • ࡡᓃ߅Ҧ௴፬ࠋபɛ • Server Side Developer for 8 years. • QA Engineer @ Glu Mobile • SDET @ Real Networks • SDE @ Medio Systems.
8. TODAY’S TOPIC
9. Software Quality
10. Why Software Quality is Important?
11. When I am busy working on features
12. and finding customers
13. Because it is a feature
14. Because it is a strategic weapon
15. Trust me, I can build a data center for you
16. VMWare: We know our service.
17. software is all about trust.
18. Users need to trust the cloud won’t explode and bring down their business
19. slashdot effect - overnight success turns into a disaster.
20. How to build trust?
21. Trust is faith, Faith is a religion. But engineering is a science not religion.
23. Trust is Quality
24. What is Quality • Availability • Manageability • Performance • Reliability • Scalability • Security
25. Availability • of your software • of your IT / cloud infrastructure • of your network environment • of your external dependencies • of your client side network environment
26. Calculate Availability source: Economics of High Availability for Telecommunications Systems. An Intel® Primer
27. Network Availability • 2001/02/09 ʕߕऎ᝙ቊ௡୵ʵॎ • 2003/10/02 ʕߕऎ᝙ቊ௡୵ʵॎ • 2006/12/26 ܩ݆ήቤ • 2009/08/13 ୽זдჱࠬ • 2010/03/04 ৷ඪ̀͠ήቤ • 2011/11/14 ऎ᝙ආБၪࡌʬ඄
28. Availability of External Dependencies
30. SOA in reality
31. Availability • Service Level Agreement • Scheduled down time • Unscheduled down time
32. Manageability • Expose Information for System Admin to check the condition of the online system. • Configurability • Monitorability
33. Manageability • Monitoring is must, every cloud Infrastructure fails • Amazon • Azure • Google
37. Monitoring • Service credits are not automatics. • client has to file a ticket immediately. • claim downtime of certain time period. • proof / log of outrage and area.
38. Performance • Response Time. • Throughput • Network Bandwidth
39. Reliability • Availability v.s. Reliability • Availability: system uptime • Reliability: number of errors in a given time.
40. Reliability • Crash or become unresponsive occasionally. • Output is inconsistent.
41. Scalability • Slashdot Effect. • Scale Up and Scale Out • ERP system are much more sticky. • A mid-size website may have ~200 active sessions • But a ERP system used by a company of 200 employees would have 200 active sessions.
42. Security • Secure communication. • User Authorization and Authentication. • Data Management. • Usage/Operation logging.
43. Trust is tangible evidences.
44. • Internal Audit •DEV Test •QA Process • External Audit • ISO/27001 - security management standard. • yho@bluetangstudio.com :)
45. Software Metrics
46. How do we know Service A can handle X amount of traffic before we launch the Service?
47. Load Test
48. Load Tests • Performance Test • Stress Test • Longevity Test
49. Performance Test • find • best case performance. • baseline performance.
50. Performance Test • Key figures • request per minutes • number of concurrent requests • number of machines • overall throughput
51. Performance Test • Ꮠ͜೻ό࿁ᏀɢٙˀᏐ݊νО • ஈଣඎ throughput ݊щึᎇഹ request ᅰᄣ̋Ͼᄣ̋d݊щί൴ ཀݔࡈᔷұᓃࣛdึක֐ ˀᔷɨࠥ • ਂމ͊Ը൙П݊щცࠅᄣ̋ዚኜא٫݊؁ॴዚኜٙਿ๟࠽f • ཫПɪᇞ༈Դ͜εˇዚኜ ٙ࣬ኽf
52. Important Figures • minimum • maximum • mean • standard deviation • 75 percentile • 95 percentile • 98 percentile • 99 percentile • 99.9 percentile
53. A Picture Worth A Thousand Words
55. Stress Test • Test how system perform under extreme situation. • ie: iphone5 is on sales.
56. Longevity Test • evaluates a system's ability to handle a constant, moderate work load for a long time. • find memory leak • find hidden routine that exhausts system resource.
57. Monitoring System Under Test • OS Level: CPU, Memory, Disc I/O, and Network Usage • JVM: Memory Usage, Object counts, GC Cycles. • Application Level: request count, method call time, size of queue, cache hit/miss ratio
58. Load Test is the external view of the system.
59. we shall measure the code from developer’s point of view too.
60. what we thought != reality
61. Shopping Cart • SQL Query • Page Rendering • Session Replication
62. Shopping Cart • SQL Query • Page Rendering • Session Replication It is slow!
63. Shopping Cart • SQL Query • Page Rendering • Session Replication • Load test result: 300ms.
64. Shopping Cart • SQL Query • Page Rendering • Session Replication we shall cache sql results!
65. Shopping Cart • SQL Query • Page Rendering • Session Replication • 50 ms • 20 ms • 230 ms
66. We need to measure the code.
67. To make better decision.
68. •Obverse ᝈ࿀ •Orient ሜ዆ •Decide Ӕഄ •Act Бਗ
69. • Perf4j • Java Simon • Google Guava Stopwatch • Metrics by Coda Hale. TOOLS
70. Thank you • Any Questions?