Metrics that Matter

Whenever we speak about measuring web performance and user experience, we typically refer to static events in the browsing experience. Modern websites, however, are far from static, and user interactions with these websites have continuous aspects that cannot be represented by distinct events.

* How do you measure user frustration?
* How do you tell how fast your users expect your site to be?
* How do you tell if users were frustrated or annoyed before they got to your site or if your site caused that frustration?
* What are TTVR, TTI, TTFI, FID, and which of them is the most important?
* How do you tell if your page locks up during user interaction?

In this talk, we’ll look at some new user experience metrics. We’ll find out how to measure page responsiveness, smoothness, jank, and usability. We’ll learn about things like Rage Clicks, Missed Clicks, Dead Clicks, and Cursor Thrashing. We’ll also look at real user data that we’ve collected showing how these aspects of the page affect user behaviour.

Transcript

1. Metrics that Matter Measuring the effects of website performance on users
2. Philip Tellis Principal RUM Distiller @ Akamai Author of the OpenSource boomerang RUM library twitter:@bluesmoon ⦿ github:@bluesmoon speakerdeck:@bluesmoon
3. Abbé Jean-Antoine Nollet 1700 - 1770 French Clergyman & Budding Electrician Invented one of the first Electroscopes (we now call them beacon collectors) L'Abbé Jean Antoine Nollet — Maurice Quentin de La Tour Alte Pinakothek, Munich, Germany Public Domain
4. In 1746, he conducted the first ever RUM Experiment He shot an electric current through 200 monks, and checked how quickly they jumped; thereby measuring the latency of an electric signal with… Real Users!
5. Fortunately, our methods have gotten a lot less intrusive
6. What should we measure? What are the metrics that matter to users?
7. Loading a Webpage... https://addyosmani.com/blog/usability/
8. These moments are related and trigger different expectations with the user
9. Is it happening? What visual cues does the user have that this page might actually show up? That initial period of a page load when the screen goes blank causes anxiety & doubt for the user… • Did I click correctly? • Is it loading? • Has is already billed my credit card?
10. First Paint Framework Ready Hero Images DOM Content Loaded Visually Ready https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness
11. When does the first thing show up on screen? • Register a PerformanceObserver before the event happens • OR use performance.getEntriesByType("paint") after the event https://developer.mozilla.org/en-US/docs/Web/API/PerformancePaintTiming Currently only available in Chrome and Chrome based browsers
12. What about other browsers? Internet Explorer & Pre-Chrome Edge NavigationTiming on IE includes a proprietary msFirstPaint property. DOM Events We can look at when domInteractive fires, using NavigationTiming or by listening for readystatechange. Hero Images/Elements We could use a MutationObserver to determine when a particular image or element shows up in the DOM, or use ResourceTiming to check for it later. UserTiming Page or framework authors can use the UserTiming API to specify when important items are available.
13. Last Painted Hero max(h1, (biggest_img || bg_img)) • Developed by Steve Souders • Named by Paul Irish It’s a synthetic measure of when the last of the hero elements was drawn on screen. https://twitter.com/vazac/status/979147191896784897 https://speedcurve.com/blog/last-painted-hero/
14. Is it Useful? Is this the page I’m looking for? This stage… • Removes doubt • Confirms that the user is on the right path
15. Signals to detect usefulness • First Meaningful Paint – Lighthouse only at the moment • Hero Element timing is something that can be set by page developers using User Timing or Resource Timing. • Boomerang aggregates all of these into a measure called Time to Visually Ready or ttvr. https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html https://developers.google.com/web/fundamentals/performance/user-centric-performance-metrics
16. Is it Usable? If the user tried to interact with the page, would they have a good experience? The user is ready to be delighted at this point, they’ve come to the right place... • But is the page responsive? • Is there jank? • Are there random lockups? • Is the interface confusing?
17. Not Interactive Until Here User Sees Content Time to Interactive https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness
18. Signals to detect usability After Visually Ready (ie, user thinks they can use the page), check... • Frame Rate using requestAnimationFrame • Long Tasks using PerformanceObserver or use setInterval on older browsers • Measure latency of responding to any user input (click, keypress, scroll) First period of 500ms after VR with no Long Tasks, FPS > 20, and Page Busy < 10% https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html https://developers.google.com/web/updates/2018/05/first-input-delay
19. Short Tasks Long Tasks https://developers.google.com/web/fundamentals/performance/user-centric-performance-metrics
20. Long Task Duration https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness
21. LongTasks as % of Front End Load Time https://www.slideshare.net/nicjansma/reliably-measuring-responsiveness
22. Check Frame Rate using requestAnimationFrame var frames = 0; function measureFps() { frames++; // request a callback before the next frame window.requestAnimationFrame(measureFps); } // start measuring window.requestAnimationFrame(measureFps); // report on frame rate (FPS) once a second setInterval(function() { console.log("FPS: " + frames); frames = 0; }, 1000); https://www.slideshare.net/nicjansma/measuring-continuity
23. Other important continuity metrics • Measure Battery Level using navigator.getBattery(...) • Measure memory usage using performance.memory.usedJSHeapSize • Check visibilityState and use IntersectionObserver to stop expensive tasks when not visible. • Capture JavaScript errors
24. Is it delightful? When the user does interact, is that interaction pleasant? • Were interactions smooth? • Was the interface responsive? • Was the interface consistent?
25. Signals to detect Delight Developer Signals • Long Tasks come up again • As does Frame Rate and input latency • Measure these continuously while the page is active User Signals • Rage Clicks • Cursor Thrashing / Wild Mouse • Scrandom
26. • Ask the user • Bounce / Conversion Rate / LD 50 • Rage Clicks / Cursor Thrashing / Wild Mouse • Facial analysis • Wireless Brain Interface Methods of measuring Emotion (with JavaScript)
27. Rage Clicks occur when users rapid-fire click (or tap) on your site or app. Rage clicking is the digital equivalent of cursing to release frustration. https://www.fullstory.com/resources/guide-to-understanding-frustrating-user-experiences-online/ https://www.psychologytoday.com/us/blog/hide-and-seek/201205/hell-yes-the-7-best-reasons-swearing Rage Clicks
28. Measuring Rage Clicks Rage Clicks are measured in JavaScript by looking for more than 2 clicks within the same 10 pixels in a short period of time. onMouse*, event.timestamp, and an array https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
29. Rage Clicks depend on Latency of Page Usability • Interacting with a page before onload OR interactive is the most common cause of Rage Clicks • Some Rage Clicks happen after the page is usable, possibly due to JavaScript errors or CPU intensive tasks; they could also be false positives. • In over 30% of cases, a page is interactive after onload fires, and in 15% of cases, users try interacting between onload and interactive. Data collected and analysed with boomerang and Akamai mPulse
30. Data collected and analysed with boomerang and Akamai mPulse Rage Clicking is fairly consistent if first Interaction is before the page becomes Interactive Steady drop off if first Interaction is after the page becomes Interactive Rage Clicks are most likely if first Interaction is just before onload, possibly because DOM Ready event handlers are hogging CPU. (This is true even if the page becomes Interactive before onload)
31. Data collected and analysed with boomerang and Akamai mPulse Desktop vs Mobile
32. Rage Clicks as a Function of First Interaction & Visually REady Data collected and analysed with boomerang and Akamai mPulse
33. People who are angry are more likely to use the mouse in a jerky and sudden, but surprisingly slow fashion. People who feel frustrated, confused or sad are less precise in their mouse movements and move it at different speeds. https://www.telegraph.co.uk/technology/news/12050481/Websites-could-read-emotions-by-seeing-how-fast-you-move-your-mouse.html https://conversionxl.com/blog/user-frustration/ https://www.academia.edu/3085041/Patterns_of_cursor_movement_for_different_devices Cursor Thrashing/Wild Mouse
34. Measuring Cursor Thrashing & Wild Mouse Cursor Thrashing is measured in JavaScript by looking at mouse movements over time, both in absolute terms and as a fraction of screen size. onMouseMove, setTimeout, event.clientX, event.clientY, ... https://developer.akamai.com/tools/boomerang/docs/BOOMR.plugins.Continuity.html
35. We’re still working on correlating this with performance and business metrics
36. Optimizing user centric Metrics • Remove render blocking scripts from the HEAD • Inline critical CSS • Reduce work done in JavaScript • Move non-critical JavaScript to after onload • Split up long tasks • If you can, move expensive JavaScript over to Web Workers
37. Summary • Measure what affects your user’s emotional state. • Reduce the amount of work you do in onload and readyState handlers. • Measure Responsiveness and Smoothness of your UI. Pacific Islander Navigation Map, Museum of Fine Arts, Boston https://www.flickr.com/photos/bluesmoon/1266590108/
38. • Addy Osmani on Usability • Reliably Measuring Responsiveness in the Wild • Last Painted Hero • Boomerang Continuity Plugin • User Centric Performance Metrics • First Input Delay • Measuring Continuity • Understanding Frustrating User Experiences • The 7 Best Reasons for Swearing • Reading Emotions by Measuring Mouse Movements • Your users are Frustrated • Patterns of Cursor Movement for Different Devices • Tracking CPU with the Long Tasks API References • Paint Timing API • Navigation Timing • Resource Timing • User Timing • MutationObserver • PerformanceObserver • msFirstPaint • readystatechange • requestAnimationFrame • setInterval / requestIdleCallback • Lighthouse • Long Tasks
39. Thank You!