CRO does not equal A/B Testing

Transcript

1. CRO ≠ A/B Testing
2. Eidra Commercial Presentation 2 We are a consultancy collective helping leaders create great change. The world is becoming even more complex for business leaders due to economic uncertainty, emerging technologies, geopolitical risk, climate change, new consumer behaviors and talent demands. We believe a new partner is needed to rise to the demands of today and tomorrow. Eidra is a consultancy collective, built upon the foundational idea that great change requires the combination of management consulting with best in breed expertise in creativity, engineering and data. We believe we are equipped to perform the task with the agility and speed needed to turn business challenges into opportunity.
3. Eidra Commercial Presentation 3 We can help you create great change. Framing the role of your business and brand to set the direction for meaningful change. Rethink positioning & purpose Finding relevant pockets of growth and acquire the capabilities to address them. Identify new growth opportunities Addressing challenges and seizing opportunities in the sustainability transition. Scale sustainability Organizing for innovation to build, scale and ship more relevant products and services. Accelerate innovation Designing the interface between your business and customers for more affinity. Optimize customer experience Optimizing market stance and boosting productivity with strategic models, enhanced sales, and cost efficiency. Define business strategy Elevating your digital presence by integrating new tools for enhanced intelligence and efficiency. Advance with emerging tech Designing your brand to better articulate who you are and where you’re going. Build your brand Optimizing how you organize and operate for better return on brand & marketing investments. Increase commerce efficiency Developing your cross-funnel communication and content and driving and measuring the results. Craft your creative communication
4. © Eidra Our expertise. Eidra Commercial Presentation 4 Strategy & Change Brand & Marketing Innovation & Experience Technology & Product Data & Commerce Define, design and build unforgettable brands and experiences that resonate with your business’ desires and unlock growth. Make clients grow by defining when, where and how to evolve their business through business or product innovation. Drive transformation, growth and competitive differentiation through technology. Grow your business through data, AI and insights, optimizing your sales and marketing channels and operations. Future-proof your business through strategy and organizational change.
5. © Eidra How we deliver new, more and better business results. Eidra Commercial Presentation 5 Venturing into uncharted business territories by launching new products, services, and business models. Redefining value propositions through technology and capturing growth potential, finding the core of tomorrow. • Products & Services • Technologies & Innovations • Brands & Business models • Markets & Eco-systems • Ventures & Acquisitions New Growing core operations by maximizing current customer value, optimizing offerings, and enhancing experiences. Leveraging technology and creativity to boost engagement, growth, and competitiveness of current business. • Customers & Segments • Loyalty & CLV • Assortment & Pricing • Experiences & Features • Revenues & Gross profit More Creating better business value by enhancing efficiency, profitability, and sustainability of operations. Optimizing processes and integrating innovative solutions to drive impactful and lasting change. • Tech-stack & Platforms • Operations & Processes • Data & Insights • Sustainability & Footprint • Profitability & Valuation Better EXECUTION DIRECTION Innovation & Experience Technology & Product Strategy& Change Brand & Marketing Data & Commerce Orchestration &Collaboration
6. © Eidra Baboee Eidra provides R&D and AI strategies to transform work processes, ensuring seamless information access across Royal BAM Group projects. Eidra’s Data & Insights Lab established a data pipeline and and AI classification engine, analyzing a million support emails to deliver crucial business insights to guide leadership forward with precision. The Eidra team created a specialized AI solution that facilitates visual searches within a database of 100,000 paintings. The Eidra AI Bloodline architecture enables the subscription-based furnishing furnishing service NORNORM to get accurate AI driven forecasts, enabling sales team to identify when companies are are planning to move and are open to exploring interior solutions.
7. © Eidra Growth Advisor 2025-05-26 7
8. © Eidra 2025-05-26 8 CRO DATA SEO
9. eidra.com CRO ≠ A/B Testing
10. © Eidra WTF Simon? 2025-05-26 10
11. © Eidra Alright, what is CRO? 2025-05-26 11
12. © Eidra 2025-05-26 12 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app.
13. © Eidra 2025-05-26 13 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app.
14. © Eidra 2025-05-26 14 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app.
15. © Eidra 2025-05-26 15 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app.
16. © Eidra 2025-05-26 16 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app. A/B testing is one method within the CRO toolbox, where two or more versions of a webpage or element are tested against each other to see which performs better. However, CRO goes beyond just testing.
17. © Eidra 2025-05-26 17 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app. A/B testing is one method within the CRO toolbox, where two or more versions of a webpage or element are tested against each other to see which performs better. However, CRO goes beyond just testing.
18. © Eidra 2025-05-26 18 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app. A/B testing is one method within the CRO toolbox, where two or more versions of a webpage or element are tested against each other to see which performs better. However, CRO goes beyond just testing.
19. © Eidra 2025-05-26 19 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app. A/B testing is one method within the CRO toolbox, where two or more versions of a webpage or element are tested against each other to see which performs better. However, CRO goes beyond just testing.
20. © Eidra 2025-05-26 20 Great question! Conversion Rate Optimization (CRO) is often confused with A/B testing, but they’re not the same thing. CRO is a broad strategy focused on improving the rate at which website visitors complete a desired action, like making a purchase, signing up for a newsletter, or downloading an app. A/B testing is one method within the CRO toolbox, where two or more versions of a webpage or element are tested against each other to see which performs better. However, CRO goes beyond just testing.
21. eidra.com What is in that CRO Toolbox?
22. © Eidra 2025-05-26 22 Hierarchy of Evidence
23. © Eidra 2025-05-26 23 Hierarchy of Evidence Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
24. © Eidra 2025-05-26 24 Hierarchy of Evidence Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
25. © Eidra 2025-05-26 25 Hierarchy of Evidence Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
26. © Eidra 2025-05-26 26 Hierarchy of Evidence Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
27. © Eidra 2025-05-26 27 Hierarchy of Evidence Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
28. © Eidra 2025-05-26 28 Hierarchy of Evidence Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website. Risk of Bias Quality of Evidence Lower Strong Weak Higher
29. © Eidra 2025-05-26 29 Hierarchy of Evidence Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website. Not evidence Anecdotes, gut feeling, forums, competitors, Linkedin Lunatics, A/B test results websites, Awards, etc. Risk of Bias Quality of Evidence Lower Strong Weak Higher
30. © Eidra 2025-05-26 30 The progress you want to make Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website. Not evidence Anecdotes, gut feeling, forums, competitors, Linkedin Lunatics, A/B test results websites, Awards, etc. Risk of Bias Quality of Evidence Lower Strong Weak Higher
31. © Eidra The maturity levels for each driver tend to vary and can enable or hinder your growth. The weaker areas act as bottlenecks of your optimization efforts. 2025-05-26 31 Tools Process Organization Culture Focus
32. © Eidra 2025-05-26 32 From MacGyver to Booking.com/Spotify/etc. Level 1 Basic Level 2 Beginner Level 3 Intermediate Level 4 Expert Level 5 Master Focus Specific Person Team Level Digital Management Level Management Level Entire Company Culture Gut feeling & Loose ideas Random observations from web analytics data Data driven when possible Continuous data-driven decisions & Prioritizations Company-wide data-driven experimentation in your DNA Resources Local Hero Responsibilities in Team Part-time CRO Specialist Full-time CRO Specialist CRO Team Knowledge Online Marketing Basics Simple A/B Testing & Basic Analysis Deeper CRO with regular A/B Testing & statistics Excellence in CRO with advanced A/B Testing & Advanced analytics Excellence in CRO, A/B testing & analytics with full business perspective Process None Ad-Hoc Prioritized test hypothesis test execution plan Documentation of results & learnings Aligned processes used by team and long term CRO goals & strategy Site reliability engineering integrated with company processes Tools Barely any tools Basic tools & Begging IT for help Tools required for now & Workarounds to get IT help Complete set of quantitative & qualitative tools & IT friends Lean, mean optimization machine with direct IT access
33. © Eidra 2025-05-26 33 Hierarchy of Evidence – Level 1 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
34. © Eidra Highest Paid Person’s Opinion The CEO, CMO, CxO… HiPPO 2025-05-26 34
35. © Eidra Zero Evidence But Really Arrogant The Sales Director, Tech Lead, Marketing Manager ZEBRA 2025-05-26 35
36. © Eidra When was the last time you bought something on your website? Identify all the obvious issues you run into and fix them. Eat Your Own Dog Food. 2025-05-26 36
37. © Eidra Fix the obvious issues. 2025-05-26 37 • 404s • Internal 301s • Check if the main conversions actually work • Check your website: • In incognito • On mobile • With a 3G connection. • And witness the cookie banner, push notifications, newsletter popup, coupon code modal, and chat bubble. Did It Spark Joy? And why did the 3rd parties load without consent?
38. © Eidra 2025-05-26 38 Hierarchy of Evidence – Level 1 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trails Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
39. © Eidra 2025-05-26 39 Hierarchy of Evidence – Level 2 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
40. © Eidra Finally some SEO! 2025-05-26 40 • Keyword data • Google Trends • AlsoAsked • CrUX Dataset
41. © Eidra Site Speed Is it an SEO issue? Or more CRO? 2025-05-26 41
42. © Eidra Is it an SEO issue? Or more CRO? Site Speed 2025-05-26 42
43. © Eidra Is it an SEO issue? Or more CRO? Interaction to Next Paint (INP) Site Speed 2025-05-26 43
44. © Eidra Is it an SEO issue? Or more CRO? Interaction to Next Paint (INP) I don’t care. Let’s fix it. Site Speed 2025-05-26 44
45. © Eidra Are there now 4 pillars of SEO? 2025-05-26 45 1. Tech 2. Content 3. Links
46. © Eidra Are there now 4 pillars of SEO? 2025-05-26 46 1. Tech 2. Content 3. Links 4. Experience
47. © Eidra Are there now 4 pillars of SEO? 2025-05-26 47 1. Tech 2. Content 3. Links 4. Experience SEO Brand UX/UI CRO
48. © Eidra 2025-05-26 48 Hierarchy of Evidence – Level 2 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trails Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
49. © Eidra 2025-05-26 49 Hierarchy of Evidence – Level 3 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trails Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
50. © Eidra Quantitative & Qualitative data 50 • Google Analytics 4 • Hotjar or MS Clarity • Google Search Console and Bing Webmaster Tools • Text analysis on Customer Support data => has never been easier, thanks to AI. 2025-05-26
51. © Eidra 2025-05-26 51 From Learn. 01. Ask questions
52. © Eidra 2025-05-26 52 From Learn. To Earn. 01. Ask questions 02. Build solutions
53. © Eidra 2025-05-26 53 From Learn. To Earn. 01. Ask questions
54. © Eidra 2025-05-26 54 From Learn. To Earn. 01. Ask questions 02. Build solutions
55. © Eidra 2025-05-26 55 Hierarchy of Evidence – Level 3 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trails Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
56. © Eidra 2025-05-26 56 Hierarchy of Evidence – Level 4 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trails Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
57. © Eidra Stick to the process You are doing science now 2025-05-26 57 Hypothesis Design & Build QA Test live Analysis Learnings & Conclusions Research Thoughts Implementation
58. © Eidra 2025-05-26 58 Growth through Experimentation by increasing test velocity and quality of experiments Tools Process Organization Culture Focus
59. © Eidra 2025-05-26 59 Hierarchy of Evidence – Level 4 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
60. © Eidra 2025-05-26 60 Hierarchy of Evidence – Level 5 Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
61. 2025-05-26
62. 2025-05-26
63. © Eidra Relax. Step by step. 63
64. © Eidra 2025-05-26 64 Relax. Step by step. Tools
65. © Eidra 2025-05-26 65 Relax. Step by step. Tools Process
66. © Eidra 2025-05-26 66 Relax. Step by step. Tools Process Organization
67. © Eidra 2025-05-26 67 Relax. Step by step. Tools Process Organization Culture
68. © Eidra 2025-05-26 68 Relax. Step by step. Tools Process Organization Culture Focus
69. © Eidra 2025-05-26 69 Relax. Step by step. Tools Process Organization Culture Focus
70. © Eidra 2025-05-26 70 Relax. Step by step. Identify your level and start improving. Level 1 Basic Level 2 Beginner Level 3 Intermediate Level 4 Expert Level 5 Master Focus Specific Person Team Level Digital Management Level Management Level Entire Company Culture Gut feeling & Loose ideas Random observations from web analytics data Data driven when possible Continuous data-driven decisions & Prioritizations Company-wide data-driven experimentation in your DNA Resources Local Hero Responsibilities in Team Part-time CRO Specialist Full-time CRO Specialist CRO Team Knowledge Online Marketing Basics Simple A/B Testing & Basic Analysis Deeper CRO with regular A/B Testing & statistics Excellence in CRO with advanced A/B Testing & Advanced analytics Excellence in CRO, A/B testing & analytics with full business perspective Process None Ad-Hoc Prioritized test hypothesis test execution plan Documentation of results & learnings Aligned processes used by team and long term CRO goals & strategy Site reliability engineering integrated with company processes Tools Barely any tools Basic tools & Begging IT for help Tools required for now & Workarounds to get IT help Complete set of quantitative & qualitative tools & IT friends Lean, mean optimization machine with direct IT access
71. © Eidra 2025-05-26 71 Let’s build some pyramids! Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website.
72. © Eidra 2025-05-26 72 Let’s build some pyramids! Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trails Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website. Not evidence Anecdotes, gut feeling, forums, competitors, Linkedin Lunatics, A/B test results websites, Awards, etc. Risk of Bias Quality of Evidence Lower Strong Weak Higher
73. © Eidra 2025-05-26 73 Let’s build some pyramids! Meta RCT’s Meta-analysis of your experiments Randomized Controlled Trials Your own A/B tests or MVT experiments Quantitative & Qualitative data Web analytics analysis, on-site surveys, usability testing, heat- & scroll maps, 1-1 interviews, data mining at sales and/or customer service. Market Research Heuristic evaluations, external reports/sources, meta-research Expert opinion & case studies Asking just one person (not a customer), HiPPO or ZEBRA if (s)he likes the website. Not evidence Anecdotes, gut feeling, forums, competitors, Linkedin Lunatics, A/B test results websites, Awards, etc. Risk of Bias Quality of Evidence Lower Strong Weak Higher
74. Thank you! Simon Vreeman simon.vreeman@eidra.com