Causal Impact Analysis for SEO: From Theory to Practice

SMX Munich 2024 - Analytics and Data Literacy track

Transcript

1. Causal Impact Analysis for SEO: From theory to practice Giulia Panozzo
2. #SMXMunich Giulia Panozzo | @SequinsNsearch What is Causal Impact?
3. #SMXMunich Giulia Panozzo | @SequinsNsearch Powerful package to analyse data and infer the cumulative impact of a change in a time series
4. #SMXMunich Giulia Panozzo | @SequinsNsearch Based on BSTS (Bayesian Structural Time Series) statistical model
5. #SMXMunich Giulia Panozzo | @SequinsNsearch Uses past data to predict the outcome in the absence of the treatment (the counterfactual)
6. #SMXMunich Giulia Panozzo | @SequinsNsearch Defines the impact by measuring the deviation of the actual VS predicted outcome
7. #SMXMunich Giulia Panozzo | @SequinsNsearch
8. #SMXMunich Giulia Panozzo | @SequinsNsearch Example: Title change ‘Let’s add a price point to the title!’
9. #SMXMunich Giulia Panozzo | @SequinsNsearch Example: Title change ‘Let’s add a price point to the title!’
10. #SMXMunich Giulia Panozzo | @SequinsNsearch Change implemented
11. #SMXMunich Giulia Panozzo | @SequinsNsearch
12. #SMXMunich Giulia Panozzo | @SequinsNsearch Pre- implementation data Post- implementation data
13. #SMXMunich Giulia Panozzo | @SequinsNsearch
14. #SMXMunich Giulia Panozzo | @SequinsNsearch Actual Prediction Deviation
15. #SMXMunich Giulia Panozzo | @SequinsNsearch
16. #SMXMunich Giulia Panozzo | @SequinsNsearch
17. #SMXMunich Giulia Panozzo | @SequinsNsearch Because Testing is Hard!
18. #SMXMunich Giulia Panozzo | @SequinsNsearch
19. #SMXMunich Giulia Panozzo | @SequinsNsearch
20. #SMXMunich Giulia Panozzo | @SequinsNsearch
21. #SMXMunich Giulia Panozzo | @SequinsNsearch Example ‘Let’s add a price point to the title tag!’
22. #SMXMunich Giulia Panozzo | @SequinsNsearch ‘But will this help bring more traffic in?’
23. #SMXMunich Giulia Panozzo | @SequinsNsearch ‘Well…’
24. #SMXMunich Giulia Panozzo | @SequinsNsearch
25. #SMXMunich Giulia Panozzo | @SequinsNsearch However, with Causal Impact… X
26. #SMXMunich Giulia Panozzo | @SequinsNsearch Causal Impact gives you the confidence to leverage statistically significant results and drive changes at scale
27. #SMXMunich Giulia Panozzo | @SequinsNsearch ‘But will this help bring more traffic in?’
28. #SMXMunich Giulia Panozzo | @SequinsNsearch YES (Most likely) NO (Most likely)
29. #SMXMunich Giulia Panozzo | @SequinsNsearch You can use Causal Impact on a number of domains, not only on SEO tests!
30. #SMXMunich Giulia Panozzo | @SequinsNsearch Impact of feature changes on app installs
31. #SMXMunich Giulia Panozzo | @SequinsNsearch Impact of influencer campaigns
32. #SMXMunich Giulia Panozzo | @SequinsNsearch Impact of offline events and campaigns
33. #SMXMunich Giulia Panozzo | @SequinsNsearch
34. #SMXMunich Giulia Panozzo | @SequinsNsearch May 25th: Trailer release July 9th: Barbie premiere
35. #SMXMunich Giulia Panozzo | @SequinsNsearch And almost any time series data
36. #SMXMunich Giulia Panozzo | @SequinsNsearch If you run Causal Impact on R Studio, it’s free and open-source
37. #SMXMunich Giulia Panozzo | @SequinsNsearch
38. #SMXMunich Giulia Panozzo | @SequinsNsearch How can it help us in marketing?
39. #SMXMunich Giulia Panozzo | @SequinsNsearch Example of a clear winner from a title tag change Clicks: +58% CTR: +38% Position: -15%
40. #SMXMunich Giulia Panozzo | @SequinsNsearch It can validate proposed strategy changes in case of any doubts
41. #SMXMunich Giulia Panozzo | @SequinsNsearch
42. #SMXMunich Giulia Panozzo | @SequinsNsearch It’s great to clearly show stakeholders the impact of our team’s work
43. #SMXMunich Giulia Panozzo | @SequinsNsearch It can help forecast the direction of changes at scale and help make a case for more resources
44. #SMXMunich Giulia Panozzo | @SequinsNsearch How can it help us in marketing? Example of a clear loser from a title tag change
45. #SMXMunich Giulia Panozzo | @SequinsNsearch By clearly identifying a winner or loser, we can understand what works and doesn’t work for our audience
46. Demo time: How to run a Causal Impact Analysis
47. #SMXMunich Giulia Panozzo | @SequinsNsearch 1. Download R Studio Download R first https://cran.r-project.org/ Download RStudio https://www.rstudio.com/products/rstudio/download/
48. #SMXMunich Giulia Panozzo | @SequinsNsearch 1. Download R Studio
49. #SMXMunich Giulia Panozzo | @SequinsNsearch 1.1 Install Causal Impact
50. #SMXMunich Giulia Panozzo | @SequinsNsearch 2. Prepare the data
51. #SMXMunich Giulia Panozzo | @SequinsNsearch 2. Prepare the data
52. #SMXMunich Giulia Panozzo | @SequinsNsearch The first column is always your test group. Other columns can be used as control groups if they are a good fit
53. #SMXMunich Giulia Panozzo | @SequinsNsearch The pre-period should be at least twice as long as the post-period, to allow the model to plot the actual VS predicted outcome
54. #SMXMunich Giulia Panozzo | @SequinsNsearch Isolated 0 in data set Multiple 0s VS
55. #SMXMunich Giulia Panozzo | @SequinsNsearch 3. Run the script!
56. #SMXMunich Giulia Panozzo | @SequinsNsearch Choose file to import
57. #SMXMunich Giulia Panozzo | @SequinsNsearch Check preview
58. #SMXMunich Giulia Panozzo | @SequinsNsearch Set pre and post periods
59. #SMXMunich Giulia Panozzo | @SequinsNsearch It’s a winner!
60. #SMXMunich Giulia Panozzo | @SequinsNsearch
61. #SMXMunich Giulia Panozzo | @SequinsNsearch
62. #SMXMunich Giulia Panozzo | @SequinsNsearch Now give it a go! Access this script here
63. #SMXMunich Giulia Panozzo | @SequinsNsearch What I’ve learned from (several) trials and errors…
64. #SMXMunich Giulia Panozzo | @SequinsNsearch The date column should always be removed when using this script
65. #SMXMunich Giulia Panozzo | @SequinsNsearch Column titles can error out if they contain special characters, spaces, capitalised letters
66. #SMXMunich Giulia Panozzo | @SequinsNsearch Start small, then expand your datasets with additional controls and features once you’re comfortable with the script
67. Statistics VS The World
68. #SMXMunich Giulia Panozzo | @SequinsNsearch
69. #SMXMunich Giulia Panozzo | @SequinsNsearch
70. #SMXMunich Giulia Panozzo | @SequinsNsearch 1. External events can impact your data
71. #SMXMunich Giulia Panozzo | @SequinsNsearch Google algo updates
72. #SMXMunich Giulia Panozzo | @SequinsNsearch Tools tracking failures
73. #SMXMunich Giulia Panozzo | @SequinsNsearch Engineering releases
74. #SMXMunich Giulia Panozzo | @SequinsNsearch Create a document to map internal changes & external events Algo updates Other releases / Bugs 4th July, Black Friday, etc.
75. #SMXMunich Giulia Panozzo | @SequinsNsearch 2. Mind the Outliers!
76. #SMXMunich Giulia Panozzo | @SequinsNsearch
77. #SMXMunich Giulia Panozzo | @SequinsNsearch Outliers can originate from…
78. #SMXMunich Giulia Panozzo | @SequinsNsearch New product launches (within the test group)
79. #SMXMunich Giulia Panozzo | @SequinsNsearch Holidays and seasonal events
80. #SMXMunich Giulia Panozzo | @SequinsNsearch Results only from one page Page 1 Page 2 Page 3 Page 4 Page 5
81. #SMXMunich Giulia Panozzo | @SequinsNsearch Tracking bugs
82. #SMXMunich Giulia Panozzo | @SequinsNsearch You can spot outliers and rule them out by…
83. #SMXMunich Giulia Panozzo | @SequinsNsearch Increasing the size of the test group
84. #SMXMunich Giulia Panozzo | @SequinsNsearch Adding control groups
85. #SMXMunich Giulia Panozzo | @SequinsNsearch Pre-processing your raw data
86. #SMXMunich Giulia Panozzo | @SequinsNsearch
87. #SMXMunich Giulia Panozzo | @SequinsNsearch
88. #SMXMunich Giulia Panozzo | @SequinsNsearch
89. #SMXMunich Giulia Panozzo | @SequinsNsearch 3. Beware of confirmation bias
90. #SMXMunich Giulia Panozzo | @SequinsNsearch
91. #SMXMunich Giulia Panozzo | @SequinsNsearch
92. #SMXMunich Giulia Panozzo | @SequinsNsearch In that case, you can run the test a little longer, or repeat the test with bigger groups
93. #SMXMunich Giulia Panozzo | @SequinsNsearch If it’s still inconclusive or a loser, it’s probably best to revert the change and focus on other tests
94. #SMXMunich Giulia Panozzo | @SequinsNsearch References and useful resources • How to Measure the Impact of your SEO Changes with Causal Impact – Giulia Panozzo (step by step guide) • How we use causal impact analysis to validate campaign success - Part and Sum • Measuring No-ID Campaigns with Causal Impact - Remerge & Alicia Horsch • Causal Impact – Data Skeptic • R Studio on GitHub • The Comprehensive R Archive Network • Causal Impact for App Store Analysis - William Martin
95. Thank You! speakerdeck.com/giuliapanozzo @SequinsNSearch Questions? Find me here: Neuroscientive.com