Measure the whole SERP at any scale #brightonSEO

Slides from my brightonSEO talk on November 11, 2023 in San Diego, CA.

The era of traditional rank tracking is over.

In this era of SEO, we need to understand the value that can be derived from, instead, monitoring and measuring the entire SERP.

The shape of the modern SERP is evolving quickly.

It is critical to understand both the ""the how and the why"" of this data, which can be leveraged not only by SEOs but also by Paid, Content, and Product teams.

There are numerous challenges involved in collecting accurate data at scale and even more challenges in turning this data into signals that are relevant for business teams.

Scale is not only about technical capture but also ensuring maximum impact and we discuss how to ensure that both criteria are met.

We also introduce some key KPIs and metrics that can be used to measure the whole SERP, including visual ranking, pixel depth, pixel height, fold visibility, and more.

Transcript

1. Measuring the whole SERP at any scale Ray Grieselhuber DemandSphere Speakerdeck.com/raygrieselhuber @raygrieselhuber
2. The era of traditional rank tracking is over Speakerdeck.com/raygrieselhuber @raygrieselhuber Up Down 23 12
3. But the shape of the SERP is changing faster than ever Speakerdeck.com/raygrieselhuber @raygrieselhuber
4. Impacts: • Fold visibility • CTR • Clicks / Sessions • Conversions • Revenue 600+ distinct elements across the SERPs Paid Content • Pixel depth: 120px • Pixel height: 320px • Visual position: 1 • Nested position: 2 Organic Result • Pixel depth: 440px • Pixel height: 330px • Visual position: 3 • Title changed
5. Daily layout changes on Google SERPs Paid Content • Position (order of items) • Product Name • Price • URL • Pixel Data Organic Results • Organic Position • Visual Position • Title • Meta Description • Product URL • Pixel Data Popular Products • Position (order of items) • Product Name • Price • URL • Pixel Data
7. Speakerdeck.com/raygrieselhuber @raygrieselhuber Scale driver #1: AI + content marketing is driving explosive growth in content on the web
8. Speakerdeck.com/raygrieselhuber @raygrieselhuber Scale driver #2: The cost of money is affecting investment in ads Money supply SPIKED but now is decreasing Money is not as cheap anymore
9. Speakerdeck.com/raygrieselhuber @raygrieselhuber Typical clients: 75K-300K+ SERPs Larger clients: 1M+ SERPs Let’s define scale (volume) for our purposes
10. What is the right volume to monitor? Speakerdeck.com/raygrieselhuber @raygrieselhuber 1%-10% of your indexed pages is a good place to start
11. The bigger question is: Speakerdeck.com/raygrieselhuber @raygrieselhuber Who uses this data and for what?
12. Old answer: Speakerdeck.com/raygrieselhuber @raygrieselhuber SEOs
13. Better answer: Speakerdeck.com/raygrieselhuber @raygrieselhuber Good SEO is good product management, and vice-versa.
14. Even better answer: exploding use cases Speakerdeck.com/raygrieselhuber @raygrieselhuber Complexity increases Developer • APIs • Data lakes SEO Analyst • SERP data • Crawl data • GSC • GA • Exploration Content Team • Content & topics • Content performance • Visibility trends Product • Technical • SERP rankings • Indexation • PAA + Knowledge Graph Executive / Reporting • Permissions • Dashboards • Speed of answers • Forecasting • Predictive Performance Team • Ad data • Hotel Ads • Shopping • Google for Jobs • etc.
15. Speakerdeck.com/raygrieselhuber @raygrieselhuber How to meet the needs of all of these groups?
16. Speakerdeck.com/raygrieselhuber @raygrieselhuber Step 1: understand the data pipeline
17. Typical data pipelines Speakerdeck.com/raygrieselhuber @raygrieselhuber Fetching (HTML) ETL (JSON & DB) Analytics (DB) Dashboards, etc. SERPs (HTML) GSC GA4 Build? Buy? Build? Buy? Keywords
18. Where do development bottlenecks occur? Speakerdeck.com/raygrieselhuber @raygrieselhuber Fetching (HTML) ETL (JSON & DB) Analytics (DB) Dashboards, etc. SERPs (HTML) GSC GA4 Build? Buy? Build? Buy? Keywords
19. Speakerdeck.com/raygrieselhuber @raygrieselhuber Step 2: examine parallels from other problem domains
20. Similar use cases Speakerdeck.com/raygrieselhuber @raygrieselhuber Data Operations Observability (log files + search index / database) Problem domains Asset / Technology Parallels to SEO • Log files: SERPs • Parsed log files: Parsed SERPS (JSON, etc.) • Index: ??? Data Science BI Datasets ML Models Dashboards & Visualizations • SERPs • GSC • GA4 • Keyword sets, etc. • Looker Studio • Superset • DOMO • etc.
21. Speakerdeck.com/raygrieselhuber @raygrieselhuber Step 3: use tools with a track record of acceleration
22. Tools that can help accelerate and support many use cases Speakerdeck.com/raygrieselhuber @raygrieselhuber Data Lakes BI / Dashboards / Data apps Cloud Data Warehouse • S3-compatible object storage • BigQuery • Snowflake • Redshift • Looker Studio • Superset • Jupyter & other notebooks • Python data apps (Dash, Streamlit)
23. SQL is a decent lingua franca for turning SERP data into BI Speakerdeck.com/raygrieselhuber @raygrieselhuber select search_domain, url, title, desc, position, people_also_asked_snippets from results.organic_results where fetch_date = “2023-11-08” limit 100000
24. We mapped every single element on the SERP and put it into a data warehouse Speakerdeck.com/raygrieselhuber @raygrieselhuber
25. As data science matures, we’re seeing many Python-based tools become available Speakerdeck.com/raygrieselhuber @raygrieselhuber • Pandas / Dask / Pola.rs • Data Science notebooks • Numpy • PyTorch • Scrapy • Advertools • Plotly • spaCy • NLTK
26. Simplified access to SERP data Speakerdeck.com/raygrieselhuber @raygrieselhuber Developers SEO Team Content Team Product Team Executive / Reporting Performance Team APIs Data Warehouse Data Lakes Data Warehouse Data Warehouse SEO & Content Tools BI
27. Let’s talk! Speakerdeck.com/raygrieselhuber @raygrieselhuber • Scan the QR code to interact with these slides on Slido • Follow us at x.com/demandsphere • Connect with me at linkedin.com/in/raygrieselhuber