Managing Googlebot's Greed

Slides from my May 2025 talk at Hive MCR in Manchester. I spoke about the technical SEO foundations of Googlebot crawling, some theories on different levels of crawl queues, and tips on maximising crawl efficiency.

Transcript

1. Managing Googlebot’s Greed Optimise for Efficient Crawling and Indexing Barry Adams 15 May 2025
2. Barry Adams ➢ Active in SEO since 1998 ➢ Specialist in SEO for News Publishers ➢ Newsletter: SEOforGoogleNews.com ➢ Co-founder of the News & Editorial SEO Summit
3. Google’s Three Main Processes Crawling Indexing Serving
4. Technical SEO Crawling Indexing
5. Today’s Talk Crawling
6. Google’s Model Crawl Queue Crawling Processing Index Render Queue Rendering Serving Rendered HTML URL HTML
7. Three Layers of Index Storage Crawl Queue Crawling Processing Index Render Queue Rendering Index Index Serving
8. Three Layers of Index Storage 1. RAM storage; ➢Pages that need to be served quickly and frequently ➢Includes news articles, popular content, high- traffic URLs 2. SSD storage; ➢Pages that are regularly served in SERPs but aren’t super popular 3. HDD storage; ➢Pages that are rarely (if ever) served in SERPs
9. Three ‘layers’ of Googlebot? Crawl Queue Crawling Processing Index Render Queue Rendering Index Index Serving Crawling Crawling Crawl Queue Crawl Queue
10. Three Indices… Three Crawl Queues? Priority crawl queue Regular crawl queue Legacy crawl queue RAM storage SSD storage HDD storage
11. Priority Crawl Queue • Crawls VIPs ➢ Very Important Pages; Webpages that have a high change frequency and/or are seen as highly authoritative News website homepages & key section pages Highly volatile classified portals (jobs, properties) Large volume ecommerce (Amazon, eBay, Etsy) • Main purpose = discovery of valuable new content; ➢ i.e., news articles, new product pages, new classified listings • Rarely re-crawls newly discovered URLs; ➢ New URLs can become VIPs over time
12. Regular Crawl Queue • Google’s main crawling; ➢ Does most of the heavy lifting • Less frantic; ➢ More time for crawl selection, de-duplication, sanitisation of the crawl queue • Recrawls URLs first crawled by the Priority crawl queue; ➢ Checks for changes ➢ Updates relevant signals for next crawl prioritisation
13. Legacy Crawl Queue • Crawls VUPs; ➢ Very Unimportant Pages; URLs that have very little link value and/or are very rarely updated ➢ Recrawls URLs that serve 4XX errors; -Likely also occasionally checks old redirects
14. It’s probably more complicated Priority crawl queue Regular crawl queue Legacy crawl queue RAM storage SSD storage HDD storage
15. Crawl Sources • Site crawl • Feeds & XML sitemaps • Inbound links • DNS records • Domain registrations • Browsing data?
16. URLs are Sacred • Search Engines don’t crawl, index, rank pages or content… • They crawl, index, and rank URLs. • One piece of content = one URL
17. Crawlable Links https://developers.google.com/search/docs/crawling- indexing/links-crawlable
18. Crawl Management • Robots.txt Disallow; ➢ Strongest crawl management signal ➢ Evaporates crawl budget
19. https://developers.google.com/search/docs/crawling- indexing/large-site-managing-crawl-budget
20. Don't use robots.txt to temporarily reallocate crawl budget for other pages; use robots.txt to block pages or resources that you don't want Google to crawl at all. Google won't shift this newly available crawl budget to other pages unless Google is already hitting your site's serving limit. https://developers.google.com/search/docs/crawling- indexing/large-site-managing-crawl-budget
22. Robots.txt prevents crawling… … but not indexing! • Links to blocked URLs are still crawled • Their anchor texts carry relevancy for indexing
24. • Canonicals & noindex are NOT crawl management; ➢ Google needs to see meta tags before it can act on them ➢ That means Googlebot still crawls those URLs Crawl Management vs Index Management
25. What about ‘rel=nofollow’? https://developers.google.com/search/blog/2019/09/ evolving-nofollow-new-ways-to-identify
26. What about ‘rel=nofollow’? All the link attributes—sponsored, ugc, and nofollow—are treated as hints about which links to consider or exclude within Search. https://developers.google.com/search/blog/2019/09/ evolving-nofollow-new-ways-to-identify
27. Optimise Crawling • Server Response Time
28. Optimise Crawling • Server Response Time
29. GSC Crawl Stats
30. Page Resource Load
31. Page Resources
32. Multiple Hostnames
34. Crawl Budget Per Hostname • Every hostname has its own allocation of crawl budget; ➢ ‘www.website.com’ is independently crawled from ‘cdn.website.com’ • Offload page resources to a subdomain; ➢ Frees up crawl budget on your main domain
35. Googlebot & AdsBot
36. AdsBot does not obey ‘User-Agent: *’
37. Sitewide Changes
38. Sitewide Changes • Googlebot detects large scale changes on a website; ➢ Crawl rate will temporarily be increased ➢ Ensures the changes are rapidly reflected in the index • Enable your hosting to handle increased crawl rate; ➢ Temporarily increase server capacity after a sitewide change ➢ Sitewide changes include: -Redesign -Site migrations -Large number of new URLs
39. https://developers.google.com/search/docs/crawling- indexing/http-network-errors
40. Optimise Crawling; 3xx
41. Optimise Crawling; 3xx
42. GSC hack: Inspect URL follows redirects
43. Optimise Crawling; 4xx
44. Optimise Crawling; 5xx
45. Summarised
46. Manage Googlebot’s Greed • ALL web resources are crawled by Googlebot; ➢ Not just HTML pages ➢ Reduce amount of HTTP requests per page • Link equity (PageRank) impacts crawling; ➢ More link value = more crawling ➢ Elevate key pages to VIPS • Each hostname has its own crawl budget; ➢ Offload resources to subdomains • AdsBot can use up crawl requests; ➢ Double-check your Google Ads campaigns • Serve correct HTTP status codes; ➢ Googlebot will adapt accordingly
47. Thank You barry@polemicdigital.com https://www.linkedin.com/in/barryadams/ https://www.SEOforGoogleNews.com/