Crawling and Indexing (and Ranking) in Google in 2024

Slides from my presentation at Pubcon 2024 where I look into the processes involved in crawling, indexing, and ranking pages in Google search.

Transcript

1. #Pubcon #Pubcon Crawling and Indexing (and Ranking) in Google Barry Adams March 2024
2. #Pubcon Barry Adams ➢ Doing SEO since 1998 ➢ Specialist in SEO for News Publishers ➢ Newsletter: SEOforGoogleNews.com ➢ Co-founder of the News & Editorial SEO Summit
3. #Pubcon I’ve Worked With…
4. #Pubcon #Pubcon How does Google work?
5. #Pubcon #Pubcon Google’s Three Main Processes Crawling Indexing Ranking
6. #Pubcon #Pubcon Technical SEO Crawling Indexing
7. #Pubcon
8. #Pubcon #Pubcon 1. Crawling Crawling
9. #Pubcon Three ‘layers’ of Googlebot?
10. #Pubcon Three ‘layers’ of Googlebot 1. Priority crawl queue 2. Regular crawl queue 3. Legacy content crawl queue
11. #Pubcon Priority Crawl Queue • Crawls VIPs ➢ Very Important Pages; Webpages that have a high change frequency and/or are seen as highly authoritative News website homepages & key section pages Highly volatile classified portals (jobs, properties) Large volume ecommerce (Amazon, eBay, Etsy) • Main purpose = discovery of valuable new content; ➢ i.e., news articles • Rarely re-crawls newly discovered URLs; ➢ New URLs can become VIPs over time
12. #Pubcon Regular Crawl Queue • Google’s main crawler; ➢ Does most of the hard work ➢ Less frantic; -More time for crawl selection, de-duplication, sanitisation of the queue
13. #Pubcon Legacy Content Crawl Queue • Crawls VUPs; ➢ Very Unimportant Pages; URLs that have very little link value and/or are very rarely updated ➢ Recrawls URLs that serve 4XX errors -Likely also occasionally checks old redirects
14. #Pubcon Crawlable Links https://developers.google.com/search/docs/crawling- indexing/links-crawlable
15. #Pubcon Crawl Management • Robots.txt Disallow; ➢ Strongest crawl management signal ➢ Evaporates crawl budget
16. #Pubcon https://developers.google.com/search/docs/crawling- indexing/large-site-managing-crawl-budget
17. #Pubcon Don't use robots.txt to temporarily reallocate crawl budget for other pages; use robots.txt to block pages or resources that you don't want Google to crawl at all. Google won't shift this newly available crawl budget to other pages unless Google is already hitting your site's serving limit. https://developers.google.com/search/docs/crawling- indexing/large-site-managing-crawl-budget
18. #Pubcon Robots.txt prevents crawling… … but not indexing! • Links to blocked URLs are still crawled • Their anchor texts carry relevancy for indexing
19. #Pubcon
20. #Pubcon • Canonicals & noindex are NOT crawl management; ➢ Google needs to see meta tags before it can act on them ➢ That means Googlebot still crawls those URLs Crawl Management vs Index Management
21. #Pubcon
22. #Pubcon Optimise Crawling • Server Response Time
23. #Pubcon Optimise Crawling • Server Response Time
24. #Pubcon GSC Crawl Stats
25. #Pubcon Page Resource Load
26. #Pubcon Page Resources
27. #Pubcon Googlebot & AdsBot
28. #Pubcon AdsBot does not obey ‘User-Agent: *’
29. #Pubcon Optimise Crawling • Serve correct HTTP status codes; ➢ 200 OK ➢ 3xx Redirects ➢ 4xx Errors -429 Too Many Requests ➢ 5xx Errors
30. #Pubcon https://developers.google.com/search/docs/crawling- indexing/http-network-errors
31. #Pubcon Optimise Crawling; 3xx
32. #Pubcon Optimise Crawling; 3xx
33. #Pubcon GSC hack: Inspect URL follows redirects
34. #Pubcon Optimise Crawling; 4xx
35. #Pubcon Optimise Crawling; 5xx
36. #Pubcon Optimise Crawling • ALL web resources are crawled by Googlebot; ➢ Not just HTML pages ➢ Reduce amount of HTTP requests per page • AdsBot can use up crawl requests; ➢ Double-check your Google Ads campaigns • Link equity (PageRank) impacts crawling; ➢ More link value = more crawling ➢ Elevate key pages to VIPS • Serve correct HTTP status codes; ➢Googlebot will adapt accordingly
37. #Pubcon #Pubcon 2. Indexing Indexing
38. #Pubcon Indexing • HTML lexer; ➢ Cleaning & tokenising the HTML • Index selection; ➢ De-duplication prior to indexing • Indexing; ➢ First-pass based on HTML ➢ Potential rendering (not guaranteed) • Index integrity; ➢ Canonicalisation & de-duplication
39. #Pubcon Two Stages* of Indexing Crawler Indexer Ranker 1 2 * There are MANY more; indexing is a collection of interconnected processes
40. #Pubcon Rendering
41. #Pubcon Evergreen Googlebot
42. #Pubcon Possible Rendering Issues in GSC
43. #Pubcon Rendering Issues • Inaccessible Resources; ➢ Make sure all page resources can be crawled
44. #Pubcon Rendering Issues • JavaScript inserts invalid HTML in the <head>; ➢ <body> tags in the <head> break Google’s processing of meta tags
45. #Pubcon https://developers.google.com/search/docs/ crawling-indexing/valid-page-metadata
46. #Pubcon Rendering Issues • HTML vs Render mismatch; ➢ Different content in raw HTML vs fully rendered page
47. #Pubcon https://chrome.google.com/webstore/detail/view-rendered- source/ejgngohbdedoabanmclafpkoogegdpob
48. #Pubcon Google Tools *ALWAYS* Render
49. #Pubcon Optimise Indexing • Don’t rely on Google’s rendering; ➢ Use Server Side Rendering & CDN caching • Reduce page resources; ➢ Fewer page resources = more efficient crawling faster load speed & CWV less chance of rendering issues • Improve internal linking; ➢ More PageRank = higher chance of indexing
50. #Pubcon What about the Index itself?
51. #Pubcon Three Crawlers… Three Indices? Priority crawler Regular crawler Legacy content crawler RAM storage SSD storage HDD storage
52. #Pubcon Three Layers of Index Storage 1. RAM storage; ➢ Pages that need to be served quickly and frequently Includes news articles but also popular content 2. SSD storage; ➢ Pages that are regularly served in SERPs but aren’t super popular 3. HDD storage; ➢ Pages that are rarely (if ever) served in SERPs
53. #Pubcon It’s probably more complicated Priority crawler Regular crawler Legacy content crawler RAM storage SSD storage HDD storage
54. #Pubcon #Pubcon 3. Ranking Ranking
55. #Pubcon Search Intent – first BERT, then MUM
56. #Pubcon
57. #Pubcon https://searchengineland.com/how-google-search- ranking-works-pandu-nayak-435395
58. #Pubcon Turns out… Slide from Trial Exhibit UPX0203
59. #Pubcon It’s clicks after all! Slide from Trial Exhibit UPX0228
60. #Pubcon Google is all about Keywords, Links, and Clicks 1. Keywords: ➢ Allows Google to understand what your content should rank for 2. Links: ➢ Gets you onto the 1st page of Google 3. Clicks: ➢ Determines whether you stay there (and rise) or drop off
61. #Pubcon Can You Cheat This? Yes. For a while.
62. #Pubcon Building Long-Term Value 1. Create content optimised for ranking; ➢ Use keywords in the right places 2. Make this content link-worthy; ➢ And keep making more of it 3. Have a website people like engaging with; ➢ Good UX and all that jazz
63. #Pubcon Building Long-Term Value SEO as usual
64. #Pubcon #Pubcon Thank You barry@polemicdigital.com https://www.linkedin.com/in/barryadams/ https://www.SEOforGoogleNews.com/