How Search Happens: Before & After you Click Enter @ SearchAfricon

Google & others don't make it easy but there's a lot of misconceptions about what happens when you make a search and how content get's in to Search Engines' Indexes. Dave goes through the process and what it means for us as SEOs

Transcript

1. How Search Happens: Before & After you Click Enter @davetheseo https://davetheseo.com /daves-decks/ https://www.linkedin. com/in/davecousin/
2. The Basics
3. A simplified view of how most Search Engines work: Crawl Index Ranking Algorithm Search Send Fetch Results Query
4. A simplified view of how most Search Engines work: Crawl Crawlers or Bots follow links to find new pages & content
5. A simplified view of how most Search Engines work: Send Fetch Index s & If indexable pages & content added to a database (index). Sea sco
6. A simplified view of how most Search Engines work: Results Query Search s. User searches: query expanded, context processed, ‘terms’ weighted.
7. A simplified view of how most Search Engines work: Send Fetch Results Query Ranking Algorithm Search matched to pages in index. Matches scored, 100s of other factors decide positions. Us pro
8. SEOs can (must) influence every stage Crawlability Indexability Rank/ Position Relevance • Page Discoverability • Internal Linking • Robots Blocking • Mobile Version • Redirects & Status Codes • Site Hierarchy /Structure • Navigation • Sitemaps • Cookie handling • Page Indexability • Robots Tags • Duplicate Handling • International Targeting • Content Indexability • JS / SPA reliant • Mobile Visibility • Blocked Resources • E-E-A-T Content • Content Length • Content Quality • Content Surfacing & backlinks • Experience & Expertise • Topical Relevancy • Quantity of content • Backlinks • Internal Linking • User Engagement • CTR • SERPs Features • Page Experience • Keyword Matching • Keyword Research • Opp. Analysis • Content Ideation • On Page Optimisation • Semantic Relevancy • Content Quality • Writing Style • Markup • Localisation • Market level KWR • Localised content • Targeting
9. SEOs can (must) influence every stage Crawlability • Page Discoverability • Internal Linking • Robots Blocking • Mobile Version • Redirects & Status Codes • Site Hierarchy /Structure • Navigation • Sitemaps • Cookie handling • P • • •
10. SEOs can (must) influence every stage Indexability • Page Indexability • Robots Tags • Duplicate Handling • International Targeting • Content Indexability • JS / SPA reliant • Mobile Visibility • Blocked Resources • E • T
11. SEOs can (must) influence every stage Rank/ Position • E-E-A-T Content • Content Length • Content Quality • Surfacing & Links • Experience+Expertise • Topical Relevancy • Quantity of content • Backlinks • Internal Linking • User Engagement • CTR • SERPs Features • Page Experience • K • • • •
12. SEOs can (must) influence every stage Relevance e • Keyword Matching • Keyword Research • Opp. Analysis • Content Ideation • On Page Optimisation • Semantic Relevancy • Content Quality • Writing Style • Localisation • Market level KWR • Localised content • Targeting
13. Before: Indexing & Crawling
14. What a Search Engine Sees What users see What search engines see
15. The Crawling Process 1. Start Crawler finds first page on a new site via external links, sitemap or submission 2. Download Crawler goes to first page and downloads the HTML file, finding links within it 3. Discover Crawler follows each link and continues to find new links on each page 4. Stop Crawler stops when all pages are found or crawl budget is used up 5. Render Indexer returns to pages to render pages including running supported scripts 6. Index Page content is added to the index (additional links found may be crawled)
16. The Crawling Process 1. Start Crawler finds first page on a new site via external links, sitemap or submission
17. The Crawling Process irst page on a new site nks, sitemap or 2. Download Crawler goes to first page and downloads the HTML file, finding links within it Crawler follows each lin continues to find new lin each page
18. The Crawling Process 3. Discover Crawler follows each link and continues to find new links on each page
19. The Crawling Process ws each link and ind new links on 4. Stop Crawler stops when all pages are found or crawl budget is used up Indexer returns to pages render pages including running supported scrip
20. The Crawling Process 5. Render Indexer returns to pages to render pages including running supported scripts
21. The Crawling Process to pages to ncluding ted scripts 6. Index Page content is added to the index (additional links found may be crawled)
22. Links Matter Without links, search engines might never find this page This page is 9 clicks/links from the home page
23. Javascript & CSS Checking Indexable Content • Mobile Friendly Tool • Google Search Console • Screaming Frog JS & CSS Issues • Too Slow to Load (timeout) • Blocked (by robots.txt) • Issue for Baidu & Yandex • Non-Static Links • Cookie Reliant • Forwarding based on IP • Web Apps
24. After: Information Retrieval
25. Weighted expanded Google’s Information Retrieval System 25 Index Match Score Ranking Algorithm Query Handling Search Candidate Docs Fetch Results Query 3 : 2 Scored Pairs Feedback Loop
26. Query Expansion •We used to keep our searches simple •Got good at using the right word or words
27. Query Expansion •Google started matching synonyms and we had to worry less
28. Query Expansion •Google Hummingbird Update switch the focus from exact keyword matching to intent
29. Query Expansion •Google started adding Machine Learning in 2015 •Now uses LLMs to add synonyms and contextually relevant terms
30. Query Expansion Contextual & Intentual Expansion •best gifts for curious kids •affordable STEM toys •gifts that encourage learning •inspiring young scientists •gifts to spark curiosity •birthday ideas for science lovers •recommended by teachers / parents •cool gifts for smart kids •popular 2024 STEM gifts Interest #2: Astronomy & STEM •telescope •stargazing kit •planetarium projector •space-themed gifts •constellation map •science toys •solar system model •NASA merchandise •space puzzles •astronomy books Interest #1: Reading •books •novels •science fiction •young adult literature •educational books •bookworm •fantasy novels •graphic novels •astronomy- themed fiction •reading gifts Synonyms & Related Phrases (Gift & Recipient) •present •gift idea •birthday present •unique gifts •teenager •teen •young adult •14-year-old girl / boy / nonbinary •middle schooler •age-appropriate •gifts to spark curiosity •birthday ideas for science lovers •recommended by teachers / parents •cool gifts for smart kids •popular 2024 STEM gifts what can I give as a gift to a 14 year old who loves reading and astronomy?
31. Term Weighting •Not all parts of a search term are equal Nike Running Shoes Nike Running Shoes Running Shoes
32. Term Weighting •Which words are important? •Which sequences of words are important? •Non Weighted: • Match MORE terms what can I give as a gift to a 14 year old who loves reading and astronomy?
33. Term Weighting •Which sequences of words are important? •Weighted: • Match IMPORTANT terms what can I give as a gift to a 14 year old who loves reading and astronomy?
34. Candidate Documents •All the pages that match in some way and need to be ranked
35. Candidate Documents •Now based on a database of c. 400 Billion
36. Candidate Documents •With Expanded Queries being matched too
37. Matched Scored Pairs •How well does the page match the query? 3 : 2 Page A. The best phone for photo editing Page B. Top smartphones for photography lovers 2025 "best phone for photos" 95 10
38. Matched Scored Pairs •How well does the page match the expanded queries? 3 : 2 Page A. The best phone for photo editing Page B. Top smartphones for photography lovers 2025 "best phone for photos" 40 72
39. Matched Scored Pairs Better match for upweighted terms? = Higher Score 3 : 2
40. Matched Scored Pairs Gives an initial ranking based solely on relevance 3 : 2
41. Ranking Algorithm All Other Ranking Factors get added on top of initial Score
42. After: Ranking
43. Key Content Ranking Factors Expertise & Experience Trustworthy Relevance User Experience Authority Does the writer show expertise and/or 1st person experience? Can the content be trusted? Is the page relevant to the user’s query? Do we trust this site to provide users with a good experience? Is this page an authoritative source among other relevant pages? Content content Content Quality Content Optimisation User Signals Backlinks Topical Authority Use of data, anecdotes, qualification to cover, other content by author. Is content accurate & well researched? Does it contain the searched keywords & match the users’ intent? Are users engaging with your content and finding it useful? Are 3rd party influential websites linking to your content? Are websites relevant to the topic of your site linking to your content? Copy Focus
44. Scoring •The Ranking Algorithm is a series of processes
45. Scoring •Further Scoring added based on main ranking factors
46. Scoring •Processes such as re- ranking, for context, location etc. shake up results
47. Weighting Factors by Context Experience Freshness Authority Detail Location Relevant Expertise Established Trust Clarity General
48. After: Adding in SERPs Features
49. Forming the final SERPs •Context Counts
50. Forming the final SERPs •Context Counts
51. Forming the final SERPs •Options weighted for query +Availability
52. Forming the final SERPs •Options weighted for query +Availability
53. After: Feedback Loop
54. Feedback Loop •Google measure Result Quality •CTR, Dwell Time, Next Action etc.
55. Feedback Loop •Data feeds back to elements including term- weighting & main ranking process
56. Thank You @davetheseo https://davetheseo.com /daves-decks/ https://www.linkedin. com/in/davecousin/