Jori Ford - Chasing the Googlebot Trail

Transcript

1. Jori Ford CMO at FoodBoss Chasing the Googlebot Trail Building the Foundation: Using Crawl Maps & Crawl Analysis to Guide Your Optimization Strategy October 17 - 18 2024 CAM Raleigh 409 W. Martin Street Raleigh, NC 27603
2. Part 01 - Understanding the Crawl Stats Reports Part 02 - Mapping Googlebot’s Entry Points Today’s Agenda 01 02 03 04 Part 03 - Building Your Crawl Map Part 04 - Opportunities & Strategic Analysis Chasing the Googlebot Trail
3. Chasing the Googlebot Trail I'm Jori, a Search Geek & Resident CMO @ FoodBoss I’m a seasoned search veteran. Armed with log files, a touch of humor and a serious knack for data-driven strategies I’ve spent that last 15+ years driving results for brands like Kohl’s, OfficeMax, Verizon, Progressive, G2 and now FoodBoss. When I’m not working, I’m either building houses, sailing boats, submerged in anime or kickin’ with with my family (they are kinda a big deal) Did I forget to mention I’m a Geek? About Me
4. Part 01 Understanding the Crawl Stats Report Chasing the Googlebot Trail
5. REMINDER: Rendering is part of crawling Chasing the Googlebot Trail Crawling is the 1st stage of Google Search T E C H N I C A L S E O A D V I S O R Y VITAL CONTENT
6. Crawl Stats Report in Search Console • Total crawl requests: The number of requests made to your site • Total download size: The amount of data downloaded during crawling • Average response time: How quickly your server responds to crawl requests* • Host Status: whether Google encountered issues when crawling • Detailed breakdowns: Crawl requests by response, file type, purpose, and Googlebot type Chasing the Googlebot Trail
7. Total Crawl Requests What to look for: A high number of requests can indicate strong interest from Googlebot, but it may also suggest issues like duplicate content or infinite crawl spaces. Tip: You want this to remain stable unless you’ve increase the number of pages. Remember that duplicate URL reads count. Chasing the Googlebot Trail
8. Crawl Request: Analysis 1. Look at day of count (get an average using 30 days) 2. Use the discovery to refresh ratio (let’s assume 7:3) 3. Compare to indexation count 4. Compare to the XML Sitemap Submission Count Chasing the Googlebot Trail
9. Crawl Request Calculation Avg Daily Crawl 10,000 Discovery/Refresh 7:3 Indexation 65,000 XML Sitemap 145,000 Est.Time to Full Indexation: 49 days Chasing the Googlebot Trail
10. Total Download Size What to look for: Large download sizes can impact your server resources and crawl budget. Crawl efficiency management is a balance of page load vs response time (not page speed). Chasing the Googlebot Trail
11. Average Response Time What to look for: Longer response times can affect crawl frequency, impacting overall crawl budget and indexing. Average Response Time = Number of page requests Sum of all page response times Chasing the Googlebot Trail
12. Chasing the Googlebot Trail
13. Host Status What to look for: 5% or greater errors in last 90 days. Check your error table and fix related errors. Chasing the Googlebot Trail
14. Take note of the date, review errors for that day. Chasing the Googlebot Trail
15. Part 02 Mapping Googlebot’s Entry Points Chasing the Googlebot Trail
16. Cross Analyze: Top Crawled, Sitemaps, URL Inspector What You Will Need: • Discovery & Refresh URL Breakdown • Access Logs (Optional, If Available) • XML Sitemaps Report (Search Console API) • URL Inspection Report (Search Console API) • Backlinks Report (Ahrefs/Search Console) Chasing the Googlebot Trail
17. URL Discove red Refresh ed In XML Index ed Crawl Requests Referring URLs Back links Last Crawl Status https://yourdomain.com/sl ug Yes Yes Yes Yes 100 https://yourdomain.com/sitemap.xm l, https://someothersite.com/slug 8 2024-10-02 200 OK https://yourdomain.com/sl ug2 No Yes Yes Yes 20 https://yourdomain.com/sitemap1.x ml, https://yourdomain.com/slug5 3 2024-09-13 200 OK https://yourdomain.com/sl ug3 Yes Yes No Yes 45 https://thatdomainoverthere.com, http://yourdomain.com/slug12 1 2024-10-01 200 OK https://yourdomain.com/sl ug4 Yes No Yes Yes 10 0 2024-09-01 200 OK https://yourdomain.com/sl ug5 No No No No 0 http://anothersite.com/forum=123 2 2023-01-22 404 Not Found https://yourdomain.com/sl ug6 Yes Yes No Yes 5 https://somesitesomewhere.com?p =1 1 2024-07-05 200 OK https://yourdomain.com/sl ug7 No No No No 0 0 2022-08-02 Blocked by robots.txt Chasing the Googlebot Trail
18. Part 3: Building Your Crawl Map Making it simpler to see the past the forest & survey the trees Chasing the Googlebot Trail
19. Tools, Tasks or Technical Create your map based on your resources. Three ways to visualize map to make use of data: • Tools: Log Analyzers & Crawl Maps - Screaming Frog or JetOctopus • Tasks: Export Data & Group Templates & apply recognized patterns • Technical: Run a Data Visualization Script Conference title here
20. Screaming Frog Crawl Log Analyzer PROS: • Simple • Solid Visuals • Can Filter & Sort Every Way Imaginable • FREE & Paid Version CONS: • Visuals are limited • Malformed Logs can cause issues • Almost too much ability to analyze TOOLS (Recommended) Chasing the Googlebot Trail
21. JetOctopus Log Analyzer PROS: • Amazing Visuals • Grouped by Problem • Supports Multiple Data Sources • Direct Log Integration/Monitoring CONS: • 14 Day Trial (No FREE Version) • Works best with DevOps/Engineering Assistance • Learning Curve TOOLS (Recommended) Chasing the Googlebot Trail
22. Manual Downloads of Files (Daily/Weekly) TASK Chasing the Googlebot Trail
23. Manual Charting (Sorted by URL Depth) ../product-123 /Category A (25) Home (150) /Category B (10) ../product-456 ../partner-111 ../partner-222 Crawled but not indexed Crawled, Indexed Low Crawl, Indexed Problem w/ Crawl or Index Chasing the Googlebot Trail
24. Technical: Python & Pandas Conference title here Chasing the Googlebot Trail
25. Chasing the Googlebot Trail
26. Building Blocks for Strategy 1. Identify missed opportunities for XML sitemaps 2. Expose templates/patterns of health vs. harm 3. Understand influence of crawling on traffic 4. See powerful backlinks & whether backlinks even matter to your business 5. Centralize crawling nodes (create hubs) 6. See blockages, crawl budget waste, and infinite spaces And so much more… Chasing the Googlebot Trail
27. Generate Comparison Charts Chasing the Googlebot Trail New Insights Available 1 Chart - Tons of opportunities Start understanding how your updates are impacting performance outcomes & show it!
28. Example Insights 1. Spike in discovered URLs but flat traffic: This might indicate that while Googlebot is finding your new content, it’s not ranking well enough to drive traffic, suggesting potential keyword or optimization issues. 2. Increase in refreshed URLs and traffic: This could suggest that updating older content (perhaps through content refreshes, SEO tweaks, or adding new sections to existing pages) is working well to attract users and Googlebot. 3. High traffic with low indexing activity: If traffic is increasing but both discovered and refreshed URLs are low, it might indicate that fewer but high-quality pages are driving most of the user activity. Conference title here
29. Content Hubs & Duds Connect the Dots Blockages & Hubs Galore! Is your strategy working to drive crawlers where you want people to go? • Use crawl frequency to highlight URLs a specific color • Use Traffic to increase the depth of color • Use Red to indicate “noindex” or “discovered - not crawled” Conference title here
30. Chasing the Googlebot Trail
31. Q&A Got questions? I might have some answers… Chasing the Googlebot Trail
32. Thank you! Email me at jori.ford@gmail.com if you have more questions.