Lauren Green - AI Bots on Your Site: What They See, Why It Matters, and Where to Draw the Line

1. TECH SEO CONNECT • LAUREN GREEN AI Bots: What They See Why It Matters Where to Draw the Line Lauren Green
2. ABOUT ME Lauren Green Technical AI SEO Manager at Microsoft Six years in SEO AISEO strategy at T-Mobile and Microsoft linkdin.com/in/lauren-ngreen
3. WHAT THE EVIDENCE SAYS Training data impacts citations 2.6 yrs 87% 8 mo Avg Page Age - ChatGPT Citations AI crawl volume involves training After page deletion, still cited Lauren Green
4. Lauren Green Ahrefs content freshness study - 2025
5. Lauren Green Cloudflare AI insights
6. Lauren Green Otterly.ai Dead Citation Experiment – July 2026
7. Discoverability comes with responsibility Lauren Green
8. AI Bots: What They See Lauren Green
9. Let’s talk developer docs https://developers.openai.com/api/docs/ https://support.claude.com/en/articles/12138966-release-notes Lauren Green
10. What They See OAI-Searchbot & GPTBot “We may use the results from just one crawl for both use cases to avoid duplicative crawling.” OpenAI Dev Docs Lauren Green
11. WHY IT MATTERS Crawling efficiency Lauren Green
12. What They See Offline Web “OpenAI’s systems may update some indexed content quickly, but there is no refresh SLA for a specific URL. A particular page may be refreshed on a different schedule depending on the site, crawl access, caching, popularity, and other technical factors.” OpenAI Dev Docs Lauren Green
13. What They See Google Bot “URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our systems.” Google Crawling Docs Lauren Green
14. WHY IT MATTERS Search may impact crawls & freshness Lauren Green
15. What They See ClaudeBot: Bot Knowledge History “Site's future materials should be excluded from our AI model training datasets.” Claude Support Docs Lauren Green
16. Lauren Green Otterly.ai Dead Citation Experiment – July 2026
17. What They See Claude’s Exception “Claude has an exception where you can submit a claim asking to block access to a URL that already appears in Claude outputs.” Claude Support Docs Lauren Green
18. WHY IT MATTERS A disallow may not erase the past Lauren Green
19. What They See OAI-SearchBot “When fetching robots.txt files, we may use a user-agent string with an additional robots.txt marker.” ChatGPT Dev Docs Lauren Green
20. WHY IT MATTERS Sitemaps must be fresh Lauren Green
21. Parametric memory vs RAG Lauren Green
22. WHAT THE MODEL ANSWERS FROM Retrieval (RAG) Fetches evidence at inference time Retrieves relevant information, reviews, responds Lauren Green
23. WHAT THE MODEL ANSWERS FROM Parametric memory Training-time knowledge Static; can be stale or hallucinated without verification. Lauren Green
24. Lauren Green
25. Lauren Green
26. WHAT YOU CAN CHANGE Your live page is the input you control Lauren Green
27. AI Bots: Where to Draw the Line Lauren Green
28. WHAT YOU WILL LEARN Four actions Official facts Refresh, redirect or retire. International bots & pages Review the customer experience. Community/UGC Choose access by purpose. Robots & Sitemaps Regular audits & updates Lauren Green
29. ACTION 1 Official facts Product launches, price changes, statistics Lauren Green
30. Refresh content with the latest information Live article page Published 2+ years ago? AI may use older training data No Yes Where is the traffic from? AI only REVIEW & UPDATE with current info Lauren Green Protect accuracy and brand trust Google / Bing (with or without AI) Yes Any organic traffic? (Google, Bing or AI) KEEP Under 2 years KEEP
31. Redirect if there is a related & relevant page Live article page Published 2+ years ago? Yes Any organic traffic? (Google, Bing or AI) Redirect when there’s a match No Yes Relevant live page to send to? Lauren Green REDIRECT (301) to closest relevant page
32. Retire when there is no useful replacement Live article page Published 2+ years ago? Yes Any organic traffic? (Google, Bing or AI) No Big cleanup, small traffic impact Relevant live page to send to? No Lauren Green RETIRE Remove post (404/410) No reader value left
33. AIO references old news article Search Date: Fall 2025 🔍 t mobile deals ✦ AI Overview T-Mobile Newsroom – January 2024 T-Mobile’s currently offering “TradeContacted In & Save”: get upblog to $800 off thetoiPhone the team sunset the article. Trade In & Save launches this fall 16 pickup when you trade in today.” • Offer applies to new lines on any current and new customer • Credits are applied over 24 monthly bills Lauren Green
34. Make the 404 page fun bugsink.com/clearly-made-up-url-by-chatgpt?utm_source=chatgpt.com Lauren Green
35. Assess, then refresh, redirect, or retire Live article page No Published 2+ years ago? WHY REFRESH, REDIRECT, OR RETIRE KEEP Under 2 years Google / Bing (with or without AI) Yes Yes Any organic traffic? (Google, Bing or AI) Where is the traffic from? AI only REVIEW & UPDATE with current info No Yes Protect accuracy and brand trust KEEP Redirect when there’s a match Big cleanup, small traffic impact REDIRECT (301) to closest relevant page No reader value left Relevant live page to send to? Lauren Green AI may use older training data No RETIRE Remove post (404/410)
36. ACTION 2 INTERNATIONAL LOVE International Bots & Pages Reduce unnecessary crawls Lauren Green (WITH CORRECT HREFLANG & ROBOTS DIRECTIVES)
37. INTERNATIONAL LOCALES Review whether AI bots should only crawl their locale Security Limit Server Load Hreflang & MLLM Apple.com/robots.txt Lauren Green
38. INTERNATIONAL LOCALES “Microsoft research has found that 46% of web content used to train large language models is English” Lauren Green English dominates AI Training (July 2025)
39. Tripadvisor Hreflang or MLLM Lauren Green ChatGPT citations over the past 3 months. Location = France. Tripadvisor.fr vs tripadvisor.com citation share Source: Ahrefs.
40. ACTION 3 Community Forums Limit UGC Lauren Green
41. AIO referenced Reddit Lauren Green Source: X
42. COMMUNITY Preserve human and search value People Useful stories, complaints and feedback. Search A route to original community context. Official facts Clear product information alongside it. Lauren Green
43. COMMUNITY • RECOMMENDATION Consider training restrictions. Decide by content and purpose. Lauren Green
44. COMMUNITY Lauren Green Proposed AI Training Directive (NEW)
45. ACTION 4 Robots.txt & Sitemaps Review error codes Lauren Green
46. ROBOTS.TXT & SITEMAPS Review if robots.txt & sitemaps point to 3xx & 4xx Lauren Green
47. ROBOTS.TXT & SITEMAPS – RECOMMENDATION Regular audits & dynamic sitemaps Lauren Green
48. VERIFY Our due diligence helps customers navigate AI answers. It protects them (and us) from misinformation. Discoverability comes with responsibility. Lauren Green
49. What to do when you get back to your desk Goal: LLMs get the accurate, most recent information to answer customers. Takeaway Do this week Make it a habit Why it matters 0 Developer docs Bookmark OpenAI & Claude docs Read release notes every Monday Bot rules change without notice 1 Official facts Refresh, redirect or retire dated pages Check last-modified dates Stop false answers about your brand 2 International bots & pages Check logs for malicious bots & update robots.txt Limit crawls of duplicate locales Right locale, fewer wasted crawls 3 Community Decide AI access by content & purpose Review UGC training access Train on facts, not opinions 4 Robots & XML sitemaps Review robots.txt token in OAI-Searchbot user agent strings Audit robots.txt & 3xx/4xx URLs Fresh sitemaps: only 200 status codes
50. VERIFY You set the rules. Verify that the bots follow them. Lauren Green
51. Thank you! linkdin.com/in/lauren-ngreen Connect with me!