Hidden Traps with Robots.txt @BrightonSEO 2024

You risk losing traffic, revenue, and potentially even information security issues with your robots.txt. You will learn why, and what to do to safeguard yourself from trouble.

1. Hidden Traps with Robots.txt and GSC Gianna Brachetti-Truskawa DEEPL SE Speakerdeck.com/giannabrachetti @gianna-brachetti-truskawa @tentaclequing
2. 4 What‘s the fastest way to get your domain dexindexed? BrightonSEO | @gianna-brachetti-truskawa
3. 5 Have your robots.txt respond with a 5xx! 5 BrightonSEO | @gianna-brachetti-truskawa
4. 6 Today, I want you to look beyond syntax … 6 BrightonSEO | @gianna-brachetti-truskawa
5. 7 Sometimes it‘s not what‘s in your robots.txt 7 BrightonSEO | @gianna-brachetti-truskawa
6. 8 … but what‘s outside of it 8 BrightonSEO | @gianna-brachetti-truskawa
7. 9 What you think it is: • A security guard • A law bots will have to obey • A tool to prevent testing environments from being accessible via the internet 9 BrightonSEO | @gianna-brachetti-truskawa
8. 10 What it really is: • An optional set of directives that crawlers can use to save on resources for crawl efficiency • A security risk 10 BrightonSEO | @gianna-brachetti-truskawa
10. 12 And it can cost you a lot of money
11. 13 1. What if you don‘t have a robots.txt? 2. What if you have one but it becomes unavailable? 3. Why can‘t you use it as a security guard? 4. What can you do to save yourself trouble? 13 BrightonSEO | @gianna-brachetti-truskawa What we‘re going to do today:
12. 14 Without a robots.txt, crawlers assume they can access all 14 BrightonSEO | @gianna-brachetti-truskawa
13. 15 Fine for small sites – for large sites, it helps to manage crawl budget more efficiently 15 BrightonSEO | @gianna-brachetti-truskawa
14. 16 But how would I reference my XML sitemap? 16 BrightonSEO | @gianna-brachetti-truskawa
15. 17 You‘re better off keeping it a secret! 17 BrightonSEO | @gianna-brachetti-truskawa
16. Vulnerability reduction 18 Why would you hide your XML sitemap? Crawl control 18 BrightonSEO | @gianna-brachetti-truskawa
17. 19 Why would you hide your XML sitemap? Control who accesses your XML sitemap Use case: • Competitive intelligence in eCommerce 19 BrightonSEO | @gianna-brachetti-truskawa
18. 20 Why would you hide your XML sitemap? Make it harder to use the sitemap as a vector for attacks Use cases: • Scraping • (soft) DDoS attacks 20 BrightonSEO | @gianna-brachetti-truskawa
19. 21 What if your robots.txt becomes unavailable?
20. 22 22 BrightonSEO | @gianna-brachetti-truskawa Let‘s go back to this shocker:
21. 23 The devil is in the details documentation 23 BrightonSEO | @gianna-brachetti-truskawa
22. Search Central Documentation: How Google interpretes the robots.txt specification² 24 Documentation Specified Robots.txt Web Standards and Google Documentation RFC 9309: Robots Exclusion Protocol (REP)1 Search Console Help: robots.txt report³ 1 https://www.rfc-editor.org/rfc/rfc9309.html 2 https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt 3 https://support.google.com/webmasters/answer/6062598?hl=en#crawl_errors Last access 30.09.2024
23. 25 25 BrightonSEO | @gianna-brachetti-truskawa Documentation Specified Robots.txt Web Standards and Google Documentation
24. 26 Google may cache your robots.txt for up to 24h 26 BrightonSEO | @gianna-brachetti-truskawa
25. 27 … or shorter if you limit caching 27 BrightonSEO | @gianna-brachetti-truskawa
26. 28 If you make changes, it can still take up to 24h until they‘re fetched 28 BrightonSEO | @gianna-brachetti-truskawa
27. 29 Robots.txt redirect chains can lead to it being ignored 29 BrightonSEO | @gianna-brachetti-truskawa
28. • For all 4xx but 429 • Treat as allow all 429 (Too many requests) will be treated as 5xx 30 Client Errors 4xx What happens if Google cannot fetch your robots.txt anymore? • Status codes 400 – 499 • MAY be treated as allow all • First 12h: Stops crawling domain • <= 30 days: use last cached version • > 30 days: Check if the site is available in general, treat as allow all Confidential REP Search Console Help Search Central Documentation 30 BrightonSEO | @gianna-brachetti-truskawa
29. If you have time- sensitive info, it might not be fetched on time 31 31 BrightonSEO | @gianna-brachetti-truskawa
30. 32 A 429 can lead to your domain being deindexed! 32 BrightonSEO | @gianna-brachetti-truskawa
31. 33 Server Errors 5xx What happens if Google cannot fetch your robots.txt anymore? • Treat as complete disallow • > 30 days: preferably use last cached version unless unavailable – else treat as 4xx (= allow all) • Treat 4xx and 5xx all the same: allow all REP Search Console Help Search Central Documentation 33 BrightonSEO | @gianna-brachetti-truskawa
32. 34 Server Errors 5xx Contradictions between known sources REP Search Console Help Search Central Documentation Gary Illyes 34 BrightonSEO | @gianna-brachetti-truskawa DISALLOW ALL ALLOW ALL DEINDEX ALL
33. 35 DNS errors, connection timeouts will be treated the same! 35 BrightonSEO | @gianna-brachetti-truskawa
34. 36 So which is it now, Google? 36 BrightonSEO | @gianna-brachetti-truskawa
35. 37 Why you can‘t use robots.txt as a security guard
36. 38 Error codes can lead to your robots.txt become a liability 38 BrightonSEO | @gianna-brachetti-truskawa
37. 39 … esp. if you 39 BrightonSEO | @gianna-brachetti-truskawa to hide secrets used it
38. 40 Robots.txt might tell us where the most interesting files might be 40 BrightonSEO | @gianna-brachetti-truskawa
39. You’re relying on your robot.txt to keep things out of the internet 41 How your robots.txt can become a liability (even if it‘s a 200!) You’re exposing vulnerabilities of your website or servers You’re not monitoring the content or uptime of your robots.txt 41 BrightonSEO | @gianna-brachetti-truskawa
40. 42 Exposing sensitive information can become an expensive GDPR issue 42 BrightonSEO | @gianna-brachetti-truskawa
41. 43 Robots.txt emerged as a practical solution to real-world problems in the early web 43 BrightonSEO | @gianna-brachetti-truskawa
42. 44 But … Robots.txt is not legally binding! 44 BrightonSEO | @gianna-brachetti-truskawa
43. 45 Google may ignore it in some cases 45 BrightonSEO | @gianna-brachetti-truskawa
44. 46 46 BrightonSEO | @gianna-brachetti-truskawa Example Click tracking via parameters in URLs in prominent places https://www.deepl.com/pro?cta=header-prices
45. 47 47 BrightonSEO | @gianna-brachetti-truskawa
46. 48 If in doubt, Google may choose indexation over restriction 48 BrightonSEO | @gianna-brachetti-truskawa
47. 49 Best practices to avoid risks
48. 50 Limit access with more reliable methods! 50 BrightonSEO | @gianna-brachetti-truskawa
49. 51 Want to remove leaked data from the web? 51 BrightonSEO | @gianna-brachetti-truskawa
50. 52 Remove leaked data from the web: 410 (gone) X-Robots-Tag: none 52 BrightonSEO | @gianna-brachetti-truskawa
51. 53 Want to protect data from leaking into the web? 53 BrightonSEO | @gianna-brachetti-truskawa
52. 54 Protect data from the web: HTTP Authentication X-Robots-Tag: none Avoid internal links 54 BrightonSEO | @gianna-brachetti-truskawa
53. 55 Want to make it hard to spy on you or train genAI with your content? 55 BrightonSEO | @gianna-brachetti-truskawa
54. 56 Rate limiting Path obfuscation Serve different versions 56 BrightonSEO | @gianna-brachetti-truskawa
55. 57 Example by @pandraus, 03.07.2024 57 BrightonSEO | @gianna-brachetti-truskawa
56. 58 Monitor content and uptime! 58 BrightonSEO | @gianna-brachetti-truskawa
57. 59 GSC shows you versions and status of your robots.txt 59 BrightonSEO | @gianna-brachetti-truskawa
58. 60 Uptime / Status Code Monitoring 60 BrightonSEO | @gianna-brachetti-truskawa
59. 61 Monitor Search Console error classes 61 BrightonSEO | @gianna-brachetti-truskawa
60. 62 Set filters for WNC- and the error classes most relevant for you 62 BrightonSEO | @gianna-brachetti-truskawa
61. 63 If you do want to test syntax, use a parser, eg. by Will Critchlow: www.realrobotstxt.com 63 BrightonSEO | @gianna-brachetti-truskawa
62. 64 Go multi-level to successfully control which content can be accessed by AI crawlers 64 BrightonSEO | @gianna-brachetti-truskawa
63. 65 65 • Multilingual Tech SEO Strategist of 15 years • Turned PM Driving Growth in Global Markets • Ask me about complex tech issues, B2B SaaS SEO, and… • … my paper for the IAB Workshop on AI Control. Current role: PM Search at DeepL 65 BrightonSEO | @gianna-brachetti-truskawa
64. Thank you! 66
65. 67 Read on here: https://www.robotstxt.org/faq/legal.html
66. 68 How your robots.txt can become a liability You’re exposing vulnerabilities of your website or servers • Outdated folders with sensitive data • Server vulnerabilities (Apache server status) • Admin login paths in your CMS • Internal services 68 BrightonSEO | @gianna-brachetti-truskawa
67. 69 How your robots.txt can become a liability You’re relying on your robot.txt to keep things out of the internet • Mitigate scraping attacks • Restrict access to personal data • Avoid duplicate content issues (why not target the root cause?) • Get things out of the index you accidentally leaked 69 BrightonSEO | @gianna-brachetti-truskawa
68. 70 How your robots.txt can become a liability You’re not monitoring the content or uptime of your robots.txt • Inhouse teams make changes without aligning with you • CMS plugins can change it without that being their purpose / disclosure in their release notes 70 BrightonSEO | @gianna-brachetti-truskawa