Jori Ford - Beyond Googlebot: Evidence-Based Retrieval Experiments with AI Crawlers

Transcript

1. JORI FORD CMO at FoodBoss Beyond Googlebot: Evidence-Based Retrieval Experiments with AI Crawlers December 4-5 2025 Durham, NC
2. Beyond Googlebot: Evidence-Based Retrieval Experiments with AI Crawlers Welcome to the Kingdom of Retrieval. Get Excited!
3. The Problem: It’s Not Sorcery, It’s Science The Experiments: Setting Things Ups Today’s Talk 01 02 03 04 Results & Learnings Now What? Evidence-Based Retrieval Experiments with AI Crawlers
4. Assume Nothing. Test Everything P.S. Logs still don’t lie
5. August Wilhelm von Hoffman Credit: Heinrich von Angeli “I will listen to any hypothesis but on one condition—that you show me a method by which it can be tested.”
6. The Experiments JoriCon vs. JoriFest Evidence-Based Retrieval Experiments with AI Crawlers
7. The First Test HTML + Embedded JSON-LD Integrated 01 Standalone JSON Endpoint Raw Endpoint 02 A page that links to the JSON Routing 03 Evidence-Based Retrieval Experiments with AI Crawlers Hypothesis: The bot will follow the link (route) to get the cleaner data.
8. Logs Video Slide
11. "Treat failure like a scientist. Each attempt is an experiment. Each mistake is a clue. You're not failing. You're refining." — James Clear
12. Test Test Name Methodology / Description Hypothesis Expected Outcome T-01 The Routing Test (Variant C) Provide a URL (/both-variant.html) that contains embedded JSON-LD and a <link> to a raw JSON endpoint. Observe logs. The bot will recognize the rel="alternate" link and fetch the JSON endpoint for higher fidelity. REJECTED. The bot extracts data from the HTML and ignores the link to save HTTP resources (Efficiency wins). T-02 The Accuracy Floor (Control) prompt the bot to extract specific facts (Price, Dates) from an unstructured text paragraph (/control.html). The bot will struggle with precision or formatting compared to structured variants. TIE (Simple): Accurate for basic facts. FAIL (Complex): Fails on reasoning/filtering tasks. T-03 The Accuracy Ceiling (Variant A) Prompt the bot to extract facts from the HTML page containing embedded JSON-LD (/page-variant.html). The bot will retrieve data with 100% accuracy and prefer this format for complex queries. CONFIRMED. High accuracy, low latency, 1 HTTP request. The "Gold Standard." T-04 Endpoint Utility (Variant B) Direct the bot specifically to the raw JSON file (/endpoint.json). The bot can parse raw JSON files as a valid data source if explicitly provided. CONFIRMED. Accurate, but requires the user/system to know the specific endpoint URL. Phase 1: The Efficiency & Format Experiments Goal: To determine the optimal method for serving data to RAG bots.
13. Test ID Test Name Methodology / Description Hypothesis Expected Outcome T-05 The Logic Test (Time) Ask a constraint-based question: "I am busy in July. Can I attend JoriCon?" (JoriCon is in July). Unstructured text (Control) will fail to reason. Structured Data (Variant A) will allow date comparison. CONFIRMED. Variant A correctly answers "No." Control Page answers "I don't know" or hallucinates. T-06 The Comparative Test Ask the bot to compare data points from two different domains: "Which is cheaper, JoriCon or JoriFest?" The bot will fetch both URLs, extract the price integer, and perform a mathematical comparison. CONFIRMED. Successful data fusion and comparison, provided both sources are discovered/provided. T-07 The Negation Test Ask the bot to filter data: "Tell me everything about the event EXCEPT the special guest." Structured data allows the bot to identify the performer field and programmatically exclude it. CONFIRMED. The bot lists all JSON-LD fields but omits the requested exclusion. Phase 2: The Reasoning & Logic Experiments Goal: To prove that Structured Data is for logic, not just reading
14. Test ID Test Name Methodology / Description Hypothesis Expected Outcome T-08 The Fog of War (Blindness) Ask a specific question about the entity without providing a URL: "How much are tickets to JoriCon?" Without a URL or search index entry, the bot will hallucinate or fail. CONFIRMED. Result is "I don't know" or hallucination. Proves "Retrieval Trigger" is required. T-09 Autonomous Discovery Provide the Homepage URL (index.html) which contains a link to the Variant page. Ask for the price. The bot will spider from the Homepage -> Link -> Variant Page to find the answer. CONFIRMED. Discovery works via Internal Linking (JoriFest test). T-10 Context Bias (Memory) Step 1: Ask bot to visit Control Page. Step 2: Ask bot to find "best price" on the domain (where Variant A exists). The bot will lazily reuse the data from Step 1 (Control) rather than crawling for a better source. CONFIRMED. "Session Stickiness" overrides site architecture. The bot cheats using short-term memory. Phase 3: Discovery & Behavior Experiments Goal: To understand how bots find (or fail to find) content in the "Stone World."
15. Test ID Test Name Methodology / Description Hypothesis Expected Outcome T-11 The Dungeon Run (Depth) Create a chain of 4 URLs (A->B->C->D). Place the fact on Page D. Ask bot to find it starting at A. The bot will abandon the crawl chain after a specific number of hops due to latency/cost limits. CONFIRMED. Appetite Limit is ~3 Hops. Content deeper than 3 clicks is invisible. T-12 The Battery Test (Throttle) Run the same complex retrieval prompt 5 times in rapid succession within the same session. The system will throttle compute resources, leading to increased latency or hallucination. CONFIRMED. Response time spikes and accuracy degrades after repeated heavy queries ("Session Throttle"). Phase 4: The Crawl Appetite (Stress) Experiments Goal: To measure the physical limits (Stamina) of the RAG bot.
16. Results & Learnings
17. Results Highlights: 5 Key Findings 1. Efficiency Beats Format: RAG bots prioritize HTTP efficiency over data purity; if the answer is in the container (HTML), they won't open the package (JSON)*. 2. "Retrieval Appetite" limit: content buried deeper than 3 clicks from the entry point is effectively invisible to a live RAG agent. - It’s too lazy! 3. Context Bias Overrides Architecture . Session Stickiness (Short-Term Memory) is a stronger signal to the AI than live site architecture. 4. Structure Enables Reasoning, Not Just Retrieval Only the JSON-LD variant allowed the bot to perform date math and negative filtering. 5. The "Session Throttle" Exists Repeatedly executing complex retrieval prompts in a single session caused a measurable spike in latency. This indicates that RAG systems have a compute-based throttle similar to a crawl delay, triggered by high inference costs rather than just server load.
18. The Strategy: 5 Key Learnings 1. Optimize for "Retrieval Appetite," Not Just Crawl Budget flattening your site architecture is no longer just a best practice—it is a survival requirement for AI visibility. Critical data must be surface-level. 2. Schema is the API for AI Logic Technical SEOs often view Schema as a tool for Rich Snippets (visuals). To win on complex, comparative, or "assistant-style" queries you must provide structured data that supports mathematical and logical operations. 3. Embed, Don't Link The failure of the Routing Test (T-01) teaches us to reduce friction. Do not rely on the bot to follow a breadcrumb trail to a better data source. Embed the JSON-LD directly in the HTML 4. "Clean Beaker" Testing is Mandatory The Context Bias finding (T-10) invalidates any RAG test performed in a "dirty" session. Technical SEOs must adopt strict testing protocols: Always start a new chat/session for every crawl test. If you don't, you are testing the model's memory. 5. The "No Index" Danger Zone without a URL provided by the user OR a search index to pull from, the AI is blind.
19. 11 for 12 on the hypothesis…. 92% Celebrate the wins…but Learn from the Fails!
20. Micro-Tests Matter Changes happen everyday & AI is a new part of our jobs! Just one more item to juggle. Don’t rely on the test results of others when deciding what’s best for your business, your clients, etc. TEST EVERYTHING! IMPORTANT
21. Thank you! @chicagoseopro