Darwin Santos - AI and the Rise of the Agentic Web

2. TECH SEO CONNECT · DURHAM · FRIDAY, NOVEMBER 6, 2026 AI and the Rise of the Agentic Web Technical SEO for a web that answers and acts Darwin Santos AI SEARCH PRODUCT GROWTH, HUBSPOT · NODEANDLINK.COM 01 / 70
3. 01 — WELCOME 02 / 70 Thank you. For building the room we kept asking for. Matthew Kay Mike King Tech SEO Connect JR Oakes SOURCES Tech SEO Connect ↗ Patrick Stox
4. 01 — WELCOME · ABOUT ME 03 / 70 Darwin Santos AI SEARCH PRODUCT GROWTH · HUBSPOT Named the Great Decoupling of Search ↗ May 2025, on X Wrote it up for Search Engine Land ↗ The Great Decoupling of Search and the agentic web Keeps the timelines at Node & Link ↗ Sourced events you can download Builds tools when something slows him down ↗ GSC Guardian: algorithm context in Search Console nodeandlink.com ↗ x.com/DarwinSantosNYC ↗ SOURCES @DarwinSantosNYC · naming the Great Decoupling (May 31, 2025) ↗ · Darwin Santos · The Great Decoupling of Search (Search Engine Land, Jul 2025) ↗ Node & Link · The Great Decoupling Timeline ↗ · Node & Link · GSC Guardian ↗ ·
5. 01 — WELCOME · GOOGLE I/O · MAY 2024 04 / 70 “When did we last have a technical SEO conference?” THE QUESTION GOOGLE I/O · MOUNTAIN VIEW SOURCES Google · AI Overviews U.S. rollout (May 14, 2024) ↗ BACK TO THE HOTEL
6. 01 — WELCOME 05 / 70 RALEIGH · OCT 2024 DURHAM · DEC 2025 DURHAM · NOV 2026 And now here we are. SOURCES Aleyda Solís · Tech SEO Connect, CAM Raleigh (Oct 17–18, 2024) ↗ · Tech SEO Connect 2025 · Durham (Dec 4–5, 2025) ↗ · Tech SEO Connect ↗
7. 01 — WELCOME Where were you when you first tried ChatGPT? Take a moment. 06 / 70
8. 01 — WELCOME · MY FIRST CHATGPT MOMENT 07 / 70 November 30, 2022 around 1:00 a.m. Kids asleep. I had already played with davinci. This felt different. It took me two weeks to take it in. DRAWN IN CODE FROM A FAMILY PHOTO SOURCES OpenAI · Introducing ChatGPT (Nov 30, 2022) ↗
9. 02 — HOW FAST? 02 How fast? Letʼs put the time in perspective. 08 / 70
10. 02 — HOW FAST? 09 / 70 How long each has existed YEARS, AS OF NOVEMBER 2026 · ↓ = TIME SINCE THE PREVIOUS LAUNCH The internet SOURCES Kleinrock · The first message transmission (ICANN) ↗ 57.0 yrs Oct 29, 1969
11. 02 — HOW FAST? 10 / 70 How long each has existed YEARS, AS OF NOVEMBER 2026 · ↓ = TIME SINCE THE PREVIOUS LAUNCH The internet 57.0 yrs ↓ 21.8 yrs The Web 35.3 yrs ↓ 2.4 yrs Yahoo SOURCES CERN · A short history of the Web ↗ Aug 6, 1991 32.8 yrs · History of Yahoo ↗ Jan 1994 Oct 29, 1969
12. 02 — HOW FAST? 11 / 70 How long each has existed YEARS, AS OF NOVEMBER 2026 · ↓ = TIME SINCE THE PREVIOUS LAUNCH The internet 57.0 yrs ↓ 21.8 yrs The Web 35.3 yrs ↓ 2.4 yrs Yahoo 32.8 yrs 28.2 yrs ↓ 10.7 yrs Bing Google · Our story ↗ Jan 1994 ↓ 4.7 yrs Google SOURCES Aug 6, 1991 17.4 yrs · Jun 2009 Microsoft · Bing launch (May 28, 2009) ↗ Sep 4, 1998 Oct 29, 1969
13. 02 — HOW FAST? 12 / 70 How long each has existed YEARS, AS OF NOVEMBER 2026 · ↓ = TIME SINCE THE PREVIOUS LAUNCH The internet 57.0 yrs ↓ 21.8 yrs The Web 35.3 yrs ↓ 2.4 yrs Yahoo 32.8 yrs 28.2 yrs ↓ 10.7 yrs Bing SOURCES Jan 1994 ↓ 4.7 yrs Google ChatGPT Aug 6, 1991 17.4 yrs ↓ 13.5 yrs 3.9 yrs Nov 30, 2022 OpenAI · Introducing ChatGPT (Nov 30, 2022) ↗ Jun 2009 Sep 4, 1998 Oct 29, 1969
14. 02 — HOW FAST? 13 / 70 How long each has existed YEARS, AS OF NOVEMBER 2026 · ↓ = TIME SINCE THE PREVIOUS LAUNCH The internet 57.0 yrs ↓ 21.8 yrs The Web 35.3 yrs 32.8 yrs 28.2 yrs AI Mode SOURCES Sep 4, 1998 ↓ 10.7 yrs Bing AI Overviews Jan 1994 ↓ 4.7 yrs Google SGE Aug 6, 1991 ↓ 2.4 yrs Yahoo ChatGPT Oct 29, 1969 17.4 yrs Jun 2009 ↓ 13.5 yrs 3.9 yrs Nov 30, 2022 ↓ 5 months 3.5 yrs May 10, 2023 ↓ 12 months 2.5 yrs May 14, 2024 ↓ 12 months 1.5 yrs May 20, 2025 Google · SGE in Search Labs (May 10, 2023) ↗ · Google · AI Overviews U.S. rollout (May 14, 2024) ↗ · Google · AI Mode (May 20, 2025) ↗
15. 02 — HOW FAST? 14 / 70 Decades → months. Four launches in thirty months. CHATGPT AI OVERVIEWS SGE 5 MO INTERNET AI MODE 12 MO YAHOO WEB 22 YRS SOURCES OpenAI · Introducing ChatGPT (Nov 30, 2022) ↗ · Google · AI Mode (May 20, 2025) ↗ BING GOOGLE 11 YRS · 12 MO Google · SGE in Search Labs (May 10, 2023) ↗ 13 YRS · Google · AI Overviews U.S. rollout (May 14, 2024) ↗
16. 02 — HOW FAST? · ANSWERS ARRIVE 15 / 70 Not that long ago. SEARCH ENGINE ROUNDTABLE MAY 11, 2023 The Google Search Generative Experience: all-new AI Google search engine seroundtable.com/google-search-generative-experience-ai-35360.html ↗ SOURCES SERoundtable · SGE (May 11, 2023) ↗
17. 02 — HOW FAST? · ANSWERS ARRIVE 16 / 70 Not that long ago. SEARCH ENGINE ROUNDTABLE MAY 11, 2023 The Google Search Generative Experience: all-new AI Goog… SEARCH ENGINE LAND MAR 2024 Google starts testing AI overviews from SGE in main Google search interface searchengineland.com/google-starts-testing-ai-overviews-from-sge-in-main-google-search-interface-438680 ↗ SOURCES Search Engine Land · AI overviews in main search (Mar 2024) ↗
18. 02 — HOW FAST? · ANSWERS ARRIVE 17 / 70 Not that long ago. SEARCH ENGINE ROUNDTABLE MAY 11, 2023 The Google Search Generative Experience: all-new AI Goog… SEARCH ENGINE LAND MAR 2024 Google starts testing AI overviews from SGE in main Goog… SEARCH ENGINE LAND MAY 2024 Google rolls out AI Overviews in US with more countries coming soon searchengineland.com/google-rolls-out-ai-overviews-in-us-with-more-countries-coming-soon-440418 ↗ SOURCES Search Engine Land · AI Overviews in the US (May 2024) ↗
19. 02 — HOW FAST? · ANSWERS ARRIVE 18 / 70 Not that long ago. SEARCH ENGINE ROUNDTABLE MAY 11, 2023 The Google Search Generative Experience: all-new AI Goog… SEARCH ENGINE LAND MAR 2024 Google starts testing AI overviews from SGE in main Goog… SEARCH ENGINE LAND MAY 2024 Google rolls out AI Overviews in US with more countries … SEARCH ENGINE LAND MAY 2025 Google launches AI Mode to all U.S. searchers with new features searchengineland.com/google-ai-mode-us-searchers-455654 ↗ SOURCES Search Engine Land · AI Mode for all US searchers (May 2025) ↗
20. 02 — HOW FAST? · SCALE 19 / 70 Then it scaled. MAY 2025 OCT 2025 MAY 2026 AI Mode launches for every U.S. countries and territories, 35 monthly AI Mode users, per searcher new languages Google at I/O U.S. SOURCES 200+ 1B+ Google · AI Mode in 200+ countries (Oct 7, 2025) ↗ · Google · Search at I/O 2026 (May 19, 2026) ↗
21. 02 — HOW FAST? It’s a lot to take in. Feeling behind? So is everyone here. The tools changing search can help us figure them out. 20 / 70
22. 03 — THE GREAT DECOUPLING 03 The Great Decoupling of Search Being found, and being visited. 21 / 70
23. 03 — THE GREAT DECOUPLING 22 / 70 The journey we built around. Search Find Visit Learn a question or a product a result a website about the offer Never the only path. But it gave us a way to think about discovery and to measure it. SOURCES Darwin Santos · The Great Decoupling of Search (Search Engine Land, Jul 2025) ↗
24. 03 — THE GREAT DECOUPLING · UNDER THE HOOD 23 / 70 Under the hood: how a page reaches the results. CRAWLING INDEXING SERVING Discover Crawl Render Index Serve Links and sitemaps surface the URL Googlebot downloads text, images, video Runs the JavaScript in a recent Chrome Analyzes and stores. Not every page. Ranks and builds the results page Googleʼs three stages: crawling (rendering happens here), indexing, serving. Every technical audit we run maps to one of these steps. SOURCES Google · In-depth guide to how Google Search works ↗
25. 03 — THE GREAT DECOUPLING Being found can separate from being visited. 24 / 70 THE ANSWER PERSON YOUR FACTS They read your facts somewhere else. YOUR SITE SOURCES Darwin Santos · The Great Decoupling of Search (Search Engine Land, Jul 2025) ↗
26. 03 — THE GREAT DECOUPLING · THE CHART 25 / 70 Impressions ↑ Clicks ↓ +0.43 → −0.35 Correlation of impressions and clicks, late 2024 → early 2025. Ahrefs. IF YOU USE GSC DATA AHREFS · ORIGINAL PUBLISHED EXAMPLE, JUNE 2025 · CLICKS BLUE, IMPRESSIONS PURPLE Impressions keep rising while clicks fall. One chart doesnʼt explain every journey. It tells you where to start digging. SOURCES Ahrefs · The Great Decoupling (Jun 17, 2025) ↗ · Google · Search Console data anomalies ↗ Google says impressions were not reported accurately May 13, 2025  Apr 27, 2026. Clicks were not affected.
27. 03 — THE GREAT DECOUPLING · THE DATA 26 / 70 What people did when an AI summary appeared. PEW RESEARCH CENTER · GOOGLE SEARCHES BY U.S. ADULTS · PUBLISHED JULY 22, 2025 Clicked a traditional result · no AI summary 15% Clicked a traditional result · with an AI summary 8% Clicked a link inside the AI summary 1% About half the clicks on traditional results when a summary is on the page. SOURCES Pew Research · AI summaries and clicks (Jul 22, 2025) ↗
28. 03 — THE GREAT DECOUPLING 27 / 70 It needed a name. SERoundtable · Jun 3 Charts were already going around. Ahrefs · Jun 17 On May 31, 2025, I called it the Great Decoupling. Googler on stage · Jun 18 The Great Decoupling SEO decks Newsletters Podcasts CHARTS ALREADY GOING AROUND Glossaries SOURCES @DarwinSantosNYC · naming the Great Decoupling (May 31, 2025) ↗ · SERoundtable · The Great Decoupling (Jun 3, 2025) ↗ Decoupling (Jun 17, 2025) ↗ · SERoundtable · A Googler explains the chart (Jun 18, 2025) ↗ · Ahrefs · The Great
29. 03 — THE GREAT DECOUPLING What did the person actually get? What did the person learn? Was the answer useful, current and in context? What happened next? Did they decide, visit, buy, or ask something else? SOURCES Darwin Santos · The Great Decoupling of Search (Search Engine Land, Jul 2025) ↗ 28 / 70
30. 03 — THE GREAT DECOUPLING · THE TIMELINE I keep the timeline. Sources, corrections, and the difference 29 / 70 NOV 2022 ChatGPT research preview FEB 2023 AI-powered Bing preview MAY 2023 SGE in Search Labs MAY 2024 AI Overviews, U.S. MAY 2025 AI Mode, U.S. · the decoupling gets a between an announcement and a rollout. nodeandlink.com/the-great-decoupling-timeline ↗ name SOURCES Node & Link · The Great Decoupling Timeline ↗ OCT 2025 AI Mode in 200+ countries MAY 2026 AI Mode passes 1B monthly users
31. 03 — THE GREAT DECOUPLING · THE DATA 30 / 70 Every event, sourced. Download the data. 28 events, Nov 2022  Jul 2025, each linked to its source DOWNLOAD DATA → EVENTS.CSV date · category · company · title · description · link Filter by company, event type and year. Put these dates on your own Search Console chart. nodeandlink.com/the-great-decoupling-timeline ↗ SOURCES Node & Link · The Great Decoupling Timeline ↗ · Node & Link · timeline data (events.csv) ↗
32. 04 — THE RISE OF THE AGENTIC WEB 04 The rise of the agentic web From answers to action. 31 / 70
33. 04 — THE RISE OF THE AGENTIC WEB 32 / 70 From answers to actions: the stack grows. Search THE WEB WE KNOW Crawl Index Rank Ten blue links Person visits Query Fan-out searches Retrieve passages Generate Cite links Person asks Fetch for the person Read and use tools Act Confirm AI answers AI OVERVIEWS · AI MODE Agents GOOGLE-AGENT · CHATGPT AGENT Same pages. Each layer adds a reader with a different job. SOURCES Google · In-depth guide to how Google Search works ↗ · Google · AI features and your website ↗ · Google · user-triggered fetchers ↗
34. 04 — ACTING 33 / 70 Agents learned to use the web. OCT 22, 2024 Anthropic: computer use in API beta JAN 23, 2025 OpenAI Operator JUL 17, 2025 ChatGPT agent: research + browser action PHASED ROLLOUT AUG 20, 2025 Brave: Comet prompt-injection proof of concept SECURITY REPORT OCT 21, 2025 OpenAI Atlas browser, agent-mode preview JUL 9, 2026 ChatGPT Work: longer, scheduled tasks STAGED ROLLOUT JUL 9, 2026 Atlas retirement announced RETIRED AUG 9 SEP 8, 2026 Meta Muse personal agent U.S. ROLLOUT SOURCES Anthropic · computer use, public beta (Oct 22, 2024) ↗ DEVELOPER BETA U.S. PRO PREVIEW · OpenAI · Introducing Operator (Jan 23, 2025) ↗ MACOS LAUNCH · Node & Link · Rise of the Agentic Web ↗
35. 04 — DEEP DIVE · ACTING 34 / 70 How OpenAI’s agent grew up in 18 months. JAN 23, 2025 JUL 17, 2025 JUL 9, 2026 Operator ChatGPT agent ChatGPT Work Its own browser. Works from screenshots: clicks, types, scrolls. Visual browser, text browser, terminal and connectors. Stays with complex projects for hours. Hands control back for logins, payments and CAPTCHAs. Asks permission before actions of consequence. Scheduled tasks: once, on a schedule, or on an event. U.S. Pro preview. Pro first, then Plus and Team. Plugins for Drive, Slack, email, calendars, CRMs. OSWORLD · REAL COMPUTER TASKS · SUCCESS RATE (OPENAI, JAN 2025) Previous best Operatorʼs model CUA 22.0% 38.1% Humans SOURCES OpenAI · Computer-Using Agent (Jan 23, 2025) ↗ · OpenAI · Introducing Operator (Jan 23, 2025) ↗ ↗ · OpenAI · ChatGPT for your most ambitious work (Jul 9, 2026) ↗ 72.4% · OpenAI · Introducing ChatGPT agent (Jul 17, 2025)
36. 04 — DEEP DIVE · ACTING 35 / 70 Muse keeps working after you close the app. Asks before it sends or buys. Keeps an audit trail. Pays with one-time cards through Stripe Link. ASKS Person Muse Sentinel The web sets a goal own cloud VM, own a second agent approves its your site browser traffic 10M+ SOURCES Meta · Introducing Muse ↗ Muse accounts in the U.S. and Canada by October 21, about six weeks after launch. Meta, via AP. · AP · Muse accounts top 10 million (Oct 21, 2026) ↗
37. 04 — CONNECTING 36 / 70 Then they got shared protocols. NOV 25, 2024 Anthropic open-sources MCP APR 9, 2025 Google: Agent2Agent A2A) draft SEP 8, 2025 Official MCP Registry DEC 9, 2025 Agentic AI Foundation: MCP, goose, AGENTS.md JAN 26, 2026 MCP Apps: interactive UI in the conversation MAR 12, 2026 A2A v1.0 STABLE SPEC JUN 9, 2026 WebMCP origin trial in Chrome ORIGIN TRIAL JUN 17, 2026 Agentic Resource Discovery ARD JUN 18, 2026 Enterprise-Managed Authorization for MCP JUL 28, 2026 MCP July revision: stateless core SOURCES Anthropic · Model Context Protocol (Nov 25, 2024) ↗ RELEASED DRAFT SPEC PREVIEW · Node & Link · Rise of the Agentic Web ↗ LINUX FOUNDATION OFFICIAL EXTENSION OPEN SPEC STABLE EXTENSION FINAL SPEC
38. 04 — DEEP DIVE · CONNECTING 37 / 70 MCP: one plug for tools and data. MCP SERVERS OFFER HOST Claude, ChatGPT, IDEs MCP client JSON-RPC 2.0 Tools functions the model can call Resources data and context Prompts reusable templates NOV 25, 2024 MAY 21, 2025 SEP 8, 2025 DEC 9, 2025 Anthropic open-sources MCP OpenAI joins the steering Official MCP Registry Linux Foundationʼs AAIF · committee (preview) 10,000+ servers SOURCES Anthropic · Model Context Protocol (Nov 25, 2024) ↗ · Linux Foundation · Agentic AI Foundation (Dec 9, 2025) ↗ · MCP specification (2024-11-05) ↗ · OpenAI · New tools in the Responses API (May 21, 2025) ↗
39. 04 — TRANSACTING 38 / 70 Then they started to transact. MAY 6, 2025 Coinbase x402: payments over HTTP 402 SEP 16, 2025 Google Agent Payments Protocol AP2 PUBLIC SPEC SEP 29, 2025 ChatGPT Instant Checkout + ACP LIMITED U.S. JAN 11, 2026 Universal Commerce Protocol UCP PUBLIC PROTOCOL MAR 24, 2026 OpenAI shifts to merchant-owned checkout STRATEGY CHANGE SEP 16, 2026 Coach & Kate Spade: live UCP checkout on Google SOURCES Node & Link · Rise of the Agentic Web ↗ PUBLIC PROTOCOL MERCHANT-REPORTED
40. 04 — DEEP DIVE · TRANSACTING 39 / 70 Four ways agents can pay, and one change of course. x402 AP2 ACP UCP COINBASE · MAY 2025 Pay per request over HTTP 402, in stablecoins. GOOGLE · SEP 2025 Signed mandates prove what the person authorized. OPENAI + STRIPE · SEP 2025 Agents place orders. The merchant stays the merchant of record. GOOGLE + SHOPIFY, ETSY, WAYFAIR, TARGET, WALMART · JAN 2026 Discovery, checkout and post-purchase in one standard. MAR 2026 · CHANGE OF COURSE SEP 2026 · LIVE OpenAI moves to merchant-owned checkout. ACP shifts to discovery. Coach and Kate Spade: “Buyˮ in AI Mode and Gemini. The shopper approves every order. SOURCES Coinbase · x402 (May 6, 2025) ↗ · · Universal Commerce Protocol (Jan 11, 2026) ↗ (Sep 16, 2026) ↗ Google Cloud · Agent Payments Protocol (Sep 16, 2025) ↗ · OpenAI · Buy it in ChatGPT (Sep 29, 2025) ↗ · Google · OpenAI · Powering product discovery in ChatGPT (Mar 24, 2026) ↗ · Tapestry · Coach and Kate Spade in Gemini
41. 04 — A PERSONAL INFLECTION POINT 40 / 70 My practical inflection point. NOV 24, 2025 LATE 2025 JAN 2026 Claude Opus 4.5 ↗ Clawd → Clawdbot ↗ Moltbot, then OpenClaw ↗ The tools started finishing work I Peter Steinbergerʼs WhatsApp Agents I could run for my own gave them. relay grows up. work. When agents got real for me. SOURCES Anthropic · Claude Opus 4.5 (Nov 24, 2025) ↗ · Peter Steinberger · Introducing OpenClaw ↗
42. 04 — PERSONAL AGENTS 41 / 70 Personal agents have names now. MAY 19, 2026 SEP 8, 2026 SEP 29, 2026 OCT 21, 2026 Gemini Spark Muse dots Claude Duet GOOGLE META OPENAI ANTHROPIC Four introductions in one year. Previews and availability differ. Judge each by what it actually does. SOURCES Google · Gemini Spark ↗ · Meta · Introducing Muse ↗ · OpenAI · Introducing dots ↗ · Anthropic · Claude Duet ↗
43. 04 — THE RISE OF THE AGENTIC WEB 42 / 70 The need remains. Some of the work moves. LOOK A person can ask another system to do part of the looking, the interpreting and the acting. PERSON AGENT INTERPRET ACT SOURCES Darwin Santos · The Great Decoupling of Search (Search Engine Land, Jul 2025) ↗ · OpenAI · Introducing Operator (Jan 23, 2025) ↗
44. 05 — FOR TECHNICAL SEOS 05 What does this change for technical SEOs? Three kinds of visitors. One job. 43 / 70
45. 05 — THREE KINDS OF VISITORS 44 / 70 For years, we optimized for one kind of visitor. THE WEB WE KNOW THE GREAT DECOUPLING THE AGENTIC WEB 01 Search crawlers Find, render and index pages. ? Googlebot · Bingbot OAI-SearchBot · Claude-SearchBot PerplexityBot ? WHAT I CARE ABOUT Crawl access, rendering, indexing, sitemaps, status codes. SOURCES OpenAI · crawlers and user agents ↗ · Anthropic · how Anthropic crawls and how to block it ↗ · Perplexity · Perplexity crawlers ↗
46. 05 — THREE KINDS OF VISITORS 45 / 70 The decoupling added a second. THE WEB WE KNOW THE GREAT DECOUPLING 01 02 Search crawlers Training crawlers Find, render and index pages. Collect content that may train future THE AGENTIC WEB models. Googlebot · Bingbot GPTBot · ClaudeBot OAI-SearchBot · Claude-SearchBot Google-Extended (a robots.txt token) ? PerplexityBot WHAT I CARE ABOUT WHAT I CARE ABOUT Crawl access, rendering, indexing, sitemaps, status codes. A policy per section, with an owner. Blocking later wonʼt undo what a model already learned. SOURCES OpenAI · crawlers and user agents ↗ · Anthropic · how Anthropic crawls and how to block it ↗ · Google · Common crawlers and Google-Extended ↗
47. 05 — A QUESTION I ASKED IN 2023 46 / 70 A question I asked in 2023. Which user agent do you block to keep OpenAIʼs crawler out? Lily Ray answered. Today the major AI companies document their crawlers, by purpose. My question on X ↗ SOURCES Lily Ray’s reply ↗ @DarwinSantosNYC · crawler question (Jan 31, 2023) ↗ · Lily Ray · reply to the 2023 crawler question ↗
48. 05 — THREE KINDS OF VISITORS 47 / 70 The agentic web adds a third. THE WEB WE KNOW THE GREAT DECOUPLING THE AGENTIC WEB 01 02 03 Search crawlers Training crawlers Live user agents Find, render and index pages. Collect content that may train future Fetch or act for a person, right now. models. Googlebot · Bingbot GPTBot · ClaudeBot ChatGPT-User · Claude-User OAI-SearchBot · Claude-SearchBot Google-Extended (a robots.txt token) Perplexity-User · Google-Agent PerplexityBot WHAT I CARE ABOUT WHAT I CARE ABOUT WHAT I CARE ABOUT Crawl access, rendering, indexing, sitemaps, status codes. A policy per section, with an owner. Blocking later wonʼt undo what a model already learned. Access, speed, verification, tools they can call. May not follow robots.txt. SOURCES OpenAI · crawlers and user agents ↗ · Anthropic · how Anthropic crawls and how to block it ↗ · Google · user-triggered fetchers ↗
49. 05 — WHAT I CHECK FOR EACH 48 / 70 What I check for each visitor. Search crawlers Training crawlers Live user agents THE QUESTION Can it find, render and index the page? Do we allow this use, for this section? Can it get the facts and finish the task? MY CONTROLS robots.txt, sitemaps, canonicals, status codes robots.txt rules per bot or token, and a written policy Verification, speed, WebMCP tools, no blanket blocks WHERE I LOOK Search Console, Bing Webmaster Tools, logs Logs: crawl volume by bot over time Logs, referrals, and task tests SOURCES OpenAI · crawlers and user agents ↗ · Google · user-triggered fetchers ↗ · Google · Common crawlers and Google-Extended ↗
50. 05 — WHAT WE CAN DO ON MONDAY 49 / 70 What can we do on Monday? Steps from my own work. Each one fits in an afternoon. 01 02 03 Get log access Verify the visitor Compare and split visitors by purpose DNS, IP lists, signatures what you publish vs. what agents get 04 05 06 Test the task Measure Give agents tools with a real agent your own decoupling with WebMCP
51. 05 — THREE QUESTIONS 50 / 70 Three questions for technical SEO. 01 02 03 What can it receive? Can it understand the oer? Can it use the interface? Response, rendering, access The right facts tied to the right A supported path, permission, and context. product. a verified result. SOURCES Google · AI features and your website ↗
52. 05 — WHY LOGS FIRST 51 / 70 It’s rising. And we can barely see it. ILLUSTRATIVE PROJECTION · THE SHAPE MANY OF US SEE IN OUR LOGS, NOT MEASURED DATA TODAY LIVE USER FETCHES ChatGPTUser · Claude-User · Perplexity-User · Google-Agent MCP AND WEBMCP TOOL CALLS Agents using the tools you publish Check your own logs. Youʼll likely see the same climb, without the journeys behind it. WHAT PLATFORM REPORTS SHOW US TODAY: VERY LITTLE 2024 SOURCES 2025 OpenAI · crawlers and user agents ↗ 2026 · Google · user-triggered fetchers ↗ 2027 · Bing · AI Performance in Webmaster Tools (Feb 2026) ↗
53. 05 — SAMPLE · WHO IS VISITING 52 / 70 Start in the logs. Logs are our first-party analytics for AI visitors. Get access. access.log · count AI visitors by user agent # search / index OAI-SearchBot Claude-SearchBot PerplexityBot # training collection Split AI visitors by purpose: search, training, or a personʼs live request. GPTBot User agents can be spoofed. Verify them next. grep -oiE "(OAI-SearchBot|GPTBot|ChatGPT-User| SOURCES OpenAI · crawlers and user agents ↗ · ClaudeBot # a person’s live request ChatGPT-User Claude-User Perplexity-User Claude(Bot*-User*-SearchBot)|Perplexity(Bot*-User))" access.log | sort | uniq -c | sort -rn Anthropic · how Anthropic crawls and how to block it ↗ · Google · user-triggered fetchers ↗
54. 05 — SAMPLE · VERIFY THE VISITOR 53 / 70 Verify before you count. 1 Reverse DNS, then forward verify.sh · is this really Googlebot? Googlebot: *.googlebot.com · Bingbot: # 1. reverse lookup *.search.msn.com host 66.249.66.1 # 2 Match published IP ranges OpenAI, Anthropic, Perplexity and Google publish JSON lists. 3 Check the signature crawl-66-249-66-1.googlebot.com # 2. forward lookup must return the same IP host crawl-66-249-66-1.googlebot.com # 66.249.66.1 # 3. or match against a published list curl -s https:*/openai.com/chatgpt-user.json \ | jq -r '.prefixes[].ipv4Prefix' ChatGPT agent signs requests. So do some Google-Agent requests. SOURCES Google · Verifying Googlebot and other Google crawlers ↗ · Bing · How to verify that Bingbot is Bingbot ↗ · OpenAI · crawlers and user agents ↗ Anthropic · how Anthropic crawls and how to block it ↗ · Perplexity · Perplexity crawlers ↗ · OpenAI · ChatGPT agent allowlisting (signed requests) ↗ · Google · Web Bot Auth for Google-Agent ↗ ·
55. 05 — SAMPLE · WHO IS BEHIND THE BOTS 54 / 70 Who is behind each visitor? USER AGENTS OPERATOR PEOPLE BEHIND IT IP LIST ChatGPT-User OAI-SearchBot OpenAI 900M weekly ChatGPT users Mar 2026 openai.com/chatgpt-user.json ↗ Google-Agent Googlebot Google 2.5B AI Overviews · 1B AI Mode, monthly May 2026 user-triggered-agents.json ↗ Claude-User Claude-SearchBot Anthropic No official figure claude.com/crawling/bots.json ↗ Perplexity-User PerplexityBot Perplexity No official figure perplexity.ai/perplexity-user.json ↗ GIVE THIS TO YOUR AGENT › Using nodeandlink.com/ai-bots, split last month’s access logs by bot purpose, verify each IP against the operator’s published ranges, and list anything that fails. SOURCES OpenAI · 900M+ weekly ChatGPT users (Mar 31, 2026) ↗ · Google · I/O 2026 keynote (AI Overviews, AI Mode users) ↗ · Google · Gemini app passes 1B monthly users (Aug 11, 2026) ↗ · Anthropic · how Anthropic crawls and how to block it ↗ · Perplexity · Perplexity crawlers ↗ · Node & Link · AI bot directory ↗
56. 05 — SAMPLE · WHAT IT RECEIVES 55 / 70 Compare what you publish with what it gets. fetch.sh · do the facts survive without rendering? Check the raw response, what needs JavaScript, and what depends on a # 1. raw HTML, as a non-rendering client sees it curl -sL -A "ChatGPT-User" "$URL" > raw.html session or location. # 2. are the facts the task needs in the response? for f in "SSO" "per month" "Enterprise"; do Record URL, time, client and the exact printf "%-14s %s\n" "$f" \ "$(grep -c "$f" raw.html)" response, so someone else can done reproduce it. # 3. log it: URL, UTC time, user agent, status, bytes SOURCES Google · AI features and your website ↗ · Google · user-triggered fetchers ↗
57. 05 — SAMPLE · RAW VS RENDERED 56 / 70 What exists only after JavaScript? Fetch the page twice: raw, and rendered by a browser. Anything that only appears in the rendered copy may never reach a client that doesnʼt run JavaScript. diff.sh · raw vs rendered # raw: what a non-rendering client gets curl -sL -A "ChatGPT-User" "$URL" > raw.html # rendered: what a browser builds google-chrome *-headless *-dump-dom "$URL" \ > rendered.html # facts that exist only after rendering diff <(grep -o '[^*]*per month[^<]*' raw.html) \ <(grep -o '[^*]*per month[^<]*' rendered.html) BLOCKED BY YOUR OWN CDN? If your CDN blocks spoofed agents, it blocks your test too. Ask your SRE or CDN team to allowlist your test IP. Then test as the agent. SOURCES Google · AI features and your website ↗ · # example output > $49 per month, billed annually Google · user-triggered fetchers ↗
58. 05 — CAN IT UNDERSTAND THE OFFER? 57 / 70 Can it understand the oer? Identity and version Price and availability Which product, which plan? Current, and where? ONE PLAN Applicable conditions Terms, eligibility, limits Attributes have to belong to the same product. SOURCES Google · AI features and your website ↗ Updated When was this true?
59. 05 — SAMPLE · TEST THE TASK 58 / 70 Can an agent actually finish the job? ASK · CAN IT REPEAT YOUR FACTS? “What does the Pro plan cost per month?ˮ ACT · CAN IT DO THE JOB? 1 Reach the page 2 Use the pricing calculator 3 Fill the demo form 4 Subscribe to the newsletter 5 Stop at checkout confirmation “Does it include SSO?ˮ “Compare [you] and [competitor] for [use case].ˮ Check: accurate, current, attached to the right plan. Run it with ChatGPT agent, Claude, Gemini or OpenClaw. Protect forms from spam, but donʼt block the person behind the agent. Advocate for letting agents use what we publish.
60. 05 — SAMPLE · MEASURE YOUR DECOUPLING 59 / 70 Measure your own decoupling. bigquery · searchconsole bulk export Monthly impressions, clicks and CTR from the Search Console bulk export. SELECT DATE_TRUNC(data_date, MONTH) AS month, SUM(impressions) AS impressions, Mark May 13, 2025  Apr 27, 2026 on the SUM(clicks) AS clicks, SAFE_DIVIDE(SUM(clicks), SUM(impressions)) AS ctr chart: impressions were over-reported in FROM `project.searchconsole.searchdata_site_impression` that window. WHERE data_date *= '2024-01-01' GROUP BY month ORDER BY month *- flag 2025-05-13 → 2026-04-27 before you compare SOURCES Google · Search Console bulk data export ↗ · Google · Search Console data anomalies ↗
61. 05 — SAMPLE · WHAT THE QUERY GIVES YOU 60 / 70 What that query gives you. ILLUSTRATIVE OUTPUT · THE SHAPE TO LOOK FOR, NOT REAL DATA Read it IMPRESSIONS OVER-REPORTED MAY 13, 2025 – APR 27, 2026 Impressions up, IMPRESSIONS clicks flat: the decoupling. Then subtract the shaded window before you conclude anything. CLICKS JAN 2024 SOURCES JAN 2025 Google · Search Console bulk data export ↗ JAN 2026 · Google · Search Console data anomalies ↗
62. 05 — THE SURFACE IS GROWING 61 / 70 The surface we optimize is growing. Tools agents can call WEBMCP · ORIGIN TRIAL Crawlable, indexable pages THE WEB WE KNOW Agent capability catalog /.WELL-KNOWN/ARD.JSON YOUR SITE Answers built from your facts THE DECOUPLING Product feeds and checkout ACP · UCP SOURCES Node & Link · Rise of the Agentic Web ↗ · Darwin Santos · The Great Decoupling of Search (Search Engine Land, Jul 2025) ↗
63. 05 — SAMPLE · WEBMCP 62 / 70 WebMCP: tell the agent what it can do here. pricing.html · declarative newsletter.js · imperative <form toolname="estimatePrice" await document.modelContext tooldescription="Monthly cost for .registerTool({ a plan and seat count"> name: "subscribe-newsletter", <select name="plan" description: "Subscribe an email toolparamdescription="Plan tier"> to the weekly newsletter", <input name="seats" type="number"> inputSchema: { type: "object", */form> properties: { email: { type: "string" } } }, async execute({ email }) { ** … */ } }); DRAFTED BY STATUS CHROME OPENAI Google and Microsoft W3C community draft, not a Origin trial from Chrome 149, Support in ChatGPTʼs desktop engineers standard Jun 2026 browser, Aug 2026 SOURCES W3C Web Machine Learning CG · WebMCP draft ↗ WebMCP Challenge ↗ · Chrome · WebMCP origin trial (Jun 9, 2026) ↗ · Chrome · WebMCP declarative API ↗ · OpenAI ·
64. 05 — RUN A SMALL INVESTIGATION 63 / 70 Pick one real task. Test it this week. One task. One gap. Before and after. Choose one real task Repeat the same check You donʼt need to audit the whole site. Compare facts and response The new tools make this fast: describe the test in a task box, or spin up a test page with ChatGPT Sites ↗ Bring it to the owner SOURCES OpenAI Academy · ChatGPT Sites ↗ Find the gap
65. 06 — USE THE SAME TECHNOLOGY 64 / 70 Use the same technology to investigate. 1 Synthesize Bring sources together. Spot the question worth checking. 2 Inspect Read the source yourself. Challenge the explanation. 3 Build Fix your own friction. GSC Guardian puts algorithm context inside Search Console. We still check the sources. We just donʼt have to hold every update in our heads before we start. SOURCES Node & Link · GSC Guardian ↗ GSC GUARDIAN
66. 06 — ASK FOR ACCESS 65 / 70 We need access to see these journeys. Citation Visit Action Outcome Four different measures. A report should say which one it counts. ASK THE PLATFORMS ASK YOUR OWN COMPANY Reporting that defines its terms. Bingʼs AI Performance report (preview, Feb 2026) is a first step. Check what it counts and what it leaves out. Log access. Agents act on behalf of real customers. Without the logs, we optimize for them blind. SOURCES Bing · AI Performance in Webmaster Tools (Feb 2026) ↗
67. 06 — PEOPLE WHO SHARE THEIR WORK 66 / 70 Follow the people who share. This is how we move forward as one community. You? share your method Lily Ray @lilyraynyc ↗ Aleyda Solís @aleyda ↗ Metehan Yeşilyurt @metehan777 ↗ Mike King @iPullRank ↗ Suganthan Mohanadasan @suganthan ↗ Natzir Turrado @natzir9 ↗ Full list: nodeandlink.com/people-i-follow ↗ Today’s speakers: techseoconnect.com ↗ SOURCES Tech SEO Connect 2026 · speakers ↗ · Victor Pan @victorpan ↗ Node & Link · People I follow ↗ · Metehan Yeşilyurt · @metehan777 ↗ Dan Petrovic @dejanseo ↗
68. 06 — CLOSE The human need remains. 67 / 70 A2A AI MODE GOOGLE CHECKOUT SGE CHATGPT PERSON People still need the products, services and knowledge our businesses provide. MCP INTERNET DECOUPLING WEB BING AI OVERVIEWS AGENTS SOURCES Darwin Santos · The Great Decoupling of Search (Search Engine Land, Jul 2025) ↗
69. TECH SEO CONNECT · 2026 Thank you. Keep testing. Keep sharing. nodeandlink.com/the-great-decoupling-timeline ↗ nodeandlink.com/rise-of-the-agentic-web ↗ nodeandlink.com/ai-bots ↗ nodeandlink.com/people-i-follow ↗ x.com/DarwinSantosNYC ↗ 68 / 70
70. APPENDIX — SOURCES (1 OF 2) 69 / 70 Tech SEO Connect ↗ SERoundtable · The Great Decoupling (Jun 3, 2025) ↗ Google I/O 2023 keynote (May 10, 2023) ↗ SERoundtable · A Googler explains the chart (Jun 18, 2025) ↗ OpenAI · Introducing ChatGPT (Nov 30, 2022) ↗ Google · Search Console data anomalies ↗ Kleinrock · The first message transmission (ICANN) ↗ Node & Link · The Great Decoupling Timeline ↗ CERN · A short history of the Web ↗ Node & Link · Rise of the Agentic Web ↗ History of Yahoo ↗ Anthropic · computer use, public beta (Oct 22, 2024) ↗ Google · Our story ↗ Anthropic · Model Context Protocol (Nov 25, 2024) ↗ Microsoft · Bing launch (May 28, 2009) ↗ OpenAI · Introducing Operator (Jan 23, 2025) ↗ Microsoft · AI-powered Bing preview (Feb 7, 2023) ↗ Anthropic · Claude Opus 4.5 (Nov 24, 2025) ↗ Google · SGE in Search Labs (May 10, 2023) ↗ Peter Steinberger · Introducing OpenClaw ↗ Google · AI Overviews U.S. rollout (May 14, 2024) ↗ Google · Gemini Spark ↗ Google · AI Mode (May 20, 2025) ↗ Meta · Introducing Muse ↗ Google · AI Mode in 200+ countries (Oct 7, 2025) ↗ OpenAI · Introducing dots ↗ Google · Search at I/O 2026 (May 19, 2026) ↗ Anthropic · Claude Duet ↗ SERoundtable · SGE (May 11, 2023) ↗ OpenAI · crawlers and user agents ↗ Search Engine Land · AI overviews in main search (Mar 2024) ↗ Anthropic · how Anthropic crawls and how to block it ↗ Search Engine Land · AI Overviews in the US (May 2024) ↗ Google · user-triggered fetchers ↗ Search Engine Land · AI Mode for all US searchers (May 2025) ↗ Google · AI features and your website ↗ Darwin Santos · The Great Decoupling of Search (Search Engin… ↗ Google · Search Console bulk data export ↗ Pew Research · AI summaries and clicks (Jul 22, 2025) ↗ Bing · AI Performance in Webmaster Tools (Feb 2026) ↗ Ahrefs · The Great Decoupling (Jun 17, 2025) ↗ Node & Link · GSC Guardian ↗ @DarwinSantosNYC · naming the Great Decoupling (May 31, 2025) ↗ Lily Ray · reply to the 2023 crawler question ↗
71. APPENDIX — SOURCES (2 OF 2) 70 / 70 @DarwinSantosNYC · crawler question (Jan 31, 2023) ↗ OpenAI · 900M+ weekly ChatGPT users (Mar 31, 2026) ↗ Google · In-depth guide to how Google Search works ↗ Google · Gemini app passes 1B monthly users (Aug 11, 2026) ↗ Node & Link · timeline data (events.csv) ↗ Google · I/O 2026 keynote (AI Overviews, AI Mode users) ↗ OpenAI · Computer-Using Agent (Jan 23, 2025) ↗ W3C Web Machine Learning CG · WebMCP draft ↗ OpenAI · Introducing ChatGPT agent (Jul 17, 2025) ↗ Chrome · WebMCP origin trial (Jun 9, 2026) ↗ OpenAI · ChatGPT for your most ambitious work (Jul 9, 2026) ↗ Chrome · WebMCP declarative API ↗ AP · Muse accounts top 10 million (Oct 21, 2026) ↗ OpenAI · WebMCP Challenge ↗ MCP specification (2024-11-05) ↗ Agentic Resource Discovery · specification ↗ OpenAI · New tools in the Responses API (May 21, 2025) ↗ OpenAI Academy · ChatGPT Sites ↗ MCP Registry preview (Sep 8, 2025) ↗ Node & Link · AI bot directory ↗ Linux Foundation · Agentic AI Foundation (Dec 9, 2025) ↗ Node & Link · People I follow ↗ Coinbase · x402 (May 6, 2025) ↗ Tech SEO Connect 2026 · speakers ↗ Google Cloud · Agent Payments Protocol (Sep 16, 2025) ↗ Aleyda Solís · Tech SEO Connect, CAM Raleigh (Oct 17–18, 2024) ↗ OpenAI · Buy it in ChatGPT (Sep 29, 2025) ↗ Tech SEO Connect 2025 · Durham (Dec 4–5, 2025) ↗ Google · Universal Commerce Protocol (Jan 11, 2026) ↗ Google · Common crawlers and Google-Extended ↗ OpenAI · Powering product discovery in ChatGPT (Mar 24, 2026) ↗ Metehan Yeşilyurt · @metehan777 ↗ Tapestry · Coach and Kate Spade in Gemini (Sep 16, 2026) ↗ Google · Verifying Googlebot and other Google crawlers ↗ Google · Web Bot Auth for Google-Agent ↗ OpenAI · ChatGPT agent allowlisting (signed requests) ↗ Perplexity · Perplexity crawlers ↗ Bing · How to verify that Bingbot is Bingbot ↗