Lessons Learnt from Crawling 1000+ Websites

@charlesmeaden #brightonseo

@charlesmeaden #brightonseo Lets Go Back 30 Years

@charlesmeaden #brightonseo This Was The Height of SEO <meta name="keywords"
content="Robert Jordan, house, flat, bungalow, farm, apartment, country house, terraced house, semi-detached house, detached house, condominium, rental, rent, landlord, tenant, property management, homes, leases, agreements, landlords insurance, accommodation, rooms, management lettings, residential, renters,">

@charlesmeaden #brightonseo Good Luck Checking At Scale

@charlesmeaden #brightonseo Along Came Link Sleuth

@charlesmeaden #brightonseo Now We’re Spoilt for Choice

@charlesmeaden #brightonseo I’m Not Talking About • Tool Specific Functions
• One tool vs another

@charlesmeaden #brightonseo What I Am Talking About • 12 Useful
tips based on my success and failures

@charlesmeaden #brightonseo 1: Make Friends With The Devs • They
know all the tricks to getting your crawl to proceed • Speak their language

@charlesmeaden #brightonseo 2: Never Assume • Work on the premise
that somewhere, someone has got it wrong • Always Be Curious

@charlesmeaden #brightonseo 3: Ask Some, Not Many Questions • Just
enough for the basic facts • Go in with an open mind

@charlesmeaden #brightonseo 4: Eyeball The Site

@charlesmeaden #brightonseo Some Handy Chrome Extensions SEO Toggle Robots Exclusion
Checker

@charlesmeaden #brightonseo Some Handy Chrome Extensions Detailed SEO Extension Rendering
Difference Engine

@charlesmeaden #brightonseo 5: Do A Quick First Pass Crawl You’ll
quickly see what you can safely exclude from the whole crawl

@charlesmeaden #brightonseo 6: Check What Appears On The Page •
Data from a crawl does a great job of telling you about the page structure • What about the content on the page?

@charlesmeaden #brightonseo A Case Study • USA Hotel chain site
with 900 hotels, each with their own page(s) • Former agency had crawled for 18 months and found no issues

@charlesmeaden #brightonseo Something Seemed Wrong Google Search console wasn’t showing
any searches for swimming pools

@charlesmeaden #brightonseo 33% of On Page Content Not Rendered No
facilities or menu content rendered

@charlesmeaden #brightonseo One Line in The Robots.txt Incorrect use of
a wildcard stopped the render process

@charlesmeaden #brightonseo Enable JavaScript Screenshots Both Screaming Frog and Sitebulb
have this feature

@charlesmeaden #brightonseo 7: Every Web Site is Slightly Different Content
management systems and JavaScript frameworks handle things differently

@charlesmeaden #brightonseo 8: Compare and Contrast • Run a text
and JavaScript crawl • You may be surprised by the difference

@charlesmeaden #brightonseo 8: Keep Your Crawls • Always handy when
a change has been made • You’ve been tasked with working out what went wrong

@charlesmeaden #brightonseo 9: Automate Regular Crawls Handy way to check
for things that can and will go wrong

@charlesmeaden #brightonseo Examples • Lighthouse tests • Checking for 301
and 404 • XML Orphan Pages • Overly large images

@charlesmeaden #brightonseo Case Study – Embargoed Products • Resellers who
have restrictions on the brands they can show in Google • Run a weekly crawl to check for pages that mention it

@charlesmeaden #brightonseo 10: Extract Useful Data Xpath and Regex allow
you to extract content at scale

@charlesmeaden #brightonseo Things We’ve Extracted • Stock levels / Out
of stock • Categories and taxonomies • Schemas • Video embeds

@charlesmeaden #brightonseo 11: Combine With Other Data Crawl data is
even more effective when combined

@charlesmeaden #brightonseo Crawl and Analytics Data Use traffic data to
understand which changes will make the most impact

@charlesmeaden #brightonseo Crawl and Your Data Combine crawl data with
your own data to create actionable reports

@charlesmeaden #brightonseo Case Study – Crawl and Stock Data •
Ecommerce client combines crawl data with stock data • Determines which category pages get updated first

@charlesmeaden #brightonseo 12: Always Check for New Features Tools are
constantly bringing out new functions

@charlesmeaden #brightonseo ScreamingFrog – LLM Integration Sending crawl data to
LLM opens a whole new world

@charlesmeaden #brightonseo Sitebulb: Code & CSS Coverage Discover where you’re
loading CSS and JavaScript when it’s not needed

@charlesmeaden #brightonseo ScreamingFrog N-Grams Incredibly useful for spotting patterns

@charlesmeaden #brightonseo Thank You Find this deck at https://speakerdeck.com/charlesmeaden/ Connect
with me https://www.linkedin.com/in/charlesmeaden

Lessons Learnt from Crawling 1000+ Websites

Lessons Learnt from Crawling 1000+ Websites

More Decks by Charles Meaden

Other Decks in Marketing & SEO

Featured

Transcript