JR Oakes - Leveraging LLMs To Extract Insights From Analytics Data

Transcript

1. CONFIDENTIAL Context is the Analytics King? Tech SEO Connect 1
2. JR Oakes 2 JR Oakes is the VP of Strategy for LOCOMOTIVE Agency. He has been an SEO since 2011 and was formerly an architectural glass artist. His focus areas are in SEO, machine learning, language, and user experience. jroakes locomotive.agency
3. CONFIDENTIAL My Interests Have Broadened a Bit 3 Causal ML Generative AI Personalization Engines Technical SEO
4. CONFIDENTIAL CONFIDENTIAL Interestingness 4 Section 2
5. CONFIDENTIAL Efficient Infinite Context Transformers with Infini-attention 5 Source
6. CONFIDENTIAL Mixed Bag 6
7. CONFIDENTIAL Better Model 7 Brand Expertise Trust Transparency Engagement Real Experience
8. CONFIDENTIAL LLMs Love the Word Delve 8 Source
9. CONFIDENTIAL 9 Source
10. CONFIDENTIAL Top-10 Best SEO Agencies* * According to ChatGPT models 10
11. CONFIDENTIAL Top-10 Best AI Writing Tools* * According to ChatGPT models 11 Shocking how consistent this is
12. CONFIDENTIAL Reasons LLM Brand Tracking is Messy 12 Temperature Memory/Projects Random Seeds RAG Search Slight differences in wording Different Models
13. CONFIDENTIAL LibreChat 13
14. CONFIDENTIAL CONFIDENTIAL Trends 14 Section 1
15. CONFIDENTIAL Watching Emerging Trends 15
16. CONFIDENTIAL 16 People are looking at the problems with AI, but not the possibilities.
17. CONFIDENTIAL 17 OR
18. CONFIDENTIAL 18 MORE AI CONTENT
19. CONFIDENTIAL 19 AI Everyone
20. CONFIDENTIAL Causal Reasoning and Large Language Models “…we find that LLMs can generate text corresponding to correct causal arguments with high probability, surpassing the best-performing existing methodsˮ 20 Source
21. CONFIDENTIAL ▪ LLMs can generate causal graphs from metadata alone ▪ Capable of counterfactual reasoning in natural language ▪ Determine necessary/sufficient causes from text descriptions ▪ Capture relevant context and common sense for causal judgments ▪ Complement existing statistical causal methods ▪ Automate parts of causal analysis previously requiring human experts ▪ Enable flexible, natural language interaction for causal tasks 21 Source Causal Reasoning and Large Language Models
22. CONFIDENTIAL CONFIDENTIAL We Code 22 Section 3
23. CONFIDENTIAL 23 This Past Year We Have Been Building.. ▪ Built a tool called Npath, that turns GA4 sequence data into user cohorts, ideals paths, and insights. ▪ Built a tool that takes a companyʼs ICP information and turns it into tens of thousands of categorized keywords with competitive metrics. ▪ Formalized all competitive gaps to aggregate data based on subjects rather than keywords. ▪ Finalizing a tool to detect anomalies buried in thousands of URLs and segments of URLs.
24. CONFIDENTIAL 24 I had a thought
25. CONFIDENTIAL Can we just throw URL page data into a big bucket each month and have LLMs compare and share insights on it? 25
26. CONFIDENTIAL 26 A Cloud Robot Looking for Errors in My URLs
27. CONFIDENTIAL 27 We Need to Put Some Pieces Together
28. CONFIDENTIAL Can we fit all the data and have a dependable output? 28
29. Gemini 29 - 2M Tokens - Schema specification - MIME type specification (e.g. application/json)
30. Tech Debt 30 You kinda get into dependency hell with LLMs. Will it work if they upgrade the model?
31. CONFIDENTIAL Can we afford it? 31
32. CONFIDENTIAL 32 Gemini API Costs Input Pricing $1.25 / 1 million tokens Output Pricing $5.00 / 1 million tokens 1M tokens = ~130 pages 1M tokens = ~700 pages
33. CONFIDENTIAL What if it is not perfect? 33
34. CONFIDENTIAL 34 Models Will Get Better The pace of improvement compared with human-level performance continues to expand across a wide-range of tasks. Source
35. CONFIDENTIAL 35 And More Accurate Models continue to get better at controlling hallucinations. Source
36. CONFIDENTIAL 36 It’s Yours, Change it This is open source, so tweak the prompts your your needs.
37. CONFIDENTIAL Can LLMs do math? 37
38. Ok. There Are Issues Here…. 38 …position improve from 7.85 to 9, an increase of 92.31%...
39. We Have to Be a Bit Creative 39 Just provide current and prior values. We calculate the rest.
40. CONFIDENTIAL What context can we provide? 40
41. CONFIDENTIAL 41 Metrics Content Analysis: • Clean Content • Metadata • Word Count • Heading Structure • Image Count • Schema Markup Page Speed Insights: • Largest Contentful Paint • Cumulative Layout Shift • Interaction to Next Paint • First Contentful Paint • Time to Interactive • Speed Index • Total Blocking Time • Performance Score • First Meaningful Paint Search Engine Performance: • Clicks (from Google Search Console) • Impressions (from Google Search Console) • Click-Through Rate CTR • Average Position (in search results) • Ranking Keywords • Top No-Click Queries
42. CONFIDENTIAL 42 Metrics Traffic and User Behavior: • Organic Sessions • Organic Users • Organic New Users • Bounce Rate • Average Time on Page • Engagement Rate • Revenue • User Demographics • Device Categories • Pages Visited Prior • Pages Visited Next • Referring Sites
43. CONFIDENTIAL CONFIDENTIAL SEODP 43 Section 4
44. CONFIDENTIAL An eye in the cloud watching my content 44
45. CONFIDENTIAL 45 SEO Data Platform on Github
46. Requirements 46 1. A developer :-) 2. A few API keys (most are free or very cheap)
47. Modular 47 1. Very clean and extensible extractor workflow. 2. Intuitive to find and update code.
48. Customizable 48 You tell it what to report on.
49. Output 49 Insights in your inbox. Find it here: SEODP
50. Examples 50
51. Examples 51
52. Bonus 52 pip install repocoder
53. CONFIDENTIAL CONFIDENTIAL The End 53 Section null