Paul Shapiro - Rolling Your Own Rank Tracking Solution

Transcript

1. Rolling Your Own Rank Tracking Solution Paul Shapiro | @fighto | searchwilderness.com Rankenberry: The Open-Source SEO Rank Tracking Revolution
2. Paul Shapiro • Currently: Uber, Web Intelligence • Previously: Shopify, Conde Nast, GroupM • From: New Jersey • Likes: Board Games, Coffee, Horror Movies
3. Limited Customization: Most rank-tracking platforms offer predefined workflows, limiting flexibility for SEOs who need to tailor data to their unique needs. Data Silos: Rank-tracking data is often isolated from other key metrics like Google Search Console, preventing a holistic view of SEO performance. The Problem with Most Rank Tracking Solutions Vendor Lock-In: High costs and long-term contracts lock users into restrictive ecosystems, limiting control over their data and analytics.
4. Full Data Control Complete ownership of your SEO data, end-to-end; Analyze and integrate it however you need. Customizable & Flexible Tailor the platform to fit your unique SEO strategy. Community-Driven Innovation As an open-source platform, Rankenberry evolves constantly with community contributions and a collaborative development roadmap. Meet Rankenberry.
5. Rankenberry Github Repo https:/ /github.com/pshapiro/rankenberry
6. searchwilderness.com/spaceserp Affiliate Link
7. Technical Foundations and Architecture
8. Backend Overview
9. Python Has Emerged THE Language for Data Data Science Dominance • Python is the most widely used language for data analysis. https://www.developernation.net/developer-reports/dn26/ Set up sot the future data analysis that you’re ultimately going to want AND it will be able to be added right within the Rankenberry backend. • No additional data pipelining needed.
10. Backend Overview FastAPI as the Orchestrator • Manages requests from the frontend and external APIs. • Central hub for processing data between APIs and the database. Managing Background Jobs with AsyncIOScheduler • Handles scheduled tasks such as fetching data from APIs at regular intervals. • Manages background processes to ensure smooth, asynchronous operations without blocking the main application. Database: SQLite for Local Storage • Stores keyword data, project information, Google Search Console data, and SERP data. • Will migrate to PostgreSQL for cloud or multi-user environments in the future.
11. Backend Overview External APIs • SpaceSERP, Google Search Console, GrepWords • FastAPI fetches search analytics, ranking data, and search volume from APIs. • Easily integrates new APIs via FastAPI’s modular architecture.
12. FastAPI for Flexibility Handling Asynchronous Requests Efficiently • FastAPI's built-in support for asynchronous requests ensures high performance and scalability. • Ideal for handling frequent API calls, processing large amounts of SEO data, and minimizing latency. Automatic API Creation for Rapid Development • FastAPI automatically turns backend functions into fully-documented APIs. ◦ http://localhost:5001/docs API-Driven Architecture for Future Growth • FastAPI provides a robust backend architecture with API endpoints for seamless communication between the frontend and backend.
14. Frontend Overview
15. Why Vue? • Reactive data binding ensures smooth user interaction and immediate feedback. • Plays well with FastAPI's built-in support for asynchronous requests.
16. Regarding Vite… Development Mode: • In most tools (like Webpack), your entire application is bundled (turned into a single file or multiple chunks) before it can be served to the browser. • Vite skips that bundling step during development. Instead, it uses the browser’s native ability to understand ES modules (modern JavaScript) to load code directly. Fast Updates with Hot Module Replacement (HMR): • When you make a small change to your code (like editing a Vue component), Vite only updates that piece of code in the browser without rebuilding the entire app. This makes your changes show up instantly—this feature is called Hot Module Replacement (HMR). Production Mode: • When it’s time to deploy your project, Vite bundles everything (just like Webpack) using Rollup. So, for development, you get speed, and for production, you still get an optimized build.
17. Pinia for State Management State in Frontend Apps • Data like user inputs, API responses, and selected items. Seamless with Vite • Hot Module Replacement (HMR) ensures instant updates without page reloads. When Apps Grow, State Gets Complex • Vue’s built-in tools (like props) aren’t enough for large apps. • Components need to share data and stay in sync with APIs.
18. Core Features
20. Core Features Overview Scheduling & On-Demand Fetches • Automate keyword tracking at scheduled intervals or perform on-demand fetches for individual keywords, tags, or projects. Tag Management System • Organize keywords using custom tags for more flexible tracking and reporting. Summary Data • By Domain/Project or Tag Share of Voice (SOV) Over Time • Analyze how your keywords perform relative to competitors with SOV charts that track changes over time.
25. Core Features Overview Google Search Console (GSC) Integration • GSC + 3P Rank Data: Merge Google Search Console data with 3P rank checking metrics to gain deeper insights into your performance. • Business Impact Analysis: Estimate potential traffic and revenue from ranking changes using CTR models and keyword conversion data. Time Series Views • See how rankings have changed over time Full SERP Access (Top 100 Results) • Retrieve and analyze the entire top 100 search results for detailed competitive analysis.
30. Challenges and Limitations
31. Current API and Feature Limitations SpaceSERP API Limitations • No Featured Snippets or AI Overviews: Cannot track advanced SERP features. • Incomplete Related Search Data: Limited or missing related search results. • Single API Provider: Current setup only supports SpaceSERP. Path to Future Flexibility • Modular architecture will allow adding or swapping APIs (e.g., Serper.dev, SerpAPI, ValueSerp). • Future-proofing the platform with multiple API integrations.
32. Data Storage and Development Limitations SQLite Database Constraints • Local development only: SQLite works well for individual use but isn’t scalable. • No multi-user support: Requires future migration to PostgreSQL or a cloud-based solution. Solo Development Effort • Slower feature development: Managing the project solo limits how quickly new features can be implemented. • The Power of Open Source: Contributions from the community will accelerate growth and improve the platform. Dashboard Customization • Customization via code: Dashboards require modifying the codebase—there’s no UI-based customization yet.
33. Scaling and Open-Source Growth
34. Scaling Rankenberry Beyond a Single Creator The Challenge of Solo Development • Currently developed and managed by a single creator. • Slows down feature releases and platform improvements. The Power of Open Source • Open-source collaboration enables faster development. • Users can submit bug reports, feature requests, or contribute code. • Encourages a shared development roadmap driven by community needs. Goal • Create a self-sustaining community of contributors and maintainers.
35. Scaling Rankenberry Beyond a Single Creator What is AGPL-3? • AGPL-3 (Affero General Public License v3) is a strong copyleft license designed to ensure that software, including modifications, remains open-source. • It is an extension of the GPL license, but with a focus on network-based software. How AGPL-3 Protects Open-Source Projects • Sharing Requirement: Any changes made to the software, even when used on private or hosted servers, must be shared with the public. • Prevents Closed Forks: Companies cannot take the software, modify it, and offer it as a proprietary solution without sharing their changes. Why AGPL-3 Fits Rankenberry • Ensures Future Contributions: If someone builds on top of Rankenberry’s code, their improvements must be shared back with the community. • Protects the Open-Source Spirit: Prevents private or closed versions of the software from emerging. • Encourages Collaboration: Developers are motivated to contribute to the official project rather than fork it privately.
36. The Future Roadmap— Upcoming Features
37. Algorithm Update Overlays Overlay Google Algorithm Updates: • Highlight algorithm changes (e.g., Core Updates) on your keyword ranking graphs. • Correlate fluctuations in rankings with these updates. Impact: • Understand whether algorithm changes caused ranking shifts. • Identify patterns to adjust SEO strategies accordingly. https://status.search.google.com/products/rGHU1u87FJnkP6W2GwMi/history
38. Anomaly Detection Use Statistical Models (e.g., Z-Score): • Identify sudden shifts in rankings or traffic. • Detect outliers beyond expected trends. Why It Matters • React quickly to unexpected drops or surges. • Spot potential SEO issues or competitor activity before it's too late. https://detectoutliers.com/2023/03/20/understanding-and-implem enting-z-score-for-anomaly-detection/ Wrote about CTR anomaly detection in 2016: https://searchengineland.com/alert-abnormal-organic-ctr-detected-automatic-detection-poorly-performin g-meta-data-280290
39. Forecasting Use Libraries like Facebook Prophet • Predict future ranking trends and traffic growth. • Plan SEO strategy based on data-backed projections. Impact • Forecast ranking shifts for better planning. • Adjust SEO campaigns in advance based on expected trends. https://facebook.github.io/prophet/
40. Keyword Clustering Leverage Full SERP Data • Since we have data from the top 100 search results, we have the data needed to power robust clustering Clustering Methods • Centroid Clustering, Agglomerative Clustering, BERT, Word2Vec, or hybrid techniques.
41. Scaling Infrastructure for Multi-User Support SQLite to PostgreSQL Transition: • SQLite is great for local use but not scalable for multi-user environments. • Migrate to PostgreSQL for cloud-based or shared setups. User Authentication and Team Access: • Add login systems and multi-user support for collaboration. ◦ Enable running on a server Impact: • Enable teams to use Rankenberry simultaneously, with shared access to dashboards and data.
42. Thank You Get started with rankenberry and own your own rank tracking destiny git clone https://github.com/pshapiro/rankenberry.git SpaceSERP API Lifetime Deal: searchwilderness.com/spaceserp Connect X/Twitter: @fighto LinkedIn: https:/ /www.linkedin.com/in/paulnshapiro Blog: https:/ /searchwilderness.com