From Bottlenecks to Breakthroughs: Our Rails Horizontal Sharding Journey

In this insightful talk at Ruby Meetup Kathmandu, Prepsa Kayastha shares how we tackled database scaling challenges with horizontal sharding in Rails. Dive into real-world lessons, performance improvements, and key takeaways!

Transcript

1. From Bottlenecks to Breakthroughs: Our Rails Horizontal Sharding Journey Prepsa Kayastha 28 March, 2025 Ruby Meetup 2025
2. Background Introduction to Horizontal Sharding When to use Horizontal Sharding? Implementing Horizontal Sharding Demo Challenges / Things to consider Table of Content
3. Two Separate Environments: Staging Environment Production Environment Sync Content from Staging to Production: After finalizing contents in staging and being satisfied with the results, the client should be able to sync all or specific changes to the production environment. Types of syncing required: Individual record sync Whole model sync Batch sync Background
4. Introduction to Sharding Source: Hazelcast. What is Database Sharding? An Overview of Sharding. https://hazelcast.com/foundations/distributed-computing/sharding/
5. Introduction to Horizontal Sharding Practice of dividing data into smaller, more manageable pieces (‘shards’) and distributing them across multiple databases. Data is split by rows, the schema is same. Example 1: Shard 1: Contains users with IDs from 1 to 100,000. Shard 2: Contains users with IDs from 100,001 to 200,000. Shard 3: Contains users with IDs from 200,001 to 300,000. Source: https://architecturenotes.co/database-sharding-explained/ Example 2: Shard 1: Contains products of category ‘clothing’ Shard 2: Contains products of category ‘electronics’ Shard 3: Contains products of category ‘books’.
6. When to Use Horizontal Sharding Large datasets: When data grows too large for a single database to handle effectively, sharding distributes the load across multiple servers. High traffic websites: When high traffic or queries from users increase, horizontal sharding can distribute the load, allowing for better performance and quicker responses. Scalability: Horizontal sharding allows you to scale out by adding more servers as needed, improving the ability to handle more data and users. Custom requirements
7. Implementing Horizontal Sharding a. Configure database.yml
8. b. Connect models to shards
9. b. Connect models to shards
10. c. Switching shards (i) Output:
11. c. Switching shards (ii) Output:
12. c. Switching shards (iii) Output:
13. Scenario Recap
14. Demo
15. Challenges / Things to consider Tracking records across shards to avoid duplication Callbacks Handling delete operation
16. a. Tracking records across shards to avoid duplication How to avoid duplication while syncing? How to track which records to sync?
17. b. Handling callbacks (i) How to handle if there are callbacks?
18. b. Handling callbacks (ii) Thread-safe version
19. c.Handling delete operations How to track any record deletion in primary shard and sync in secondary shard? Use soft delete? Simultaneous delete?
20. Multiple Databases with Active Record — Ruby on Rails Guides. (n.d.). Ruby on Rails Guides. https://guides.rubyonrails.org/active_record_multiple_databases.html# horizontal-sharding Xuanchien. (2019, August 5). Skip ActiveRecord callbacks. The Lazy Log. https://thelazylog.com/skip-activerecord-callbacks/ Soma. (2024b, January 20). What is Database Sharding? Scaling Your Data Horizontally. Medium. https://medium.com/javarevisited/what- is-database-sharding-scaling-your-data-horizontally-1dc12b33193f Yusuf, M. (2022, December 12). Database Sharding explained. Architecture Notes — System Design & Software Development. https://architecturenotes.co/p/database-sharding-explained References
21. Let’s connect Thank you