Scale the Google Way

A discussion of issues with Google scaling technologies

Transcript

1. @tpryan Scale the Google Way Terry Ryan Developer Advocate
2. @tpryan Who are you?
10. @tpryan First Cluster History
11. @tpryan First Cluster •Every machine had a role History
12. @tpryan First Cluster •Every machine had a role •Every machine had a name History
13. @tpryan First Rack History
14. @tpryan First Rack •Machines belong to groups History
15. @tpryan First Rack •Machines belong to groups •Slack capacity goes to waste History
16. November 1998 December 1998 January 1999 February 1999 March 1999 April 1999
17. November 1998 December 1998 January 1999 February 1999 March 1999 April 1999 December 1998 50,000
18. November 1998 December 1998 January 1999 February 1999 March 1999 April 1999 December 1998 50,000 January 1999 150,000
19. November 1998 December 1998 January 1999 February 1999 March 1999 April 1999 December 1998 50,000 January 1999 150,000 April 1999 500,000
20. @tpryan Today’s Numbers History
21. @tpryan Today’s Numbers •Over 3 Billion searches a day History
22. @tpryan Today’s Numbers •Over 3 Billion searches a day •On a universe of over 60 Trillion urls. History
23. @tpryan Today’s Numbers •Over 3 Billion searches a day •On a universe of over 60 Trillion urls. •We index over 20 billion a day History
24. @tpryan Today’s Numbers •Over 3 Billion searches a day •On a universe of over 60 Trillion urls. •We index over 20 billion a day •Average time to deliver under 250ms History
25. @tpryan Handling this growth took fundamental changes
26. Networking
27. 70 POP’s 33 Countries B4 WAN
28. Jupiter Superblock
29. Jupiter Superblock • 1 Petabit/sec
30. Jupiter Superblock • 1 Petabit/sec • 100,000 Servers at 10GB/s
31. Jupiter Superblock • 1 Petabit/sec • 100,000 Servers at 10GB/s • 10 Libraries of Congress a second
32. Storage
33. Colossus Datacenter wide storage
34. Processing
35. Productionized Containers
36. Productionized Containers • Developed as only practical way to manage Google-scale compute
37. Productionized Containers • Developed as only practical way to manage Google-scale compute • Everything at Google runs in a container
38. Productionized Containers • Developed as only practical way to manage Google-scale compute • Everything at Google runs in a container • We launch over 2 Billion containers per week.
39. @tpryan What does this mean for you?
40. @tpryan
41. @tpryan Big Table
42. @tpryan Big Table
43. @tpryan Big Table MapReduce
44. @tpryan Big Table MapReduce
45. @tpryan Big Table Dremel MapReduce
46. @tpryan Big Table Dremel MapReduce
47. @tpryan Big Table Borg Dremel MapReduce
48. @tpryan Big Table Borg Dremel MapReduce
49. @tpryan
50. @tpryan Google Cloud Platform
51. @tpryan Google Cloud Platform Compute
52. @tpryan Google Cloud Platform Compute Storage
53. @tpryan Google Cloud Platform Compute Connectivity Storage
54. @tpryan Google Cloud Platform Compute Connectivity Big Data Storage
55. @tpryan Google Cloud Platform Compute Connectivity Developer Tools Mobile Big Data Storage Management
56. @tpryan Conclusions
59. Your App Here
60. @tpryan Thank You! terrenceryan.com @tpryan This preso: http://bit.ly/tpryan-scale This work is licensed under a Creative Commons Attribution 2.0 Generic License.