Field Notes from Expeditions in the Cloud

Keynote from Cloud Expo 2014, in NYC.

Transcript

1. from Dr. Matt Wood matthew@amazon.com General Manager, Data Science a presentation by Matt Wood N E S W @mza FIELD NOTES EXPEDITIONS IN THE CLOUD
2. RETAIL 280
3. RETAIL MERCH 280
4. RETAIL MERCH AWS 280
5. 14th March 2006 S3 25th AUGUST 2006 EC2 22nd OCTOBER 2009 RDS 15th FEBRUARY 2013 Redshift 18th JANUARY 2012 DYNAMODB RETAIL MERCH AWS 280
6. 14th March 2006 S3 25th AUGUST 2006 EC2 22nd OCTOBER 2009 RDS 15th FEBRUARY 2013 Redshift 18th JANUARY 2012 DYNAMODB RETAIL MERCH AWS 2008 2009 2010 2011 2012 2013 159 82 61 48 24 280
7. 14th March 2006 S3 25th AUGUST 2006 EC2 22nd OCTOBER 2009 RDS 15th FEBRUARY 2013 Redshift 170 launches in 2014 18th JANUARY 2012 DYNAMODB RETAIL MERCH AWS 2008 2009 2010 2011 2012 2013 159 82 61 48 24 280
8. LOTS OF DATA
9. LOTS OF DATA LOTS OF USES
10. LOTS OF DATA LOTS OF USES LOTS OF USERS
11. LOTS OF DATA LOTS OF USES LOTS OF USERS LOTS OF LOCATIONS
12. LOTS OF DATA LOTS OF USES LOTS OF USERS LOTS OF LOCATIONS COMPLEXITY
13. SOLAR PANELS Notes on CAN we make A better solar panel?” “
14. SOLAR PANELS Notes on CAN we make A better solar panel?” “ Synthesize PURIFY ANALYZE PHOTO CELL?
15. SOLAR PANELS Notes on CAN we make A better solar panel?” “ Synthesize PURIFY ANALYZE PHOTO CELL? GRAD STUDENT EQUIPMENT +
16. SOLAR PANELS Notes on CAN we make A better solar panel?” “ 205,000 organic molecules Synthesize PURIFY ANALYZE PHOTO CELL? GRAD STUDENT EQUIPMENT +
18. 1.21 PFLOPS 16,788 instances
19. 1.21 PFLOPS 16,788 instances 264 years of compute time <18 hours
20. 1.21 PFLOPS 16,788 instances 264 years of compute time <18 hours $33k
21. 1.21 PFLOPS 16,788 instances 264 years of compute time <18 hours $33k COST MODELS ENABLE INNOVATION UTILIZATION
22. 1.21 PFLOPS 16,788 instances 264 years of compute time <18 hours $33k COST MODELS ENABLE INNOVATION UTILIZATION RESERVED CAPACITY
23. 1.21 PFLOPS 16,788 instances 264 years of compute time <18 hours $33k COST MODELS ENABLE INNOVATION UTILIZATION RESERVED CAPACITY ON DEMAND
24. 1.21 PFLOPS 16,788 instances 264 years of compute time <18 hours $33k COST MODELS ENABLE INNOVATION UTILIZATION RESERVED CAPACITY ON DEMAND
25. VIDEO GAMES Notes on
26. VIDEO GAMES Notes on “WHO BUYS VIDEO GAMES?”
27. VIDEO GAMES Notes on “WHO BUYS VIDEO GAMES?” 3.5 billion records 13 TB of click stream logs 71 million unique cookies C luster
28. VIDEO GAMES Notes on “WHO BUYS VIDEO GAMES?” 3.5 billion records 13 TB of click stream logs 71 million unique cookies C luster HUGE TVs
29. 500% return on AD SPEND 17000% decrease in procurement time
30. RETARGETING Notes on
31. RETARGETING Notes on HOW DO WE SERVE THE BEST POSSIBLE AD?” ”
32. RETARGETING Notes on HOW DO WE SERVE THE BEST POSSIBLE AD?” ” AD SERVERS
33. RETARGETING Notes on HOW DO WE SERVE THE BEST POSSIBLE AD?” ” AD SERVERS BIDDING ENGINE
34. RETARGETING Notes on HOW DO WE SERVE THE BEST POSSIBLE AD?” ” AD SERVERS BIDDING ENGINE DYNAMODB
35. RETARGETING Notes on HOW DO WE SERVE THE BEST POSSIBLE AD?” ” AD SERVERS BIDDING ENGINE 10ms 20MS 20MS 20MS HARD LIMIT: 100ms 10k advertisers 100 countries 1.5PB in S3 DYNAMODB
36. 100 countries
37. 100 countries
38. 100 countries HUNDREDS OF MILLIONS OF RECORDS 10ks WRITES PER SECOND in sync at WITH APACHE STORM
39. DYNAMODB THROUGHPUT AVERAGE LATENCY 11/1/2013 11/7/2013
40. DYNAMODB THROUGHPUT AVERAGE LATENCY # REQUESTS 11/1/2013 11/7/2013
41. DYNAMODB THROUGHPUT AVERAGE LATENCY # REQUESTS “THE LAW OF LARGE NUMBERS IS YOUR FRIEND UNTIL YOU REACH LARGE NUMBERS” 11/1/2013 11/7/2013
42. DYNAMODB THROUGHPUT “THE LAW OF LARGE NUMBERS IS YOUR FRIEND UNTIL YOU REACH LARGE NUMBERS” PRECISE DESCRIPTION FORMAL METHODS TEST, TEST, TEST AVERAGE LATENCY # REQUESTS 11/1/2013 11/7/2013
43. SOCIAL GAMINg Notes on
44. SOCIAL GAMINg Notes on CAN PHYSICAL AND VIRTUAL WORLDS INTERACT?” ” SOCIAL EXPERIENCE BUILT RIGHT INTO THE GAMES
45. SOCIAL GAMINg Notes on CAN PHYSICAL AND VIRTUAL WORLDS INTERACT?” ” SOCIAL EXPERIENCE BUILT RIGHT INTO THE GAMES LOW LATENCY HIGH AVAILABILITY ANALYTICS
46. JAPAN US EUROPE
47. JAPAN US EUROPE S3 CLOUDFRONT VPC
48. JAPAN US EUROPE S3 CLOUDFRONT VPC EC2 EC2 EC2 AZ1 AZ2 AZ3
49. JAPAN US EUROPE REDSHIFT S3 CLOUDFRONT VPC EC2 EC2 EC2 AZ1 AZ2 AZ3
50. GENOMICS Notes on
51. Chromosome 11 GENOMICS Notes on
52. Chromosome 11 Chromosome X GENOMICS Notes on
53. Chromosome 11 Chromosome X GENOMICS Notes on Chromosome 19
54. Chromosome 11 Chromosome X Chromosome 15 GENOMICS Notes on Chromosome 19
55. Chromosome 11 Chromosome X Chromosome 15 GENOMICS Notes on Chromosome 19
56. Chromosome 11 Chromosome X Chromosome 15 GENOMICS Notes on Chromosome 19
57. DNA SEQUENCING COSTS < $1000 PER GENOME
58. DNA SEQUENCING COSTS < $1000 PER GENOME
59. DNA SEQUENCING COSTS < $1000 PER GENOME S3 COLLABORATION and TOOLING
61. CLINICAL TRIALs Notes on
62. CLINICAL TRIALs Notes on BUSINESS DRIVERS Speeding server provisioning for R&D apps Extending capacity for internal grid environments Slowing internally hosted compute infrastructure growth On-boarding security, validation services and compliance Reducing cost while extending capabilities
63. CLINICAL TRIAL SIMULATIONS Individual Clinical Trial Simulation Run Time (Min) Total number of trials Number of servers Number of CPUS TOTAL ANALYSIS RUN TIME (hours) COST INTERNAL AWS 56 56 2000 2000 2 32 60 ??? $336 1.2 2048 256
64. REDUCED BURDEN ON pEDIATRIC SUBJECTS NUmber of subjects NUmber of blood samples per subject length of stay per subject length of study total study cost traditional DESIGN DESIGN OPTIMIZED USING CLINICAL TRIAL SIMULATION 60 40 12 5 72 hours 2.5 years $700k $250k 1.7 years 26 hours
65. CONNECTED DEVICES GENOMICS CLINICAL TRIALS GAMING MOBILE DEVELOPMENT ADVERTISING HOSPITALITY RETAIL E-COMMERCE MANUFACTURING TRAVEL FREE STEAK SOCIAL APPS FINANCIAL ANALYTICS MEDIA STREAMING DISASTER RECOVERY ARCHIVAL INTERACTIVE TV APPS WEB APPS MARS EXPLORATION
66. WHERE TO FOCUS? EARLY STAGE START-UP BILLION DOLLAR MULTINATIONAL
67. AGILITY REASON #1:
68. AGILITY EARLY STAGE START-UP REASON #1:
69. AGILITY EARLY STAGE START-UP BILLION DOLLAR MULTINATIONAL REASON #1:
70. FLY WHEELS FOR AGILITY SECURITY SCALE PERFORMANCE RELIABILITY COST
71. ADOPTION PATTERNS OF How are enterprises using the cloud in 2014?
72. DEVELOPMENT & TESTING SHAREPOINT & SAP ORACLE SAP
73. NEW WORKLOADS GLOBAL DEALS ENGINE PRODUCT PROTOTYPING MOBILE MUSIC DISCOVERY HOTEL BOOKING VIDEO STREAMING
74. SUPPLEMENT EXISTING WORKFLOWS operational applications equipment leasing analytics with the cloud
75. SUPPLEMENT EXISTING WORKFLOWS with on-premises infrastructure AWS serves up application content & data Integration back to Samsung Data Centers for financial transactions
76. MIGRATING EXISTING APPLICATIONS 500 web properties in 5 months CLINICAL TRIALS 1,700 web properties
77. ALL IN “WE HAVE TO BE GREAT AT A NUMBER OF THINGS... OPERATING DATA CENTERS IS NOT ONE OF THOSE THINGS” “WE ARE A HOSPITALITY MANAGEMENT COMPANY, NOT AN IT SERVICES COMPANY” “FRIENDS DON’T LET FRIENDS BUILD DATA CENTERS” INFOR CLOUDSUITE micro-vertical BUSINESS APPS
78. IN SUMMARY... DATA DRIVES INNOVATION WHEN COMBINED WITH UTILITY COMPUTING Broad and deep services enable new business AGILITY REMAINS THE #1 REASON FOR ADOPTING THE AWS CLOUD ENTERPRISES LEAD WITh DEV, TEST & NEW WORKLOADS MIGRATION AND SUPPLEMENTS QUICKLY FOLLOW Increasingly ‘ALL IN’
79. @mza MATTHEW@AMAZON.COM THANK YOU
80. More sessions from AWS today 11.00am: Transforming Applications into SaaS 11.40am: Big Data Is Best Served Fresh: Real-time Analytics in the Cloud 1.55pm: What is the Internet of Things – and how can it help my business? 4.00pm: Moving Enterprises to the Cloud the Right Way and for the Right Reasons 4.40pm: Creating a Cloud Based DR Solution for your Enterprise