Parallel Computing in the Cloud

GreenButton at Multicore World 2013

Transcript

1. Parallel Computing in the Cloud Dave Fellows (CTO) Ivan Towlson (Chief Architect)http://www.greenbutton.com/ dave@greenbutton.com ivan@greenbutton.com
3. www.greenbutton.com Parallel Architectures • Shared memory: lots of cores in one machine • HPC cluster: lots of cores with fast communication between them • “Army of Davids” – Server farm or data centre populated with commodity machines – Crowdsourcing (Folding@Home etc.)
4. www.greenbutton.com Parallel Cloud • Recruiting an army, even of Davids, is expensive and time-consuming • Then you have to provide them with pay and rations even if they’re just sitting around • Cloud benefits: – Zero lead time – Zero capital requirement – Zero costs when not in use
5. www.greenbutton.com Parallel Cloud at GreenButton • Architecture • Tools • Experiences and Lessons Learned
6. www.greenbutton.com
7. www.greenbutton.com Web Service Job submission Controller Job allocation Instance provisioning Status monitoring Processor Runs compute tasks Billing Monitoring Notification Authorisation GreenButton.com Mission Control Web app Web services (user/account) Asset streaming Apps
8. www.greenbutton.com Web Service Controller Processor Tasks placed on queue Job record written to cloud storage Requests next job Monitors job list Starts up and shuts down processor instances according to demand Tasks pulled off queue Task processor Job splitter
9. www.greenbutton.com Parallel Cloud at GreenButton • Architecture • Tools • Experiences and Lessons Learned
10. www.greenbutton.com
11. www.greenbutton.com
12. www.greenbutton.com $1360/hr
13. www.greenbutton.com Parallel Cloud at GreenButton • Architecture • Tools • Experiences and Lessons Learned
14. www.greenbutton.com Animation and Rendering • Bandwidth challenges – Data can be very large for TV/Movie quality • Environmental complexities • Licensing and dynamic provisioning – Animation software is high-value and proprietary • Security (sometimes!)
15. www.greenbutton.com Scientific Computing • MPI challenges – Inter-node bandwidth and latency – Managing dynamic nodes as a virtual cluster instead of independent actors – Managing failure • check pointing • Live migration with virtualization – Virtualization overhead?
16. www.greenbutton.com Financial Computing • There’s an awful lot of Excel in finance! – Deployment challenge – Partitioning challenges • Data sensitivity
17. www.greenbutton.com Oil & Gas
18. www.greenbutton.com
19. www.greenbutton.com Oil & Gas • Bandwidth, bandwidth and more bandwidth – 50 terabyte files defeat online upload – Scalable file system (Gluster) • Seismic imaging – visualization – Entire workflow moving to cloud
20. API / Controller VIZ Glacier
21. www.greenbutton.com Parallel Cloud Futures • High memory instances – We have lots of cores – RAM is harder to come by • Infiniband / RDMA – Virtualisation challenges • GPUs • “Bare metal cloud”
22. www.greenbutton.com Conclusion • Challenges – Bandwidth in and out of cloud – Latency between cloud nodes – Limits on node sizing – Licensing – Immediacy – spin-up time – Cloud lock-in – Governance – operational budget control
23. www.greenbutton.com Conclusion • Huge benefits for many parallel workloads • Choose candidate applications wisely • Don’t roll your own cloud solution http://www.greenbutton.com/ dave@greenbutton.com ivan@greenbutton.com