You are not alone - Open Parallel

An overview of Open Parallel's projects, from the integration of Intel's Threading Building Blocks (TBB) with Facebook's HipHop, to SKA (Square Kilometre Array radio-telescope) middleware scaling. The talk will present capabilities and track record plus current projects including functional programming in relation to Big Data and the Financial Industry

Transcript

1. You Are Not Alone
2. YOU ARE NOT ALONE • Beginning • Languages • Web • Infrastructure • Cows & Stars • Consulting • Financial Industry • Big Data & Cloud • Beginnings
3. The Humble Programmer “As long as were no machines, programming was no problem at all” “With gigantic computers, programming has become an equally gigantic problem” “The electronic industry ... has created the problem of using its products” “The increased power of hardware and the dramatic increase in its reliability, made solutions feasible that the programmer had not dared to dream a few years before...now, he HAD to transform dreams into reality!” “Is it a wonder that we found ourselves in a software crisis?”
4. “The Humble Programmer” Edsger W. Dijkstra Turing Award Lecture August 1972
5. Software changes (eats?) industries Multicore and Manycore processors allow you to do things faster + things that you would not have thought possible before 705,024 (November 2011) K Computer, built by Fujitsu (Japan) has 705,024 SPARC64 cores
6. Projects and Work Made & Managed in / from New Zealand by (many of them new) world-class New Zealanders (since 2005)
7. Clients and Partners
8. Languages New and “old”
9. COBOL original approaches to extend COBOL with parallel constructs Given that more than half of the world businesses run on lines of code written in COBOL -that are operational today, this is a strategic effort that could benefit many organisations (NZ - IRD?) Richard O'Keefe (Otago) (tomorrow 11:55 am) David Eyers (Otago) Greg Smith (Open Parallel)
10. PHP – Perl
11. Open Parallel solves PHP performance and scalability problems in large websites We take an existing PHP application and port it over to the HipHop engine. HipHop is a PHP to C++ compiler developed by Facebook and intended to large sites that run on PHP. “How to build large scale applications using PHP” - LCA2011 Lenz Gschwendtner / Sam Vilain (Open Parallel)
12. Threading Perl using TBB threads::tbb module to CPAN White paper – May 2011 YAPC::NA - June 2011 Sam Vilain (now at SAY Media)
13. We explored the new language X10 designed to support programming at scale in the multicore era (NZ eResearch Symposium – July 2011) IBM Research since 2004 + (~30 universities, VUW & ANU) - Part of DARPA's program on High Productivity Computer Systems, designed to unveil Power7's potential Applications: financial risk computing, signal processing, botnet detection Jim Cheetham (Otago) Paul McKenney, Vijay Saraswat, Michael Hinds (IBM) Vivek Sarkar (Rice)
14. We know • Web Performance Optimization bringing TBB to web application developers working on PHP and Perl – LCA2012 • OpenCV – Vision capabilities: optimisations with ARM's Ne10 framework using auto-vectorisation for the Cortex-A9 - LCA2013 • Corvid – cloud-based software, automatically generates rosters for medical staff – Tomorrow 2:00 pm • OpenSIM - Scalability and Performance analysis of OpenSimulator (3D internet) – Japan, 2008 • Lenz Gschwendtner / Zane Ashby / Remo Williams / Richard Parrat / Beau Johnston / Christian Rolf / Mariusz Nowostawski / Nigel Tucker
15. “Parallelism is not yet a paradigm, but may become so if enough people adopt it as the standard way of thinking about computation” “The Resurgence of Parallelism” (Peter Denning and Jack Dennis) Communications of the ACM – June 2010
16. Networking & multicore server optimization (so yesterday...) • UltraSPARC Accelerators for Networking Stack • Software-only networking solutions • Bare-metal networking / No OS overheads • Parallel Snort: 24x the throughput by using all your CPUs • High-throughput offload engines, Custom high-performance network solutions, High-speed inter-domain communication • Generic TCP/IP stack, Specialised RDMA protocols • Mariusz Nowostawski, Zhiyi Huang (Otago) – World45 – SUN CoE
17. What happened? “...one of the biggest hurdles I recall was the lack of interest by the applications world to re-architect to take advantage” Former SVP with operational oversight for Sun Microsystems' >$1 billion OEM systems business (personal correspondence, Feb' 2013)
18. More recently... We looked at the problems of Cluster Filesystems vs. conventional file servers (NZ eResearch Symposium – July 2012) with OpenFlow which is an enabler of SDN where scalable multicore technology is enabling energy-efficiency and performance in networking applications David Eyers (tomorrow 10:10) / Mariusz Nowostaski (Otago) Josh Bailey (Google) Ariel Hendel (Broadcom)
19. Big (and ambitious) Projects (I) SKA – algorithms, middleware • Developing and supporting massively parallel replacements for commonly used middleware is part of Open Parallel's core business. • The types of scalability changes required for SKA will need to go beyond tweaking the existent libraries • These developments will have immediate spillover effect into other areas in science and industry -in New Zealand and globally • Open Parallel plans to use the expertise developed for SKA into Big Data, Internet of Things, and other problems • Lenz Gschwendtner / Gastón Hillar / Christian Rolf (Open Parallel) • David Eyers / Mariusz Nowostawski (Otago)
20. Big (and ambitious) Projects (II) Can multicore improve NZ dairy industry? • New DNA instruments now sequence in a few days and at low cost, compared to the years of effort and billions of dollars spent to sequence the first human genome • Distributed and parallel computing advance the state of the art in bioinformatics and genomics • Ismo Stranden (Agrifood Research, Finland) - "Parallel computing applied to breeding value estimation in dairy cattle" • Bevin Harris (LIC) - “Breeding dairy cows for the future in New Zealand” • Can we improve Fonterra's $20bn revenue in 3%? • Christian Rolf (Open Parallel) / Mariusz Nowostawski (Otago)
21. Consulting • Multicore NZ: Multicore Computing Systems Business Case – NZTE, University of Otago (2008) • IT trends for NZ businesses 2010 - 2015 – University of Otago (2009) • Markets for New Technologies / Multicore Strategies – Fx2, ANII (2010 – 2011) • SKA – Multicore focus – NZTE, CSIRO – Australia (2012) • Salud.uy – EoI for eHealth system for Uruguay – AGESIC, Fx2, Orion Health (2013)
22. Financial Industry Problems where functional programming provides its best value • Prototyping complex algorithms that will run on huge amounts of data • Difficulties to scale across multiple cores, processors, servers and clusters. • Difficulties to move to the Cloud. • Performance problems, difficulties to find errors for specific cases when performing computations, applying statistical analysis, and/or processing complex rules in huge amounts of data. • Solutions? www.OpenParallel.com • Gastón Hillar (Open Parallel) / Christian Rolf (Open Parallel)
23. Big Data @ Open Parallel (I) • Bridge between business intelligence and modern business analysis • In real-time using high-performance parallel algorithms designed for the cloud • Functional programming provide the best scalability for cloud computing
24. Big Data @ Open Parallel (II) • Open Parallel ensures that big data analysis is both based on the right models and scales well. • With massively parallel execution and our business experience and access to mathematical talent, we fast-track your company into the big data age. • Christian Rolf / Pete Salerno (Open Parallel) • Mariusz Nowostawski (Otago)
25. Open Parallel contributes the framework on software and innovative management for breakthrough projects
26. You are not alone: Multicore World Community + Open Parallel Nicolás Erdödy - 19 February 2013