IPython: tools for the lifecycle of research computing

Talk delivered at SciPy 2012 in Austin, TX.
Video: http://www.youtube.com/watch?v=iwVvqwLDsJo.

Note: the middle section of the talk consisted of interactive demos with the IPython notebook.

Transcript

1. IPython: tools for the lifecycle of research computing Fernando Pérez @fperez_org http://ipython.org SciPy 2012 July 19, 2012
2. The lifecycle of a scientific idea (the toy version) 1 Individual exploratory work 2 Collaborative development 3 Production work (HPC, cloud, parallel) 4 Publication (with reproducible results!) 5 Education 6 Goto 1. The Problem with most tools Barriers and discontinuities in worfklow in between all the steps FP (UC Berkeley) IPython 7/19/2012 2 / 24
3. The lifecycle of a scientific idea (the toy version) 1 Individual exploratory work 2 Collaborative development 3 Production work (HPC, cloud, parallel) 4 Publication (with reproducible results!) 5 Education 6 Goto 1. The Problem with most tools Barriers and discontinuities in worfklow in between all the steps FP (UC Berkeley) IPython 7/19/2012 2 / 24
4. IPython’s goal: Fluid transitions in all these steps
5. Demos
6. Important: The notebook format JSON but version control-friendly Easy for machine processing, fixable by hand if need be. Lots of hooks for metadata Produce Markdown, reST, L A TEX, HTML, etc... An open format for sharing, publishing and archiving executable computational work FP (UC Berkeley) IPython 7/19/2012 5 / 24
7. The Notebook: the tip of the spear FP (UC Berkeley) IPython 7/19/2012 6 / 24
8. The rest of the spear FP (UC Berkeley) IPython 7/19/2012 7 / 24
9. An architecture for interactive computing FP (UC Berkeley) IPython 7/19/2012 8 / 24
10. Text console with visualization
11. Qt console: inline plots, html, multiline editing, ... Evan Patterson (Enthought)
12. Browser-based notebook Brian Granger, James Gao (Berkeley), rest of the team
13. High-level parallel computing Min Ragan-Kelley, Brian Granger
14. A growing ecosystem of applications
15. Microsoft Visual Studio 2010 integrated console Dino Viehland and Shahrokh Mortazavi; http://pytools.codeplex.com
16. A vim client to control an IPython kernel/console Paul Ivanov, https://github.com/ivanov/vim-ipython
17. Notebooks on Windows Azure Cloud Shahrokh Mortazavi, B.G., F.P.: http://bit.ly/JQeojD.
18. Star Cluster: IPython parallel+Notebook on Amazon EC2 Justin Riley: http://web.mit.edu/star/cluster
19. One-click single notebook on Amazon EC2 Carl Smith: https://notebookcloud.appspot.com.
20. Canopy: Enthought’s new Python environment (EPD 8)
21. An Emacs Notebook Client! Takafumi Arakaki: http://tkf.github.com/emacs-ipython-notebook.
22. vIPer: a dedicated notebook client Damián Avila: https://github.com/damianavila/vIPer
23. (Incomplete) Cast of Characters Brian Granger - Physics, Cal State San Luis Obispo Min Ragan-Kelley - Nuclear Engineering, UC Berkeley Matthias Bussonnier - Physics, Institut Curie, Paris Jonathan March- Enthought Thomas Kluyver - Biology, U. Sheffield Jörgen Stenarson - Elect. Engineering, Sweden. Paul Ivanov - Neuroscience, UC Berkeley. Robert Kern - Enthought Evan Patterson - Physics, Caltech/Enthought Brad Froehle - Mathematics, UC Berkeley Stefan van der Walt - UC Berkeley John Hunter - TradeLink Securities, Chicago. Prabhu Ramachandran - Aerospace Engineering, IIT Bombay. Satra Ghosh- MIT Neuroscience Gaël Varoquaux - Neurospin (Orsay, France) Ville Vainio - CS, Tampere University of Technology, Finland Barry Wark - Neuroscience, U. Washington. Ondrej Certik - Physics, U Nevada Reno Darren Dale - Cornell Justin Riley - MIT Mark Voorhies - UC San Francisco Nicholas Rougier - INRIA Nancy Grand Est Thomas Spura - Fedora project Many more! (~150 commit authors)
24. Support Thank you! Enthought, Austin, TX: Lots! Microsoft: WinHPC support, Visual Studio integration, Azure (thanks to Shahrokh Mortazavi). DoD/DRC Inc: funding through Sept. 2012 (thanks to Jose Unpingco and Chris Keees). NIH: via NiPy grant NSF: via Sage compmath grant Google: summer of code 2005, 2010. Tech-X Corporation, Boulder, CO: Parallel/notebook (previous versions) We REALLY need more stable funding...
25. Support Thank you! Enthought, Austin, TX: Lots! Microsoft: WinHPC support, Visual Studio integration, Azure (thanks to Shahrokh Mortazavi). DoD/DRC Inc: funding through Sept. 2012 (thanks to Jose Unpingco and Chris Keees). NIH: via NiPy grant NSF: via Sage compmath grant Google: summer of code 2005, 2010. Tech-X Corporation, Boulder, CO: Parallel/notebook (previous versions) We REALLY need more stable funding...
26. Plug: Reproducibility in Computational and Experimental Mathematics December 10-14 2012, Brown, Rhode Island. http://icerm.brown.edu/tw12-5-rcem