IPython & Project Jupyter: A language-independent architecture for open computing and data science

An overall view of the current state of the IPython/Jupyter project, its vision and future directions.

The video of the live presentation is available here: https://sea.ucar.edu/event/project-jupyter-language-independent-architecture-open-computing-and-data-science

PDF slides for my keynote at the NCAR SEA 2015 conference.

Companion notebooks for demo are hosted here: https://github.com/fperez/talk-1504-boulder

Transcript

1. Fernando Pérez (@fperez_org & fperez@lbl.gov) LBL & UC Berkeley A language-independent architecture for open computing and data science IPython & Project Jupyter
2. –Hamming'62 “The purpose of computing is insight, not numbers”
3. The Lifecycle of a Scientific Idea (schematically) 1. Individual exploratory work 2. Collaborative development 3. Parallel production runs (HPC, cloud, ...) 4. Publication & communication (reproducibly!) 5. Education 6. Goto 1.
4. IPython: CU Boulder, 2001 or how to best procrastinate on a Physics dissertation
5. November 2001: "Just an afternoon hack" ❖ 259 Line Python script. ❖ sys.ps1 -> In [N]. ❖ sys.displayhook -> Out[N], caches results. ❖ Plotting, Numeric, etc. Now (Openhub stats) ❖ 19,279 commits ❖ 442 contributors ❖ Total Lines: 187,326 ❖ Number of Languages : 7 (JS, CSS, HTML, ...)
6. Real credit goes to whole team Plus ~ 500 more Open source contributors!
7. Current funding
8. Beyond the Terminal… ❖ The REPL as a network protocol ❖ Kernels ❖ execute code ❖ Clients ❖ Read input ❖ Present output Simple abstractions enable rich, sophisticated clients
9. 2011: The IPython Notebook ❖ Rich web client ❖ Text & math ❖ Code ❖ Results ❖ Share, reproduce.
10. The Notebook: “Literate Computing” Computational Narratives ❖ Computers deal with code and data. ❖ Humans deal with narratives that communicate. Literate Computing (not Literate Programming) narratives anchored in a live computation, that communicate a story based on data and results. Cf: Mathematica, Maple, MuPad, Sage…
11. Demo: IPython Notebook
12. From IPython to Project Jupyter
13. A simple and generic architecture
14. Not just about Python: Kernels in any language ❖ IPython "Official", we ship it. ❖ IJulia ❖ IRKernel ❖ IHaskell ❖ IFSharp ❖ Ruby ❖ IScala ❖ IErlang ❖ Lots more! ~37 and counting
15. “Why is it called IPython, if it can do Julia, R, Haskell, Ruby, … ?”
16. IPython ❖ Interactive Python shell at the terminal ❖ Kernel for this protocol in Python ❖ Tools for Interactive Parallel computing ❖ Network protocol for interactive computing ❖ Clients for protocol ❖ Console ❖ Qt Console ❖ Notebook ❖ Notebook file format & tools (nbconvert...) ❖ Nbviewer
17. IPython … Jupyter ❖ Interactive Python shell at the terminal ❖ Kernel for this protocol in Python ❖ Tools for Interactive Parallel computing ❖ Network protocol for interactive computing ❖ Clients for protocol ❖ Console ❖ Qt Console ❖ Notebook ❖ Notebook file format & tools (nbconvert...) ❖ Nbviewer Language Agnostic
18. What’s in a name? ❖ Inspired by the open languages of science: ❖ Julia, Python & R ❖ not an acronym: all languages equal class citizens. ❖ Astronomy and Scientific Python: ❖ A long and fruitful collaboration ❖ Galileo's notebooks: ❖ the original, open science, data-and-narrative papers ❖ Authorea: “Science was Always meant to be Open”
19. Demo: Jupyter Notebooks
20. The Jupyter Notebook Ecosystem
21. nbviewer: seamless notebook sharing ❖ Zero-install reading of notebooks ❖ Just share a URL ❖ nbviewer.ipython.org
22. Reproducible Research
23. Paper, Notebooks and Virtual Machine
24. Scientific Blogging Jake van der Plas @ UW http://blogs.scientificamerican.com/ sa-visual/2014/09/16/visualizing-4- dimensional-asteroids
25. Executable books ❖ Springer hardcover book ❖ Chapters: IPython Notebooks ❖ Posted as a blog entry ❖ All available as a Github repo Python for Signal Processing, by José Unpingco
26. More authors creating books this way By Cameron Davidson-Pilon By Matthew Russell
27. University Courses These are just some we are aware of!
28. A collaborative MOOC on OpenEdX http://lorenabarba.com/news/announcing-practical-numerical-methods-with-python-mooc ❖ Lorena Barba at George Washington University, USA. ❖ Ian Hawke at Southampton, UK ❖ Carlos Jerez at Pontifical Catholic University of Chile. ❖ All materials on Gihtub.
29. Books about IPython IPython Interactive Computing and Visualization Cookbook Learning IPython for Interactive Computing and Data Visualization Cyrille Rossant cyrille.rossant.net
30. Changing the scientific culture http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261
31. Executable papers: the future? http://www.nature.com/news/ipython-interactive-demo-7.21492?article=1.16261
32. Notebook Workflows: The Big Picture Image credit: Joshua Barratt
33. Lots more! The IPython Gallery https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks
34. Jupyter as Infrastructure OSS and commercial adoption
35. Microsoft: Python Tools for Visual Studio Shahrokh Mortazavi, Dino Viehland, Wenming Ye, Dennis Gannon.
36. Microsoft Azure: Notebooks in the Cloud
37. Google CoLaboratory Kayur Patel, Kester Tong, Mark Sanders, Corinna Cortes @ Google Matt Turk @ NCSA/UIUC
38. IBM Watson
39. Quantopian: algorithmic trading Karen Rubin Dir. Product Management at Quantopian Quantopian Research Post Fortune.com
40. Authorea: notebooks in papers https://www.authorea.com/users/3/articles/3904/_show_article
41. New directions
42. Full-page text editor
43. In-browser terminal (real-time sync)
44. Google CoLab: next steps ❖ Google Research funding a postdoc @ Berkeley. Thanks! ❖ Integrate real-time collaboration into Jupyter architecture. ❖ First, supported on Google Drive. ❖ Then, generalize, support other real-time backends. Matthias Bussonnier @ Berkeley Kester Tong @ Google
45. JupyterHub: multiuser support ❖ Out of the box ❖ Unix accounts ❖ Local single-user notebooks ❖ Customizable ❖ Authentication: OAuth, LDAP, etc. ❖ Subprocess control: Docker, VMs, etc.
46. JupyterHub in Education @ Berkeley https://developer.rackspace.com/blog/deploying-jupyterhub-for-education ❖ Computationally intensive course, ~220 students ❖ Fully hosted environment, zero-install ❖ Homework management and grading (w B. Granger) Jess Hamrick @ Cal K. Kelley Rackspace M. Ragan-Kelley Cal B. Granger Cal Poly
47. Thank You @fperez_org fperez@lbl.gov @ProjectJupyter @IPythonDev Try it out at try.jupyter.org