Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Python's Data Science Stack (JSM 2016)

Python's Data Science Stack (JSM 2016)

The Python language was not originally designed with scientific computing in mind, but its beauty and ease-of-use have inspired the development of a powerful and mature ecosystem of scientific and data-focused computing tools. This talk will give a broad introduction to the essential tools for data analysis and visualization in Python, as well as a look at recent developments and new tools on the horizon – including Altair, the new declarative visualization library for Python built on Vega-Lite.

Jake VanderPlas

July 31, 2016
Tweet

More Decks by Jake VanderPlas

Other Decks in Programming

Transcript

  1. #JSM2016
    Jake VanderPlas
    Python’s Data
    Science Stack
    Jake VanderPlas @jakevdp
    JSM, July 31, 2016

    View full-size slide

  2. #JSM2016
    Jake VanderPlas
    Python is not a statistical
    computing language!

    View full-size slide

  3. #JSM2016
    Jake VanderPlas
    Python is not a statistical
    computing language!
    . . . and this may be its
    greatest strength as a
    language for statistical
    computing.

    View full-size slide

  4. #JSM2016
    Jake VanderPlas
    A Quick Tour of Python’s
    Data Science Stack

    View full-size slide

  5. #JSM2016
    Jake VanderPlas
    Python’s Data Science Stack

    View full-size slide

  6. #JSM2016
    Jake VanderPlas

    View full-size slide

  7. #JSM2016
    Jake VanderPlas
    NumPy = Numerical Python
    Efficient array storage, manipulation, and computation

    View full-size slide

  8. #JSM2016
    Jake VanderPlas
    NumPy = Numerical Python
    Efficient array storage, manipulation, and computation

    View full-size slide

  9. #JSM2016
    Jake VanderPlas

    View full-size slide

  10. #JSM2016
    Jake VanderPlas
    Cython = C + Python
    Super-set of the Python language that allows easy
    interfacing with C & Fortran libraries (e.g. BLAS,
    LAPACK, etc.) and also fast Python code.
    Drives many of the packages in the data science stack.

    View full-size slide

  11. #JSM2016
    Jake VanderPlas

    View full-size slide

  12. #JSM2016
    Jake VanderPlas
    IPython / Jupyter
    Terminal, development environment, Notebooks, and
    more for efficient use of Python in day-to-day work

    View full-size slide

  13. #JSM2016
    Jake VanderPlas
    IPython / Jupyter
    Terminal, development environment, Notebooks, and
    more for efficient use of Python in day-to-day work

    View full-size slide

  14. #JSM2016
    Jake VanderPlas

    View full-size slide

  15. #JSM2016
    Jake VanderPlas

    View full-size slide

  16. #JSM2016
    Jake VanderPlas
    SciPy
    Provides an interface to common scientific computing
    Tasks, including wrappers of many NetLib packages.

    View full-size slide

  17. #JSM2016
    Jake VanderPlas
    SciPy
    Provides an interface to common scientific computing
    Tasks, including wrappers of many NetLib packages.
    List from http://docs.scipy.org/doc/scipy/reference/
    ● Special functions (scipy.special)
    ● Integration (scipy.integrate)
    ● Optimization (scipy.optimize)
    ● Interpolation (scipy.interpolate)
    ● Fourier Transforms (scipy.fftpack)
    ● Signal Processing (scipy.signal)
    ● Linear Algebra (scipy.linalg)
    ● Sparse Eigenvalue Problems with ARPACK
    ● Compressed Sparse Graph Routines (scipy.sparse.csgraph)
    ● Spatial data structures and algorithms (scipy.spatial)
    ● Statistics (scipy.stats)
    ● Multidimensional image processing (scipy.ndimage)
    ● File IO (scipy.io)

    View full-size slide

  18. #JSM2016
    Jake VanderPlas

    View full-size slide

  19. #JSM2016
    Jake VanderPlas
    Sympy
    Library for symbolic computation: algebraic operations,
    differentiation & integration, optimization, etc.

    View full-size slide

  20. #JSM2016
    Jake VanderPlas
    Sympy
    Library for symbolic computation: algebraic operations,
    differentiation & integration, optimization, etc.

    View full-size slide

  21. #JSM2016
    Jake VanderPlas

    View full-size slide

  22. #JSM2016
    Jake VanderPlas
    matplotlib
    Matlab-inspired plotting and visualization

    View full-size slide

  23. #JSM2016
    Jake VanderPlas
    matplotlib
    Matlab-inspired plotting and visualization
    From http://matplotlib.org/gallery.html

    View full-size slide

  24. #JSM2016
    Jake VanderPlas

    View full-size slide

  25. #JSM2016
    Jake VanderPlas
    Pandas
    R-inspired DataFrames & associated functionality
    (data munging & cleaning, group-by & transformations,
    and much more)

    View full-size slide

  26. #JSM2016
    Jake VanderPlas
    Pandas
    R-inspired DataFrames & associated functionality
    (data munging & cleaning, group-by & transformations,
    and much more)

    View full-size slide

  27. #JSM2016
    Jake VanderPlas

    View full-size slide

  28. #JSM2016
    Jake VanderPlas

    View full-size slide

  29. #JSM2016
    Jake VanderPlas
    Scikit-Learn
    Machine Learning in Python, built on NumPy and SciPy

    View full-size slide

  30. #JSM2016
    Jake VanderPlas
    Scikit-Learn
    Machine Learning in Python, built on NumPy and SciPy

    View full-size slide

  31. #JSM2016
    Jake VanderPlas

    View full-size slide

  32. #JSM2016
    Jake VanderPlas
    Python’s Scientific Ecosystem (and
    many,
    many
    more)

    View full-size slide

  33. #JSM2016
    Jake VanderPlas
    Recent-ish Developments
    - Dask: Parallelization of Data & Computation
    - Numba: LLVM compilation of Python code
    - Jupyter Lab: interactive & extensible polyglot
    development environment
    - Altair: Declarative Visualization based on
    Vega-Lite

    View full-size slide

  34. #JSM2016
    Jake VanderPlas
    Dask: Parallel Computation for
    Distributed Arrays & DataFrames
    With minimal changes to your NumPy & Pandas
    expressions, parallelize your computations over
    distributed data!
    http://dask.pydata.org/

    View full-size slide

  35. #JSM2016
    Jake VanderPlas
    http://dask.pydata.org/
    Dask: Parallel Computation for
    Distributed Arrays & DataFrames
    A straightforward NumPy computation:

    View full-size slide

  36. #JSM2016
    Jake VanderPlas
    http://dask.pydata.org/
    Dask: Parallel Computation for
    Distributed Arrays & DataFrames
    Dask uses the same expressions . . .

    View full-size slide

  37. #JSM2016
    Jake VanderPlas
    http://dask.pydata.org/
    Dask: Parallel Computation for
    Distributed Arrays & DataFrames
    With minimal changes to your NumPy & Pandas
    expressions, parallelize your computations over
    distributed data!
    “Task Graph”

    View full-size slide

  38. #JSM2016
    Jake VanderPlas
    http://dask.pydata.org/
    Dask: Parallel Computation for
    Distributed Arrays & DataFrames

    View full-size slide

  39. #JSM2016
    Jake VanderPlas
    With a simple decorator, Python is compiled to
    LLVM and executes at near C/Fortran speed!
    http://numba.pydata.org/
    Still some features missing, but very promising
    (see my blog posts for some examples).
    Numba: JIT-compilation of
    Python code

    View full-size slide

  40. #JSM2016
    Jake VanderPlas
    Numba: JIT-compilation of
    Python code
    With a simple decorator, Python is compiled to
    LLVM and executes at near C/Fortran speed!
    http://numba.pydata.org/
    Still some features missing, but very promising
    (see my blog posts for some examples).
    20x speedup!

    View full-size slide

  41. #JSM2016
    Jake VanderPlas
    Jupyter Lab
    Jupyter beyond notebooks: extensible cross-platform
    interactive computing environment (release soon!)
    http://jupyter.org
    Link to Animation

    View full-size slide

  42. #JSM2016
    Jake VanderPlas
    Altair: Declarative Visualization
    based on Vega-Lite

    View full-size slide

  43. #JSM2016
    Jake VanderPlas
    The Visualization story in Python is
    somewhat confusing . . .
    - Matplotlib
    - Bokeh
    - Plotly
    - Seaborn
    - Holoviews
    - VisPy
    - ggplot
    - pandas plot
    - Lightning
    Each library has strengths, but
    arguably none is yet the “killer
    viz app” for Data Science.

    View full-size slide

  44. #JSM2016
    Jake VanderPlas
    Most Useful for Data Science is
    Declarative Visualization
    Declarative
    - Specify What should be
    done
    - Details determined
    automatically
    - Separates Specification
    from Execution
    Imperative
    - Specify How something
    should be done.
    - Must manually specify
    plotting steps
    - Specification &
    Execution intertwined.
    Declarative visualization lets you think about data
    and relationships, rather than incidental details.

    View full-size slide

  45. #JSM2016
    Jake VanderPlas
    Enter Altair.
    Declarative statistical visualization library for Python,
    driven by Vega-Lite
    http://github.com/ellisonbg/altair
    Collaboration with Brian Granger (Jupyter team), myself,
    and University of Washington’s Interactive Data Lab

    View full-size slide

  46. #JSM2016
    Jake VanderPlas
    Example: Cars Dataset

    View full-size slide

  47. #JSM2016
    Jake VanderPlas
    Matplotlib is an imperative API:

    View full-size slide

  48. #JSM2016
    Jake VanderPlas
    Altair is a declarative API:
    http://github.com/ellisonbg/altair

    View full-size slide

  49. #JSM2016
    Jake VanderPlas
    Altair is a declarative API:
    Altair itself contains no renderers,
    but simply outputs a Vega-Lite
    visualization specification:
    - Portable JSON serialization (Vega-Lite spec)
    - Interest from other viz libraries (matplotlib,
    Bokeh, Plotly) in supporting this serialization.
    - Potential for cross-language compatibility
    http://github.com/ellisonbg/altair

    View full-size slide

  50. #JSM2016
    Jake VanderPlas
    Vega-Lite schema is well-defined; allows
    round-trip between spec and code:

    View full-size slide

  51. #JSM2016
    Jake VanderPlas
    Altair/Vega-Lite supports many plot types:

    View full-size slide

  52. #JSM2016
    Jake VanderPlas
    Altair/Vega-Lite supports many plot types:

    View full-size slide

  53. #JSM2016
    Jake VanderPlas
    Altair/Vega-Lite supports many plot types:

    View full-size slide

  54. #JSM2016
    Jake VanderPlas
    Altair/Vega-Lite supports many plot types:

    View full-size slide

  55. #JSM2016
    Jake VanderPlas
    Altair/Vega-Lite supports many plot types:

    View full-size slide

  56. #JSM2016
    Jake VanderPlas
    Altair/Vega-Lite supports many plot types:

    View full-size slide

  57. #JSM2016
    Jake VanderPlas
    or
    $ conda install altair --channel conda-forge
    $ pip install altair
    $ jupyter nbextension install --sys-prefix --py vega
    Try Altair:
    http://github.com/ellisonbg/altair/
    For a Jupyter notebook tutorial, type
    import altair
    altair.tutorial()

    View full-size slide

  58. #JSM2016
    Jake VanderPlas
    Email: [email protected]
    Twitter: @jakevdp
    Github: jakevdp
    Web: http://vanderplas.com
    Blog: http://jakevdp.github.io
    Thank You!

    View full-size slide

  59. #JSM2016
    Jake VanderPlas

    View full-size slide

  60. #JSM2016
    Jake VanderPlas
    Bar Chart: d3
    var margin = {top: 20, right: 20, bottom: 30, left: 40},
    width = 960 - margin.left - margin.right,
    height = 500 - margin.top - margin.bottom;
    var x = d3.scale.ordinal()
    .rangeRoundBands([0, width], .1);
    var y = d3.scale.linear()
    .range([height, 0]);
    var xAxis = d3.svg.axis()
    .scale(x)
    .orient("bottom");
    var yAxis = d3.svg.axis()
    .scale(y)
    .orient("left")
    .ticks(10, "%");
    var svg = d3.select("body").append("svg")
    .attr("width", width + margin.left + margin.right)
    .attr("height", height + margin.top + margin.bottom)
    .append("g")
    .attr("transform", "translate(" + margin.left + "," + margin.top +
    ")");
    d3.tsv("data.tsv", type, function(error, data) {
    if (error) throw error;
    x.domain(data.map(function(d) { return d.letter; }));
    y.domain([0, d3.max(data, function(d) { return d.frequency; })]);
    svg.append("g")
    .attr("class", "x axis")
    .attr("transform", "translate(0," + height + ")")
    .call(xAxis);
    svg.append("g")
    .attr("class", "y axis")
    .call(yAxis)
    .append("text")
    .attr("transform", "rotate(-90)")
    .attr("y", 6)
    .attr("dy", ".71em")
    .style("text-anchor", "end")
    .text("Frequency");
    svg.selectAll(".bar")
    .data(data)
    .enter().append("rect")
    .attr("class", "bar")
    .attr("x", function(d) { return x(d.letter); })
    .attr("width", x.rangeBand())
    .attr("y", function(d) { return y(d.frequency); })
    .attr("height", function(d) { return height - y(d.frequency);
    });
    });
    function type(d) {
    d.frequency = +d.frequency;
    return d;
    }

    View full-size slide

  61. #JSM2016
    Jake VanderPlas
    Bar Chart: Vega
    {
    "width": 400,
    "height": 200,
    "padding": {"top": 10, "left": 30, "bottom": 30, "right": 10},
    "data": [
    {
    "name": "table",
    "values": [
    {"x": 1, "y": 28}, {"x": 2, "y": 55},
    {"x": 3, "y": 43}, {"x": 4, "y": 91},
    {"x": 5, "y": 81}, {"x": 6, "y": 53},
    {"x": 7, "y": 19}, {"x": 8, "y": 87},
    {"x": 9, "y": 52}, {"x": 10, "y": 48},
    {"x": 11, "y": 24}, {"x": 12, "y": 49},
    {"x": 13, "y": 87}, {"x": 14, "y": 66},
    {"x": 15, "y": 17}, {"x": 16, "y": 27},
    {"x": 17, "y": 68}, {"x": 18, "y": 16},
    {"x": 19, "y": 49}, {"x": 20, "y": 15}
    ]
    }
    ],
    "scales": [
    {
    "name": "x",
    "type": "ordinal",
    "range": "width",
    "domain": {"data": "table", "field": "x"}
    },
    {
    "name": "y",
    "type": "linear",
    "range": "height",
    "domain": {"data": "table", "field": "y"},
    "nice": true
    }
    ],
    "axes": [
    {"type": "x", "scale": "x"},
    {"type": "y", "scale": "y"}
    ],
    "marks": [
    {
    "type": "rect",
    "from": {"data": "table"},
    "properties": {
    "enter": {
    "x": {"scale": "x", "field": "x"},
    "width": {"scale": "x", "band": true, "offset": -1},
    "y": {"scale": "y", "field": "y"},
    "y2": {"scale": "y", "value": 0}
    },
    "update": {
    "fill": {"value": "steelblue"}

    View full-size slide

  62. #JSM2016
    Jake VanderPlas
    Bar Chart: Vega-Lite
    {
    "description": "A simple bar chart with embedded data.",
    "data": {
    "values": [
    {"a": "A","b": 28}, {"a": "B","b": 55}, {"a": "C","b": 43},
    {"a": "D","b": 91}, {"a": "E","b": 81}, {"a": "F","b": 53},
    {"a": "G","b": 19}, {"a": "H","b": 87}, {"a": "I","b": 52}
    ]
    },
    "mark": "bar",
    "encoding": {
    "x": {"field": "a", "type": "ordinal"},
    "y": {"field": "b", "type": "quantitative"}
    }
    }

    View full-size slide