Upgrade to Pro — share decks privately, control downloads, hide ads and more …

spotlight-python

Avatar for Lynn Root Lynn Root
November 06, 2014

 spotlight-python

How Spotify uses Python - presented at Python Brazil 2014

Avatar for Lynn Root

Lynn Root

November 06, 2014
Tweet

More Decks by Lynn Root

Other Decks in Programming

Transcript

  1. whoami Lynn Root Backend Engineer @ Spotify PyLadies of San

    Francisco Founder Board Member of the 
 Python Software Foundation
  2. November 3, 2014 About Spotify • Available in 58 countries

    • Launched in Brazil in May 2014 • Over 10 million paying subscribers • Over 40 million monthly active users • Over 20 million songs • Paid over $1 billion to rights holders to date • Main engineering, analytics, data & machine learning offices located in Stockholm & New York.
  3. November 3, 2014 About Spotify: In numbers • 14 TB

    of user & service-related log data per day • 170 TB of data generated in Hadoop each day • 900 node Hadoop cluster • 7,800 Hadoop jobs per day
  4. November 3, 2014 About Spotify: Our tech • Python, Java,

    C++, Protobuf, ZeroMQ • Debian, Ubuntu, Puppet, Docker & Helios • Hadoop, Cassandra, Postgres • PHP & MySQL • git, Gerrit, GHE, JIRA
  5. November 3, 2014 About Spotify: Tech culture • Agile methodology

    that works for each squad • Autonomous squads that are loosely coupled but tightly aligned • Squads, Tribes, Chapters, and Guilds.
  6. November 3, 2014 About Python • 23 years old, latest

    releases: 2.7.8 and 3.4.2 • Scripting, web dev, networking, scientific computing, database management, 2/3D animation, natural language processing, infosec, artificial intelligence (what else is there?)
  7. November 3, 2014 These are some of my favorite things

    • SimpleHTTPServer, profile, pdb • IPython & notebook • Scapy, tox, pyenv, Twisted, Sphinx + RTDs
  8. November 3, 2014 Who uses Python? • Google • Red

    Hat • Dropbox • Rackspace • Twitter • Facebook • Raspberry Pi • NASA • CERN • ITA • Yahoo! • Walt Disney • IBM • Reddit • YouTube
  9. November 3, 2014 Libraries & Packages • Python’s stdlib is

    very comprehensive • PyPI - over 51,000 packages
  10. November 3, 2014 Community • Rich ecosystem of third-party packages

    • Documentation is king • “Zen” of Python widely adopted
  11. November 3, 2014 Fast where it counts • Code reviews

    are fast • Writing less code • Startup time is instant • Quicker overall development cycle • We’re not CPU-bound, more I/O bound
  12. November 3, 2014 Where Spotify uses Python • Systems: •

    Users: account creation and login • Social graph: (followers, suggestions, etc) • Payments system • All analytics jobs • Service daemon & request handler • Build scripts, internal cloud management • Internal PyPI • DNS & service discovery • Content management (data & songs supplied by labels)
  13. November 3, 2014 What Spotify uses • Django, Pyramid, Flask

    • Gevent, Twisted, Celery • requests • SQLAlchemy • lxml • Kazoo • Sphinx, nose, python-pagerduty • dnspython
  14. How we’ve given back • Luigi & Snakebite • dh-virtualenv

    • gevent patches • Community diversity
  15. November 3, 2014 Issues Spotify has faced with Python •

    A lot of home-grown solutions • No good HTTP client - every library is missing something • memcache client doesn’t work well with our gevent setup • Debian packaging for everything.
  16. November 3, 2014 Issues Python face • The Great Debate:

    Python 2 versus 3 • The GIL • Python on Windows
  17. November 3, 2014 Looking Forward • Moving to Ubuntu -

    revisit Python 3 • Java’s popularity internally is growing since difficult to hire Python developers • Wanting to incorporate more Python into our desktop client • Looking at IPython for data visualization and analysis