Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Beating Python's GIL to Max Out Your CPUs

Beating Python's GIL to Max Out Your CPUs

Alternative title: "Scaling Python to 3,000 Cores." Among the #1 complaints of Python in a data analysis context is the presence of the Global Interpreter Lock, or GIL. At its core, it means that a given Python program cannot easily utilize more than one core of a multi-core machine to do computation in parallel. However, fear not! To beat the GIL, you just need to be willing to adopt a little magic -- and this talk will tell you how.

Andrew Montalenti

November 09, 2015
Tweet

More Decks by Andrew Montalenti

Other Decks in Programming

Transcript

  1. What happens when you have 153 TB of compressed customer

    data that may need to be reprocessed at any time, and it’s now growing at 10-20TB per month?
  2. Is the GIL a feature, not a bug?! In one

    Python process, at any one time, only one Python bytecode instruction is executing at once.
  3. Python State Code Server 1 Core 2 Core 1 Server

    2 Core 2 Core 1 Server 3 Core 2 Core 1 from urllib.parse import urlparse urls = ["http://arstechnica.com/", "http://ars.to/1234", "http://ars.to/5678", ...]
  4. Python State Code Server 1 Core 2 Core 1 Server

    2 Core 2 Core 1 Server 3 Core 2 Core 1 map(urlparse, urls) from urllib.parse import urlparse urls = ["http://arstechnica.com/", "http://ars.to/1234", "http://ars.to/5678", ...]
  5. Python State Code Server 1 Core 2 Core 1 Server

    2 Core 2 Core 1 Server 3 Core 2 Core 1 executor = ThreadPoolExecutor() executor.map(urlparse, urls)
  6. Python State Code Server 1 Core 2 Core 1 Server

    2 Core 2 Core 1 Server 3 Core 2 Core 1 executor = ProcessPoolExecutor() executor.map(urlparse, urls) Python subprocess State Code Python subprocess State Code pickle.dumps() os.fork()
  7. Python State Code Server 1 Core 2 Core 1 Server

    2 Core 2 Core 1 Server 3 Core 2 Core 1 par = Parallel(n_jobs=2) do_urlparse = delayed(urlparse) par(do_urlparse(url) for url in urls) Python subprocess State Code Python subprocess State Code pickle.dumps() os.fork()
  8. Python State Code Server 1 Core 2 Core 1 Server

    2 Core 2 Core 1 Server 3 Core 2 Core 1 rc = Client() rc[:].map_sync(urlparse, urls) Python State Code Python State Code ipengine Python State Code Python State Code Python State Code ipengine ipengine ipcontroller Python State Code pickle.dumps() pickle.dumps()
  9. Python State Code Server 1 Core 2 Core 1 Server

    2 Core 2 Core 1 Server 3 Core 2 Core 1 consumer = ... # balanced while True: msg = consumer.consume() msg = json.loads(msg) urlparse(msg["url"]) Python State Code Python State Code Python State Code Python State Code Python State Code pykafka.producer Python State Code
  10. Python State Code Server 1 Core 2 Core 1 Server

    2 Core 2 Core 1 Server 3 Core 2 Core 1 Python State Code Python State Code Python State Code Python State Code pykafka.producer Python State Code multi-lang json protocol class UrlParser(Topology): url_spout = UrlSpout.spec(p=1) url_bolt = UrlBolt.spec(p=4, input=url_spout)
  11. Python State Code Server 1 Core 2 Core 1 Server

    2 Core 2 Core 1 Server 3 Core 2 Core 1 Python State Code Python State Code Python State Code Python State Code pyspark.SparkContext sc = SparkContext() file_rdd = sc.textFile(files) file_rdd.map(urlparse).take(1) cloudpickle py4j and binary pipes
  12. Parse.ly "Batch Layer" Topologies with Spark & S3 Parse.ly "Speed

    Layer" Topologies with Storm & Kafka Parse.ly Dashboards and APIs with Elasticsearch & Cassandra Parse.ly Raw Data Warehouse with Streaming & SQL Access Technology Component Summary