Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Celluloid Ecosystem - RubyConf 2012

tarcieri
November 03, 2012

The Celluloid Ecosystem - RubyConf 2012

tarcieri

November 03, 2012
Tweet

More Decks by tarcieri

Other Decks in Programming

Transcript

  1. Going Faster (The Old Way) •Clock speed grows exponentially •Faster

    clock speed = faster program! •Threads are expensive!
  2. Going Faster (The New Way) •POWER WALL! Clock speed is

    “stuck” •Number of cores grows exponentially •Threads are increasingly cheap
  3. “I thought of objects being like biological cells and/or individual

    computers on a network, only able to communicate with messages” - Alan Kay, creator of Smalltalk, on the meaning of "object oriented programming"
  4. Actor Model •Actors are computational entities that can receive messages

    •Each actor has a unique address •If you know an actor’s address, you can send it messages •Actors can create new actors
  5. require 'celluloid' class Launcher include Celluloid def launch 3.downto(1) do

    |count| puts "#{count}..." sleep 1 end puts "BLASTOFF!" end end
  6. require 'celluloid' class Launcher include Celluloid def launch 3.downto(1) do

    |count| puts "#{count}..." sleep 1 end puts "BLASTOFF!" end end
  7. &DOOHU &HOOXORLG $FWRU3UR[\ &$// 5HFHLYHU &HOOXORLG 0DLOER[ 5(63216( &HOOXORLG 0DLOER[

    &HOOXORLG$FWRU &HOOXORLG&DOO &HOOXORLG5HVSRQVH Synchronous Calls
  8. >> launcher1, launcher2 = Launcher.new, Launcher.new => [#<Celluloid::Actor(Launcher:0x3fc3f1c1c0d8)>, #<Celluloid::Actor(Launcher:0x3fc3f1c1b82c)>] >>

    launcher1.async.launch; launcher2.async.launch => nil >> 3... 3... 2... 2... 1... 1... Easy Parallelism
  9. >> launcher1, launcher2 = Launcher.new, Launcher.new => [#<Celluloid::Actor(Launcher:0x3fc3f1c1c0d8)>, #<Celluloid::Actor(Launcher:0x3fc3f1c1b82c)>] >>

    launcher1.async.launch; launcher2.async.launch => nil >> 3... 3... 2... 2... 1... 1... BLASTOFF!!! BLASTOFF!!! Easy Parallelism
  10. >> worker = FibonacciWorker.new => #<Celluloid::Actor(FibonacciWorker:0x3ffdcaed65cc)> >> future = worker.future.fib(40)

    => #<Celluloid::Future:0x007ffb9201f0a0> >> future.value Blocks until complete
  11. &DOOHU &HOOXORLG $FWRU3UR[\ 5HFHLYHU &HOOXORLG 0DLOER[ &HOOXORLG$FWRU &HOOXORLG&DOO &$// &HOOXORLG

    0DLOER[ )8785( &HOOXORLG )XWXUH &HOOXORLG5HVSRQVH 9$/8(" 9$/8( Futures
  12. >> pool = FibonacciWorker.pool => #<Celluloid::Pool(FibonacciWorker:0x3ffdcaed65cc)> >> (32...40).map { |n|

    pool.future.fib(n) }.map(&:value) => [2178309, 3524578, 5702887, 9227465, 14930352, 24157817, 39088169, 63245986]
  13. Built on 0MQ •Fast, brokerless message queue •DCell uses push/pull

    sockets •Built on Celluloid::ZMQ/ffi-rzmq
  14. •Distributed process orchestration (e.g. Chef, Puppet) •Multi-tier Web Applications •Asynchronous

    background jobs Use Cases •Distributed process orchestration (e.g. Chef, Puppet) •Multi-tier Web Applications / SOA •Asynchronous background jobs
  15. Why do this? )URQW(QG )URQW(QG )URQW(QG 5(67 &OLHQW 5(67 &OLHQW

    5(67 &OLHQW 5(67 6HUYLFH 5(67 6HUYLFH 5(67 6HUYLFH 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW
  16. Instead of this? )URQW(QG )URQW(QG )URQW(QG 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW

    'RPDLQ 2EMHFW 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW 'RPDLQ 2EMHFW
  17. "Objects can message objects transparently that live on other machines

    over the network, and you don't have to worry about the networking gunk, and you don't have to worry about finding them, and you don't have to worry about anything. It's just as if you messaged an object that's right next door." --Steve Jobs describing NeXT Portable Distributed Objects
  18. zk gem •Used by the DCell Zookeeper backend •High level

    API to Zookeeper •Abstract leader election API https://github.com/slyphon/zk
  19. $ rake zookeeper:start *** Starting Zookeeper cd zookeeper && bin/zkServer.sh

    start JMX enabled by default Using config: bin/../conf/zoo.cfg Starting zookeeper ... STARTED
  20. “itchy” commands >> require 'dcell' => true >> DCell.start :id

    => 'itchy', :addr => 'tcp:// 127.0.0.1:7777' I, [2012-05-09T10:20:46.999000 #52416] INFO -- : Connected to itchy => #<Celluloid::Supervisor(DCell::Group):0x836> Note itchy is on port 7777
  21. “scratchy” commands >> require 'dcell' => true >> DCell.start :id

    => 'scratchy', :addr => 'tcp:// 127.0.0.1:7778' I, [2012-05-09T10:26:42.322000 #52555] INFO -- : Connected to itchy I, [2012-05-09T10:26:42.331000 #52555] INFO -- : Connected to scratchy => #<Celluloid::Supervisor(DCell::Group):0x838> Note scratchy is on port 7778
  22. >> info_service = DCell::Node['itchy'][:info] => #<Celluloid::Actor(DCell::InfoService:0x83c) @platform="java" @ruby_platform="jruby 1.6.7" @cpu_type="Intel(R)

    Core(TM) i7-2635QM CPU" @ruby_version="1.9.2" @ruby_engine="jruby" @cpu_vendor=:intel @distribution="Mac OS X 10.7.3 (11D50b)" @cpu_count=8 @hostname="wintermute.local" @cpu_speed=2.0 @os="darwin" @cpu_arch="x86_64" @os_version="11.3.0"> >> info_service.uptime => 42 >> info_service.load_averages => [1.08, 0.93, 0.84] >> info_service.distribution => "Mac OS X 10.7.3 (11D50b)" “scratchy” commands
  23. Evented IO •Large numbers of connections (>10,000) •Mostly idle connections

    •Mostly IO-bound problems •Websockets are an ideal case
  24. nio4r •Quasi-inspired by Java NIO •Smallest API possible •libev C

    extension for CRuby/rbx •Java extension for JRuby •Pure Ruby version too! http://github.com/tarcieri/nio4r/
  25. Celluloid::IO::TCPSocket •Uses fibered I/O •“Duck-type” of ::TCPSocket •Evented inside Celluloid::IO

    actors •Blocking IO elsewhere (Ruby Threads, normal Celluloid actors)
  26. class EchoServer include Celluloid::IO def initialize(host, port) puts "*** Starting

    echo server on #{host}:#{port}" # Since we included Celluloid::IO, we're actually making a # Celluloid::IO::TCPServer here @server = TCPServer.new(host, port) run! end def finalize @server.close if @server end def run loop { handle_connection! @server.accept } end def handle_connection(socket) _, port, host = socket.peeraddr puts "*** Received connection from #{host}:#{port}" loop { socket.write socket.readpartial(4096) } rescue EOFError puts "*** #{host}:#{port} disconnected" socket.close end end supervisor = EchoServer.supervise("127.0.0.1", 1234) trap("INT") { supervisor.terminate; exit } sleep
  27. Hello World Benchmark # httperf --num-conns=50 --num-calls=1000 Ruby Version Throughput

    Latency ------------ ---------- ------- JRuby HEAD 5650 reqs/s (0.2 ms/req) Ruby 1.9.3 5263 reqs/s (0.2 ms/req) JRuby 1.6.7 4303 reqs/s (0.2 ms/req) rbx HEAD 2288 reqs/s (0.4 ms/req)
  28. Hello World Comparison Web Server Throughput Latency ---------- ---------- -------

    Goliath (0.9.4) 2058 reqs/s (0.5 ms/req) Thin (1.2.11) 7502 reqs/s (0.1 ms/req) Node.js (0.6.5) 11735 reqs/s (0.1 ms/req) Ruby servers on 1.9.3
  29. Goals •Built out of parts of Rails •Webmachine for resources

    •Multithreaded development mode •Easy scatter/gather for SOA