Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ruby loves Unix: Applying beautiful Unix idioms...

Ruby loves Unix: Applying beautiful Unix idioms to build a Ruby prefork server

The Unix programming model has existed for more than 40 years now. Unix programming techniques and principles have transcended programming languages and frameworks du jour. Also, Ruby was built for unix hacking. Implementing unix system calls in its standard library was the best perlism Ruby adopted.

Learning Unix programming and its idioms will make you a better Ruby programmer. Your ruby programs will be friendly citizens on your *nix server making them easy to administer. You will be able to daemonize your code at will. Your concurrent ruby code will be easy to grok and debug. You will have deeper understanding into the inner working of popular tools like the Resque job scheduler and
the Unicorn web server.

This talk is divided into two parts. The first part will be a dive into the basic building block of the unix programming model: processes. We will learn the application of the myriad unix system calls such as kill(2), fork(2), execve(2), pipe(2) and friends to our processes and understand various important idioms.

The next part will involve using the knowledge we've gained to build a Ruby preforking server. Our daemon will follow the classical master/worker prefork model wherein the master will fork and monitor its children and use the operating system to load balance work amongst available children. Our daemon will demonstrate classical unix idioms like daemonizing via double fork(2), administration via signals, IPC via signals and pipes,
concurrency via fork(2) and many more.

Sahil Muthoo

June 23, 2013
Tweet

More Decks by Sahil Muthoo

Other Decks in Programming

Transcript

  1. Processes have parents % irb >> Process.pid => 705 >>

    Process.ppid => 99930 Process #705 Parent process #99930
  2. Process lineage % pstree -s irb -+= 00001 root /sbin/launchd

    \-+= 00136 sahilm /sbin/launchd \-+= 08735 sahilm /Applications/iTerm.app \-+= 08739 sahilm -/bin/bash \--= 08830 sahilm irb
  3. Processes have friendly names % irb >> $PROGRAM_NAME => "irb"

    >> $PROGRAM_NAME="Ponies!" => "Ponies!" Process #705 Parent process #99930 Name irb Ponies!
  4. Process names are useful % bin/kaanta [kaanta master (PID: 4402)]

    INFO -- Listening on 0.0.0.0: 8080 [kaanta master (PID: 4402)] INFO -- Spawning 3 workers [kaanta worker 0 (PID: 4429)] INFO -- up [kaanta worker 1 (PID: 4430)] INFO -- up [kaanta worker 2 (PID: 4431)] INFO -- up
  5. Process names are useful % bin/kaanta [kaanta master (PID: 4402)]

    INFO -- Listening on 0.0.0.0: 8080 [kaanta master (PID: 4402)] INFO -- Spawning 3 workers [kaanta worker 0 (PID: 4429)] INFO -- up [kaanta worker 1 (PID: 4430)] INFO -- up [kaanta worker 2 (PID: 4431)] INFO -- up
  6. Process names are useful % bin/kaanta [kaanta master (PID: 4402)]

    INFO -- Listening on 0.0.0.0: 8080 [kaanta master (PID: 4402)] INFO -- Spawning 3 workers [kaanta worker 0 (PID: 4429)] INFO -- up [kaanta worker 1 (PID: 4430)] INFO -- up [kaanta worker 2 (PID: 4431)] INFO -- up
  7. Processes have resources % irb >> STDIN.fileno => 0 >>

    STDOUT.fileno => 1 >> STDERR.fileno => 2 Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2
  8. file == resource == file % irb >> socket =

    TCPServer.open => #<TCPServer:fd 3> >> socket.fileno => 3 Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2 fd3
  9. Processes can fork % irb >> fork { puts "Oh

    hai!" } => 5306 Oh hai! Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2 fd3
  10. % irb >> fork { puts "Oh hai!" } =>

    5306 Oh hai! Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2 fd3 => Oh hai! Process #5306 Parent process #705 Name Ponies! Resources fd0-2 fd3
  11. % irb >> fork { puts "Oh hai!" } =>

    5306 Oh hai! Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2 fd3 => Oh hai! Process #5306 Parent process #705 Name Ponies! Resources fd0-2 fd3
  12. fork returns twice if fork puts "forked" else puts "didn't

    fork" end => forked => didn't fork Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2 fd3
  13. fork returns twice puts "Parent: #{Process.pid}" if fork puts "#{Process.pid}

    forked" else puts "#{Process.pid} didn't fork" end => Parent 705 => 705 forked => 5306 didn't fork Process #705 Parent process #99930 Name irb Ponies! Resources fd0-2 fd3
  14. homework = Tempfile.new('') pid = fork do homework.write "5 *

    5 = 25" end if File.zero?(homework) puts "Y U NO HOMEWORK!" else puts "Have a cookie" end
  15. homework = Tempfile.new('') pid = fork do homework.write "5 *

    5 = 25" end if File.zero?(homework) puts "Y U NO HOMEWORK!" else puts "Have a cookie" end Process.waitpid(pid)
  16. Processes have exit statuses 3.times do fork {exit([0,1].sample)} end 3.times

    do pid, status = Process.wait2 if status.exitstatus == 1 puts "#{pid} failed!" end end
  17. wait2 returns status 3.times do fork {exit([0,1].sample)} end 3.times do

    pid, status = Process.wait2 if status.exitstatus == 1 puts "#{pid} failed!" end end
  18. Nonzero exit statuses indicate errors 3.times do fork {exit([0,1].sample)} end

    3.times do pid, status = Process.wait2 if status.exitstatus == 1 puts "#{pid} failed!" end end
  19. % bin/kaanta [kaanta master (PID: 13685)] INFO -- Listening on

    0.0.0.0: 8080 [kaanta master (PID: 13685)] INFO -- Spawning 3 workers [kaanta worker 0 (PID: 13712)] INFO -- up [kaanta worker 1 (PID: 13713)] INFO -- up [kaanta worker 2 (PID: 13714)] INFO -- up [kaanta master (PID: 13685)] INFO -- reaped worker 0 (PID:13712) status: 0 [kaanta worker 0 (PID: 13725)] INFO -- up % kill 13712
  20. % bin/kaanta [kaanta master (PID: 13685)] INFO -- Listening on

    0.0.0.0: 8080 [kaanta master (PID: 13685)] INFO -- Spawning 3 workers [kaanta worker 0 (PID: 13712)] INFO -- up [kaanta worker 1 (PID: 13713)] INFO -- up [kaanta worker 2 (PID: 13714)] INFO -- up [kaanta master (PID: 13685)] INFO -- reaped worker 0 (PID:13712) status: 0 [kaanta worker 0 (PID: 13725)] INFO -- up % kill 13712
  21. Processes can receive signals % irb >> Process.pid => 14576

    >> Terminated: 15 % % kill -SIGTERM 14576
  22. kill(2) isn't only for killing puts Process.pid i=0 loop do

    print i+=1 sleep 1 end % ruby kill.rb 15068 1234... % kill -SIGSTOP 15068 [2]+ Stopped ruby kill.rb
  23. kill(2) isn't only for killing puts Process.pid i=0 loop do

    print i+=1 sleep 1 end % kill -SIGCONT 15068 % 5678...
  24. Processes can trap signals % irb >> Process.pid => 15369

    >> trap(:TERM){puts "nope"} => "DEFAULT" >> nope % kill -SIGTERM 15369
  25. Refuse to die % irb >> Process.pid => 15369 >>

    trap(:TERM){puts "nope"} => "DEFAULT" >> nope % kill -SIGTERM 15369
  26. A more useful example pid = fork do loop do

    print "." sleep 1 end end trap(:TERM) do Process.kill(:TERM, pid) end Process.waitpid(pid) % kill -SIGTERM 15369
  27. exited = 0 trap(:CHLD) do exited +=1 exit if exited

    == 3 end 3.times do fork do sleep 1 end end # tight loop loop {}
  28. def handler(exited) loop do Process.waitpid(-1, Process::WNOHANG) || break exited +=1

    exit if exited == 3 end exited rescue Errno::ECHILD end exited = 0 trap(:CHLD) { exited = handler(exited) } 3.times do fork do sleep 3 end end # tight loop loop {}
  29. def handler(exited) exited rescue Errno::ECHILD end exited = 0 trap(:CHLD)

    { exited = handler(exited) } 3.times do fork do sleep 3 end end # tight loop loop {} loop do Process.waitpid(-1, Process::WNOHANG) || break exited +=1 exit if exited == 3 end
  30. def handler(exited) exited end exited = 0 trap(:CHLD) { exited

    = handler(exited) } 3.times do fork do sleep 3 end end # tight loop loop {} loop do Process.waitpid(-1, Process::WNOHANG) || break exited +=1 exit if exited == 3 end rescue Errno::ECHILD
  31. Processes can communicate reader, writer = IO.pipe fork do writer.close

    puts reader.gets reader.close end reader.close writer.write("Wake up!") writer.close Process.wait
  32. Pipes are one-way reader, writer = IO.pipe fork do writer.close

    puts reader.gets reader.close end reader.close writer.write("Wake up!") writer.close Process.wait
  33. select(2) tells you when stuff is ready reader, writer =

    IO.pipe fork do writer.puts("Do this") sleep 2 writer.puts("Do that") end loop do ret = IO.select([reader], nil, nil, 1) if ret puts reader.gets else puts "no work :(" end end Do this no work :( Do that no work :(
  34. def start $PROGRAM_NAME="kaanta master" @master_pid = Process.pid @socket = TCPServer.open(Config.host,

    Config.port) spawn_workers end def spawn_workers worker_number = -1 until (worker_number += 1) == Config.workers @workers.value?(worker_number) && next worker = Kaanta::Worker.new(@master_pid, @socket, tempfile, worker_number,logger) pid = fork { worker.start } @workers[pid] = worker end end
  35. def start $PROGRAM_NAME="kaanta master" @master_pid = Process.pid @socket = TCPServer.open(Config.host,

    Config.port) spawn_workers end def spawn_workers worker_number = -1 until (worker_number += 1) == Config.workers @workers.value?(worker_number) && next worker = Kaanta::Worker.new(@master_pid, @socket, tempfile, worker_number,logger) pid = fork { worker.start } @workers[pid] = worker end end
  36. while alive && @master_pid == Process.ppid do if ret begin

    client = @socket.accept_nonblock rescue Errno::EAGAIN end end ret = begin IO.select([@socket], nil, nil, Config.timeout / 2) || next rescue Errno::EBADF end end
  37. loop do reap_workers case (mode = @sig_queue.shift) when nil kill_runaway_workers

    spawn_workers when 'QUIT', 'TERM', 'INT' # we don't handle gracefully stopping workers # to demonstrate that workers go down quickly # after the master quits by tracking their # Process.ppid. break when 'TTIN' Config.workers += 1 when 'TTOU' unless Config.workers <= 0 Config.workers -= 1 kill_worker('QUIT', @workers.keys.max) end end
  38. % bin/kaanta [kaanta master (PID: 729)] INFO -- Listening on

    0.0.0.0: 8080 [kaanta master (PID: 729)] INFO -- Spawning 3 workers [kaanta worker 0 (PID: 756)] INFO -- up [kaanta worker 1 (PID: 757)] INFO -- up [kaanta worker 2 (PID: 758)] INFO -- up [kaanta worker 3 (PID: 862)] INFO -- up % kill -TTIN 729 13712
  39. Hand each worker an unlinked temporary file def spawn_workers worker_number

    = -1 until (worker_number += 1) == Config.workers @workers.value?(worker_number) && next tempfile = Tempfile.new('') tempfile.unlink tempfile.sync = true worker = Kaanta::Worker.new(tempfile) pid = fork { init_worker(worker) } @workers[pid] = worker end end
  40. Workers must chmod that tempfile periodically while alive && @master_pid

    == Process.ppid do tempfile.chmod(i += 1) if ret begin client = @socket.accept_nonblock command = client.gets logger.info("Executing: #{command}") client.write `#{command}` client.flush client.close rescue Errno::EAGAIN end end tempfile.chmod(i += 1) ret = begin IO.select([@socket], nil, nil, Config.timeout / 2)||next rescue Errno::EBADF end end
  41. Kill workers that failed to chmod in time def kill_runaway_workers

    now = Time.now @workers.each_pair do |pid, worker| (now - worker.tempfile.ctime) <= Config.timeout && next kill_worker('KILL', pid) end end
  42. Spawn replacements in your master loop loop do reap_workers case

    (mode = @sig_queue.shift) when nil kill_runaway_workers spawn_workers when 'QUIT', 'TERM', 'INT' break when 'TTIN' Config.workers += 1 when 'TTOU' unless Config.workers <= 0 Config.workers -= 1 kill_worker('QUIT', @workers.keys.max) end end end
  43. def reap_workers loop do pid, status = Process.waitpid2(-1, Process::WNOHANG)||break reap_worker(pid,

    status) end rescue Errno::ECHILD end def reap_worker(pid, status) worker = @workers.delete(pid) worker.tempfile.close rescue nil logger.info "reaped worker #{worker.number} " \ "(PID:#{pid}) " \ "status: #{status.exitstatus}" end
  44. while alive && @master_pid == Process.ppid do tempfile.chmod(i += 1)

    if ret begin client = @socket.accept_nonblock command = client.gets logger.info("Executing: #{command}") client.write `#{command}` client.flush client.close rescue Errno::EAGAIN end end tempfile.chmod(i += 1) ret = begin IO.select([@socket], nil, nil, Config.timeout / 2) || next rescue Errno::EBADF end end
  45. Fin