Software Engineering Manager at CaringBridge MNPHP Organizer Open Source Contributor (Zend Framework and various others) Where you can find me: • Twitter: mwillbanks G+: Mike Willbanks • IRC (freenode): mwillbanks Blog: http://blog.digitalstruct.com • GitHub: https://github.com/mwillbanks Housekeeping…
Concepts How it really works • Quick Start Get it up and running and start playing. • The Details How can it be a tech talk without details? • Some use cases How you might use it. • Questions Although you can bring them up at anytime! Agenda
work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages.” Official Statement
At the core, they are simply small programs. •The daemon handles the negotiation of work Workers and Clients •The worker does the work •The client requests work to be done What The Hell? Tell me!
download •Download the Binary •Unpack the binary •./configure && make && make install •Bam! You’re off! For more advanced configuration see ./configure –help •Starting gearmand -d Installation
ourselves… Yes, I wrote a layer on top of Zend Framework called Zend_Gearman; wow unique. https://github.com/mwillbanks/Zend_Gearman PHP – Zend Framework
this as the default is ideal; however, does not work in all environments. •Persistent Queues Libdrizzle Libsqlite3 Libmemcached Postgres TokyoCabinet MySQL Redis Persistence
gearman. •Additionally, arguments to the gearman daemon need to be passed to talk to the specific persistence layer. •Each persistence layer is actually built as a plugin to gearmand http://bazaar.launchpad.net/~tangent- org/gearmand/trunk/files/head:/libgearman- server/plugins/queue/ Getting Up and Running with Persistence
is called the workload; it can be anything that can become a string. Utilize an open format; it will make life easier if you chose to use a different language for processing • XML, JSON, etc. • Yes, you can serialize objects if you wanted to… not recommended although. Clients
the work •Generally they will run as a daemon in the background •Workers register a function that they perform They should ONLY be doing a single task. This makes them far easier to manage. •The worker does the work and “can” return results If you are doing the work asynchronously you generally do not return the result. Synchronous work you will return the result. Workers
connection • Must have the ability to reconnect to the database. • Watch for connection timeouts •Handling Memory Leaks Watch the amount of memory and detect leaks then kill the worker. •Request Languages PHP for instance, sometimes slows down after hundreds of executions, kill it off if you know this will happen. Workers – special notes
to boot them back up after a restart Utilizing a service to watch your workers and ensure they are always running is a GOOD thing. •Supervisord Can watch processes, restart them if they die or get killed Can manage multiple processes of the same program Can start and stop your workers. •When running workers, BE SURE to handle KILL signals such as SIGKILL. Keeping the Daemon Running
socket interface… telnet on port 4730 Write “STATUS” • Gives you the registered functions, number of workers and items in the queue. •Gearman Monitor – PHP Project NOTE: I’ve never actually attempted this; BUT it is referenced on gearman.org so it must be doing something! https://github.com/yugene/Gearman-Monitor Monitoring
Web servers should serve, not process images. Images require a lot of memory AND processing power • They are best to be processed on their own! •Processing in the Background Generally will require a change to your workflow and checking the status with XHR to see if the job has been completed. • This allows you to process them as you have resources available. • Have enough workers to process them “quickly enough” Images
take up time, time that is better spent getting the user to the next page. •The feedback on the mail doesn’t really make a difference so it is great to send it to the background. Email
•Process the logs to produce analytical data •Impression / Click Tracking •Why run a cron over your logs nightly? Real-time data is where it is at! Log Analysis / Aggregation