Queue N Task Queue 1 Broker Client 2 Worker 1 Worker 2 Task Result Storage distribute tasks distribute tasks send tasks send tasks store task results store task results get task result get task result
Celery apps Exists until it has been acknowledged Result of the tasks can be stored or ignored States: PENDING, STARTED, SUCCESS, FAILURE, RETRY, REVOKED Periodic task (cron jobs) 8
the earliest time at which task will be executed countdown set eta by seconds into the future expires set task’s expire time serializer pickle (default), json, yaml and msgpack compression compress the messages using gzip or bzip2 queue route the tasks to different queues 11
result.successful() true if the task executed successfully result.result the return value of the task or exception result.get() blocks until the task is complete, return result or exception 12
control of the workers internally, called “bootsteps” Customize the worker components, e.g: ConsumerStep Add new components Bootsteps http://celery.readthedocs.org/en/latest/ userguide/extending.html 31
and history • Show task details (arguments, start time, runtime, and more) • Graphs and statistics • Shutdown, restart worker instances • Control worker pool size, autoscaling settings • … 32