PyCon Ireland 2013: Marconi: Queuing and Notification service for OpenStack

(Via http://python.ie/pycon/2013/speakers/flavio_percoco/)
Speaker: Flavio Percoco

Marconi is a multi-tenant cloud queuing system written in Python as part of the OpenStack project. As every message bus, it’s main goals are: performance, availability, durability, fault-tolerance and scalability. Besides providing support for queuing and notification services through OpenStack, Marconi aims to ease the design of distributed systems and allow for asynchronous work distribution without creating yet another message broker.

This talk aims to give the audience as much information as possible about Marconi’s architecture, design, patterns, performance, issues it faced and how / why it was built from scratch.

Transcript

1. @flaper87 at PyConIE 2013 Queuing & Notification Service … made by and for OpenStack
2. @flaper87 at PyConIE 2013 What it is not What it is Architecture Features
3. @flaper87 at PyConIE 2013 Who's this dude? @flaper87 (a.k.a, Flavio Percoco): Very proud Red Hatter, who's also part of the MongoDB Master team and has contributed to Open Source since... forever. One of those who thinks about programming when he's not programming. member of
4. @flaper87 at PyConIE 2013 Replacement for QPID, RabbitMQ, ZMQ It is NOT a ...
5. @flaper87 at PyConIE 2013 It is NOT a ... Task Manager / Workflow engine
6. @flaper87 at PyConIE 2013 Queuing service Queues Messages Claims
7. @flaper87 at PyConIE 2013 Messaging Service Req / Rep Pub / Sub
8. @flaper87 at PyConIE 2013 Very young project Bugs WIP Ideas Missing pieces
9. @flaper87 at PyConIE 2013 Lego Bricks Transports Storage
10. @flaper87 at PyConIE 2013 Stevedore, FTW
11. @flaper87 at PyConIE 2013 Transports HTTP ZMQ TCP
12. @flaper87 at PyConIE 2013 HTTP Falcon No container
13. @flaper87 at PyConIE 2013 Storage SQLite MongoDB will be replaced with sqlalchemy.core
14. @flaper87 at PyConIE 2013 Storage Redis ElasticSearch
15. @flaper87 at PyConIE 2013 Authentication Basic HTTP Keystone
16. @flaper87 at PyConIE 2013 Third-Party Modules Tests Base ABC Classes
17. @flaper87 at PyConIE 2013
18. @flaper87 at PyConIE 2013 SIMPLER CRUD-like ROUTING
19. @flaper87 at PyConIE 2013 QUEUES Metadata 64b Name
20. @flaper87 at PyConIE 2013 MESSAGES TTL Echo
21. @flaper87 at EuroPython 2013 CUSTOM Max Size Max Bulk ...
22. @flaper87 at PyConIE 2013 FIFO Guaranteed
23. @flaper87 at PyConIE 2013 Storage Partitions
24. @flaper87 at PyConIE 2013 Fits in your stack
25. @flaper87 at PyConIE 2013 Easy to Scale™
26. @flaper87 at PyConIE 2013 Other benefits Unified API Service oriented Easy to scale
27. @flaper87 at PyConIE 2013 ROADMAP AMQP Backends support Per resource RBAC Cluster support Client libraries Kombu transport
28. @flaper87 at PyConIE 2013 TAKEAWAYS Start small Respect YAGNI Not opinionated... at all Ship tests along with your app Off-load tasks to third-party apps.
29. @flaper87 at PyConIE 2013 Q&A Thanks we're hiring http:/ /wiki.openstack.org/Marconi #openstack-marconi @ freenode