Microservices with Python and Flask

Tutorial given at PyCon 2017 on May 17th, 2017

Microservices are receiving the buzzword treatment these days, and as such, they have a cloud of hype surrounding them that makes it hard to separate substance from fluff. In this tutorial, Miguel Grinberg starts with an introduction to this architecture, including what's great and not so great about it, and then teaches you how a traditional monolithic application written in Flask can be refactored into a modern distributed system based on microservices.

Transcript

1. Microservices with Python and Flask Miguel Grinberg @miguelgrinberg
2. Agenda • First hour ◦ Introduction & demo ◦ Microservices concepts ◦ Class project design discussion • Remaining time ◦ Step-by-step development of the class project ◦ (Focus is on techniques more than on a particular implementation)
3. About the MicroFlack Project • MicroFlack is a microservices version of Flack ◦ Flack is the chat server I used in the “Flask at Scale” class I gave at PyCon 2016 • The application lives on 7 (yes, seven!) GitHub repositories • Runs on Python 3.4+ and Docker • Not tied to any specific cloud or microservice vendor technology
4. Deploying MicroFlack to your Laptop • Requirements ◦ 4GB RAM (8GB recommended) ◦ Vagrant ◦ VirtualBox ◦ Everything is installed in an Ubuntu 16.04 VM (Windows, Mac, Linux laptops are all OK!) • Deployment commands: git clone https://github.com/miguelgrinberg/microflack_admin cd microflack_admin vagrant up # to create the VM or restart it after shutdown vagrant ssh # to open a shell session on the VM vagrant halt # to shutdown the VM (without destroying it) vagrant snapshot save clean # to save a snapshot with name “clean” vagrant snapshot restore clean --no-provision # to restore the snapshot vagrant destroy # to delete the VM
5. Monolithic Flack
6. MicroFlack
7. MicroFlack Features • Five microservices, four HTTP/REST, one WebSocket • Each service is a standalone Flask app • Two of the services use MySQL databases • Services run in Docker containers • Services communicate over HTTP, message queue or service registry • All services are load balanced • Services scale independently of each other • Upgrades can be done without downtime
8. Microservice Benefits • Less complexity (maybe) ◦ Awesome for teams with varying degrees of experience • Scaling flexibility • More reliability • Less coupling • More choice • Deploy/upgrade while running So where is performance in this list?
9. Breaking up a Monolith • Going from monolith to microservices is very hard • Several strategies ◦ Microservices only going forward ◦ Break pieces of functionality into microservices over time ◦ Refactor the entire monolith into microservices • In all cases, a base platform needs to be put in place before refactoring work begins • Good automated tests are crucial in avoiding bugs when refactoring functionality into a microservice
10. The Microservices Platform
11. Load Balancer • All services are load balanced • You think you don’t need load balancing? ◦ Upgrades without downtime require a load balancer for a seamless transition ◦ Autoscaling, A/B testing, green/blue deployments, etc. become possible • Many options ◦ Open source: Nginx, HAProxy ◦ As a service: AWS, OpenStack, Azure, Google Cloud, etc. ◦ Serverless: Load balancing and scaling are implicitly done by the cloud operator ▪ The Lambda and API Gateway services on AWS are by far the best in this category ◦ Hardware: F5
12. Service Registry • A distributed data store that keeps track of running services • All running services maintain an entry in the service registry • The load balancer’s configuration is generated and updated from the registry contents • Many great open source projects to choose from: ◦ Etcd (CoreOS) ◦ Consul (HashiCorp) ◦ Zookeeper (Apache) ◦ Eureka (Netflix) ◦ SmartStack (Airbnb) ◦ and more!
13. Logging • Logs from all the services are typically consolidated to a single stream to avoid the complexity of maintaining lots of individual log files • Open source: ◦ ELK stack (elasticsearch + logstash + kibana) ◦ Logspout • Several commercial options: ◦ Papertrail ◦ Splunk ◦ Loggly ◦ and more!
14. Containers • Provide an additional layer of isolation over processes • Each container runs a virtualized host ◦ You can have containers using different Linux distros on the same host ◦ Dependencies that would conflict if installed on the same host can be installed on containers ◦ Virtualized network ports • A container maps to one main process, but there can be additional tasks • Not a required component of the stack, but very convenient
15. Application State • Stateful services are hard to manage ◦ No way to avoid them for service registry, databases, message queues, etc. • Stateless services are easily scalable, replaceable and disposable • Application-level services should ideally be stateless, and should use stateful services for storage • The state associated with a service should be private ◦ Each service must use its own database ◦ This prevents coupled services that are difficult to upgrade independently of each other ◦ Database joins across services must be done in the application
16. Retries • Distributed systems have a lot of moving parts • It is always a good idea to implement retries for all network calls • The “exponential backoff” strategy works nicely in this environment
17. The “Backwards-Compatible” Rule • Changes to services must be backwards compatible ◦ Why? Because a distributed system cannot be updated atomically without downtime • Database migrations must not break any code that may still be deployed ◦ Phased micro-upgrades can help with column renames or deletes, constraints, etc. • API changes must not break any code that may still be deployed ◦ Why? Need a way to upgrade the API consumer and producer independently • Complex changes that span several services must be “orchestrated” so that they can be applied as micro-deployments without breaking the system
18. Lifecycle of a Microservice • On startup, the microservice registers with the service registry, or is “discovered” by it • The load balancer watches the registry and updates itself to include the new microservice • The new service starts receiving traffic from the load balancer • If more than one instance of the service exist, the traffic is split among them • The service sends “keep-alive” signals, or responds to periodic health checks • When the service is stopped, or stops sending keep-alives, or fails a health check, it is removed from the registry, and in turn from the load balancer
19. Recommended reading: The Twelve-Factor App https://12factor.net • Codebase • Dependencies • Config • Backing services • Build, release, run • Processes • Port binding • Concurrency • Disposability • Dev/prod parity • Logs • Admin Processes
20. MicroFlack Design
21. Choosing a Stack • For the services: ◦ Flask and friends • For the platform: ◦ Load balancer: HAProxy ◦ Service registry: Etcd ◦ Confd (http://www.confd.io/) configures the load balancer dynamically ◦ Docker containers ◦ Logspout log consolidation ◦ MySQL databases ◦ Redis message queue
22. Leveraging Public Container Images • miguelgrinberg/easy-etcd ◦ Container image that deploys etcd clusters • miguelgrinberg/easy-lb-haproxy ◦ Load balancer + confd preconfigured • gliderlabs/logspout ◦ Consolidated log output • mysql:5.7 ◦ Official Docker image for MySQL 5.7 • redis:3.2-alpine ◦ Official Docker image for Redis 3.2
23. Administration Scripts: microflack_admin • Vagrantfile: deploy to a single-node Vagrant VM • setup-host.sh, setup-all-in-one.sh, make-db-passwords.sh: deploy scripts • mfvars: common environment variables • mfclone: clone the repositories for all the services • mfbuild: build Docker images for services • mfrun: start services • mfkill: stop services • mflogs: consolidated log stream of all services • mfupgrade: upgrade services • mfenv: generate a .env file with environment needed for development • mfdev: attach a locally running service to a deployed system for debugging • etcd-dump: dump the contents of the service registry to the console
24. Common Code: microflack_common • There is some functionality that all microservices need ◦ Service registration ◦ Unit testing helpers ◦ Inter-service communication ◦ Authentication handlers • We’ll use a Python package that services can install with pip • Easy option: install from pypi (if you don’t mind making it public) • Less easy option: private package installed from a local file ◦ We’ll use the Python wheel format for this (pip install wheel) ◦ The --find-links option in pip can install packages from a local file system directory ◦ The mkwheel script builds the wheel packages
25. Authentication: Let’s switch to JWTs • Tokens stored in a database are inconvenient ◦ Services would need to send a request to the tokens service for verification • JSON Web Tokens (JWTs) can be verified just with cryptography ◦ A JWT token stores data inside it, such as a username or id ◦ When the token is generated, a cryptographic signature is added to it ◦ Signature can only be generated or verified if you have a secret key ◦ The data in a token can be trusted only if the token has a valid signature ◦ Not everything is great with JWTs: token revocations become harder • Since tokens are opaque, switching to JWT is not a breaking change • Beware of JWT exploits: always set and check signing algorithm
26. MicroFlack Service Boundaries Microservice URL(s) User interface / /static/... Users /api/users /api/users/:id Tokens /api/tokens Messages /api/messages /api/messages/:id Socket.IO /socket.io
27. From Flack to MicroFlack
28. MicroFlack v0.1: Just the UI
29. UI Service Summary • Endpoints Method Endpoint Authentication Description GET / None Client HTML page GET /static/app.js None Main client application code GET /static/*.js None Client application code GET /static/*.css None Client application stylesheets
30. UI Service: microflack_ui • app.py, config.py, templates/, static/, requirements.txt ◦ Ordinary Flask app that serves the index HTML page plus all the JavaScript and CSS files that make up the client application ◦ To ease the transition, at this stage we’ll use an older version of the UI that does not use Socket.IO (we’ll add Socket.IO later) • .env ◦ Environment variables ◦ This file should not be added to source control, as it can contain secrets • tests.py, tox.ini ◦ Unit tests, code coverage and linting • Dockerfile, boot.sh, build.sh ◦ Docker support
31. Running the UI service • vagrant ssh (connect to the VM) • mfkill all (reset your VM to an initial state without any services) ◦ Watch the load balancer at http://192.168.33.10/stats • cd ~/microflack_ui • git checkout 1 (get version 1 of the UI service) • ./build.sh (build the service) • mfrun ui (run the service) • Connect to the application at http://192.168.33.10 ◦ Browser errors are expected, as no other services are yet running
32. Incorporating the Flack Monolith • The service registry has a simple tree structure (use etcd-dump to see it) • You can register the Flack monolith with the load balancer in the VM: ◦ curl -X PUT $ETCD/v2/keys/services/monolith/location -d value="/api" ◦ curl -X PUT $ETCD/v2/keys/services/monolith/upstream/server -d value="10.0.2.2:5000" ◦ Note: 10.0.2.2 is the IP address the host machine has inside a vagrant VM • Now the UI is served by the new microservice, while everything else comes from the old Flack+Celery application • To remove: ◦ curl -X DELETE $ETCD/v2/keys/services/monolith?recursive=true
33. MicroFlack v0.2: Users service
34. Users Service Summary • Endpoints (new endpoints in red) Method Endpoint Authentication Description POST /api/users None Register a new user GET /api/users Token Optional Get list of users GET /api/users/:id Token Optional Get user by id PUT /api/users/:id Token Modify user by id GET /api/users/me Basic Authenticate user
35. Users service: microflack_users • Same basic structure as the ui microservice • Includes User model and all /api/users endpoints from original Flack • Token authentication imported from microflack_common • Ported existing unit tests and used them as a guide to fix everything up • Add database migration support (Flask-Migrate) ◦ Databases are created by mfrun if they don’t exist yet ◦ Migrations are executed in the container startup script • Add new /api/users/me endpoint to validate username and password and return user information
36. Running the Users service • cd ~/microflack_users • git checkout 1 (select version 1 of the service) • ./build.sh (build the service) • mfrun users (run the service) ◦ The /api/users family of endpoints should now be working!
37. MicroFlack v0.3: Tokens service
38. Tokens Service Summary • Endpoints (postponed endpoints grayed out) Method Endpoint Authentication Description POST /api/tokens Basic Request a token DELETE /api/tokens Token Revoke a token
39. Token service: microflack_tokens • Just one endpoint copied from the monolithic app: /api/tokens • Authentication is relayed to the users service /me endpoint • Generated JWT token contains the numeric user id • No token revocations for now
40. Running the Tokens service • cd ~/microflack_tokens • git checkout 1 (select version 1 of the service) • ./build.sh (build the service) • mfrun tokens (run the service) ◦ The /api/tokens endpoint should now be working
41. MicroFlack v0.4: Messages service
42. Messages Service Summary • Endpoints Method Endpoint Authentication Description POST /api/messages Token Post a new message GET /api/messages Token Optional Get list of messages GET /api/messages/:id Token Optional Get message by id PUT /api/messages/:id Token Modify message by id
43. Messages Service: microflack_messages • Structure based on the user service • Models and endpoints copied from original Flack code • Removed all asynchronous functions for now ◦ We want a basic app up and running from which we can build on • Ported unit tests to verify the code works
44. Running the Messages service • cd ~/microflack_messages • git checkout 1 (select version 1 of the service) • ./build.sh (build the service) • mfrun messages (run the service) • The application should be fully functional (though not very performant yet)
45. MicroFlack v0.5: Async message rendering • Original Flack used Celery for asynchronous message rendering ◦ Unfortunately, Celery workers are by design tightly coupled with the caller process • Instead of Celery, we will use background threads for rendering ◦ Our render task is not CPU intensive, so this works very well ◦ For CPU intensive tasks, the multiprocessing module can be used instead ◦ If a very high volume of tasks must be supported, an asynchronous server can be used
46. Development Workflow: Common Package • Build the common packages locally: ◦ cd microflack_admin; source mfvars; cd .. ◦ git clone https://github.com/miguelgrinberg/microflack_common ◦ cd microflack_common ◦ ./mkwheel all ◦ cd ..
47. Development Workflow: Running a Service • Set up the source code for the desired microservice: ◦ cd microflack_admin; source mfvars; cd .. ◦ git clone https://github.com/miguelgrinberg/microflack_messages ◦ cd microflack_messages ◦ python3 -m venv venv ◦ source venv/bin/activate ◦ pip install -r requirements.txt ◦ flask run • Test by sending requests with curl, httpie, postman, etc. • For integration testing with an actual system running in a VM: ◦ Create .env file (run mfenv inside VM to get the variables you need) ◦ mfdev start messages ◦ flask run
48. Upgrading the Messages service • cd ~/microflack_messages • git checkout 2 (select version 2 of the service) • ./build.sh (build the service) • mfupgrade try messages (start upgrade) ◦ After watching the log for a few seconds, hit Ctrl-C • mfupgrade roll (rolling upgrade)
49. MicroFlack v0.6: Token revocation
50. Tokens Service Summary • Endpoints (new endpoints in red) Method Endpoint Authentication Description POST /api/tokens Basic Request a token GET /api/tokens Token Check if a token is revoked DELETE /api/tokens Token Revoke a token
51. Token revocation • We need to maintain a list of revoked tokens ◦ The best place to implement this is the tokens service ◦ Revoked tokens need to be kept in a list only until they expire ◦ We can keep the list in etcd, and write all entries with the appropriate expiration • Services need to check tokens against that list ◦ We can encapsulate this inside the verify_token function in microflack_common ◦ Option 1 (more correct): send a request to the tokens service to check revocation status ◦ Option 2 (more performant): check the list in etcd directly ◦ Improvement for both options: cache calls to verify_token
52. Upgrading the Tokens service • cd ~/microflack_tokens • git checkout 2 (select version 2 of the service) • ./build.sh (build the service) • mfupgrade roll tokens (rolling upgrade, skipping the “try” step) • Upgrade the services that work with tokens: ◦ users to version 2 ◦ messages to version 3
53. MicroFlack v1.0: Socket.IO service
54. Socket.IO Service Summary • Socket.IO client to server events • Socket.IO server to client events (new in red) Event Authentication Description ping_user Token Mark a user as online post_message Token Post a message disconnect Token (from session) Mark the user as offline Event Description updated_model Render updated user or message expired_token Ask user to log in again
55. System changes for Socket.IO • Task list for the Socket.IO service ◦ Implement “ping_user”, “post_message” and “disconnect” events ◦ Push “expired_token” notifications to clients when appropriate • Task list for the common package ◦ Add support for setting sticky sessions in the load balancer • Task list for the UI service ◦ Add Socket.IO support ◦ Handle expired tokens in Socket.IO calls (bug in old version) • Task list for users service ◦ Add “ping” and “user offline” endpoints ◦ Push “updated_model” notifications to message queue • Task list for messages service ◦ Push “updated_model” notifications to message queue
56. Socket.IO service: microflack_socketio • Same structure as a no-database service • No HTTP endpoints, only the three Socket.IO events • Needs to be an async service due to the long term WebSocket connections ◦ Don’ t understand why? Come to my talk “Asynchronous Python” on Sunday! ◦ We have several options ▪ python-socketio supports WSGI-compatible async frameworks (eventlet, gevent) and also asyncio ▪ Flask-SocketIO builds on python-socketio, but drops asyncio support ◦ While we don’t need Flask for this service, having access to Flask’s user session is handy ◦ We’ll go with Flask-SocketIO and eventlet for this service
57. Users Service Summary • Endpoints (new endpoints in red) Method Endpoint Authentication Description POST /api/users None Register a new user GET /api/users Token Optional Get list of users GET /api/users/:id Token Optional Get user by id PUT /api/users/:id Token Modify user by id GET /api/users/me Basic Authenticate user PUT /api/users/me Token Set user online DELETE /api/users/me Token Set user offline
58. Running the Socket.IO service • cd ~/microflack_socketio • git checkout 1 (select version 1 of the service) • ./build.sh (build the service) • mfrun socketio (run the service) • Upgrade the services that work with Socket.IO (order is important!): ◦ users to version 3 ◦ messages to version 4 ◦ ui to version 2
59. MicroFlack on Other Platforms
60. Kubernetes • Open-source container orchestration, from Google • Manages a cluster of nodes (container hosts) transparently • Works with Docker images • Has its own service registry and load balancer • Gives each service a DNS name (i.e. http://users connects to the Users service) • Stores secrets securely • Handles service replication, and does rolling upgrades • microflack_admin includes example Kubernetes deployment scripts in install/kubernetes
61. Amazon ECS • A cluster of Docker hosts running on EC2 instances • Access to the AWS ecosystem • MicroFlack platform and application containers can run without change • Some effort required in configuring roles and security groups (as with everything done on AWS)
62. AWS Serverless (Lambda & API Gateway) • Very different paradigm, only the application logic is uploaded to AWS ◦ No need for gunicorn, just the application code that handles the endpoints ◦ Tools like Zappa (or my own Slam) enable transparent support for WSGI apps • AWS Lambda provides automatic load balancing and auto-scaling • Access to the AWS ecosystem • Cons: ◦ No WebSocket support, since there is no server running all the time ◦ Response times are not great
63. Good ol’ Processes • Deploying MicroFlack on a host without Docker is possible • All the core components of the platform can be installed without Docker: ◦ etcd, haproxy, confd, mysql, redis • The MicroFlack application services can run as regular gunicorn processes ◦ But a network port assignment strategy needs to be implemented
64. Homework!
65. Improvement Ideas • Revoke all tokens for a user, and revoke all tokens for everybody • Use multiple chat rooms instead of just one (this will require some client-side work as well) • Protect the /stats and /logs endpoints with authentication • Add SSL termination to (or in front of) the load balancer • Deploy the ELK stack and configure the logspout container to forward logs to it • Add a secrets store component (maybe HashiCorp’s Vault) • Create a multi-host deployment, possibly with redundant load balancers • Replace the single-node Redis deployment with a Redis or RabbitMQ cluster • Replace the single-node MySQL with a Galera cluster • Replace MySQL with NoSQL databases of your choice • With all services running 3+ instances, create a “chaos monkey” script • Implement a platform service that recycles application services that die or are unresponsive, and maybe even auto-scales them based on load
66. Questions?