Building Micro-services at DigitalOcean

A quick talk on the building blocks and tools of microservices at DigitalOcean, presented at the November Melbourne Ruby meet-up.

Transcript

1. BUILDING
 MICRO-SERVICES Without Reinventing The Wheel
2. Ivan Vanderbyl @ivanderbyl github.com/ivanvanderbyl I’m Ivan Vanderbyl, I’m one of the front-end engineers at DigitalOcean, where we’re building a simpler cloud for developers.
3. MICRO-SERVICE? So tonight I want to talk about micro services, and the tools we extracted from our own services, and our move towards breaking our monolithic rails applications in to smaller services. So what is a micro service?
4. …the microservice architectural style is an approach to developing a single application as a suite of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API… http://martinfowler.com/articles/microservices.html — Martin Fowler Martin Fowler has an awesome article on some highly detailed descriptions of what is and isn’t a microservice. But essentially it’s a process which handles a singular defined piece of work, which another process sends to it over a communication channel, usually HTTP, upon which it does some work and returns a result.
5. Building blocks of a Micro-service Microservice 1 Core Service / Application Public API API Client Domain Logic Domain Services HTTP These are the building blocks of a microservice. We have a standalone process which another process can call to process some piece of data and handle the response. In our case this is over HTTP, on an internal network.
6. The Wheels Everyone reinvents these Public API API Client I won’t focus on the architecture of the service itself, rather the communication components. The wheels, if you will. These are the most hotly contacted components and nearly always get reinvented.
7. Core Application Internals of an API client External Microservice API Client Resolver Response Extraction Data Model Domain Logic Request Serialisation HTTP Interface When we take a look at the internals of the API client component, we see some common patterns, something to talk raw HTTP, something to resolve the requests for specific resources, and something to turn an object in to JSON or some other format.
8. Microservice 1 Internals of an API Server External Microservice API Server Router Response Extraction Data Model Domain Logic Request Serialisation HTTP Interface The internals of an API server are pretty much the same. You might not be handling the extraction yourself, but you’ll more than likely be handling the serialisation.
9. Router / Resolver Response Extraction Data Model Request Serialisation HTTP Interface ActiveModelSerializers, JBuilder, OAT , rabl, to_json ActionPack, Sinatra ActiveResource Faraday, Net::HTTP, Rack So across the client and server we have a mix of common tools which pretty much everyone ties together. If you’re using Rails most of this can be handled out of the box, but that’s not the case with Sinatra, for example. We were using ActiveModelSerializers, but we had to make a lot of internal modifications to support the responses we wanted to produce.
10. New Building Blocks Router / Resolver Response Extraction Data Model Request Serialisation HTTP Interface Kartograph ResourceKit Kartograph Faraday So across the client and server we have a mix of common tools which pretty much everyone ties together. If you’re using Rails most of this can be handled out of the box, but that’s not the case with Sinatra, for example.
11. { kartograph } https://github.com/digitalocean/kartograph We built kartograph kind of by chance when I was pairing with bobby tables on the west coast for a few days.
12. Object JSON Object Data Serialiser and Extractor { kartograph } It does exactly two things: Turns an object in to JSON, and converts JSON back to that Object.
13. class DropletMapping include Kartograph::DSL kartograph do root_key plural: 'droplets', singular: 'droplet', scopes: [:read] property :id, scopes: [:read] property :name, scopes: [:read, :create] end end lib/mappings/droplet_mapping.rb https://github.com/digitalocean/droplet_kit/blob/master/lib/droplet_kit/mappings/droplet_mapping.rb Kartograph does this using Mappings. Here’s an example of a really simple mapping, taken from DropletKit, our official API client for the DigitalOcean API.
14. class DropletMapping include Kartograph::DSL kartograph do root_key plural: 'droplets', singular: 'droplet', scopes: [:read] property :id, scopes: [:read] property :name, scopes: [:read, :create] end end lib/mappings/droplet_mapping.rb https://github.com/digitalocean/droplet_kit/blob/master/lib/droplet_kit/mappings/droplet_mapping.rb A kartograph mapping is simply a ruby class composed of the Kartograph DSL
15. class DropletMapping include Kartograph::DSL kartograph do root_key plural: 'droplets', singular: 'droplet', scopes: [:read] property :id, scopes: [:read] property :name, scopes: [:read, :create] end end lib/mappings/droplet_mapping.rb https://github.com/digitalocean/droplet_kit/blob/master/lib/droplet_kit/mappings/droplet_mapping.rb In this example we’re specifying two representations of our object. One for read and one for create. So you can see we only produce a root element on read, and not on create. We also don’t include the id attribute when serialising for create.
16. droplet = double("Droplet", name: "Sammy", id: 1) DropletMapping.representation_for(:create, droplet) # => {"name": "Sammy"} DropletMapping.representation_for(:read, droplet) # => {"droplet": {"name": "Sammy", "id": 1}} Serialising an object We can call `representation_for` and produce a representation for this object based on the given scope. So we get a root-less object for making requests, and an object with a root for responding to the client. So one of these would be used in the client, and one on the server.
17. Serialising a collection DropletMapping.represent_collection_for(:read, [droplet]) # => {"droplets": [{"name": "Sammy", "id": 1}]} Similarly we can also represent a collection of objects very easily, and get a pluralization of the root key name.
18. Extraction droplet_json = '{"droplet": {"name": "Sammy", "id": 1}}' DropletMapping.extract_single(droplet_json, :read) # => Droplet droplets_json = '{"droplets": [{"name": "Sammy", "id": 1}]}' DropletMapping.extract_collection(droplets_json, :read) # => [Droplet] Deserialisation, or Extraction as we call it, works exactly the same. We pass in some json, and receive an object according to our mapping and scope, in this case a Droplet object.
19. No Dependencies Caching Rails Cache. Custom Expiration. Supports complicated nested data structures Production Ready™ Powers DigitalOcean API V2, DropletKit, and internal clients for DigitalOcean DNS. { kartograph } digitalocean/kartograph And that’s kartograph. It has no dependencies, supports caching with custom expiration, handles complicated data structures, and it’s used in production.
20. gem "kartograph" Throw this in your Gemfile and bundle it.
21. ResourceKit https://github.com/digitalocean/resource_kit So that’s the basics of Kartograph. Let’s talk about ResourceKit, this was another extraction we pulled from DropletKit.
22. Provides the “How” of talking to remote APIs ResourceKit is essentially some very lightweight tools for building API clients. It provides the tools to describe how we make calls and handle responses.
23. Doesn’t actually make API calls It doesn’t make API calls itself, instead it will talk to any HTTP library you provide which responds to http verbs like get and post. Faraday works well here. You can also give it a test double.
24. Provides Test Helpers expect(MyResourceClass).to have_action(:delete). that_handles(:no_content). at_path('/users') ResourceKit also provides test helpers for RSpec to make testing your API clients much easier. I’ll show you an example shortly.
25. class DropletResource < ResourceKit::Resource resources do action :find do verb :get # get is assumed if this is omitted path '/droplets/:id' handler(200) { |response| DropletMapping.extract_single(response.body, :read) } end action :all, 'GET /droplets' do handler(200) { |body| DropletMapping.extract_collection(body, :read) } end end end lib/resources/droplet_resource.rb Here’s how we would make a resource which talks to our Droplets API endpoint. We define two actions, all and find.
26. lib/resources/droplet_resource.rb action :create, 'POST /droplets' do body { |object| DropletMapping.representation_for(:create, object) } handler(201) { |response| DropletMapping.extract_single(response.body, :read) } end We also make POST like resources easy to compose by providing a body and giving you the ability to transform it on the fly in to JSON.
27. Step 1. Provide a Connection conn = Faraday.new(url: 'http://api.digitalocean.com') do |req| req.adapter :net_http end resource = DropletResource.new(connection: conn) My favourite part, using it. So we just defined an API client for talking to the droplets endpoint, let’s use it.
28. Step 2. Use It. conn = Faraday.new(url: 'http://api.digitalocean.com') do |req| req.adapter :net_http end resource = DropletResource.new(connection: conn) all_droplets = resource.all single_droplet = resource.find(id: 123) create = resource.create(Droplet.new) The we can directly call those actions we defined earlier as methods, and behind the scenes everything is handled for us.
29. Step 3. Test It. require 'resource_kit/testing' # Tag the spec with resource_kit to bring in the helpers RSpec.describe DropletResource, resource_kit: true do it 'handles a 201 with response body' do expect(described_class).to handle_response(:create). with(status: 201, body: ‘{“droplets”:[{…}]}’) do |handled| expect(handled).to all(be_kind_of(Droplet)) end end end https://github.com/digitalocean/resource_kit/tree/master/lib/resource_kit/testing And finally the third step is to test it. We provide some easy to use helpers for declaratively writing specs.
30. BYO Data Model Bring your own data model, in DropletKit we use simple classes, you can use anything you want which has a setter and getter method.
31. gem "resource_kit" And again throw this one in your Gemfile and bundle it.
32. No Dependencies (Except Faraday) Batteries Included Use what you need Production Ready™ Powers DropletKit ResourceKit digitalocean/resource_kit ResourceKit only depends on Faraday, I’m not 100% sure it needs to, so if that becomes a problem I’m sure we could remove it. You don’t need to use all of ResourceKit, only what you need.
33. Real world example DropletKit digitalocean/droplet_kit For a complete and advanced example, check out DropletKit on the Githubs.
34. Ivan Vanderbyl @ivanderbyl github.com/ivanvanderbyl We’re Hiring careers.digitalocean.com Thanks Thanks, we are always, so if you fancy a trip to NYC, come and talk to me.