Hailo 2.0 Internal Tech Talk

This is similar to the H2 board presentation, but includes more depth aimed at selling the platform to the internal tech teams. The section on "Designing new things" is a good example of how we were thinking about these internal customers of our platform.

Transcript

1. Hailo 2.0 David Gardner, Architect at Hailo
2. H1: November 2011
3. PHP Cust API eu-west-1 PHP Driver API Java Hailo Engine MySQL PHP Driver API PHP Driver API PHP Cust API PHP Cust API Load Balancer Load Balancer MySQL Redis
4. H1.5: 2012
5. PHP Cust API eu-west-1 Java Hailo Engine MySQL PHP Driver API PHP Cust API PHP Cust API ELB ELB Java Hailo Engine PHP Driver API ELB MySQL Java Hailo Engine MySQL PHP Driver API ELB C* C* C* PHP Cust service PHP Credits service Java Pay service eu-west-1 us-east-1
6. Why H2?
7. Original ambitions •  Provide a simple framework for us to build an efficient, resilient, second generation Hailo •  Allow Hailo to scale the business along three axis: adding features to our current business, adding cities and brand new stuff •  Solve pain points in our current architecture •  Be productive
8. + features + cities + brand new + productive - pain
9. PHP Cust API eu-west-1 Java Hailo Engine MySQL PHP Driver API PHP Cust API PHP Cust API ELB ELB Java Hailo Engine PHP Driver API ELB MySQL Java Hailo Engine MySQL PHP Driver API ELB C* C* C* PHP Cust service PHP Credits service Java Pay service eu-west-1 us-east-1 Servers needed per-city + cities
10. PHP Cust API eu-west-1 Java Hailo Engine MySQL PHP Driver API PHP Cust API PHP Cust API ELB ELB Java Hailo Engine PHP Driver API ELB MySQL Java Hailo Engine MySQL PHP Driver API ELB C* C* C* PHP Cust service PHP Credits service Java Pay service eu-west-1 us-east-1 City configuration in many places + cities PHP array YAML Config service plists built into app plists built into app XML XML Config service PHP array YAML
11. PHP Cust API eu-west-1 Java Hailo Engine MySQL PHP Driver API PHP Cust API PHP Cust API ELB ELB Java Hailo Engine PHP Driver API ELB MySQL Java Hailo Engine MySQL PHP Driver API ELB C* C* C* PHP Cust service PHP Credits service Java Pay service eu-west-1 us-east-1 Coordination changing three apps + features
12. PHP Cust API eu-west-1 Java Hailo Engine MySQL PHP Driver API PHP Cust API PHP Cust API ELB ELB Java Hailo Engine PHP Driver API ELB MySQL Java Hailo Engine MySQL PHP Driver API ELB C* C* C* PHP Cust service PHP Credits service Java Pay service eu-west-1 us-east-1 Unclear responsibilities + features Eg: payment or cancellation
13. PHP Cust API eu-west-1 Java Hailo Engine MySQL PHP Driver API PHP Cust API PHP Cust API ELB ELB Java Hailo Engine PHP Driver API ELB MySQL Java Hailo Engine MySQL PHP Driver API ELB C* C* C* PHP Cust service PHP Credits service Java Pay service eu-west-1 us-east-1 Broad but inflexible services + features, + brand new
14. PHP Cust API eu-west-1 Java Hailo Engine MySQL PHP Driver API PHP Cust API PHP Cust API ELB ELB Java Hailo Engine PHP Driver API ELB MySQL Java Hailo Engine MySQL PHP Driver API ELB C* C* C* PHP Cust service PHP Credits service Java Pay service eu-west-1 us-east-1 Separate push deployment models + productive rsync conan/cap rsync conan/cap conan/cap
15. PHP Cust API eu-west-1 Java Hailo Engine MySQL PHP Driver API PHP Cust API PHP Cust API ELB ELB Java Hailo Engine PHP Driver API ELB MySQL Java Hailo Engine MySQL PHP Driver API ELB C* C* C* PHP Cust service PHP Credits service Java Pay service eu-west-1 us-east-1 Different auth models - pain none none IP whitelist plus token turned off IP whitelist plus login service
16. PHP Cust API eu-west-1 Java Hailo Engine MySQL PHP Driver API PHP Cust API PHP Cust API ELB ELB Java Hailo Engine PHP Driver API ELB MySQL Java Hailo Engine MySQL PHP Driver API ELB C* C* C* PHP Cust service PHP Credits service Java Pay service eu-west-1 us-east-1 SPOFs - pain
17. PHP Cust API PHP Cust API PHP Cust API ELB ELB Load balancing broken and complex - pain PHP Cust HAP ELB PHP Credits HAP ELB Java Pay HAP Phone ELB API ELB HAProxy Service ELB HAProxy Service us-east-1 eu-west-1 us-east-1
18. H2: 2013
19. Utopia Dystopia Static Better Cheaper Sooner Dynamic Broken Inefficient
20. Key features •  Lose PHP, adopt Go – gains in efficiency of developer time and compute resource •  Eliminate all SPOFs – adopt a cloud native approach to build a working whole out of ephemeral and often broken parts •  Scale engineering output in line with additional resource – services with few, clearly defined responsibilities reduce friction •  Increase reusability – develop features by composing fine-grained services that are agnostic to Hailo’s current operation
21. Design for global
22. us-east-1 C* C* C* eu-west-1 ELB Go “Thin” API RabbitMQ Message Bus (federated clusters per AZ) Go Service Go Service Java Service C* C* C* ELB Go “Thin” API RabbitMQ Message Bus (federated clusters per AZ) Go Service Go Service Java Service 1. 2. 3.
23. ELB Go “Thin” API 1. H1 Driver API H2 “API” Service H2 Orch. Service H2 Core Service v1-api-driver-london.elasticride.com api-driver-london.elasticride.com http RMQ RMQ = Hailo’s RabbitMQ Message Bus RMQ RMQ { "match":{ "regulatoryArea":"LON,DUB", "path":"/v1/point", "proportion”:0.5 }, "action":2 }, { "match":{ "path":"/v1/experiment", "proportion”:1.0 }, "action":2 }, We borrow the idea of “specificity” from CSS and use this concept to sort the rules and thus define how they are evaluated. Need to build better tools around this. Danger Zone!
24. 2. RMQ RMQ cluster RMQ RMQ cluster RMQ RMQ cluster Service haproxy Services always connect to localhost. HAProxy sends to the same AZ, unless that AZ is down, in which case it “fails over” to a different AZ. RMQ runs in clusters of 2, within each AZ Each exchange is federated to the other AZs
25. 2. H2 RabbitMQ Exchange •  Bespoke exchange for RabbitMQ, written in Erlang •  Based on the headers exchange •  Main difference is random distribution of messages to a single bound queue (rather than delivering messages matched on headers to every matched bound queue)
26. Handler 3. Logic Storage go-platform-layer go-service-layer Self-configuring external service adapters Library for building services that talk via RMQ // Clone makes a new city by cloning // an existing one func Clone(req *server.Request) (proto.Message, errors.Error) { // do stuff return &clone.Response{}, nil }
27. Discovery Service Binding Service Config Service Login Service •  Keeps track of every running instance of a service within a single region •  Stores this information as ephemeral nodes in Zookeeper, keeping a watch to ensure strong consistency between all instances of the discovery service within a region •  Sends heartbeats to services periodically via RMQ and removes dead instances •  Self-healing system because instances that don’t receive heartbeats will try to reconnect and failing that die
28. Discovery Service Binding Service Config Service Login Service •  Creates bindings within RabbitMQ for all running services, leveraging information in the discovery service •  Reacts to services coming up and down by creating and destroying bindings •  Bindings establish a connection between an exchange and a queue •  Stores and manages control plane data in order to provide advanced bindings such as “send 10% of traffic to this particular version of the service”
29. Discovery Service Binding Service Config Service Login Service •  Stores application configuration data as JSON •  Able to store JSON under any arbitrary key •  Can combine many keys, on request, to serve up “compiled” config
30. Discovery Service Binding Service Config Service Login Service •  Credential and session/token store for all applications •  The only thing that is able to issue and sign (with private key) tokens •  Applications can exchange a session ID for a token, which they can then use to establish authorisation
31. Designing new things
32. The RII of design •  Responsibility Design what service(s) are needed to implement a feature. Define what the responsibility is for each service. Responsibilities shouldn’t overlap. •  Interface Design the inputs and outputs of each service. •  Implementation Build the service. R > I > I: responsibility is king
33. ETA Service Routing Service Phone Service Profile Service State Service Charge Service Near Drivers Service Tow Truck Service Restau- rant Service Place Service /v1/customer/neardrivers API TIER ORCHESTRATION TIER CORE TIER
34. Tooling
35. Move fast: build a service in 30 minutes
36. Move fast: simple global deployment
37. Move fast: automatic monitoring of services
38. Move fast: automatic service health metrics
39. Move fast: powerful tools for developer insight
40. The future
41. Priorities •  Stability •  Test failure •  Test, benchmark and optimise the core platform •  Make us 100% certain we can cope with the demands of large markets •  Efficiency •  Use less AWS – make better use of what we have •  Build out tooling to make developers lives easier
42. Automate end-to-end testing •  Run tests against any environment (anticipating N environment future) •  Share knowledge so others can write end-to-end tests •  Aggregate results to get a clear and stable picture of performance This means that we have a stable and reliable picture of end-to-end performance and correctness that we can observe whenever we make changes.
43. Build environments on-demand •  Single button press to launch something that will run Hailo – something you can connect driver and customer apps to •  Focus on complete automation of Hailo, to provide a high level of repeatability •  Unlocks many advantages including: •  ability to failure test on virgin environments and then throw them away •  ability to develop and QA features in isolation •  continual assertion that we haven’t cut any corners with automation
44. Add release tooling •  Package up services and configuration into a “release” •  Environment automatically built to run a clone of production plus your changes – you can connect apps directly to this •  End-to-end tests kicked off automatically •  UI showing results of tests, summary of changes, and a huge DEPLOY button for someone in QA to “sign off” the release and push it into production