Container Platforms as Equalizers: Running Health Services Across the World

Talk at KubeCon + CloudNativeCon North America 2018

Praekelt.org creates and operates a number of health and youth-related services which are hosted on containerised clusters around the world, often in countries without an established cloud provider presence. This means that the infrastructure reliability and tooling that may typically be available are not. In addition, as a small team managing clusters in several isolated datacenters around the world, achieving commonality is challenging.

While we started using container orchestration because we wanted to increase resource utilisation and deployment agility, we have found the real value has been in our ability to abstract many of the differences between clusters.

Now, as we move towards Kubernetes, we will share lessons for shifting developers between different container orchestrators as seamlessly as possible by using Spinnaker as a common continuous deployment tool.

Transcript

1. Container Platforms as Equalizers: Running Health Services Across the World Jamie Hewland KubeCon + CloudNativeCon Seattle 11 December 2018
2. Introduction Me & Praekelt.org 00
3. About me Johannesburg, South Africa " Site Reliability Engineer (SRE) # Tech Ambassador @jayhewland @JayH5
4. Praekelt.org uses mobile technology to solve some of the world's largest social problems. Our Mission
5. We build open-source, scalable platforms that allow anyone with a mobile phone to access vital information and essential services—putting wellbeing in the palm of their hands. Our Technologies
6. Nonprofit projects • Projects developed through partnerships with funders • Several different projects with different funders at any one time • Projects vary in size • Multi-year, national-scale • Short pilot projects, studies
7. Before containers The Problem 01
8. Timeline …2014 2016 2017 2018 2019 2015
9. Youth 1.Running more sites more efficiently - Mobi-sites & social media - Education, sexual & reproductive health
10. web01 Nginx Django PostgreSQL The Internet
11. Funder Project Server/VM 1:n 1:1
12. The Internet web02 web01 web03 web04 Puppet Configuration management Replicate
13. web01 uWSGI Emperor x? Nginx web02 db01 db02 web03
14. For Youth we wanted to: • Make more efficient usage of resources • Automate recovery of downed servers • Make it easier to deploy code TOWARDS CONTAINERS & CONTAINER ORCHESTRATION
15. Health 2. Running apps closer to their users - Messaging (SMS, USSD, WhatsApp) - Maternal health & ECD
16. Vumi messaging platform • Tools to integrate with carriers, aggregators • SMS & USSD (“Star Menu”) • Develop message flows in a web UI or JavaScript • Fancy message store based on Riak in AWS in Ireland
17. ~200ms
18. Ping times to AWS data centres from Johannesburg www.cloudping.info
20. Unstructured Supplementary Service Data (USSD) • Session-based and latency sensitive • 180s max duration, typically billed per 20s bit.ly/USSDSMS
21. For Health we wanted to: • Host more services closer to users (lower latency) • Keep data in-country (as part of national health system) • Scale up our platform to support more users TOWARDS CONTAINERS & CONTAINER ORCHESTRATION
22. Containers And their orchestration 02
23. Timeline 2015 2016 2017 2018 2019 2014
24. 2015 at Praekelt.org Youth: Free Basics • Launched in many countries simultaneously • Incubator with 100 new sites Health: MomConnect • Services in SA taking off • POPI legislation
25. 2015 in Cloud Native Standardisation • Kubernetes reaches version 1.0 along with formation of CNCF • Docker at version 1.4-1.9, Open Container Project (eventually OCI) formed
26. We chose Mesos/Marathon • “Simpler” than Kubernetes • Fewer upfront decisions • Fewer independent components • No buy-in to networking model necessary • Marathon app = run n instances of container image x • Emphasis on things we wanted • Resource constraints the default • High-availability
28. Seed In-country maternal health platform 03
29. Timeline 2016 2015 2017 2018 2019 2014
30. What is a Seed project? • Government endorsement • Multi-stakeholder consortium • Using open source technologies • With room and budget for co-design • Using feedback loops to improve service delivery • Integrated into national information systems • Has potential for a national footprint within a year • With an explicit view to handover within a year of implementation
31. Container orchestration Hoped that container orchestration would help because… • High level of automation => high level of self-sufficiency? • Able to support a “national-scale” platform • Common platform between different countries
32. Container portability Containers allowed us to… • Get an MVP running in a new country quickly • Migrate easily between hosting providers • Treat different hosting environments as the same/similar
33. In retrospect Seed was hugely ambitious • Microservices • In-country hosting • Container orchestration Spent too many “innovation tokens”
34. HelloMama FamilyConnect MomConnect
35. High-availability for what? In Nigeria… • One public IP accessible from one host • Frequent network outages • Persistent storage issues (errors lead to RO-filesystems) • Windows-only remote desktop connection • System clocks changing underneath VMs
36. High-availability for what? “The cluster service was rebooted to bring up the highly available VMs.”
37. High-availability for what? In Uganda… • Physical servers hosted by partner in office • Starts with 2 hosts • Dial-up-like internet speeds • (Please try make your container images small) • Servers eventually moved to Gov. datacenter & 3rd added
38. Timeline 2015 2016 2018 2019 2014 2017
39. Peak Mesos • Johannesburg Mesos/Marathon cluster peaks at 30 nodes • ~255GB RAM, 120 cores, ~900 containers • Bare-metal. VMs on self-managed XenServer • Move to OSS Mesosphere DC/OS • Like a distribution for Mesos, with lots of extras, more stability • Team of 4 SREs managing clusters in 4 countries
40. The Nigerian & Ugandan platforms have now been handed over to local partners bit.ly/SeedRetro
41. Lessons learned Reflecting on Seed 04
42. Timeline 2015 2016 2017 2019 2014 2018
43. Infrastructure for handovers Did we do the best we could have? • Over-estimated scale of projects • Common platform benefitted us, but did it benefit those inheriting it? • When you have a container orchestrator-shaped hammer, everything looks like a nail
44. Infrastructure for handovers What would the ideal system for handing over look like? • Container orchestration? Possibly not. • Distributed system or an old-school web server? • Can we get portability without container orchestration? • How much are we willing to give up?
45. Co-designing infrastructure • Historically only did co-design with end-users If we’re developing infrastructure to hand over to others… …then the inheritors of that infrastructure are also end-users. • Shouldn’t dictate what technology others must use without their input
46. GlobalMoms
47. Kubernetes, Spinnaker, & beyond Looking forward… 05
48. Timeline 2015 2016 2017 2018 2014 2019+
49. When you have to manage every layer… …you can’t afford to add more of them
50. Where we can use cloud… And where we can’t… ? ? ? ?
51. Kubernetes • Increasingly hard to argue that it’s not the de-facto standard • The killer feature is the community & ecosystem • Global & local (South African) community • Building things we wouldn’t need to if we used Kubernetes • Docker images “the price of admission to modern platforms such as Kubernetes” — paid
52. Building too much stuff for Mesos Load-balancing HTTPS certificates Secrets & secure introduction Persistent storage Config file management …
53. Kubernetes • Still more complicated than we’d like • Many technology decisions to make • Moves very fast • No de-facto “distribution” yet • Waiting for “the Ubuntu of distributions” to (possibly) use in not-the-Cloud • Strategy: • Use managed Cloud services where we can • Use the simplest everything (no service meshes for us)
54. GitHub Docker Hub Travis CI Mesosphere DC/OS
55. GitHub Docker Hub Travis CI Mesosphere DC/OS Kubernetes
56. More cloud coming to Africa Cloud datacenter (TBC) Edge or CDN PoP (Azure, AWS, Google, Cloudflare, Fastly) x x x
57. Thank you. Want to read more about this? medium.com/mobileforgood Special thanks to the Linux Foundation @jayhewland jamie@praekelt.org @praekeltorg