Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Container Platforms as Equalizers: Running Heal...

Jamie Hewland
December 11, 2018

Container Platforms as Equalizers: Running Health Services Across the World

Talk at KubeCon + CloudNativeCon North America 2018

Praekelt.org creates and operates a number of health and youth-related services which are hosted on containerised clusters around the world, often in countries without an established cloud provider presence. This means that the infrastructure reliability and tooling that may typically be available are not. In addition, as a small team managing clusters in several isolated datacenters around the world, achieving commonality is challenging.

While we started using container orchestration because we wanted to increase resource utilisation and deployment agility, we have found the real value has been in our ability to abstract many of the differences between clusters.

Now, as we move towards Kubernetes, we will share lessons for shifting developers between different container orchestrators as seamlessly as possible by using Spinnaker as a common continuous deployment tool.

Jamie Hewland

December 11, 2018
Tweet

More Decks by Jamie Hewland

Other Decks in Technology

Transcript

  1. Container Platforms as Equalizers: Running Health Services Across the World

    Jamie Hewland KubeCon + CloudNativeCon Seattle 11 December 2018
  2. We build open-source, scalable platforms that allow anyone with a

    mobile phone to access vital information and essential services—putting wellbeing in the palm of their hands. Our Technologies
  3. Nonprofit projects • Projects developed through partnerships with funders •

    Several different projects with different funders at any one time • Projects vary in size • Multi-year, national-scale • Short pilot projects, studies
  4. Youth 1.Running more sites more efficiently - Mobi-sites & social

    media - Education, sexual & reproductive health
  5. For Youth we wanted to: • Make more efficient usage

    of resources • Automate recovery of downed servers • Make it easier to deploy code TOWARDS CONTAINERS & CONTAINER ORCHESTRATION
  6. Health 2. Running apps closer to their users - Messaging

    (SMS, USSD, WhatsApp) - Maternal health & ECD
  7. Vumi messaging platform • Tools to integrate with carriers, aggregators

    • SMS & USSD (“Star Menu”) • Develop message flows in a web UI or JavaScript • Fancy message store based on Riak in AWS in Ireland
  8. Unstructured Supplementary Service Data (USSD) • Session-based and latency sensitive

    • 180s max duration, typically billed per 20s bit.ly/USSDSMS
  9. For Health we wanted to: • Host more services closer

    to users (lower latency) • Keep data in-country (as part of national health system) • Scale up our platform to support more users TOWARDS CONTAINERS & CONTAINER ORCHESTRATION
  10. 2015 at Praekelt.org Youth: Free Basics • Launched in many

    countries simultaneously • Incubator with 100 new sites Health: MomConnect • Services in SA taking off • POPI legislation
  11. 2015 in Cloud Native Standardisation • Kubernetes reaches version 1.0

    along with formation of CNCF • Docker at version 1.4-1.9, Open Container Project (eventually OCI) formed
  12. We chose Mesos/Marathon • “Simpler” than Kubernetes • Fewer upfront

    decisions • Fewer independent components • No buy-in to networking model necessary • Marathon app = run n instances of container image x • Emphasis on things we wanted • Resource constraints the default • High-availability
  13. What is a Seed project? • Government endorsement • Multi-stakeholder

    consortium • Using open source technologies • With room and budget for co-design • Using feedback loops to improve service delivery • Integrated into national information systems • Has potential for a national footprint within a year • With an explicit view to handover within a year of implementation
  14. Container orchestration Hoped that container orchestration would help because… •

    High level of automation => high level of self-sufficiency? • Able to support a “national-scale” platform • Common platform between different countries
  15. Container portability Containers allowed us to… • Get an MVP

    running in a new country quickly • Migrate easily between hosting providers • Treat different hosting environments as the same/similar
  16. In retrospect Seed was hugely ambitious • Microservices • In-country

    hosting • Container orchestration Spent too many “innovation tokens”
  17. High-availability for what? In Nigeria… • One public IP accessible

    from one host • Frequent network outages • Persistent storage issues (errors lead to RO-filesystems) • Windows-only remote desktop connection • System clocks changing underneath VMs
  18. High-availability for what? In Uganda… • Physical servers hosted by

    partner in office • Starts with 2 hosts • Dial-up-like internet speeds • (Please try make your container images small) • Servers eventually moved to Gov. datacenter & 3rd added
  19. Peak Mesos • Johannesburg Mesos/Marathon cluster peaks at 30 nodes

    • ~255GB RAM, 120 cores, ~900 containers • Bare-metal. VMs on self-managed XenServer • Move to OSS Mesosphere DC/OS • Like a distribution for Mesos, with lots of extras, more stability • Team of 4 SREs managing clusters in 4 countries
  20. Infrastructure for handovers Did we do the best we could

    have? • Over-estimated scale of projects • Common platform benefitted us, but did it benefit those inheriting it? • When you have a container orchestrator-shaped hammer, everything looks like a nail
  21. Infrastructure for handovers What would the ideal system for handing

    over look like? • Container orchestration? Possibly not. • Distributed system or an old-school web server? • Can we get portability without container orchestration? • How much are we willing to give up?
  22. Co-designing infrastructure • Historically only did co-design with end-users If

    we’re developing infrastructure to hand over to others… …then the inheritors of that infrastructure are also end-users. • Shouldn’t dictate what technology others must use without their input
  23. Kubernetes • Increasingly hard to argue that it’s not the

    de-facto standard • The killer feature is the community & ecosystem • Global & local (South African) community • Building things we wouldn’t need to if we used Kubernetes • Docker images “the price of admission to modern platforms such as Kubernetes” — paid
  24. Building too much stuff for Mesos Load-balancing HTTPS certificates Secrets

    & secure introduction Persistent storage Config file management …
  25. Kubernetes • Still more complicated than we’d like • Many

    technology decisions to make • Moves very fast • No de-facto “distribution” yet • Waiting for “the Ubuntu of distributions” to (possibly) use in not-the-Cloud • Strategy: • Use managed Cloud services where we can • Use the simplest everything (no service meshes for us)
  26. More cloud coming to Africa Cloud datacenter (TBC) Edge or

    CDN PoP (Azure, AWS, Google, Cloudflare, Fastly) x x x