Upgrade to Pro — share decks privately, control downloads, hide ads and more …

HBaseCon 2016 West - Containerizing Apache HBas...

HBaseCon 2016 West - Containerizing Apache HBase Clusters

At Facebook, all production HBase clusters run in a containerized environment, with every daemon running inside its own LXC container. Containerization allows us to ensure isolation between services running on the same host and simplify operations, but sometimes abstractions leak and problems can't be addressed inside the container. In this talk, we will discuss how Facebook runs HBase as a stateful service inside containers and we will discuss some of the issues we've found when doing so.

https://hbase.apache.org/hbasecon-archives.html#HBaseCon_West_2016

Javier Maestro

May 24, 2016
Tweet

More Decks by Javier Maestro

Other Decks in Technology

Transcript

  1. 1964 IBM 360 1979 UNIX chroot 1982 BSD chroot 1999

    FreeBSD jail 2013 Docker 2007-8 cgroups LXC
  2. Buy vs. Lease • Our scale • Timing • Control

    / Full Ownership • Financial Infrastructure (os, network) Container Platform (Tupperware) Application Services Physical (data center, hardware)
  3. Tupperware Benefits • Configuration Spec • Deployments • Scheduler •

    Health Monitor • Logging • Canary • Web UI / CLI / API • Elasticity (auto-scaling)
  4. • High iops on /dev/sda • Synchronous logging The “Noisy

    Neighbor” From the Container HBase Container /dev/sda
  5. • Configuration Management putting load on /dev/sda • Memory pressure

    forcing paging on /dev/sda • Large configuration subscriptions • Bloated packages The “Noisy Neighbor” From the Host System HBase Container Host System /dev/sda
  6. • Increased latency and timeouts • Cyclical spikes in io-wait

    across all of the Region Servers / Datanodes Performance & The Bug From the Container
  7. • Cyclical spikes across all disks hitting 100% utilization •

    No sign of any applications accessing the disks (iops == 0) Performance & The Bug From the Host System
  8. • Log entries of a “learning cycle” every few minutes

    • Correlation to the drives locking up • Configuration mode to enable this “learning cycle” Performance & The Bug From the Hardware RAID Controller learning cycle HBase Container Host System
  9. Conclusions • Containers provide a rich suite of tools and

    technologies to create standard, consistent and repeatable services • However, there are critical decisions to be made: • Buy vs. Lease • What parts of the Container Platform to use • You still need to be aware of what is happening behind the container • The leverage of the container goes both ways
  10. Q&A