Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[2016.05 Meetup #2][TALK #1] Miguel Silva - Tes...

[2016.05 Meetup #2][TALK #1] Miguel Silva - Testing Distributed Systems

Miguel Silva's talk slides: Testing Distributed Systems

Originally published by Feedzai at:

http://www.slideshare.net/feedzai/2016-0516-testingdistributedsystemsv1-1

All rights belong to Miguel Silva / Feedzai.

DevOps Lisbon

May 16, 2016
Tweet

More Decks by DevOps Lisbon

Other Decks in Technology

Transcript

  1. Numbers 99.95 availability Max 250ms latency $2B of transactions processed

    per day 3 deployments in fortune 500 companies Growing 2 person / week (Hiring!)
  2. Docker and bash Hard to maintain Hard to expand Hard

    to CI Hard to test scripts Hard to script tests
  3. Feature highlights Set a multi-container, multi-datacenter test scenario in couple

    of lines Easy to assert cluster states, connections, applications, etc. Make deployment-like scenario system- tests like you do a unit test
  4. Feature highlights Abstracted container creation, setup, volumes complexity from the

    tests Integrated in CI environment (jenkins) Developers easily implement system- tests
  5. Bonus – production firewalls All containers are blocked by a

    firewall When testing, open only the necessary ports Ensures functionality with production security
  6. Bumps along the way Docker demon may have some concurrency

    problems and some calls fail in some tests Containers would not be removed Solved by doing retries on some docker calls
  7. Bumps along the way Docker-java leaves TCP IP sockets stuck

    (TCP-WAIT) No sockets available during a build run Workaround by enabling socket reuse at kernel level
  8. Bumps along the way Before 1.9, container IP would change

    when containers are stopped /etc/hosts updated on every docker run After 1.9, IPs are static and more predictable Containers always keep their IP
  9. Current status 128Gb RAM 24 CPU cores over SSD 315

    tests 3h build 7 parallel tests running Up to 100 parallel containers running bottlene ck