[2016.05 Meetup #2][TALK #1] Miguel Silva - Testing Distributed Systems

Miguel Silva's talk slides: Testing Distributed Systems

Originally published by Feedzai at:

http://www.slideshare.net/feedzai/2016-0516-testingdistributedsystemsv1-1

All rights belong to Miguel Silva / Feedzai.

Transcript

1. Testing Distributed Systems Presented by Miguel Silva, QA Team Leader
2. Meet Feedzai  https://feedzai.wistia.com/medias/toz8ny7ep7
3. Buzzwords
4. Numbers 99.95 availability Max 250ms latency $2B of transactions processed per day 3 deployments in fortune 500 companies Growing 2 person / week (Hiring!)
5. A typical deployment Pulse server App engine SQL Databases Zookeeper RabbitMq Cassandra 18 machines
6. The largest deployment 2 Datacenters 6 environments 150 Servers 13Tb of memory 3000 cores 250Tb of storage
7. Go beyond unit tests End to end functionality Fault-tolerance High-availability Machine / datacenter synchronization
8. Docker and bash Hard to maintain Hard to expand Hard to CI Hard to test scripts Hard to script tests
9. System-tests framework
10. System-tests framework
11. System-tests framework Write system-tests like unit tests
12. System-tests framework Write system-tests like unit tests
13. Feature highlights Set a multi-container, multi-datacenter test scenario in couple of lines Easy to assert cluster states, connections, applications, etc. Make deployment-like scenario system- tests like you do a unit test
14. Feature highlights Abstracted container creation, setup, volumes complexity from the tests Integrated in CI environment (jenkins) Developers easily implement system- tests
15. Bonus – production firewalls All containers are blocked by a firewall When testing, open only the necessary ports Ensures functionality with production security
16. Bumps along the way Docker demon may have some concurrency problems and some calls fail in some tests Containers would not be removed Solved by doing retries on some docker calls
17. Bumps along the way Docker-java leaves TCP IP sockets stuck (TCP-WAIT) No sockets available during a build run Workaround by enabling socket reuse at kernel level
18. Bumps along the way Before 1.9, container IP would change when containers are stopped /etc/hosts updated on every docker run After 1.9, IPs are static and more predictable Containers always keep their IP
19. Current status 128Gb RAM 24 CPU cores over SSD 315 tests 3h build 7 parallel tests running Up to 100 parallel containers running bottlene ck
20. Next steps Integrate more images Docker swarm integration Distributed test execution
21. Thank You
22. Visit www.feedzai.com/wickedsmart LIVE EVENTS | MEET THE PEOPLE BEHIND THE MACHINE