Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Infrastructure Testing Using Kubernetes And Gol...

Infrastructure Testing Using Kubernetes And Golang

Ran Tavory

May 06, 2019
Tweet

More Decks by Ran Tavory

Other Decks in Technology

Transcript

  1. Hello! I am Ran Tavory I work on multi-region at

    appsflyer You can find me @rantav (twitter/gmail/facebook) 3
  2. The Goal: Test Kafka Replication ⊙ Replicate 30MB - 300MB

    of traffic per second ⊙ Validate correctness ⊙ Measure latency ⊙ Run test cases (failure scenarios) ◦ Broker crash, cluster resize, packet loss... 4
  3. What does that do? ⊙ Provision VMs in AWS ⊙

    Setup VPCs, Subnets, Routing Tables ⊙ Create Security Groups ⊙ Setup Load Balancers ⊙ Install Kubernetes (etcd, masters, nodes) ⊙ Setup Monitoring (Prometheus & Grafana and install dashboard) ⊙ Install Weave Scope ⊙ Install Kafkas (and test them) ⊙ Install uReplicator (and test it) ⊙ Install test programs ⊙ And more… (ASGs, DHCP etc) 23
  4. $ kops create cluster \ --zones us-east-1a,us-east-1b,us-east-1c \ --node-count 40

    \ --node-size i3.large \ --master-size m4.large \ --master-zones us-east-1a \ --networking calico \ --cloud aws 24
  5. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 27 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster
  6. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 28 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator
  7. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 10 producers 29 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator 4 consumers
  8. $ kubectl \ --context us-east-1.k8s.local \ -n kafka-source \ scale

    \ statefulset kafka-source \ --replicas 29 40 RESIZE KAFKA CLUSTER
  9. $ kubectl \ --context eu-west-1.k8s.local \ -n ureplicator \ scale

    deployment \ ureplicator-worker \ --replicas 9 42 ADD uREPLICATOR WORKER
  10. $ kubectl --context eu-west-1.k8s.local \ -n ureplicator port-forward \ ureplicator-controller-76ff85b889-l9mzl

    9000 $ curl -X DELETE http://localhost:9000/topics/topic5 $ curl -X POST -d \ '{"topic":"topic5", "numPartitions":"300"}' \ http://localhost:9000/topics 47 ADD PARTITIONS (cont)
  11. 72

  12. Thanks! Any questions? You can find me at @rantav &

    [email protected] 73 This presentation: https://speakerdeck.com/rantav/infrastructure-testing-using-kubernetes- and-go The project: https://github.com/AppsFlyer/kafka-mirror-tester