Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Infrastructure testing using Kubernetes

Ran Tavory
February 24, 2019

Infrastructure testing using Kubernetes

Testing Kafka replication using uReplicator, using Kubernetes, Prometheus, Grafana and friends

Ran Tavory

February 24, 2019
Tweet

More Decks by Ran Tavory

Other Decks in Programming

Transcript

  1. Hello! I am Ran Tavory I work on multi-region You

    can find me @rantav (slack/twitter/email) 4
  2. The Goal: Test Kafka Replication ⊙ Replicate 30MB - 300MB

    of traffic per second ⊙ Validate correctness ⊙ Measure latency ⊙ Run test cases ◦ Broker crash, cluster resize, packet loss... 5
  3. What does that do? ⊙ Provision VMs in AWS ⊙

    Setup VPCs, Subnets, Routing Tables ⊙ Create Security Groups ⊙ Setup Load Balancers ⊙ Install Kubernetes (etcd, masters, nodes) ⊙ Setup Monitoring (Prometheus & Grafana and install dashboard) ⊙ Install Weave Scope ⊙ Install Kafkas (and test them) ⊙ Install uReplicator (and test it) ⊙ Install test programs ⊙ And more… (ASGs, DHCP etc) 23
  4. $ kops create cluster \ --zones us-east-1a,us-east-1b,us-east-1c \ --node-count 40

    \ --node-size i3.large \ --master-size m4.large \ --master-zones us-east-1a \ --networking calico \ --cloud aws 24
  5. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 27 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster
  6. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 28 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator
  7. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 10 producers 29 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator 4 consumers
  8. $ kubectl \ --context us-east-1.k8s.local \ -n kafka-source \ scale

    \ statefulset kafka-source \ --replicas 29 40 RESIZE KAFKA CLUSTER
  9. $ kubectl \ --context eu-west-1.k8s.local \ -n ureplicator \ scale

    deployment \ ureplicator-worker \ --replicas 9 42 ADD uREPLICATOR WORKER
  10. $ kubectl --context eu-west-1.k8s.local \ -n ureplicator port-forward \ ureplicator-controller-76ff85b889-l9mzl

    9000 $ curl -X DELETE http://localhost:9000/topics/topic5 $ curl -X POST -d \ '{"topic":"topic5", "numPartitions":"300"}' \ http://localhost:9000/topics 47 ADD PARTITIONS (cont)
  11. 68

  12. Thanks! Any questions? You can find me at @rantav &

    [email protected] 69 This presentation: https://speakerdeck.com/rantav/infrastructure-testing-using-kubernetes