Share
Embed
Jepsen: Mon cluster est-il cohérent ?
A short introduction to jepsen, a distributed databases testing tool by @aphyr
Arnaud Bailly
June 10, 2014
Transcript
35 slides
-
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Jepsen: Testing Cluster Consistency Arnaud Bailly <[email protected]> @abailly http://blog.foldlabs.com/ 2014-06 -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Jepsen? ▶ A tool for testing that distributed databases respect some consistency model in face of network partitions -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Jepsen? ▶ A tool for testing that distributed databases respect some consistency model in face of network partitions ▶ A set of primitives to define and run such tests -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Jepsen? ▶ A tool for testing that distributed databases respect some consistency model in face of network partitions ▶ A set of primitives to define and run such tests ▶ A Domain Specific Language for test and network partitions generators (aka. nemesis) -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Jepsen? ▶ A tool for testing that distributed databases respect some consistency model in face of network partitions ▶ A set of primitives to define and run such tests ▶ A Domain Specific Language for test and network partitions generators (aka. nemesis) ▶ Model checkers, includingLinearizability Checker (aka. Knossos) for asserting correctness of test runs -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Jepsen? ▶ A tool for testing that distributed databases respect some consistency model in face of network partitions ▶ A set of primitives to define and run such tests ▶ A Domain Specific Language for test and network partitions generators (aka. nemesis) ▶ Model checkers, includingLinearizability Checker (aka. Knossos) for asserting correctness of test runs ▶ With specific tests for RabbitMQ, etcd, ElasticSearch, DAtomic, Riak -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Jepsen? ▶ A tool for testing that distributed databases respect some consistency model in face of network partitions ▶ A set of primitives to define and run such tests ▶ A Domain Specific Language for test and network partitions generators (aka. nemesis) ▶ Model checkers, includingLinearizability Checker (aka. Knossos) for asserting correctness of test runs ▶ With specific tests for RabbitMQ, etcd, ElasticSearch, DAtomic, Riak ▶ Written by @aphyr (aka. Kyle Kingsbury) of Call Me Maybe fame -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Why I Should Care - Tools Figure : The NoSQL Database Zoo -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Why I Should Care - Usage ▶ Message Queues w/ different delivery guarantees -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Why I Should Care - Usage ▶ Message Queues w/ different delivery guarantees ▶ Locks & Mutexes -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Why I Should Care - Usage ▶ Message Queues w/ different delivery guarantees ▶ Locks & Mutexes ▶ Configuration -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Why I Should Care - Usage ▶ Message Queues w/ different delivery guarantees ▶ Locks & Mutexes ▶ Configuration ▶ Service Discovery -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Why I Should Care - Usage ▶ Message Queues w/ different delivery guarantees ▶ Locks & Mutexes ▶ Configuration ▶ Service Discovery ▶ Distributed Consensus -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Why I Should Care - Usage ▶ Message Queues w/ different delivery guarantees ▶ Locks & Mutexes ▶ Configuration ▶ Service Discovery ▶ Distributed Consensus ▶ Sessions Synchronization -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Why I Should Care - Usage ▶ Message Queues w/ different delivery guarantees ▶ Locks & Mutexes ▶ Configuration ▶ Service Discovery ▶ Distributed Consensus ▶ Sessions Synchronization ▶ … -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Mapping the CAP Landscape Figure : CAP as A Metric Space -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Mapping the CAP Landscape Figure : Consistency and Partition Tolerance -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Partitioning? -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Partitioning? -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Partitioning? -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Consistency? ▶ Quiescent Consistency -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Consistency? ▶ Quiescent Consistency ▶ Eventual Consistency -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Consistency? ▶ Quiescent Consistency ▶ Eventual Consistency ▶ Causal Consistency -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Consistency? ▶ Quiescent Consistency ▶ Eventual Consistency ▶ Causal Consistency ▶ Read-Your-Writes Consistency -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Consistency? ▶ Quiescent Consistency ▶ Eventual Consistency ▶ Causal Consistency ▶ Read-Your-Writes Consistency ▶ Sequential Consistency -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Consistency? ▶ Quiescent Consistency ▶ Eventual Consistency ▶ Causal Consistency ▶ Read-Your-Writes Consistency ▶ Sequential Consistency ▶ Linearizability -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . What is Consistency? ▶ Quiescent Consistency ▶ Eventual Consistency ▶ Causal Consistency ▶ Read-Your-Writes Consistency ▶ Sequential Consistency ▶ Linearizability ▶ … -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Sequential Consistency -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Linearizability -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Demo -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Questions? -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Links ▶ jepsen proper -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Links ▶ jepsen proper ▶ Knossos the linearizability checker -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Links ▶ jepsen proper ▶ Knossos the linearizability checker ▶ Kyle Kingsbury’s blog and the famous Call Me Maybe posts that introduced Jepsen -
. . . .. . . . .. . .
. .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . .. . . . .. . . . . .. . . . .. . . . . .. . . . .. . . . .. . Links ▶ jepsen proper ▶ Knossos the linearizability checker ▶ Kyle Kingsbury’s blog and the famous Call Me Maybe posts that introduced Jepsen ▶ ETH Zurich Distributed Systems Course