Share
Embed
Amsterdam Elasticsearch meetup: Cross Cluster Search with Elasticsearch
Luca Cavanna
May 11, 2017
Transcript
70 slides
-
Agenda 2 Why Cross Cluster Search? How does it work?
Follow-up search improvements 1 2 3 4 How does it compare to Tribe node? -
cluster Search api 8 Client node1 logs 2P posts 1P
node2 logs 3P users 1P node3 logs 1P posts 2P GET /posts/_search posts index: 2 primaries logs index: 3 primaries users index: 1 primary -
cluster Search request parsing 9 Client node1 logs 2P posts
1P node2 logs 3P users 1P node3 logs 1P posts 2P The coordinating node parses the request GET /posts/_search -
cluster Query phase 10 Client node1 logs 2P posts 1P
node2 logs 3P users 1P node3 logs 1P posts 2P GET /posts/_search The query gets executed on the relevant shards The query gets executed on the relevant shards -
cluster Reduce phase 11 Client node1 logs 2P posts 1P
node2 logs 3P users 1P node3 logs 1P posts 2P GET /posts/_search The coordinating node receives "size" hits per shard and performs reduction -
cluster Fetch phase 12 Client node1 logs 2P posts 1P
node2 logs 3P users 1P node3 logs 1P posts 2P GET /posts/_search The coordinating node fetches the top hits from the relevant shards The coordinating node fetches the top hits from the relevant shards -
cluster Search response 13 Client node1 logs 2P posts 1P
node2 logs 3P users 1P node3 logs 1P posts 2P The coordinating node returns the top hits back to the client -
15 europe eu1 posts 1P users 1P eu2 posts 3P
posts 2P usa us1 posts 2P posts 1P australia au1 posts 1P au2 logs 1P -
Register remote clusters PUT /_cluster/settings { "persistent" : { "search.remote"
: { "australia" : { "seeds": ["host_au1:9300", "host_au2:9300"] }, "usa" : { "seeds": "host_us1:9300" } } } } 16 -
17 Client europe eu1 posts 1P users 1P eu2 posts
3P posts 2P usa us1 posts 2P posts 1P australia au1 posts 1P au2 logs 1P GET /posts,usa:posts,australia:posts/_search The coordinating node parses the request -
18 Client europe eu1 posts 1P users 1P eu2 posts
3P posts 2P usa us1 posts 2P posts 1P australia The coordinating node fetches info about remote indices and their shards au2 logs 1P au1 posts 1P GET /posts,usa:posts,australia:posts/_search -
19 Client europe eu1 posts 1P users 1P eu2 posts
3P posts 2P usa us1 posts 2P posts 1P australia posts 1P The query gets executed on the relevant local shards au2 logs 1P au1 posts 1P GET /posts,usa:posts,australia:posts/_search -
20 Client europe eu1 posts 1P users 1P eu2 posts
3P posts 2P usa us1 posts 2P posts 1P australia The query gets also executed on the relevant remote shards au2 logs 1P au1 posts 1P GET /posts,usa:posts,australia:posts/_search -
21 Client europe eu1 posts 1P users 1P eu2 posts
3P posts 2P usa us1 posts 2P posts 1P australia au2 logs 1P au1 posts 1P The coordinating node receives "size" hits per shard and performs reduction GET /posts,usa:posts,australia:posts/_search -
22 Client europe eu1 posts 1P users 1P eu2 posts
3P posts 2P usa us1 posts 2P posts 1P australia The coordinating node fetches the top hits from the relevant shards au2 logs 1P au1 posts 1P GET /posts,usa:posts,australia:posts/_search -
23 Client europe eu1 posts 1P users 1P eu2 posts
3P posts 2P usa us1 posts 2P posts 1P australia The coordinating node returns the top hits back to the client au2 logs 1P au1 posts 1P GET /posts,usa:posts,australia:posts/_search -
Search response "hits" : [ { "_index" : "australia:posts", ...
}, { "_index" : "posts", ... }, { "_index" : "usa:posts", ... } ] 24 -
26 europe eu1 posts 1P PUT /_cluster/settings { "persistent" :
{ "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P eu2 posts 2P -
europe 27 eu1 posts 1P australia au1 posts 1P au2
logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P eu2 posts 2P Fetch the cluster state and select remote nodes PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } -
Disable cross cluster calls from specific nodes cat config/elasticsearch.yml #
---------------- Cluster ---------------- # # Use a descriptive name for your cluster: # cluster.name: europe # # ---------------- Node -------------------- # # Use a descriptive name for the node: # node.name: eu2 # Disable connections to remote clusters from this node search.remote.connect: false 28 -
europe 29 eu1 posts 1P eu2 posts 2P Fetch the
cluster state and select remote nodes PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P -
europe 30 eu1 posts 1P eu2 posts 2P Fetch the
cluster state and select remote nodes PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P -
europe 31 eu1 posts 1P eu2 posts 2P Fetch the
cluster state and select remote nodes PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P -
Maximum number of remote nodes to connect to cat config/elasticsearch.yml
# ---------------- Cluster ---------------- # # Use a descriptive name for your cluster: # cluster.name: europe # # ---------------- Node -------------------- # # Use a descriptive name for the node: # node.name: eu1 # Max number of remote nodes to connect to search.remote.connections_per_cluster: 3 32 -
europe 33 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search -
europe 34 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search The coordinating node fetches info about remote indices and their shards -
europe 35 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search The query gets executed on the relevant remote shards -
europe 36 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search The coordinating node fetches the top hits from the relevant shards -
Pick node attribute to tag remote nodes with cat config/elasticsearch.yml
# ---------------- Cluster ---------------- # # Use a descriptive name for your cluster: # cluster.name: europe # # ---------------- Node -------------------- # # Use a descriptive name for the node: # node.name: eu1 # Attribute that enables/disables remote connections search.remote.node.attr: gateway 37 -
cat config/elasticsearch.yml # ---------------- Cluster ---------------- # # Use a
descriptive name for your cluster: # cluster.name: australia # # ---------------- Node -------------------- # # Use a descriptive name for the node: # node.name: au5 # # Add custom attributes to the node: # node.attr.gateway: true 38 Tag selected nodes as remote gateways (2) -
europe 39 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P gateway PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } Fetch the cluster state and select remote nodes -
europe 40 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search gateway -
europe 41 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search gateway The coordinating node fetches info about remote indices and their shards -
europe 42 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search The query gets executed on the relevant remote shards gateway -
europe 43 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P Client GET /australia:posts/_search gateway The coordinating node fetches the top hits from the relevant shards -
europe 45 eu1 posts 1P eu2 posts 2P Remote nodes
selection: - is node available? - is node gateway? - is version compatible? PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P -
‹#› Upgrading to 6.0 with Rolling Restarts The hardest part
about upgrading to a new major version has been the fact that you have to do a full cluster restart to get there. No more! You will be able to upgrade from the latest 5.x version to 6.0 using rolling restarts, without any cluster downtime. https://www.elastic.co/blog/elasticsearch-6-0-0-alpha1-released -
europe 47 eu1 posts 1P eu2 posts 2P 6.0 won't
be able to talk directly to 5.4, only to 5.latest PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": ["host_au1", "host_au3"] } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 6.0 6.0 -
europe 48 eu1 posts 1P eu2 posts 2P PUT /_cluster/settings
{ "persistent" : { "search.remote" : { "australia" : { "seeds": "host_au7" } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 6.0 au7 5.latest Fetch the cluster state and select remote nodes 6.0 -
europe 49 eu1 posts 1P eu2 posts 2P PUT /_cluster/settings
{ "persistent" : { "search.remote" : { "australia" : { "seeds": "host_au7" } } } } australia au1 posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 au7 5.latest Fetch the cluster state and select remote nodes 6.0 6.0 -
europe 50 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 au7 5.latest 6.0 6.0 Client GET /australia:posts/_search -
europe 51 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 au7 5.latest 6.0 6.0 Client GET /australia:posts/_search The coordinating node fetches info about remote indices and their shards -
europe 52 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 au7 5.latest 6.0 6.0 Client GET /australia:posts/_search The query gets executed on the relevant remote shards -
europe 53 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 logs 1P au3 posts 2P au4 logs 2P au6 posts 3P au5 logs 3P 5.4 5.4 5.4 5.4 5.4 5.4 au7 5.latest 6.0 6.0 Client GET /australia:posts/_search The coordinating node fetches the top hits from the relevant shards -
europe 56 eu1 posts 1P eu2 posts 2P australia au1
posts 1P au2 posts 4P au3 posts 2P au4 posts 5P au6 posts 6P au5 posts 3P Client GET /australia:posts/_search gateway The coordinating node receives "size" hits per shard and performs reduction -
Shard count limit (5.3) PUT /_cluster/settings { "persistent" : {
"action.search.shard_count.limit" : 1000 } } 57 -
Multiple reduction phases (5.4) GET /posts,*:posts/_search?batched_reduce_size=512 { "query" : {
… }, "aggs" : { … } } 58 -
Multiple reduction phases (5.4) { "took" : 43, "timed_out" :
false, "num_reduce_phases": 6, "_shards" : { "total" : 3000, "successful" : 3000, "failed" : 0 }, "hits" : { "total" : 3987245, "max_score" : 1.0, "hits" : […] } } 59 -
Shard count limit (5.4) PUT /_cluster/settings { "persistent" : {
"action.search.shard_count.limit" : Long.MAX_VALUE } } 60 -
• Adding remote clusters requires node restart 62 Tribe node
Cross Cluster Search • Remote clusters can be dynamically registered -
• Adding remote clusters requires node restart • Receives all
cluster state updates from remote clusters 63 Tribe node Cross Cluster Search • Remote clusters can be dynamically registered • Retrieves info on demand from remote clusters -
• Adding remote clusters requires node restart • Receives all
cluster state updates from remote clusters • Doesn't support indices with same names on different clusters 64 Tribe node Cross Cluster Search • Remote clusters can be dynamically registered • Retrieves info on demand from remote clusters • No limitations on indices naming -
• Adding remote clusters requires node restart • Receives all
cluster state updates from remote clusters • Doesn't support indices with same names on different clusters • Requires an additional node (tribe) to join all the remote clusters 65 Tribe node Cross Cluster Search • Remote clusters can be dynamically registered • Retrieves info on demand from remote clusters • No limitations on indices naming • No additional nodes required -
• Bi-directional connections to every remote node 66 Tribe node
Cross Cluster Search • Uni-directional connections to selected gateway nodes -
• Bi-directional connections to every remote node • Specific type
of node with its own handling and edge cases, hard to test and maintain 67 Tribe node Cross Cluster Search • Uni-directional connections to selected gateway nodes • Isolated component integrated into a specific API: easier to test and maintain -
• Bi-directional connections to every remote node • Specific type
of node with its own handling and edge cases, hard to test and maintain • Works with almost every api 68 Tribe node Cross Cluster Search • Uni-directional connections to selected gateway nodes • Isolated component integrated into a specific API: easier to test and maintain • Specific to _search api