ES in Rome - Cross cluster search with Elasticsearch

Transcript

1. ‹#› @lucacavanna Cross cluster search with Elasticsearch
2. Agenda 2 Why cross cluster search? How does it work? How does it compare to tribe node? 1 2 3
3. ‹#› Why cross cluster search?
6. ‹#› How does cross cluster search work?
7. ‹#› Search detour
8. cluster Search api 8 Client node1 logs 2P posts 1P node2 logs 3P users 1P node3 logs 1P posts 2P GET /posts/_search posts index: 2 primaries logs index: 3 primaries users index: 1 primary
9. cluster Search request parsing 9 Client node1 logs 2P posts 1P node2 logs 3P users 1P node3 logs 1P posts 2P The coordinating node parses the request GET /posts/_search
10. cluster Query phase 10 Client node1 logs 2P posts 1P node2 logs 3P users 1P node3 logs 1P posts 2P GET /posts/_search The query gets executed on the relevant shards The query gets executed on the relevant shards
11. cluster Reduce phase 11 Client node1 logs 2P posts 1P node2 logs 3P users 1P node3 logs 1P posts 2P GET /posts/_search The coordinating node receives "size" hits per shard and performs reduction
12. cluster Fetch phase 12 Client node1 logs 2P posts 1P node2 logs 3P users 1P node3 logs 1P posts 2P GET /posts/_search The coordinating node fetches the top hits from the relevant shards The coordinating node fetches the top hits from the relevant shards
13. cluster Search response 13 Client node1 logs 2P posts 1P node2 logs 3P users 1P node3 logs 1P posts 2P The coordinating node returns the top hits back to the client
14. ‹#› Cross cluster search
15. Register remote clusters PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": "host_au:9300" }, "usa" : { "seeds": "host_us:9300" } } } } 15
16. 16 Client europe node1 posts 1P users 1P node2 posts 3P posts 2P usa node2 posts 2P posts 1P australia node1 logs 1P node2 posts 1P GET /posts,usa:posts,australia:posts/_search The coordinating node parses the request
17. 17 Client europe node1 posts 1P users 1P node2 posts 3P posts 2P usa node2 posts 2P posts 1P australia node1 logs 1P node2 posts 1P The coordinating node fetches info about remote indices and their shards GET /posts,usa:posts,australia:posts/_search
18. 18 Client europe node1 posts 1P users 1P node2 posts 3P posts 2P usa node2 posts 2P posts 1P australia node1 logs 1P node2 posts 1P The query gets executed on the relevant local shards GET /posts,usa:posts,australia:posts/_search
19. 19 Client europe node1 posts 1P users 1P node2 posts 3P posts 2P usa node2 posts 2P posts 1P australia node1 logs 1P node2 posts 1P The query gets also executed on the relevant remote shards GET /posts,usa:posts,australia:posts/_search
20. 20 Client europe node1 posts 1P users 1P node2 posts 3P posts 2P usa node2 posts 2P posts 1P australia node1 logs 1P node2 posts 1P The coordinating node receives "size" hits per shard and performs reduction GET /posts,usa:posts,australia:posts/_search
21. 21 Client europe node1 posts 1P users 1P node2 posts 3P posts 2P usa node2 posts 2P posts 1P australia node1 logs 1P node2 posts 1P The coordinating node fetches the top hits from the relevant shards GET /posts,usa:posts,australia:posts/_search
22. 22 Client europe node1 posts 1P users 1P node2 posts 3P posts 2P usa node2 posts 2P posts 1P australia node1 logs 1P node2 posts 1P The coordinating node returns the top hits back to the client GET /posts,usa:posts,australia:posts/_search
23. Search response "hits" : [ { "_index" : "australia:posts", ... }, { "_index" : "posts", ... }, { "_index" : "usa:posts", ... } ] 23
24. ‹#› Connection settings
25. Limit connections per cluster PUT /_cluster/settings { "persistent" : { "search.remote" : { "initial_connect_timeout": "30s", "australia" : { "seeds": "host_au:9300" }, "usa" : { "seeds": "host_us:9300" } } } } 25
26. Limit connections per cluster PUT /_cluster/settings { "persistent" : { "search.remote" : { "australia" : { "seeds": "host_au:9300", "connections_per_cluster": 3 }, "usa" : { "seeds": "host_us:9300", "connections_per_cluster": 1 } } } } 26
27. Select nodes to connect to PUT /_cluster/settings { "persistent" : { "search.remote" : { "node.attr": "gateway", "australia" : { "seeds": "host_au:9300" }, "usa" : { "seeds": "host_us:9300" } } } } 27
28. 28 Client europe node1 posts 1P users 1P node2 posts 3P posts 2P usa australia The coordinating node only communicates with the nodes marked as "gateway" GET /posts,usa:posts,australia:posts/_search node2 posts 1P node1 logs 1P gateway node2 posts 2P posts 1P gateway
29. ‹#› How does it compare to tribe node?
30. • Adding remote clusters requires node restart 30 Tribe node Cross cluster search • Remote clusters can be dynamically registered
31. • Adding remote clusters requires node restart • Doesn't support indices with same names on different clusters 31 Tribe node Cross cluster search • Remote clusters can be dynamically registered • No limitations on indices naming
32. • Adding remote clusters requires node restart • Doesn't support indices with same names on different clusters • Requires an additional node (tribe) to join all the remote clusters 32 Tribe node Cross cluster search • Remote clusters can be dynamically registered • No limitations on indices naming • No additional nodes required
33. • Adding remote clusters requires node restart • Doesn't support indices with same names on different clusters • Requires an additional node (tribe) to join all the remote clusters • Bi-directional connections to every remote node 33 Tribe node Cross cluster search • Remote clusters can be dynamically registered • No limitations on indices naming • No additional nodes required • Uni-directional connections to selected gateway nodes
34. • Receives all cluster state updates from remote clusters 34 Tribe node Cross cluster search • Retrieves info on demand from remote clusters
35. • Receives all cluster state updates from remote clusters • Works with almost every api 35 Tribe node Cross cluster search • Retrieves info on demand from remote clusters • Specific to search api
36. ‹#› https://www.elastic.co/guide/en/elasticsearch/reference/5.x/ modules-cross-cluster-search.html Coming soon with Elasticsearch 5.3.0
37. ‹#› Thank you