Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch Cluster Deployment Model

Elasticsearch Cluster Deployment Model

Elasticsearch クラスターの構成例を紹介します。
- 開発環境
- 最小プロダクション環境
- 専任マスターノード
- クライアントノード
- Ingest ノード
- Logstash
- ES Hadoop
- Tribe ノード
- モニタリング専用クラスター

Kosho Owa

May 30, 2016
Tweet

More Decks by Kosho Owa

Other Decks in Technology

Transcript

  1. Cluster ϓϩμΫγϣϯͰͷ࠷খߏ੒ 4 ES node Apps ES node ES node

    ES clients, Kibana, Logstash, Beats, User apps and etc.
  2. Cluster ઐ೚Ϛελʔϊʔυ 5 Data node Apps Master node Data node

    Data node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Master Node discovery.zen.minimum_master_nodes: 1 node.data: false # Data Nodes discovery.zen.minimum_master_nodes: 1 node.master: false
  3. Cluster ৑௕Ϛελʔϊʔυ 6 Data node Apps Master node Data node

    Data node Master node Master node Data node Data node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Master-eligible Nodes discovery.zen.minimum_master_nodes: 2 node.data: false # Data Nodes discovery.zen.minimum_master_nodes: 2 node.master: false
  4. Cluster ΫϥΠΞϯτϊʔυ 7 Data node Apps Master node Data node

    Data node Master node Master node Data node Data node Client node ES clients, Kibana, Logstash, Beats, User apps and etc. # config/elasticsearch.yml # Client Node node.master: false node.data: false
  5. Cluster *OHFTUϊʔυ GSPN7 8 Data node Apps Master node Data

    node Data node Master node Master node Data node Data node Client node Data shippers Ingest node Logstash, Beats and etc. Kibana, User apps and etc. # config/elasticsearch.yml # Client/Ingest Node node.master: false node.data: false
  6. Cluster Cluster Message queues Cluster ଱σʔλαʔδ 10 Data shippers Logstash

    nodes ES nodes Logstash, Beats and etc. Redis, Kafka or RabbitMQ
  7. Cluster &4)BEPPQ 11 ES nodes ES Hadoop HDFS Front-ends Storm,

    Spark, MapReduce, Cascading, Hive and Pig
  8. 5SJCFϊʔυͱ͸ • ෳ਺ͷΫϥελʔʹ·͕ͨΔΞΫηεΛఏڙ͢ΔΫϥΠΞϯτ ‒ ಡΈॻ͖ ‒ ౷߹తͳαʔν • $MVTUFS4UBUFΠϯσοΫεͷϚʔδ •

    ղܾ͢Δ໰୊ ‒ ஍ཧతʹ཭Εͨ৔ॴʹଘࡏ͢ΔΫϥελʔͷӡ༻ ‒ σʔλαΠϩ ෳ਺ͷσʔλ؅ཧऀ 13
  9. ઃఆྫ 16 # config/elasticsearch.yml node.name: tribe marvel.enabled: false tribe: on_conflicts:

    drop ldn: cluster.name: london discovery.zen.ping.unicast.hosts: [“london-master1:9300", “london-master2:9300”, “london- master3:9300”] tyo: cluster.name: tokyo discovery.zen.ping.unicast.hosts: [“tokyo-master1:9300", “tokyo-master2:9300”, “tokyo- master3:9300”]
  10. ϓϥάΠϯͱͷ࿈ܞ • .BSWFM ‒ :PVNBZJODMVEFFYDMVEFNPOJUPSJOHEBUBGSPNUIF5SJCFOPEF • 4IJFME ‒ *OTUBMMJOH4IJFMEJTOFDFTTBSZXIFOFWFSZPVSDMVTUFSTBSFQSPUFDUFECZ4IJFME 17

    • .BSWFM%PDVNFOUBUJPO$POGJHVSJOH.BSWFM$POGJHVSJOHB5SJCF/PEFUP8PSLXJUI.BSWFM IUUQTXXXFMBTUJDDPHVJEFFONBSWFMDVSSFOUDPOGJHVSBUJPOIUNMUSJCFDPOGJH • 4IJFME*OTUBMMJOH4IJFMEIUUQTXXXFMBTUJDDPHVJEFFOTIJFMEDVSSFOUJOTUBMMJOH TIJFMEIUNMUSJCFOPEF
  11. ϞχλϦϯάઐ༻Ϋϥελʔ 20 Cluster ES node marvel-agent ES node marvel-agent ES

    node marvel-agent Monitoring Cluster ES node marvel-agent Kibana marvel-ui # config/elasticsearch.yml marvel.agent.exporters: id1: type: http host: [“es-mon-1:9200”,…]
  12. ΠϯετʔϧNBSWFMBHFOU 22 $ cd /usr/share/elasticsearch/ $ sudo bin/plugin install license

    -> Installing license... $ sudo bin/plugin install marvel-agent -> Installing marvel-agent... @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.lang.RuntimePermission setFactory * javax.net.ssl.SSLPermission setHostnameVerifier See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y
  13. ΠϯετʔϧNBSWFMVJ 23 $ cd /opt/kibana $ sudo bin/kibana plugin --install

    elasticsearch/marvel Installing marvel … Plugin installation complete
  14. ϓϥάΠϯͱͷ࿈ܞ • 8BUDIFS ‒ $MVTUFSIFBMUIDIBOHFT )JHINFNPSZVTBHF )JHIDQVVTBHF )JHIGJMFEFTDSJQUPS VTBHF )JHIGJFMEEBUBDBDIFVTBHF

    /PEFTKPJOJOHPSMFBWJOHUIFDMVTUFS • 4IJFME - ,JCBOBΞΫηε࣌ͷύεϫʔυઃఆ - 3#"$ ໾ׂϕʔεͷΞΫηε੍ޚ FHDMVTUFSNPOJUPSJOHPOMZVTFST - NBSWFMBHFOUͱNPOJUPSJOHDMVTUFSؒͷ௨৴ͷ҉߸Խ 24 • 8BUDIFS&YBNQMF8BUDIFT8BUDIJOH.BSWFM%BUBIUUQTXXXFMBTUJDDPHVJEFFO XBUDIFSDVSSFOUXBUDIJOHNBSWFMEBUBIUNM • 4IJFME$POGJHVSJOH$MJFOUTBOE*OUFHSBUJPOT6TJOH.BSWFMXJUI4IJFMEIUUQT XXXFMBTUJDDPHVJEFFOTIJFMEDVSSFOUNBSWFMIUNM