Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elasticsearchに入門してチューニングや監視をした話 #elasticsearchj...
Search
Manabu Matsuzaki
October 06, 2017
Technology
1
1.3k
Elasticsearchに入門してチューニングや監視をした話 #elasticsearchjp /elasticsearch-fukuoka-2017
Elasticsearch勉強会 in 福岡 2017の発表資料です
Manabu Matsuzaki
October 06, 2017
Tweet
Share
More Decks by Manabu Matsuzaki
See All by Manabu Matsuzaki
Spring BootユーザのためのArmeria入門 #jsug / Introduce to Armeria for Spring users
matsumana
0
2.8k
Canary Release with Argo Rollouts #ふくばねてす / canary-release-with-argo-rollouts
matsumana
1
1.1k
Getting started Central Dogma with Golang #fukuokago #umedago / getting-started-central-dogma-with-golang
matsumana
0
890
Micrometer入門 #javaq / introduce-to-micrometer
matsumana
1
2.9k
ArmeriaとCentral Dogmaから学ぶ、マイクロサービスに必要な機能 #edayfuk / lean-from-armeria-and-central-dogma
matsumana
0
4.3k
SREcon19 Americas 参加レポート #srefukuoka / srecon19-americas-report
matsumana
0
870
SRE入門 & チームで取り組んでいるSRE #srefukuoka / introduce-to-sre
matsumana
0
1.3k
Introduce to Armeria and Central Dogma #GWD_Nulab / introduce-to-armeria-and-central-dogma
matsumana
0
560
Connector/JでMaster/Slave Replication構成のMySQLに接続する #mysql_casual_fukuoka /connector-j-master-slave-replication
matsumana
0
1.5k
Other Decks in Technology
See All in Technology
AIとSREの未来 / AI and SRE
ymotongpoo
2
1.4k
金融システムをモダナイズするためのAmazon Elastic Kubernetes Service(EKS)ノウハウ大全
daitak
0
130
libsyncrpcってなに?
uhyo
0
160
Zero Data Loss Autonomous Recovery Service サービス概要
oracle4engineer
PRO
2
7.2k
OSMnx Galleryの紹介
mopinfish
0
150
アプリケーションの中身が見える!Mackerel APMの全貌と展望 / Mackerel APMリリースパーティ
mackerelio
0
450
[zh-TW] DevOpsDays Taipei 2025 -- Creating Awesome Change in SmartNews!(machine translation)
martin_lover
1
650
AIの電力問題を概観する
rmaruy
1
220
Devin&Cursor、それぞれの「本質」から導く最適ユースケース戦略
empitsu
8
2.5k
単一Gitリポジトリから独立しました
lycorptech_jp
PRO
0
120
カンファレンスのつくりかた / The Conference Code: What Makes It All Work
tomzoh
8
940
大事なのは、AIの精度だけじゃない!〜1円のズレも許されない経理領域とAI〜
jun_nemoto
11
5.2k
Featured
See All Featured
Practical Orchestrator
shlominoach
188
11k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.7k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.3k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
14
1.5k
The Cost Of JavaScript in 2023
addyosmani
49
8.1k
Building Adaptive Systems
keathley
41
2.6k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
32
5.8k
The Pragmatic Product Professional
lauravandoore
35
6.7k
Embracing the Ebb and Flow
colly
85
4.7k
It's Worth the Effort
3n
184
28k
Transcript
Elasticsearchʹೖͯ͠ νϡʔχϯάࢹΛͨ͠ Elasticsearchษڧձ in Ԭ 2017 2017/10/06 @matsumana
ࣗݾհ • ໊લɿ দ࡚ ֶ • ॴଐɿ LINE Fukuokaגࣜձࣾ •
Twitterɿ @matsumana
ΞδΣϯμ • ೖ͢Δ࣌ʹؾΛ͚ͭͨࣄ • ηοτΞοϓ • νϡʔχϯά • όοΫΞοϓͱϦΧόϦ •
ϞχλϦϯά • ·ͱΊ
લఏ • ͜ͷεϥΠυͷ༰Elastic stack 5.4ʙ5.6࣌ͷใͰ͢ • LogstashͱBeatsࠓͷͱ͜Ζ͏༧ఆͳ͠ FluentdΛ͍ͬͯΔ • X-PackແঈͰ͑ΔMonitoring͚͍ͩͬͯΔ
• αʔϏεͷࢹΛطʹPrometheusͰߦ͍ͬͯΔͷͰ elasticsearch_exporterΛಋೖͯ͠PrometheusͰࢹ͢Δ
લఏ • ࠓճElasticsearchΛಋೖͨ͠ϓϩδΣΫτͷϢʔεέʔεҎԼ • શจݕࡧ༻్Ͱ͍ͬͯΔ • indexing1ʹճɺόονʹΑΔbulk indexingͷΈ • Ϣʔβsearch͔͠ߦΘͳ͍
• ͦͷϓϩδΣΫτͷαʔόαΠυͷݴޠJava
ೖ͢Δ࣌ʹؾΛ͚ͭͨࣄ • ެࣜυΩϡϝϯτɺެࣜϒϩάΛಡΉ • άάͬͯग़͖ͯͨݸਓͷϒϩάͳͲग़དྷΔ͚ͩಡ·ͳ͍ ʢݹ͍όʔδϣϯΛݩʹॻ͔Εͨهࣄʹҙʣ • ࡉ͔͍ڍಈΛΓ͍ͨ࣌ιʔεΛಡΉ
ެࣜυΩϡϝϯτ ϘϦϡʔϜ͕ଟ͍ͷͰ ಛʹେࣄͦ͏ͳՕॴ͚ͩಡΜͩ https://matsumana.info/blog/2017/06/17/elasticsearch-getting-started-docs/ Ҏલϒϩάʹ·ͱΊ͍ͯ·͢
ηοτΞοϓ • ެࣜͷAnsible playbookͰηοτΞοϓ ʢࢼͨ͠ࣄͳ͍͚ͲɺPuppet൛ɺChef൛͋ΔΈ͍ͨʣ • ศརɻ͓͢͢Ί • 2017/10/03ݱࡏͩͱɺ5.5.1ͱ͍͏tagΛ͏ͷ͕ྑ͍
ެࣜplaybookͷॏཁͦ͏ͳม es_version es_instance_name es_heap_size es_config.cluster.name es_config.node.name → "${HOSTNAME}" ͕ྑ͍ es_config.discovery.zen.ping.unicast.hosts
es_config.node.master es_config.node.data es_config.bootstrap.memory_lock → true ͕ྑ͍ es_config.network.host → 0.0.0.0 es_jvm_custom_parameters → Ճ͍ͨ͠JVM OptionΛࢦఆ es_plugins → analysis-icuanalysis-kuromojiͳͲΛࢦఆՄೳ es_enable_xpack → X-PackΛ͏߹true es_xpack_features → monitoringͳͲΛΠϯετʔϧՄೳ https://github.com/elastic/ansible-elasticsearch/blob/master/README.md
༻͍ͯ͠Δόʔδϣϯ • CentOS 7.3 (ެࣜͷAnsible playbookCentOS 6Λαϙʔτ͍ͯ͠ͳ͍) • Oracle JDK
1.8.0 • Elastic stack 5.5.1 • Elasticsearch • Kibana (X-Pack MonitoringͷμογϡϘʔυΛݟΔͨΊʹಋೖ) • X-Pack Monitoring ※Slideͷ࠷ޙʹัଊ͋Γ (ӡ༻ʹཱͭใ͕͋Δ͔ݟͯΈ͔ͨͬͨͷͰಋೖ) • Cerebro 0.6.6
νϡʔχϯά ʢαʔόαΠυʣ
• APIͰಈతʹઃఆͰ͖Δύϥϝʔλͱग़དྷͳ͍ύϥϝʔλ͕͋Δ • ಈతʹมߋͰ͖ͳ͍߲elasticsearch.ymlʹॻ͘ ʢөʹElasticsearchͷ࠶ىಈ͕ඞཁʣ • ಈతʹมߋͰ͖Δ߲APIͰมߋ͢Δ ʢElasticsearchΛ࠶ىಈ͠ͳͯ͘ө͞ΕΔʣ • elasticsearch.ymlͱAPIͷ྆ํͰಉ߲͡Λมߋͨ͠߹ɺ
APIͰͷมߋ͕༏ઌ͞ΕΔ ҙ
ઃఆͨ͠ओͳύϥϝʔλ
• Cacheؔ࿈ • indices.fielddata.cache.size • indices.queries.cache.size • indices.requests.cache.size • Thread
poolؔ࿈ ※Slideͷ࠷ޙʹัଊ͋Γ • thread_pool.index.queue_size • thread_pool.bulk.queue_size elasticsearch.ymlͰߦ͏Ϋϥελશମͷઃఆ ʢಈతʹมߋͰ͖ͳ͍ʣ
• Circuit Breakerؔ࿈ • indices.breaker.total.limit • indices.breaker.fielddata.limit • indices.breaker.request.limit •
Segments and Mergingؔ࿈ ※Slideͷ࠷ޙʹัଊ͋Γ • indices.store.throttle.max_bytes_per_sec σϑΥϧτHDD͚ʹͳ͍ͬͯΔͷͰɺ SSDΛ͏߹มߋ͢ΔΑ͏ʹDefinitive Guideʹॻ͍ͯ͋Δ APIͰߦ͏Ϋϥελશମͷઃఆ ʢಈతʹมߋͰ͖Δʣ
• number_of_shards • number_of_replicas bulk indexingத0ʹઃఆɻऴΘͬͨΒ͢ • _allΛແޮԽ • dynamic:
strict • index.indexing.slowlogؔ࿈ • index.search.slowlogؔ࿈ APIͰߦ͏index/index templateຖͷઃఆ ʢಈతʹมߋͰ͖Δʣ
νϡʔχϯά ʢΫϥΠΞϯταΠυʣ
• ࠷ۙɺElasticsearchࣾREST Clientਪ͠ͳงғؾ • Transport ClientͱҧͬͯαʔόͱΫϥΠΞϯτϥΠϒϥϦͷ όʔδϣϯΛҰகͤ͞ͳͯ͘ྑ͍ • ύϑΥʔϚϯεTransport Clientͱಉ͘͡Β͍Β͍͠
• Elasticsearch 5.6ͰJava High Level REST Client ϦϦʔε͞Εͨ REST Client
• Transport ClientͰbulkॲཧ͢Δ߹ɺ BulkProcessorΛ͏ͱྑ͍ɻҎԼͷύϥϝʔλ͕ઃఆՄೳ • bulkActions • bulkSize • flushInterval
• concurrentRequests • backoffPolicy https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-docs-bulk-processor.html Transport Client
όοΫΞοϓͱϦΧόϦ • ࠓճͷϓϩδΣΫτͰɺݩͷσʔλશͯMySQLʹอଘ͞Ε͍ͯͯ ̍ʹճόονͰElasticsearchʹશ݅builk indexing͍ͯ͠Δ ElasticsearchͷόοΫΞοϓऔ͍ͬͯͳ͍ • ElasticsearchΫϥελશମ͕յΕΔͱ͍͏࠷ѱͷέʔεͷ߹ɺ ElasticsearchΫϥελΛAnsibleͰ࠶ߏஙͯ͠ɺ͍ͭͷఆظόονͰ શ݅builk
indexing͢Δఆ https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html
ϞχλϦϯά • ͍ͬͯΔPrometheus exporter • node_exporter • jmx_exporter • elasticsearch_exporter
• άάΔͱ͍͔ͭ͘ݟ͔ͭΔ͕ɺݱࡏ͍ͬͯΔͷ͜Ε https://github.com/justwatchcom/elasticsearch_exporter • authorͱcontributorୡ͕ඞཁͳϝτϦΫε͔͠export͞Εͯ ͍ͳ͍ͷͰɺ͕ࣗඞཁͳϝτϦΫε͕ແ͚ΕPR͠Α͏
• Load Avg • CPU༻ • Disk I/O • network
trafic node_exporterͰऩूͰ͖Δ OSʹؔ͢ΔϝτϦΫεͷҰྫ
• Heap༻ • GCճ • GCॴ༻࣌ؒ jmx_exporterͰऔಘͰ͖Δ JVMʹؔ͢ΔϝτϦΫεͷҰྫ
• cluster health • Document count(primary shardͷΈ) • Document size(primary
shardͷΈ) • Thread pool queue count • Thread pool rejected count • Circuit Breaker tripped count elasticsearch_exporterͰऩूͰ͖Δ Elasticsearchʹؔ͢ΔϝτϦΫεͷҰྫ
• Fielddata cache size • Query cache size • Query
cache ώοτ (hit / total) • Request cache size elasticsearch_exporterͰऩूͰ͖Δ Elasticsearchʹؔ͢ΔϝτϦΫεͷҰྫ
elasticsearch_exporterҎ֎Ͱ औಘ͍ͯ͠Δ Elasticsearchؔ࿈ͷϝτϦΫε
• ϦΫΤετճ • ྫ֎ൃੜճ • Search latency percentile (0.5, 0.75,
0.98, 0.99, 0.999) prometheus_clientΛαʔόαΠυΞϓϦʹΈࠐΜͰ DropWizard MetricsͰऩू͍ͯ͠ΔϝτϦΫε
• Slow logൃੜճ • NorikraͰूܭ • Raw slow log •
Elasticsearchʹอଘ • Elasticsearchʹอଘͨ͠σʔλKibanaͰݟΔͷ͕Ұൠత ͩͱࢥ͏͕ɺ͍׳Ε͍ͯΔGrafanaͰҰ୴ݟΔࣄʹͨ͠ slow logΛFluentdͰऩू
slow log ՄࢹԽͷ֓ཁਤ
·ͱΊ
• ެࣜυΩϡϝϯτΛಡ͏ • ެࣜAnsible playbook͓͢͢Ί • ύϥϝʔλͷઃఆมߋɺAPIͰಈతʹग़དྷΔͷͱग़དྷͳ͍ ͷ͕͋Δ • elasticsearch_exporter͚ͩͰऩूͰ͖ͳ͍ϝτϦΫεผͷ
ํ๏Ͱɻ • αʔόαΠυΞϓϦʹDropWizard MetricsΛΈࠐΜͰܭଌ • slow logΛऩूͯ͠NorikraͰूܭ • Raw slow logผͷElasticsearchʹอଘ
ิ • X-Pack MonitoringΛΠϯετʔϧ͢ΔͱɺϝτϦΫεσʔλ Λอଘ͢Δઐ༻ͷΠϯσοΫε͕ग़དྷΔɻΠϯσοΫεͷอ ଘઌผͷElasticsearchΫϥελʹ͠·͠ΐ͏ • Thread poolؔ࿈ͷύϥϝʔλνϡʔχϯάɺqueue_size͚ͩ ૿ͯ͠ύϑΥʔϚϯε্͕ΒͣΩϡʔ͕ͪ૿͑Δ͚ͩ
ͳͷͰɺsize߹Θͤͯνϡʔχϯά͠·͠ΐ͏ https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html • indices.store.throttle.max_bytes_per_sec Elasticsearch 6.0 Ͱഇࢭ͞Ε·͢ https://www.elastic.co/guide/en/elasticsearch/reference/6.0/breaking_60_settings_changes.html
Thank you :)