Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ログ・係数集約と可視化・分析
Search
Shuhei Ozawa
April 26, 2018
Technology
0
140
ログ・係数集約と可視化・分析
Fluentd,Embulk,ElasticStack6.0
Shuhei Ozawa
April 26, 2018
Tweet
Share
More Decks by Shuhei Ozawa
See All by Shuhei Ozawa
Amebaアフィリエイト基盤の GKEアーキテクチャと マイクロサービス
ozashu
0
200
production_ready_envoy
ozashu
2
1.2k
Python for web architectures
ozashu
0
920
PyQではじめるPython
ozashu
0
430
インフラエンジニアのWEBアプリ入門
ozashu
1
8.1k
Other Decks in Technology
See All in Technology
【新卒研修資料】LLM・生成AI研修 / Large Language Model・Generative AI
brainpadpr
25
17k
Modern_Data_Stack最新動向クイズ_買収_AI_激動の2025年_.pdf
sagara
0
220
多様な事業ドメインのクリエイターへ 価値を届けるための営みについて
massyuu
1
390
業務自動化プラットフォーム Google Agentspace に入門してみる #devio2025
maroon1st
0
200
Why Governance Matters: The Key to Reducing Risk Without Slowing Down
sarahjwells
0
110
pprof vs runtime/trace (FlightRecorder)
task4233
0
170
AIAgentの限界を超え、 現場を動かすWorkflowAgentの設計と実践
miyatakoji
0
150
Findy Team+のSOC2取得までの道のり
rvirus0817
0
370
Oracle Cloud Infrastructure:2025年9月度サービス・アップデート
oracle4engineer
PRO
0
470
社内お問い合わせBotの仕組みと学び
nish01
0
450
神回のメカニズムと再現方法/Mechanisms and Playbook for Kamikai scrumat2025
moriyuya
4
590
後進育成のしくじり〜任せるスキルとリーダーシップの両立〜
matsu0228
7
2.8k
Featured
See All Featured
Testing 201, or: Great Expectations
jmmastey
45
7.7k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
9
960
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
19
1.2k
Making Projects Easy
brettharned
119
6.4k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.7k
The World Runs on Bad Software
bkeepers
PRO
71
11k
Bash Introduction
62gerente
615
210k
Writing Fast Ruby
sferik
629
62k
Docker and Python
trallard
46
3.6k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
657
61k
Intergalactic Javascript Robots from Outer Space
tanoku
273
27k
Transcript
ϩάɾूͱՄࢹԽɾੳ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 1
Agenda — Fluentd — Embulk — ElasticStack6.0 ϓϩμΫτษڧձ 2018/04/23 -
Ozawa Shuhei 2
Fluentdͱ — Fluentdγϯϓϧʹϩάऩू͕Ͱ͖Δ — σʔλ࿈ܞͷϋϒ — ֦ுੑͷߴ͍ετϦʔϛϯάϩάίϨΫλ ϓϩμΫτษڧձ 2018/04/23 -
Ozawa Shuhei 3
Fluentdͱ — ʮCNCF(Cloud Native Computing Foundation)ʯͷཧ͢ΔϓϩδΣΫτ — Kubernetes Prometheus
ͱ͍ͬͨΫϥυωΠςΟϒͳ OSS ٕज़ͷਪਐΛߦ͏ஂମ — Kubernetesڥʹ͓͚Δϩάऩूπʔϧͷ ඪ४ ͱͳͬͨ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 4
Fluentdͱtd-agent — Fluentd — ࠷৽όʔδϣϯv1.1.3 - 2018/04/03 — FluentdͷίΞιϑτΣΞ —
ϓϥάΠϯݸผʹར༻ऀଆͰΠϯετʔϧ — ࠷৽Λࢼ͍ͨ͠߹FluentdຊମΛ͏ — td-agent — Fluentdʹ֤छϓϥάΠϯRubyڥΛΈࠐΜͩύοέʔδ — ओཁڥͰgemίϚϯυʹΑΔΠϯετʔϧ͕Մೳ — LinuxͷΠϯετʔϧ༰қ — αϙʔτOSʹ੍͕͋Δ͕ґଘؔͷ͕΄΅ͳ͠ — τϨδϟʔσʔλଆͰݕূࡁͳͷͰຊ൪ɺ҆ఆՔಇ͕͍͍ͷtd-agent ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 5
Ϣʔεέʔε 1. ϩάͷऩू ϩάΛϩʔΧϧσΟεΫ͔ΒɺRDBʹ͢͜ͱ͕Ͱ͖Δ ϩάͷܽଛߴՄ༻ੑΛҡ࣋͢Δ͜ͱͰ͛Δ 2. ؆୯ͳϦΞϧλΠϜूܭ ϓϥάΠϯΛར༻͢Δ͜ͱͰɺϦΞϧλΠϜͰεςʔλείʔυΛؚΜͩ ϩάΛΕΔ ՄࢹԽπʔϧͰάϥϑԽͳͲͰ͖Δɻ
3. ηϯαʔϩάऩू ηϯαʔ(ϥζύΠ)͔ΒήʔτΣΠʹूΊͯɺϩάαʔόʹूܭ͢Δɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 6
ར༻͠ͳ͍ํ͕͍͍έʔε — ϩάͷܽଛॏෳڐͣ͞ɺ࣮֬ʹॻ͖ࠐΉඞཁ͕͋Δͱ͍ ͏έʔε — ՝ۚσʔλͳͲ ϓϩμΫτษڧձ 2018/04/23 - Ozawa
Shuhei 7
ඇಉظϝοηʔδϯάαʔϏεQoS ωοτϫʔΫ্Ͱఏڙ͢ΔػೳΛ҆ఆతʹՔಇͤ͞ΔͨΊʹߦ͏ɺαʔϏε࣭ཧٕज़ At Most Once(σϑΥϧτ) At Least Once(Φϓγϣϯ) Exactly Once(αϙʔτ͞Ε
͍ͯͳ͍) ౸ୡอূͳ͠ ౸ୡอূ͋Γ ౸ୡอূ͋Γ ͛Δଆ͕ҰͰ͛Δ͜ͱ ͛Δଆ͕ҰͰ͛Δ͜ͱ ͛Δଆɺड͚औΔଆͱʹ ҰͰ৴͞ΕΔ͜ͱ ܽଛͷՄೳੑ͋Γ ܽଛ͠ͳ͍ ܽଛ͠ͳ͍ ॏෳ͠ͳ͍ ॏෳͷՄೳੑ ॏෳ͠ͳ͍ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 8
v0.12όʔδϣϯ old stable - ϓϥάΠϯ: Input, Parser, Filter, Output, Formatter,Buffer
- ҎԼͷΑ͏ͳ͕͋Δ - ඵ୯ҐͷΈ - windowsະରԠ - multi coreະରԠ - ϓϥάΠϯ͕ශऑ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 9
v0.14Ҏ߱ͷόʔδϣϯ v0.14 v1ͷ։ൃόʔδϣϯ - ϓϥάΠϯ: Input, Parser, Filter, Output, Formatter,
Storage, Buffer - վળ - New Plugin APIs - ϛϦඵରԠ - windowsରԠ - multi coreରԠ - New Plugin Helpers & Plugin Storage ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 10
v0.14Ҏ߱ͷόʔδϣϯ v1.0v0.14ͱػೳ͕ಉ͡Ͱstableόʔδϣϯɻ ໊લΛมߋ͚ͨͩ͠ɻ ࠷৽όʔδϣϯv1.1.3 - 2018/04/03 td-agent32017ͷ12݄͔Βstable൛͕Ͱ͓ͯΓɺ Fluentd v1ϕʔεʹͳ͍ͬͯΔɻ ϓϩμΫτษڧձ
2018/04/23 - Ozawa Shuhei 11
v0.12ͱv1 — v0.12 APIΛ༻͢ΔϓϥάΠϯɺFluentd v0.14ͱv1ͷؒͰαϙʔτ͞Ε Δʢv2Ͱഇࢭ͞ΕΔ༧ఆʣ — Fluentd v1ɺىಈ࣌ʹࣗಈతʹv0.12ελΠϧΛv1.0ελΠϧʹม͢Δ ͷͰɺv0.12ͷઃఆΛv1Ͱ࠶ར༻Մೳ
— Fluentd v1.0ͷ৽ػೳɺ৽͍͠APIΛ༻͢ΔϓϥάΠϯͰͷΈ༻Մೳ — flexible chunk keys — placeholders — Fluentd v0.12.xͰ৽͍͠APIΛ༻͢ΔϓϥάΠϯಈ࡞͠ͳ͍ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 12
v0.12ͱv1ͷઃఆͷҧ͍ v1outputͷόοϑΝύϥϝʔλʹ<buffer>sectionΛ͍ͬͯΔ # v1 <match pattern> @type foo database db1
apikey foobarbaz # buffer parameters <buffer> @type file path /path/to/buffer flush_interval 10s </buffer> </match> # v0.12 <match pattern> @type foo database db1 apikey foobarbaz # buffer parameters buffer_type file buffer_path /path/to/buffer flush_interval 10s </match> ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 13
fluent-plugin-bigquery — ࠷৽v2.0.0.beta — scheme͕ؒҧ͍ͬͯΔͱແݶʹretry͍ͯͨ͠ɻ — v0.2.13Ҏ߱Ͱσʔλ͕invalidͳͷʹretry͔͚ͯҙຯແ ͍ͷͰɺretryableͳྫ֎ͷ͚࣌ͩྫ֎্͛ͯ͠ɺ ͦͷଞͷྫ֎ͷ࣌ʹretry_stateΛ࿔ͬͯϦτϥΠΛڧ੍ ఀࢭ͍ͯ͠Δɻ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 14
fluent-plugin-bigquery(v.1.2.0)ͷoutbigqueryinsert.rb def insert(project, dataset, table_id, rows, schema, template_suffix) writer.insert_rows(project, dataset,
table_id, rows, template_suffix: template_suffix) rescue Fluent::BigQuery::Error => e if @auto_create_table && e.status_code == 404 && /Not Found: Table/i =~ e.message # Table Not Found: Auto Create Table writer.create_table(project, dataset, table_id, schema) raise "table created. send rows next time." end raise if e.retryable? if @secondary # TODO: find better way @retry = retry_state_create( :output_retries, @buffer_config.retry_type, @buffer_config.retry_wait, @buffer_config.retry_timeout, forever: false, max_steps: @buffer_config.retry_max_times, backoff_base: @buffer_config.retry_exponential_backoff_base, max_interval: @buffer_config.retry_max_interval, secondary: true, secondary_threshold: Float::EPSILON, randomize: @buffer_config.retry_randomize ) else @retry = retry_state_create( :output_retries, @buffer_config.retry_type, @buffer_config.retry_wait, @buffer_config.retry_timeout, forever: false, max_steps: 0, backoff_base: @buffer_config.retry_exponential_backoff_base, max_interval: @buffer_config.retry_max_interval, randomize: @buffer_config.retry_randomize ) end raise end ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 15
Fluentd v1.2ʹretryͷڍಈ͕มΘΓͦ͏ɻ Fluentdͷoutput oluginɺchunk flushதʹ෮ؼෆՄೳͳΤ ϥʔΛൃੜ͢Δ͕ɺ ͜ΕΒͷνϟϯΫΛॲཧ͢ΔͨΊʹ retry limit ͱ
secondary Λ͍ͬͯΔɻ — ࠶։࣌ʹഁଛͨ͠filechunkΛskipͯ͠আ https://github.com/fluent/fluentd/pull/1874 — chunkͷflushதʹoutput plugin͕ճ෮ෆՄೳͳΤϥʔΛ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 16
όοϑΝઃܭ(v0.12) Input͔ΒOutputใ͕͞ΕΔΈʹ͓͍ͯ OutputଆͰBufferͱQueueͱ͍͏Έ͕͋Δɻ ͜Ε͕ϩάͷܽଛΛ͠ͳ͍Έʹ͍ͯ͠Δɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 17
όοϑΝઃܭ(v0.12) — ࠷ॳʹใ͕ೖͬͯ͘ΔBufferͱ͍͏ػೳͷ࠷େαΠζ: buffer_chunk_limit — ࣍ʹQueueͱ͍͏෦ʹchunk͕ԡ͠ग़͞ΕΔ͕QueueͰԿݸ·ͰchunkΛ͑ΒΕΔ͔: buffer_queue_limit — enqueue: buffer_chunk_limit
Λ͑ͨ߹ԡ͠ग़͞ΕΔ߹ͱ flush_interval Λܦաͨ͠߹͞ΕΔέʔε — ͦΕͧΕઃఆ͢ΕॊೈʹϩάΛૹΔ͜ͱ͕Ͱ͖Δ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 18
όοϑΝઃܭ(v0.12) — Outputͷύϥϝʔλ ύϥϝʔλ ༰ buffer_type όοϑΝͷछྨ(file,memory) buffer_path ϑΝΠϧόοϑΝͷ֨ೲઌ bufferchunklimit
chunck࠷େαΠζ bufferqueuelimit Queuechunck࠷େ flush_interval όοϑΝϑϥογϡִؒ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 19
όοϑΝઃܭ(v0.12) <match access.**> @type forward buffer_type file buffer_path /var/log/td-agent.buffer buffer_chunk_limit
8m #8MBΛอ࣋͢Δ buffer_queue_limit 64 #64ݸ·Ͱ͑Δ flush_interval 60s # Buffer͔ΒQueueʹ͞ΕΔ߹60ඵͨͬͨΒதͷchunckΛQueueʹ͢ <server> name test_server host 192.168.33.11 port 24224 </server> </match> ͑ΔDiskͷ༰ྔϝϞϦͷαΠζɹbuffer_chunk_limit x buffer_queue_limit Λ͔͚߹Θͤͨͷ͕ྖҬͱͯ͠BufferͷαΠζͱͯ͠ඞཁʹͳΔ matchͷ͚ͩ͜ͷ͔͚߹Θ͕ͤͨඞཁʹͳΔͷͰҙ͕ඞཁɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 20
όοϑΝઃܭ(v1) ෦తʹɺόοϑΝϓϥάΠϯʹɺνϟϯΫ͕ΠϕϯτͰ͍ͬͺ ͍ʹͳΔʮεςʔδʯͱɺ νϟϯΫ͕సૹલʹػ͢ΔʮΩϡʔʯͱ͍͏ 2ͭͷ͞Εͨॴ ͕͋Γ·͢ɻ ৽͘͠࡞͞Εͨͯ͢ͷνϟϯΫɺεςʔδ͔Β։࢝͠ɺ࣌ؒ ʹΩϡʔʹೖΕΒΕ·͢ʢͦͷޙɺѼઌʹసૹ͞Ε·͢ʣɻ — staged:buffering
ঢ়ଶ — queued:flushͪͷqueueʹೖ͍ͬͯΔঢ়ଶ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 21
όοϑΝઃܭ(v1) ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 22
Embulk ࠷৽όʔδϣϯ0.9.7(2018-04-16) όϧΫ൛ͷFluentd όονతͳసૹ — ΦʔϓϯιʔεͷฒྻࢄॲཧόϧΫϩʔμʔ — ϓϥάΠϯΞʔΩςΫνϟ — ༰қͳσʔλΠϯςάϨʔγϣϯͷ࣮ݱ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 23
EmbulkͷϢʔεέʔε — աڈͷใΛղੳ͍ͨ͠ — όονతʹσʔλΛసૹ͍ͨ͠ — ҟͳΔετϨʔδʹσʔλΛಉظ͍ͨ͠ — େ͖ͳ1ϑΝΠϧ͚ͩΛసૹ͍ͨ͠ ϓϩμΫτษڧձ
2018/04/23 - Ozawa Shuhei 24
FluentdͱEmbulkͷ͍͚ — Fluentd — WEB/APPαʔόͷϩάऩू — ࢹɺϞχλϦϯά — ྲྀྔͷେ͖͍ϩάऩू —
ϦΞϧλΠϜੑͷߴ͍ੳ༻్ — όονͰཷΊࠐΉͱૹΕͳ͍ͷ — Embulk — Ϛελσʔλͷಉظ — Ұ͝ͱͷσʔλҠಈ(όονత) — S3ͳͲ͔Βͷฒྻσʔλμϯϩʔυ — DWHͷσʔλϩʔυ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 25
όʔδϣϯ 0.9.0 (2018-01-30) - Java 8 - Lambda - Stream
- Time - Async File IO - FileSystem - Oracle Java SEαϙʔτɾϩʔυϚοϓ - LTS όʔδϣϯ͕ɺ3͝ͱͷϦϦʔεΛඪ - ػೳϦϦʔεɺ6ϲ݄͝ͱΛඪ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 26
όʔδϣϯ 0.9.3(2018-02-13) - JRubyϕʔεͷϓϥάΠϯ͕༻͞Ε͍ͯͳ͍߹ɺJRuby ͷॳظԽΛఀࢭ - ϓϥάΠϯͷϩʔυͱىಈ͕͘ͳ͍ͬͯΔ 0.9.7(2018-04-16) - ࠷৽όʔδϣϯ
ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 27
embulk-announce Embulkͷ৽όʔδϣϯͷϦϦʔε௨ɺޓੑʹؔ͢Δ௨ ͳͲ։ൃऀ͔ΒͷΞφϯεઐ༻ML https://t.co/w8TFtr30u0 ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 28
ElasticStack6.0 6.2.0ϦϦʔε(2018-02-06) ݕࡧͱੳͷελοΫͱͯ͠ػೳ͢ΔίϯϙʔωϯτͷΤίγ εςϜ - Kibana - Logstash - Beats
- X-Pack - Elasticsearch ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 29
֤ίϯϙʔωϯτͷׂ — Elasticsearch ͯ͢ͷσʔλΛ֨ೲ͠ɺݕࡧػೳͱੳػೳΛεέʔϥϒϧʹఏڙ — Logstash ϩάɺϝτϦοΫͳͲͷΠϕϯτσʔλΛҙͷܗࣜͰूதཧ — Beats Filebeatɺαʔόʔ͔ΒLogstashElasticsearchʹϩάϑΝΠϧΛ৴͢ΔͨΊʹߏங͞ΕͨBeat
Metricbeatɺαʔόʔ্Ͱ࣮ߦ͞Ε͍ͯΔOSαʔϏε͔ΒఆظతʹϝτϦοΫΛऩू͢Δαʔό ʔࢹΤʔδΣϯτ — kibana Elastic Searchͷࢹ֮Խπʔϧ — X-Pack Elastic StackʹηΩϡϦςΟɺࢹɺΞϥʔτɺϨϙʔτɺ͓ΑͼάϥϑػೳΛՃ ίʔυ͕ެ։ɻ ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 30
ཱ͓ͪใ·ͱΊ — Fluentd Fluentd v1 and future at techtalk ϓϥάΠϯ։ൃऀ͔ΒݟΔfluentd
v1.0ͷ׆༻๏ fluentd ͷجૅࣝ — Embulk Embulk v0.9 Embulk — Bigdam Bigdam — ElasticStack discuss.elastic.co ϓϩμΫτษڧձ 2018/04/23 - Ozawa Shuhei 31