Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch Ingest Processors

Avatar for Alexander Reelsen Alexander Reelsen
October 30, 2018
Technology
280
0

Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.

Avatar for Alexander Reelsen

Alexander Reelsen

October 30, 2018

More Decks by Alexander Reelsen

See All by Alexander Reelsen
Understanding Apache Lucene - More than just full-text search
Avatar for Alexander Reelsen spinscale
0
170
Elasticsearch: From Keyword Search To Data Science
Avatar for Alexander Reelsen spinscale
0
210
Evolving Search at an ecommerce marketplace
Avatar for Alexander Reelsen spinscale
0
250
The new generation of data stores
Avatar for Alexander Reelsen spinscale
0
320
Search Evolution - Keeping up with the hype?
Avatar for Alexander Reelsen spinscale
0
430
Mirror mirror... what am I typing next?
Avatar for Alexander Reelsen spinscale
0
530
The New Generation of Data Stores
Avatar for Alexander Reelsen spinscale
0
290
Elasticsearch: Distributed Search Under the Hood
Avatar for Alexander Reelsen spinscale
0
190
Working distributed - but how?
Avatar for Alexander Reelsen spinscale
0
250

Other Decks in Technology

See All in Technology
生成AIはソフトウェア開発の革命か、ソフトウェア工学の宿題再提出なのか -ソフトウェア品質特性の追加提案-
Avatar for kyonmm kyonmm
PRO
2
880
データモデリング通り #5オンライン勉強会： AIに『ビジネスの文脈』を教え込むデータモデリング
Avatar for データ横丁 datayokocho
0
260
ワールドカフェ再び、そしてゴール・ルール・ロール・ツール / World Café Revisited, and the Goals-Rules-Roles-Tools
Avatar for Kenji Saito ks91
PRO
0
150
ブラウザの投機的読み込みと投機ルールAPIを理解し、Webサービスのパフォーマンスを最適化する
Avatar for did0es shuta13
3
300
知ってた？JavaScriptの"正しさ"を検証するテストが5万以上もあること(Test262)
Avatar for Riya Amemiya riyaamemiya
1
190
試作とデモンストレーション / Prototyping and Demonstrations
Avatar for Kenji Saito ks91
PRO
0
200
Oracle Base Database Service 技術詳細
Avatar for oracle4engineer oracle4engineer
PRO
15
100k
なぜ、私がCommunity Builderに?〜活動期間1か月半でも選出されたワケ〜
Avatar for Yuuki Yamashita yama3133
0
120
Oracle Exadata Database Service on Cloud@Customer X11M (ExaDB-C@C) サービス概要
Avatar for oracle4engineer oracle4engineer
PRO
2
8k
SREの仕事は「壊さないこと」ではなくなった 〜自律化していくシステムに、責任と判断を与えるという価値〜 / 20260515 Naoki Shimada
Avatar for SHIFT EVOLVE shift_evolve
PRO
1
140
毎日の作業を Claude Code 経由にしたら、 ノウハウがコードになった
Avatar for Hajime KOSHIRO kossykinto
1
1.3k
世界の中心でApp Runnerを叫ぶ FINAL
Avatar for つくぼし tsukuboshi
0
260

Featured

See All Featured
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
51
52k
Java REST API Framework Comparison - PWX 2021
Avatar for Matt Raible mraible
34
9.3k
HU Berlin: Industrial-Strength Natural Language Processing with spaCy and Prodigy
Avatar for Ines Montani inesmontani
PRO
0
370
Scaling GitHub
Avatar for Zach Holman holman
464
140k
Darren the Foodie - Storyboard
Avatar for Kevinho.art khoart
PRO
3
3.3k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
Avatar for Irina Nazarova irinanazarova
9
1.3k
Making Projects Easy
Avatar for Brett Harned brettharned
120
6.6k
Avoiding the “Bad Training, Faster” Trap in the Age of AI
Avatar for Mike Taylor tmiket
0
140
Marketing to machines
Avatar for Jono Alderson jonoalderson
1
5.2k
How to Talk to Developers About Accessibility
Avatar for Jason CranfordTeague jct
2
190
Paper Plane (Part 1)
Avatar for Katie Cordina Lindsey katiecoart
PRO
0
7.3k
The AI Revolution Will Not Be Monopolized: How open-source beats economies of scale, even for LLMs
Avatar for Ines Montani inesmontani
PRO
3
3.4k

Transcript

  1. Alexander Reelsen [email protected] @spinscale Elasticsearch Ingest Processors Luca Wintergerst [email protected]

    @LucaWintergerst

  2. ‣ Update ‣ Writing your own processors ‣ Use-Cases ‣

    Discussion Agenda

  3. Update

  4. ‣ bytes (convert to human readable bytes) ‣ dissect (grok

    without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors

  5. ‣ - drop processor to fully drop an event ‣

    "drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors

  6. ‣ performance bump in geoip processor ‣ per processor metrics

    ‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others

  7. ‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI

    Future

  8. Writing your own

  9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest

    plugin

  10. Use-Cases

  11. … ask all the things! Discussion