Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elasticsearch Ingest Processors

Sponsored · SiteGround - Reliable hosting with speed, security, and support you can count on. →
Avatar for Alexander Reelsen Alexander Reelsen
October 30, 2018
Technology
270
0

Elasticsearch Ingest Processors

A BoF session held at the elastic on tour event in Frankfurt in October 2018.

Avatar for Alexander Reelsen

Alexander Reelsen

October 30, 2018

More Decks by Alexander Reelsen

See All by Alexander Reelsen
Understanding Apache Lucene - More than just full-text search
Avatar for Alexander Reelsen spinscale
0
160
Elasticsearch: From Keyword Search To Data Science
Avatar for Alexander Reelsen spinscale
0
210
Evolving Search at an ecommerce marketplace
Avatar for Alexander Reelsen spinscale
0
250
The new generation of data stores
Avatar for Alexander Reelsen spinscale
0
310
Search Evolution - Keeping up with the hype?
Avatar for Alexander Reelsen spinscale
0
420
Mirror mirror... what am I typing next?
Avatar for Alexander Reelsen spinscale
0
520
The New Generation of Data Stores
Avatar for Alexander Reelsen spinscale
0
290
Elasticsearch: Distributed Search Under the Hood
Avatar for Alexander Reelsen spinscale
0
180
Working distributed - but how?
Avatar for Alexander Reelsen spinscale
0
250

Other Decks in Technology

See All in Technology
マルチプロダクトの信頼性を効率良く保っていくために
Avatar for KNOWLEDGE WORK / 株式会社ナレッジワーク kworkdev
PRO
0
160
[OpsJAWS 40]リリースしたら終わり、じゃなかった。セキュリティ空白期間をAWS Security Agentで埋める
Avatar for sh_fk2 sh_fk2
3
240
AI時代 に増える データ活用先
Avatar for ono.takayuki takahal
0
230
基盤を育てる 外部SaaS連携の運用
Avatar for Hiroto Gamo gamonges_dresscode
1
120
国内外の生成AIセキュリティの最新動向 & AIガードレール製品「chakoshi」のご紹介 / Latest Trends in Generative AI Security (Domestic & International) & Introduction to AI Guardrail Product "chakoshi"
Avatar for NTT docomo Business nttcom
2
360
自分のハンドルは自分で握れ！ ― 自分のケイパビリティを増やし、メンバーのケイパビリティ獲得を支援する ― / Take the wheel yourself
Avatar for TAKAKING22 takaking22
1
900
EarthCopilotに学ぶマルチエージェントオーケストレーション
Avatar for なかしょ nakasho
0
290
Revisiting [CLS] and Patch Token Interaction in Vision Transformers
Avatar for yu4u yu4u
0
360
AzureのIaC管理からログ調査まで、​随所に役立つ​SkillsとCustom-Instructions​ / Boosting IaC and Log Analysis with Skills
Avatar for AEON aeonpeople
0
230
[最強DB講義]推薦システム | 基礎編
Avatar for RecSysLab recsyslab
PRO
1
170
Do Vibe Coding ao LLM em Produção para Busca Agêntica - TDC 2026 - Summit IA - São Paulo
Avatar for Jessica Pauli de C Bonson jpbonson
3
110
Eight Engineering Unit 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
3
7.3k

Featured

See All Featured
The Organizational Zoo: Understanding Human Behavior Agility Through Metaphoric Constructive Conversations (based on the works of Arthur Shelley, Ph.D)
Avatar for Dr. Kim W Petersen kimpetersen
PRO
0
310
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
Done Done
Avatar for chrislema chrislema
186
16k
HU Berlin: Industrial-Strength Natural Language Processing with spaCy and Prodigy
Avatar for Ines Montani inesmontani
PRO
0
320
Code Reviewing Like a Champion
Avatar for maltzj maltzj
528
40k
The Curse of the Amulet
Avatar for Matthew Lei leimatthew05
1
11k
技術選定の審美眼（2025年版） / Understanding the Spiral of Technologies 2025 edition
Avatar for Takuto Wada twada
PRO
118
110k
Dealing with People You Can't Stand - Big Design 2015
Avatar for Cassini Nazir cassininazir
367
27k
The SEO identity crisis: Don't let AI make you average
Avatar for Varn varn
0
440
How to Align SEO within the Product Triangle To Get 
Buy-In & Support - #RIMC
Avatar for Aleyda Solis aleyda
1
1.5k
How to train your dragon (web standard)
Avatar for Monica Dinculescu notwaldorf
97
6.6k
Practical Orchestrator
Avatar for Shlomi Noach shlominoach
191
11k

Transcript

  1. Alexander Reelsen [email protected] @spinscale Elasticsearch Ingest Processors Luca Wintergerst [email protected]

    @LucaWintergerst

  2. ‣ Update ‣ Writing your own processors ‣ Use-Cases ‣

    Discussion Agenda

  3. Update

  4. ‣ bytes (convert to human readable bytes) ‣ dissect (grok

    without regexes, much faster) ‣ pipeline processor, referring to other pipelines New processors

  5. ‣ - drop processor to fully drop an event ‣

    "drop" : { "if": "ctx.foo == 'bar'" } ‣ - scripting can invoke other processors ‣ "ctx.target_field = Processors.bytes(ctx.source_field)" ‣ if in every processor using scripting New processors

  6. ‣ performance bump in geoip processor ‣ per processor metrics

    ‣ index default pipeline: ‣ settings.index.default_pipeline: "my_pipeline" Others

  7. ‣ Aligning dissect filters in logstash/beats/ES ‣ https://github.com/elastic/dissect-specification ‣ UI

    Future

  8. Writing your own

  9. ‣ https://github.com/spinscale/cookiecutter-elasticsearch-ingest-processor ‣ https://github.com/spinscale/elasticsearch-ingest-langdetect ‣ https://github.com/spinscale/elasticsearch-ingest-opennlp Write your own ingest

    plugin

  10. Use-Cases

  11. … ask all the things! Discussion