Building a Search Engine with Python and Elasticsearch

https://us.pycon.org/2018/schedule/presentation/53/

One of the most common actions that we take when visiting any website is search. A common service that powers search for many sites is Elasticsearch - but what makes it so powerful? What can you do with Elasticsearch that you can’t with a regular database?

This tutorial starts with an introduction to Elasticsearch architecture, including what makes it great for search and not so great for other use cases. We will then build an application together with a search engine powered by Elasticsearch. We will also discuss how to optimize search queries and scale as the volume of data increases.

Transcript

1. Building a Search Engine with Python + Elasticsearch Julie Qiu @jqiu25 Jim Grandpre @jimtla #PyCon2018
2. Julie Qiu Engineering Lead, Spring @jqiu25 Jim Grandpre Director of Engineering, Spring @jimtla
8. Tutorial Goals and Structure
9. Develop an intuition for elasticsearch. Understand the most important pieces. Presentations
10. Build comfort with the ES docs, and pyES Have a starting point for PY + ES projects. Use ES & Python to make search work. Problem Sets
11. 15m Introduction to Elasticsearch & Indexing 45m Problem Set: Indexing 10m Break 5m Introduction to Searching 45m Problem Set: Searching 5m Break 5m Introduction to Analysis 45m Problem Set: Analysis Agenda
12. Work in Groups, help each other learn. Take your time. Read the docs. Problem Set Advice
13. Many questions on the problem set include “spoilers.” Try to figure things out on your own, but if you feel stuck don’t hesitate to read them. Spoilers
14. Intro to Elasticsearch
15. What is Elasticsearch? A search optimized distributed database, built on top of Lucene.
16. Database Stores structured documents. Insert, Update, Delete, and Query. Provides a friendly interface (JSON & HTTP).
17. Lucene A Java library used by Elasticsearch. Core data storage. Core query execution.
18. Distributed Elasticsearch is typically run in a cluster. Can add and remove instances any time. Data is split across instances, and queries are executed across the cluster.
19. Search Optimized Data is indexed to allow for fast searching. Query language for complex searches. Built in support for text analysis.
20. Database Structure
21. Index (~ Table) Database Structure
22. Document (~ Row) Document (~ Row) Index (~ Table) Database Structure Document (~ Row) Document (~ Row)
23. Document (~ Row) Document (~ Row) Index (~ Table) Database Structure Document (~ Row) Document (~ Row) Document Type (Schema)
24. “A Great Product” “This Cool Shirt” products_index Database Structure “Neat Pants” “Kittens” product
25. Getting data into Elasticsearch
26. ??? Getting data into Elasticsearch
27. ??? Data Getting data into Elasticsearch
28. ??? JSON { “name”: ‘shirt’, “image”:’www.shirtphoto.com’ } Getting data into Elasticsearch
29. Indexing API Getting data into Elasticsearch JSON { “name”: ‘shirt’, “image”:’www.shirtphoto.com’ }
30. { “name”: ‘shirt’, “image”:’www.shirtphoto.com’ } products_index product Database Structure
31. localhost:9200 Elasticsearch Index
32. Kibana Dev Tools Console (localhost:5601) bit.ly/pycon-es-devtools1
33. App Walkthrough
34. Indexing with ES and Python
35. .git/ .gitignore README.md lessons/ requirements.txt searchapp/ searchapp.egg-info/ setup.py venv/ Our Repository
36. searchapp/ __init__.py app/ constants.py data.py index_products.py products.json run.py Search App: Indexing
37. Elasticsearch Py Official Python library for Elasticsearch elasticsearch-py.readthedocs.io/en/master/
38. Work in Groups, help each other learn. Take your time. Read the docs. Problem Sets
39. bit.ly/pycon-es-lesson1 Problem Set: Indexing
40. Searching data in Elasticsearch
41. ??? Searching data
42. necklace Searching data ???
43. necklace Application (Flask App) Searching data ???
44. necklace Application (Flask App) Search API Searching data
45. Kibana Dev Tools Console (localhost:5601) bit.ly/pycon-es-devtools2
46. Searching with Python and ES
47. Elasticsearch DSL Python library – helps with writing and running Elasticsearch queries elasticsearch-dsl.readthedocs.io/en/latest/
48. Example Search: Raw query GET request to localhost:9200/products_index/product { "query": { "match": { "name": "necklace" } } }
49. Example Search: Comparison GET request to localhost:9200/products_index/product { "query": { "match": { "name": "necklace" } } } from elasticsearch import Elasticsearch from elasticsearch_dsl import Search client = Elasticsearch() s = Search( using=client, index=”products_index”, doc_type=”products” ) s.query( “match”, name=”necklace” ).execute()
50. Example Search: Comparison GET request to localhost:9200/products_index/products { "query": { "match": { "name": "necklace" } } } from elasticsearch import Elasticsearch from elasticsearch_dsl import Search client = Elasticsearch() s = Search( using=client, index=”products_index”, doc_type=”products” ) s.query( “match”, name=”necklace” ).execute()
51. Example Search: Comparison GET request to localhost:9200/products_index/products { "query": { "match": { "name": "necklace" } } } from elasticsearch import Elasticsearch from elasticsearch_dsl import Search client = Elasticsearch() s = Search( using=client, index=”products_index”, doc_type=”products” ) s.query( “match”, name=”necklace” ).execute()
52. Problem Set: Search bit.ly/pycon-es-lesson2 Continuing from Part 1 (also on GitHub repo) git commit -am “session1 work” git fetch git checkout session2 source venv/bin/activate python searchapp/index_products.py
53. Analysis
54. Analyzers improve search accuracy by adding additional structure to text. Analyzers
55. Analyzers Raw Walking The: Dog [‘Walking’, ‘The’, ‘Dog’] [‘walk’, ‘dog’] Tokenizer Token Filters
56. Break a string into components. Input: “Walking The: Dog” Standard: [“Walking”,“The”,“Dog”] Whitespace: [“Walking”,“The:”,“Dog”] Edge N-grams: [“W”,“Wa”,“Wal”,“Walk”, …] Tokenizers
57. Input: [“Walking”,“The”,“Dog”] Lowercase: [“walking”,“the”,“dog”] Stop Words:[“walking”,“dog”] Stemming: [“walk”,“dog”] Token Filters
58. Analyzers are configured in the mapping of your index. Custom analyzers are created in the settings of your index. Analyzing your Index
59. Analyzers are applied to both the fields of your document, and the queries against those fields. Analyzing your Index
60. Work in Groups, help each other learn. Take your time. Read the docs. Problem Set
61. bit.ly/pycon-es-lesson3 Survey: bit.ly/pycon-es-survey Problem Set and Survey
62. Thanks! Survey: bit.ly/pycon-es-survey Julie Qiu @jqiu25 Jim Grandpre @jimtla