Adequate Full Text Search

given at Elasticsearch UG in November 2014

Transcript

2. $ cat .profile GIT_AUTHOR_NAME=Florian Gilcher GIT_AUTHOR_EMAIL=florian.gilcher@asquera.de TM_COMPANY=Asquera GmbH TWITTER_HANDLE=argorak GITHUB_HANDLE=skade
3. • Backend developer • Focused on infrastructure and databases
4. • Elasticsearch Usergroup • mrgn.in meetup • Rust Usergroup (co-org) • organizer alumni eurucamp • organizer alumni JRubyConf.EU • Ruby Berlin board member
5. Adequate Full Text Search
6. The evaluation problem
7. Given almost no time and an unknown problem space, how do I evaluate "fitness for purpose"?
8. You can't
9. Given almost no time and only a glimpse of the problem space, how do I evaluate "fitness for purpose"?
10. How much of a glimpse do I need?
11. In this talk, I’ll present: • a solution unfit for purpose • a solution fit for purpose, but only in cer- tain boundaries • a comparison to a fully fledged solution
12. To the daily practitioners: I’ll gloss over a lot of points.
13. • Elasticsearch • PostgreSQL • MongoDB
14. Issue 1 Search systems are not binary. Faults in the system degrade the quality of the system, rarely break it.
15. Issue 2 Full text searchers are far more focused on inputs then on output.
16. Building Block 1 An inverted index
17. doc id content 0 "Überlin ist auf Twitter" 1 "Ich bin auf Twitter" 2 "Ich folge Überlin"
18. terms document ids uberlin 0,2 twitter 0,1 bin 1 ich 1,2 auf 0,1
19. Initial search rules are easy: if one or more of the terms to the left is searched for, find the document that matches. Count the matches.
20. Building Block 2 Textual input
21. Full text searchers generally work on real world text. Get hold of as many samples as possible. If necessary, write some on your own.
22. Don’t use an random generator. Or spend your next weeks writing a sophisticated one.
23. Your system should bring capabilities handling real world text.
24. Analysis
25. Analysis determines which terms end up at the left side of the table in the first place.
26. analysis result "ich folge Überlin" whitespace "ich" "folge" "Überlin" lowercase "ich" "folge" "überlin" normalize "ich" "folge" "uberlin" stemming "ich" "folg" "uberlin"
27. analysis result "ich folge ueberlin" whitespace "ich" "folge" "ueberlin" lowercase "ich" "folge" "ueberlin" normalize "ich" "folge" "ueberlin" stemming "ich" "folg" "uberlin"
28. This step happens both on indexing and queries.
29. Manipulating analysis is the basis for manipulating matches.
30. Can I manipulate analysis?
31. MongoDB Only choose between language presets PostgreSQL Analysis happens through normal PL/SQL functions Elasticsearch Analyser configura- tion with a wide vari- ety of choice
32. Ü
33. Does your system comfortably speak Unicode?
34. doc id field value 1 Test 2 test 3 Überlin
35. token doc ids test 1,2 uberlin 3
36. MongoDB
37. search term no. matches Test 2 test 2 Überlin 1 überlin 0
38. token doc ids test 1,2 Überlin 3
39. input result überlin überlin Überlin Überlin
40. MongoDB fails at the simplest case, lowercasing german umlauts, in german settings.
41. The exact analysis behaviour is not user-controllable, for simplicities sake.
42. The suggestion is to preprocess yourself.
44. Further down the Unicode
45. How well does you system handle "creative" codes?
46. "\u0055\u0308" "\u0075\u0308"
47. "\u0055\u0308" #=> Ü "\u0075\u0308" #=> ü
48. PostgreSQL
49. postgres=# SELECT unaccent(U&’\0075\0308’); unaccent ———- ü (1 row)
50. PostgreSQL handles UCS-2 level 1, not UTF.
51. No combining chars.
52. “ we should really reject combining chars, but can’t do that w/o breaking BC.”
53. sigh, Software
54. If you use PostgreSQL and text manipulation, you probably have a bug in the hiding there.
55. UCS-2 for all textual data is a doable constraint, though.
56. input result überlin überlin Überlin überlin \u0055 \u0308 Invalid input \u0075 \u0308 Invalid input
57. Elasticsearch
58. Elasticsearch can handle all those cases and then some, using the analysis-icu plugin.
59. Install it and use it.
60. curl -XGET ’localhost:9200/_analyze?\ tokenizer=\ icu_tokenizer\ &token_filters=\ icu_folding,icu_normalizer’\ -d ’Überlin’
61. input result überlin uberlin Überlin uberlin \u0055 \u0308 uberlin \u0075 \u0308 uberlin
62. The way the system supports you in safely inserting textual input is of paramount importance!
63. Find the worst shenanegans of you language, try it out.
64. l’elision, c’est magnifique
65. Building Block 3 Scoring
66. Search is all about relevance and combinations thereof.
67. Was the match in the title or the body of a document?
68. How many options do I have?
69. All three systems can weight matches on fields differently.
70. When can I decide those weights?
71. database index time query time MongoDB yes no PostgreSQL yes no Elasticsearch yes yes
72. Weights during index time need a rebuild of the index every time you change them.
73. If in doubt, choose query time weights.
74. Can I influence the scoring/ranking further?
75. database MongoDB no PostgreSQL yes, using PL/SQL functions Elasticsearch yes, in many fashions (geo, distance, etc.)
76. Building Block 4 Documentation
77. I glossed over a lot of details.
78. How well is the process documented, internally and interface-wise?
79. database interface internal MongoDB good almost non-existent PostgreSQL great great Elasticsearch great great
80. Can I grow beyond?
81. And this is where the fun starts and we stop.
82. What’s adequate?
83. • Allows to manipulate analysis • Assists with real world input • Allows you to build combined, extensible queries • Good documentation
84. MongoDB is not fit for purpose with holes that can only be fixed by careful preparation of that data.
85. That preparation needs lots of detail knowledge you probably don’t want to aquire.
86. PostgreSQL is adequate and in the PostgreSQL tradition of stable, well-documented features. It doesn’t win prices, but is workable and reliable.
87. A good solution if search is just a bystander. A thousand times better than LIKE.
88. Elasticsearch is based on Lucene and comes with all the goodies and also has great documentations and guides.
89. If search is at the core of your product, use a proper search engine.
90. References on the meetup group tomorrow.
91. Thank you!
93. COURSES
94. Elasticsearch for managers: http://esmanagers2014.asquera.de/
96. December 2nd
97. Getting started workshop: http://purchases.elastic- search.com/class/elasticsearch/elk-work- shop/berlin-germany/2014-12-15
99. December 15th