Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
On The Look-out For Your Data (DjangoCon Europe...
Search
Markus H
May 24, 2018
Technology
0
13k
On The Look-out For Your Data (DjangoCon Europe 2018)
My talk from DjangoCon Europe 2018
Markus H
May 24, 2018
Tweet
Share
More Decks by Markus H
See All by Markus H
🐍 ❤️ 🦀 — Python loves Rust
markush
0
220
Knock! Knock! Who's There?
markush
0
63
An Introduction To Kubernetes ☸
markush
0
88
Writing Safe Database Migrations (DjangoCon Europe 2021)
markush
0
14k
A Pony On The Move: How Migrations Work In Django 🐎
markush
0
13k
All Hands on Deck — Handling Security Issues
markush
0
13k
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon UK 2019)
markush
0
57
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon Australia 2019)
markush
1
200
Logging Rethought 2: The Actions of Frank Taylor Jr. (DjangoCon Europe 2019)
markush
0
13k
Other Decks in Technology
See All in Technology
Operating Operator
shhnjk
1
590
united airlines ™®️ USA Contact Numbers: Complete 2025 Support Guide
flyunitedhelp
1
340
第4回Snowflake 金融ユーザー会 Snowflake summit recap
tamaoki
1
290
高速なプロダクト開発を実現、創業期から掲げるエンタープライズアーキテクチャ
kawauso
2
9.4k
KubeCon + CloudNativeCon Japan 2025 Recap
ren510dev
1
390
OPENLOGI Company Profile
hr01
0
67k
「クラウドコスト絶対削減」を支える技術—FinOpsを超えた徹底的なクラウドコスト削減の実践論
delta_tech
4
170
いつの間にか入れ替わってる!?新しいAWS Security Hubとは?
cmusudakeisuke
0
130
Model Mondays S2E04: AI Developer Experiences
nitya
0
140
開発生産性を組織全体の「生産性」へ! 部門間連携の壁を越える実践的ステップ
sudo5in5k
2
7.2k
freeeのアクセシビリティの現在地 / freee's Current Position on Accessibility
ymrl
2
200
AIの全社活用を推進するための安全なレールを敷いた話
shoheimitani
2
530
Featured
See All Featured
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
Code Reviewing Like a Champion
maltzj
524
40k
Why Our Code Smells
bkeepers
PRO
336
57k
The Cost Of JavaScript in 2023
addyosmani
51
8.5k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
281
13k
Imperfection Machines: The Place of Print at Facebook
scottboms
267
13k
RailsConf 2023
tenderlove
30
1.1k
Intergalactic Javascript Robots from Outer Space
tanoku
271
27k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.5k
Six Lessons from altMBA
skipperchong
28
3.9k
Transcript
On The Look-Out For Your Data @m_holtermann #djangocon Europe 2018
I’m Markus Holtermann @m_holtermann • github.com/MarkusH • markusholtermann.eu @laterpay •
laterpay.net • Django Contributor • Software Engineer at
What Is Search?
How To Search In Django?
What Is Search?
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
Search Is Hard
Searching In Django?
from django.shortcuts import get_object_or_404, render from blog.models import Article def
article_view(request, pk): article = get_object_or_404(Article, pk=pk) return render( request, 'article.html', context={'article': article}, )
Searching Text
from django.shortcuts import get_list_or_404, render from blog.models import Article def
article_view(request): articles = get_list_or_404( Article, text__icontains=request.GET.get('query', ''), ) return render( request, 'articles.html', context={'articles': articles}, )
SELECT * FROM blog_article WHERE text ILIKE '%Looking for text%'
Trigrams
-- As superuser # CREATE EXTENSION pg_trgm; # SELECT show_trgm('I
love Django'); show_trgm ---------------------------------------------- - {" d", " i", " l", " dj", " i ", " lo", ang, dja, "go ", jan, lov, ngo, ove, "ve "}
from django.contrib.postgres.indexes import GistIndex class TrigramGistIndex(GistIndex): suffix = 'trgm_gist' sql
= 'CREATE INDEX %(name)s ON %(table)s %(using)s \ (UPPER(%(columns)s) gist_trgm_ops)%(extra)s' def create_sql(self, model, schema_editor, using=''): statement = super().create_sql(model, schema_editor, using=using) statement.template = self.sql return statement
Searching Text
Full-text Search
Word order doesn’t matter “Django Migrations” = “Migrations Django”
Stemming computer, compute, computation = comput
Ignoring Stopwords “Django is the best” = “Django best”
__search & PostgreSQL https://docs.djangoproject.com/en/ 2.0/ref/contrib/postgres/search/
External Search Tools
None
from django.db import models, transaction class Article(models.Model): # ... def
save(self, *args, **kwargs): with transaction.atomic(): transaction.on_commit( lambda: update_search(self.pk)) super().save(*args, **kwargs)
from django.db import models, transaction class Article(models.Model): # ... def
delete(self, *args, **kwargs): pk = self.pk with transaction.atomic(): transaction.on_commit( lambda: delete_search(pk)) return super().delete( *args, **kwargs)
Maintain A Complete Search Index
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
• Example: https://github.com/MarkusH/talk-django-search • Search in Django: https://docs.djangoproject.com/en/2.0/topics/db/search/ • Choosing
a PostgreSQL text search method: https://blog.2ndquadrant.com/text-search-strategies-in-postgresql/ • Trigram Extension: https://www.postgresql.org/docs/10/static/pgtrgm.html • Full-text search: https://www.postgresql.org/docs/10/static/textsearch-tables.html
Thank you! @m_holtermann
import blog.indexes from django.contrib.postgres.operations import TrigramExtension from django.db import migrations
class Migration(migrations.Migration): dependencies = [('blog', '0002_auto_20180503_1925')] operations = [ TrigramExtension(), migrations.AddIndex( model_name='entry', index=blog.indexes.TrigramGistIndex( fields=['body'], name='body_trgm_gist')), ]
-- Creates extension pg_trgm CREATE EXTENSION IF NOT EXISTS "pg_trgm";
-- Create index body_trgm_gist on field(s) -- body of model entry CREATE INDEX "body_trgm_gist" ON "blog_entry" USING gist (UPPER("body") gist_trgm_ops);