Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
On The Look-out For Your Data (DjangoCon Europe...
Search
Markus H
May 24, 2018
Technology
0
13k
On The Look-out For Your Data (DjangoCon Europe 2018)
My talk from DjangoCon Europe 2018
Markus H
May 24, 2018
Tweet
Share
More Decks by Markus H
See All by Markus H
🐍 ❤️ 🦀 — Python loves Rust
markush
0
220
Knock! Knock! Who's There?
markush
0
63
An Introduction To Kubernetes ☸
markush
0
88
Writing Safe Database Migrations (DjangoCon Europe 2021)
markush
0
14k
A Pony On The Move: How Migrations Work In Django 🐎
markush
0
13k
All Hands on Deck — Handling Security Issues
markush
0
13k
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon UK 2019)
markush
0
57
Logging Rethought 2: The Actions of Frank Taylor Jr. (PyCon Australia 2019)
markush
1
200
Logging Rethought 2: The Actions of Frank Taylor Jr. (DjangoCon Europe 2019)
markush
0
13k
Other Decks in Technology
See All in Technology
united airlines ™®️ USA Contact Numbers: Complete 2025 Support Guide
flyunitedhelp
1
340
United airlines®️ USA Contact Numbers: Complete 2025 Support Guide
unitedflyhelp
0
310
Yahoo!しごとカタログ 新しい境地を創るエンジニア募集!
lycorptech_jp
PRO
0
110
事業成長の裏側:エンジニア組織と開発生産性の進化 / 20250703 Rinto Ikenoue
shift_evolve
PRO
2
22k
OPENLOGI Company Profile for engineer
hr01
1
34k
開発生産性を組織全体の「生産性」へ! 部門間連携の壁を越える実践的ステップ
sudo5in5k
2
7.2k
NewSQLや分散データベースを支えるRaftの仕組み - 仕組みを理解して知る得意不得意
hacomono
PRO
2
160
SEQUENCE object comparison - db tech showcase 2025 LT2
nori_shinoda
0
140
React開発にStorybookとCopilotを導入して、爆速でUIを編集・確認する方法
yu_kod
1
280
【5分でわかる】セーフィー エンジニア向け会社紹介
safie_recruit
0
27k
品質と速度の両立:生成AI時代の品質保証アプローチ
odasho
1
360
B2C&B2B&社内向けサービスを抱える開発組織におけるサービス価値を最大化するイニシアチブ管理
belongadmin
1
7k
Featured
See All Featured
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.3k
Done Done
chrislema
184
16k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
950
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
35
2.4k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.9k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Code Reviewing Like a Champion
maltzj
524
40k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Balancing Empowerment & Direction
lara
1
430
Embracing the Ebb and Flow
colly
86
4.7k
Transcript
On The Look-Out For Your Data @m_holtermann #djangocon Europe 2018
I’m Markus Holtermann @m_holtermann • github.com/MarkusH • markusholtermann.eu @laterpay •
laterpay.net • Django Contributor • Software Engineer at
What Is Search?
How To Search In Django?
What Is Search?
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
Search Is Hard
Searching In Django?
from django.shortcuts import get_object_or_404, render from blog.models import Article def
article_view(request, pk): article = get_object_or_404(Article, pk=pk) return render( request, 'article.html', context={'article': article}, )
Searching Text
from django.shortcuts import get_list_or_404, render from blog.models import Article def
article_view(request): articles = get_list_or_404( Article, text__icontains=request.GET.get('query', ''), ) return render( request, 'articles.html', context={'articles': articles}, )
SELECT * FROM blog_article WHERE text ILIKE '%Looking for text%'
Trigrams
-- As superuser # CREATE EXTENSION pg_trgm; # SELECT show_trgm('I
love Django'); show_trgm ---------------------------------------------- - {" d", " i", " l", " dj", " i ", " lo", ang, dja, "go ", jan, lov, ngo, ove, "ve "}
from django.contrib.postgres.indexes import GistIndex class TrigramGistIndex(GistIndex): suffix = 'trgm_gist' sql
= 'CREATE INDEX %(name)s ON %(table)s %(using)s \ (UPPER(%(columns)s) gist_trgm_ops)%(extra)s' def create_sql(self, model, schema_editor, using=''): statement = super().create_sql(model, schema_editor, using=using) statement.template = self.sql return statement
Searching Text
Full-text Search
Word order doesn’t matter “Django Migrations” = “Migrations Django”
Stemming computer, compute, computation = comput
Ignoring Stopwords “Django is the best” = “Django best”
__search & PostgreSQL https://docs.djangoproject.com/en/ 2.0/ref/contrib/postgres/search/
External Search Tools
None
from django.db import models, transaction class Article(models.Model): # ... def
save(self, *args, **kwargs): with transaction.atomic(): transaction.on_commit( lambda: update_search(self.pk)) super().save(*args, **kwargs)
from django.db import models, transaction class Article(models.Model): # ... def
delete(self, *args, **kwargs): pk = self.pk with transaction.atomic(): transaction.on_commit( lambda: delete_search(pk)) return super().delete( *args, **kwargs)
Maintain A Complete Search Index
What Is Search? Try to find something by looking or
otherwise seeking carefully and thoroughly. — Oxford English Dictionary
• Example: https://github.com/MarkusH/talk-django-search • Search in Django: https://docs.djangoproject.com/en/2.0/topics/db/search/ • Choosing
a PostgreSQL text search method: https://blog.2ndquadrant.com/text-search-strategies-in-postgresql/ • Trigram Extension: https://www.postgresql.org/docs/10/static/pgtrgm.html • Full-text search: https://www.postgresql.org/docs/10/static/textsearch-tables.html
Thank you! @m_holtermann
import blog.indexes from django.contrib.postgres.operations import TrigramExtension from django.db import migrations
class Migration(migrations.Migration): dependencies = [('blog', '0002_auto_20180503_1925')] operations = [ TrigramExtension(), migrations.AddIndex( model_name='entry', index=blog.indexes.TrigramGistIndex( fields=['body'], name='body_trgm_gist')), ]
-- Creates extension pg_trgm CREATE EXTENSION IF NOT EXISTS "pg_trgm";
-- Create index body_trgm_gist on field(s) -- body of model entry CREATE INDEX "body_trgm_gist" ON "blog_entry" USING gist (UPPER("body") gist_trgm_ops);