Searchnos & Search on Nostr

Nostrasia day 3
2023-11-03

Transcript

1. Searchnos & Search on Nostr @darashi npub1q7qyk7rvdga5qzmmyrvmlj29qd0n45snmfuhkrzsj4rk 0sm4c4psvqwt9c 2023-11-03 Nostrasia day 3
2. Let me introduce myself,
3. with my works...
4. Articles:
5. I wrote "NIP-01 を読む (Reading NIP-01)" and full Japanese translation of NIP-01
6. for ...
8. "Hello, Nostr!" Fanzine https://nip-book.nostr-jp.org/
9. And the second issue is in press now,
11. "Hello, Nostr! Yo Bluesky!" Fanzine will be available on 12th November 2023, at Techbookfest 15, Ikebukuro https://nip-book.nostr-jp.org/
12. I wrote a short summary of NIP-01 updates from the first issue, and
13. the Japanese translation of the latest version of NIP-01.
14. I also wrote " 作ってわかる Nostr プロトコル (Understanding the Nostr Protocol by Writing Code)" for the series for Nostr,
16. in Software Design, November 2023 issue
17. My article is 5th of the series,
18. written by members of Japanese Nostr community.
20. You can buy Software Design magazine at bookstores in Japan.
21. Good for souvenir :)
22. Softwares:
23. murasaki: Nostr to Speech a client reads notes with Text-to-Speech.
24. Mapnos: Map Notes and Other Stuff shows geotagged kind 1 notes on a map. https://mapnos.vercel.app/
26. nostrbuzzs: buzzphrase detector for Nostr detects trending phrases in real time https://nostrbuzzs.deno.dev/
28. This is a kind of "algo" you might not like.
29. The point is that anyone can implement their own algo with Nostr.
30. I just think it's fun to see what's going on in Nostr,
31. especially at least in this early stage of Nostr.
32. nos.today: Web client for NIP-50 search
33. Searchnos: NIP-50 relay
34. Today, I'm going to talk about "Searchnos".
35. Searchnos is a NIP-50 relay, having Elasticsearch as its backend.
36. It's an OSS and available on GitHub. https://github.com/darashi/searchnos
37. Motivation:
38. I want to search on Nostr in Japanese. (for nostrbuzzs)
39. As far as I know, at the time I started developing Searchnos,
40. relay.nostr.band was the only public relay that supported the NIP-50.
41. relay.nostr.band works very well in many cases,
42. but I noticed sometime unexpected results are returned when querying in Japanese.
43. I'm not sure but maybe due to tokenization.
44. Typical full text search approach is to tokenize the text into words:
45. "One beer, please." -> ["one", "beer", "please"]
46. Query "please bear" (AND search) should matches with the text "One beer, please."
47. In Japanese, words are not separated by spaces:
48. ビールを一杯ください。 bii-ru o ippai kudasai (One beer, please.)
49. Using a technique called morphological analysis, it is possible to break them into words.
50. ビールを一杯ください。 -> [" ビール", " を", " 一杯", " ください"]
51. This analysis depends on dictionaries,
52. and it's not always correct.
53. Especially weak to new words.
54. Another approach is to use N-gram indexing.
55. ビールを一杯ください。 -> [" ビー", " ール", " ルを", " を一", " 一杯", " 杯く", " く だ", " さい", " い。"] (bi-gram)
56. If we query " ルを ビー" (this doesn't make sense), it will be tokenized as [" ル を", " ビー"],
57. If we treat these tokens in the same way as English words,
58. it can result in false positives, because
59. [" ビー", " ール", " ルを", " を 一", " 一杯", " 杯く", " くだ", " さい", " い。"] ⊇ [" ルを", " ビー"]
60. We need to use N-gram indexing and consider the position of the tokens.
61. Today I won't go into details more ...
62. Any way, some effort is needed.
63. In order to tackle Japanese language specific problems,
64. it seemed like a good idea to have my own relay implementation.
65. So I made Searchnos.
66. Architecture:
68. In order simplify the implimenation,
69. Searchnos continuously polls Elasticsearch after EOSE.
70. Sequence diagram:
71. Source Relay Indexer Elasticsearch Searchnos Relay Source Relay Indexer Elasticsearch Searchnos Relay loop loop Client REQ 1 query 2 response 3 EVENT (if matched) 4 EVENT (if matched) 5 EOSE 6 EVENT 7 EVENT 8 index request 9 wait 10 query 11 response 12 EVENT (if matched) 13 CLOSE 14 Client
72. I'm running Searchnos at wss://search.nos.today
73. Some details & future works:
74. (1) Stop polling Elasticsearch for events after EOSE
75. Doing this can reduce the latency of search results and load on Elasticsearch.
76. We need to implement "filter evaluator",
77. and it'll be duplicated with the procedure using Elasticseach.
78. (2) Different index lifetime by kind
79. Currently Searchnos indexes all events on a daily basis
80. and keeps 30 sub indices (in 30 days; configurable).
81. Elasticsearch can search multiple indices transparently,
82. and can also delete an index quite efficiently.
83. In this way, Searchnos put TTLs on events.
84. This design was choosen because I mainly targetted to real time search at first.
85. But it would be better to have longer TTLs for some kinds,
86. for example kind 0 and 30023 (NIP-23 long form contents).
87. So I want to make it possible to configure TTLs according to kinds.
88. (3) Support policy plugins (like strfry)
89. Searchnos uses indexer to filter events to be indexed.
90. Indexer sends events to Serchnos relay using NIP- 01,
91. to the special administrative endpoint
92. for spam prevention
93. If we have a policy plugin system, Searchnos can recieve events from users directly.
94. (4) Support more sophisticated queries.
95. Currently Searchnos treat queries as AND search separated by spaces.
96. (4-a) Logical operations: AND, OR, NOT, ...
97. In order to achive this, search query parser needs to be implemented.
98. (4-b) Language specific search
99. User may want to search like "nostrasia lang:ja".
100. Searchnos internally detects languages using Elasticsearch's language detector.
101. But how to treat "lang:ja"?
102. Where to parse the query?
103. Relay? or Client?
104. I'm not sure how should do this.
105. (4-c) Search events from a specific user
106. User may want to query "nostrasia from:darashi"
107. This is more difficult than the language specific search,
108. because we need to join with kind 0 with search results.
109. If we know the pubkey of the user, NIP-01 can filter the events from the user
110. But who should convert the query "user:darashi" into the NIP-01 filter?
111. Relay? or Client?
112. It's not realistic to expect all clients to implement this.
113. How about making libraries for clients?
114. It's not a bad idea, but we need to immplement in many programming languages.
115. On the other hand, if we implement on the relay side,
116. implementation differences between search relays may lead to inconsistent results.
117. Then how should we implement this?
118. Hybrid approach may be a solution.
119. What about converting the query to an intermediate representation at the client,
120. and sending it to the relay?
121. Anyway,
122. I think the beauty of Nostr is
123. that each part can be implemented with a little effort.
124. Complicated specs easily destroy that.
125. Do you have any good idea?
126. Conclusion
127. I made Searchnos and it's working.
128. But there are still many things to do.
129. Building a relay is open to everyone, and it's fun. Try it! Thank you.
130. (Please help me if you can translate the questions and answers.)
131. npub1q7qyk7rvdga5qzmmyrvmlj29qd0n45snmfuhkrzsj4rk 0sm4c4psvqwt9c