howto for CJK indexing

diff --git a/howto.mdwn b/howto.mdwn
index 23fde44aab3b3d725dbe2fbf73ecb5856c1e0aaa..9d029d8a000e6a3e524f218cad79d2214fa1d197 100644 (file)
--- a/howto.mdwn
+++ b/howto.mdwn
@@ -145,6 +145,33 @@ in a scenario where you have encrypted your hard disk anyway and are
 comfortable with the security implications (and until notmuch can index
 encrypted email itself).
 
 comfortable with the security implications (and until notmuch can index
 encrypted email itself).
 

+## <span id="special_tags">**Index and search emails written in CJK scripts**</span>
+
+CJK (Chinese, Japanese and Korean) languages do not use spaces for word
+separation. The full-text indexer (Xapian) must first perform word segmentation
+on the sentence in its TermGenerator. Otherwise, large amount of long terms
+will be included in the database, leading to extremely slow indexing and
+ineffective searching with CJK search terms.
+
+Xapian supports [N-gram](https://xapian.org/docs/sourcedoc/html/classXapian_1_1TermGenerator.html)
+term generator [since 2011](https://u7fa9.org/memo/HEAD/archives/2012-06/2012-06-01.rst)
+to as a simple substitute for word segmentation. It can be turned on by
+setting the environment variable
+
+        $ export XAPIAN_CJK_NGRAM=1
+        $ notmuch new
+
+For existing databases, one can reindex the database (since notmuch 0.26)
+with
+
+        $ export XAPIAN_CJK_NGRAM=1
+        $ notmuch reindex '*'
+
+Xapian has an on-going [pull-request](https://github.com/xapian/xapian/pull/114)
+that adds support for real CJK word-segmentation based on the ICU library.
+When it gets merged, one probably will gets better indexing and searching
+results with this new method.
+

 ## Translations
 
 - A translation of this page into [[Russian|howto-ru]]
 ## Translations
 
 - A translation of this page into [[Russian|howto-ru]]