notmuch performance: chert vs flint, compacting

diff --git a/index.mdwn b/index.mdwn
index 172015e6ecc24f2311335112a14fe2affef359d0..9924de86312d169fdb13c34e411773d1d905e051 100644 (file)
--- a/index.mdwn
+++ b/index.mdwn
@@ -52,6 +52,7 @@ contain a reference for users.  This is what we have so far:
   * [[Initial_Tagging]]
   * [[Tips for using notmuch within Emacs|emacstips]]
   * [[Remote Usage|remoteusage]]
   * [[Initial_Tagging]]
   * [[Tips for using notmuch within Emacs|emacstips]]
   * [[Remote Usage|remoteusage]]

+  * [[Performance|performance]]

 
 Apart from the wiki, help is available via email.  Join the mailing
 list.  Read the archives.  Ask questions.
 
 Apart from the wiki, help is available via email.  Join the mailing
 list.  Read the archives.  Ask questions.

diff --git a/performance.mdwn b/performance.mdwn
new file mode 100644 (file)
index 0000000..dcd70e0
--- /dev/null
+++ b/performance.mdwn
@@ -0,0 +1,103 @@
+# Optimize notmuch's performance
+
+## Xapian 1.2.x database format: chert vs. flint
+
+Xapian 1.2 (in contrast to 1.0.x) uses a new database format called
+`chert` which is a lot faster than the old format `flint`.
+
+Check whether you are on chert or flint:
+
+    $ ls $(notmuch config get database.path)/.notmuch/xapian/iam*
+    iamchert
+
+If it does not say `iamchert`, but `iamflint`, than you should migrate
+(read on).
+
+There might be a file named `flintlock`, it is no indicator for your
+database format, but also used in case of `chert`.
+
+### Migrate from flint to chert
+
+1. make sure you are using xapian 1.2.x (depends on your distribution)
+
+XXX: add examples
+
+2. Backup your notmuch tags
+
+    $ cd
+    $ notmuch dump notmuch-dump-b4chert
+
+3. Move away your old database
+
+    $ mv $(notmuch config get database.path)/.notmuch/ notmuch-db-b4chert
+
+4. Find and import your messages, a new and fresh database will be
+created.
+
+    $ XAPIAN_PREFER_CHERT=1 notmuch new
+
+I did not need the `XAPIAN_PREFER_CHERT`, but am not sure whether
+there are situations making it necessary (certain compile-time options
+to xapian?). This note is just here to be extended or removed by
+somebody more knowledgeable.
+
+After the initial import it is definitely not necessary to define
+`XAPIAN_PREFER_CHERT` for subsequent calls of `notmuch new`.
+
+5. Restore your tags
+
+    $ notmuch restore notmuch-dump-b4chert
+
+6. Check whether you are on chert now.
+
+    $ ls $(notmuch config get database.path)/.notmuch/xapian/iam*
+    iamchert
+
+If it says `iamflint`, most likely you are still using xapian 1.0.x.
+
+7. Clean-up
+
+If you are sure everything is fine, you can delete the old things. The
+dump of your tags you might want to keep anyway as a backup - on the
+other hand you should have some mechanism in place to make regular
+backups of your notmuch database.
+
+    $ rm notmuch-dump-b4chert
+    $ rm -R notmuch-db-b4chert
+
+### Compact your database
+
+In order to greatly reduce the size of your database, you should
+compact it. Carl says "This functionality is not yet available in the
+Xapian library interface or else I would probably make notmuch call it
+after building the database." and Pieter "[compacting] will cause
+writes to be a bit more expensive in the short term, but will
+significantly reduce the db's size, which your SSD will surely
+appreciate."
+
+This needs xapian 1.2
+
+Backup your database:
+
+    $ cd
+    $ notmuch dump notmuch-dump-b4compact
+
+Compact it:
+
+    $ cd $(notmuch config get database.path)/.notmuch
+    $ xapian-compact --no-renumber xapian xapian-compacted
+
+Remove old and renamed compacted, if compacting went well:
+
+    $ rm -R xapian
+    $ mv xapian-compacted xapian
+
+
+### More information about this in the email archives:
+
+* [id:xyz, Carl Worth](http://notmuchmail.org/pipermail/notmuch/2010/001105.html)
+
+* [id:xyz, Sebastian Spaeth](http://notmuchmail.org/pipermail/notmuch/2011/004048.html)
+
+* [id:xyz, Pieter Praet](http://notmuchmail.org/pipermail/notmuch/2011/004175.html)
+