+[[!img notmuch-logo.png alt="Notmuch logo" class="left"]]
# Optimize notmuch's performance
-## Xapian 1.2.x database format: chert vs. flint
+[[!toc levels=2]]
-Xapian 1.2 (in contrast to 1.0.x) uses a new database format called
-`chert` which is a lot faster than the old format `flint`.
+## Use of ecryptfs slows notmuch down.
-Check whether you are on chert or flint:
+I noticed that my `notmuch new` was really slow (several minutes) for
+less then 20 new emails (around 20K in my database).
- $ ls $(notmuch config get database.path)/.notmuch/xapian/iam*
- iamchert
-
-If it does not say `iamchert`, but `iamflint`, than you should migrate
-(read on).
+The performance tests in the notmuch source didn't show any real
+issue, even on a non-encrypted ext4 fs.
-There might be a file named `flintlock`, it is no indicator for your
-database format, but also used in case of `chert`.
+After investigation and running different tests, a switch to a
+LUKS-btrfs based filesystem to replace the ext4-ecryptfs resolved the
+issue. It now takes some seconds to index hundreds of messages.
-### Migrate from flint to chert
+## Xapian database format: chert vs. flint vs. glass
-1. make sure you are using xapian 1.2.x (depends on your distribution)
+Depending on the Xapian version that was used during the initial notmuch
+database creation, you might use another database format than the most recent
+one, which is often faster.
-XXX: add examples
+Check whether you are on chert, flint or glass:
-2. Backup your notmuch tags
-
- $ cd
- $ notmuch dump notmuch-dump-b4chert
+ $ ls $(notmuch config get database.path)/.notmuch/xapian/iam*
+ iamglass
-3. Move away your old database
+If it does not say `iamglass`, but `iamchert` or `iamflint`, than you should
+migrate (read on).
- $ mv $(notmuch config get database.path)/.notmuch/ notmuch-db-b4chert
+There might be a file named `flintlock`, it is no indicator for your database
+format, but also used in case of `chert`.
-4. Find and import your messages, a new and fresh database will be
-created.
+### Migrate from an older format to the most recent one (glass)
- $ XAPIAN_PREFER_CHERT=1 notmuch new
+Make sure you are using xapian 1.4.x (depends on your distribution)
-I did not need the `XAPIAN_PREFER_CHERT`, but am not sure whether
-there are situations making it necessary (certain compile-time options
-to xapian?). This note is just here to be extended or removed by
-somebody more knowledgeable.
+XXX: add examples for different distributions
-After the initial import it is definitely not necessary to define
-`XAPIAN_PREFER_CHERT` for subsequent calls of `notmuch new`.
+Backup your notmuch tags:
-5. Restore your tags
+ $ cd
+ $ notmuch dump --output=notmuch-dump-oldformat
- $ notmuch restore notmuch-dump-b4chert
+Move away your old database:
-6. Check whether you are on chert now.
+ $ mv $(notmuch config get database.path)/.notmuch/ notmuch-db.bak
- $ ls $(notmuch config get database.path)/.notmuch/xapian/iam*
- iamchert
+Find and import your messages, a new and fresh database will be
+created:
-If it says `iamflint`, most likely you are still using xapian 1.0.x.
+ $ notmuch new
-7. Clean-up
+Restore your tags:
-If you are sure everything is fine, you can delete the old things. The
-dump of your tags you might want to keep anyway as a backup - on the
-other hand you should have some mechanism in place to make regular
-backups of your notmuch database.
+ $ notmuch restore --input=notmuch-dump-oldformat
- $ rm notmuch-dump-b4chert
- $ rm -R notmuch-db-b4chert
+Check whether you are on glass now:
-### Compact your database
+ $ ls $(notmuch config get database.path)/.notmuch/xapian/iam*
+ iamglass
-In order to greatly reduce the size of your database, you should
-compact it. Carl says "This functionality is not yet available in the
-Xapian library interface or else I would probably make notmuch call it
-after building the database." and Pieter "[compacting] will cause
-writes to be a bit more expensive in the short term, but will
-significantly reduce the db's size, which your SSD will surely
-appreciate."
+ If it says `iamflint` or `iamchert`, most likely you are still using an old
+ xapian version.
-This needs xapian 1.2
+Clean-up: if you are sure everything is fine, you can delete the old
+things. The dump of your tags you might want to keep anyway as a
+backup - on the other hand you should have some mechanism in place to
+make regular backups of your notmuch database:
-Backup your database:
+ $ rm notmuch-dump-oldformat
+ $ rm -R notmuch-db.bak
- $ cd
- $ notmuch dump notmuch-dump-b4compact
+### Compact your database
-Compact it:
+In order to greatly reduce the size of your database, you should compact it.
+This can be easily achieved with `notmuch compact`:
- $ cd $(notmuch config get database.path)/.notmuch
- $ xapian-compact --no-renumber xapian xapian-compacted
+ $ notmuch compact
-Remove old and renamed compacted, if compacting went well:
+Note that this process can take a long time for large database and a write lock
+will be held for the duration to ensure data integrity. The process will create
+a new compacted database in a temporary directory and finally replace the
+original database if compaction is successful. For added safety you can provide
+`--backup=<directory>` option to preserve a copy of the original database. For
+more details on this process see `notmuch-compact` man page.
- $ rm -R xapian
- $ mv xapian-compacted xapian
+### More information about this in the email archives:
+* [Carl Worth](https://notmuchmail.org/pipermail/notmuch/2010/001105.html)
-### More information about this in the email archives:
+* [Sebastian Spaeth](https://notmuchmail.org/pipermail/notmuch/2011/004048.html)
-* [Carl Worth](http://notmuchmail.org/pipermail/notmuch/2010/001105.html)
+* [Pieter Praet](https://notmuchmail.org/pipermail/notmuch/2011/004175.html)
-* [Sebastian Spaeth](http://notmuchmail.org/pipermail/notmuch/2011/004048.html)
+## Performance test suite
-* [Pieter Praet](http://notmuchmail.org/pipermail/notmuch/2011/004175.html)
+Notmuch ships with a [[performance test suite|perf-test]] which uses a downloadable
+[[corpus]] of email.
projects / notmuch-wiki / blobdiff