+[[!img notmuch-logo.png alt="Notmuch logo" class="left"]]
# Optimize notmuch's performance
[[!toc levels=2]]
-## Xapian 1.2.x database format: chert vs. flint
+## Use of ecryptfs slows notmuch down.
-Xapian 1.2 (in contrast to 1.0.x) uses a new database format called
-`chert` which is a lot faster than the old format `flint`.
+I noticed that my `notmuch new` was really slow (several minutes) for
+less then 20 new emails (around 20K in my database).
-Check whether you are on chert or flint:
+The performance tests in the notmuch source didn't show any real
+issue, even on a non-encrypted ext4 fs.
+
+After investigation and running different tests, a switch to a
+LUKS-btrfs based filesystem to replace the ext4-ecryptfs resolved the
+issue. It now takes some seconds to index hundreds of messages.
+
+## Xapian database format: chert vs. flint vs. glass
+
+Depending on the Xapian version that was used during the initial notmuch
+database creation, you might use another database format than the most recent
+one, which is often faster.
+
+Check whether you are on chert, flint or glass:
$ ls $(notmuch config get database.path)/.notmuch/xapian/iam*
- iamchert
+ iamglass
-If it does not say `iamchert`, but `iamflint`, than you should migrate
-(read on).
+If it does not say `iamglass`, but `iamchert` or `iamflint`, than you should
+migrate (read on).
-There might be a file named `flintlock`, it is no indicator for your
-database format, but also used in case of `chert`.
+There might be a file named `flintlock`, it is no indicator for your database
+format, but also used in case of `chert`.
-### Migrate from flint to chert
+### Migrate from an older format to the most recent one (glass)
-Make sure you are using xapian 1.2.x (depends on your distribution)
+Make sure you are using xapian 1.4.x (depends on your distribution)
XXX: add examples for different distributions
Backup your notmuch tags:
$ cd
- $ notmuch dump --output=notmuch-dump-b4chert
+ $ notmuch dump --output=notmuch-dump-oldformat
Move away your old database:
- $ mv $(notmuch config get database.path)/.notmuch/ notmuch-db-b4chert
+ $ mv $(notmuch config get database.path)/.notmuch/ notmuch-db.bak
Find and import your messages, a new and fresh database will be
created:
- $ XAPIAN_PREFER_CHERT=1 notmuch new
-
-I did not need the `XAPIAN_PREFER_CHERT`, but am not sure whether
-there are situations making it necessary (certain compile-time options
-to xapian?). This note is just here to be extended or removed by
-somebody more knowledgeable.
-
-After the initial import it is definitely not necessary to define
-`XAPIAN_PREFER_CHERT` for subsequent calls of `notmuch new`.
+ $ notmuch new
Restore your tags:
- $ notmuch restore --input=notmuch-dump-b4chert
+ $ notmuch restore --input=notmuch-dump-oldformat
-Check whether you are on chert now:
+Check whether you are on glass now:
$ ls $(notmuch config get database.path)/.notmuch/xapian/iam*
- iamchert
+ iamglass
- If it says `iamflint`, most likely you are still using xapian 1.0.x.
+ If it says `iamflint` or `iamchert`, most likely you are still using an old
+ xapian version.
Clean-up: if you are sure everything is fine, you can delete the old
things. The dump of your tags you might want to keep anyway as a
backup - on the other hand you should have some mechanism in place to
make regular backups of your notmuch database:
- $ rm notmuch-dump-b4chert
- $ rm -R notmuch-db-b4chert
+ $ rm notmuch-dump-oldformat
+ $ rm -R notmuch-db.bak
### Compact your database
-In order to greatly reduce the size of your database, you should
-compact it. Carl says "This functionality is not yet available in the
-Xapian library interface or else I would probably make notmuch call it
-after building the database." and Pieter "[compacting] will cause
-writes to be a bit more expensive in the short term, but will
-significantly reduce the db's size, which your SSD will surely
-appreciate."
+In order to greatly reduce the size of your database, you should compact it.
+This can be easily achieved with `notmuch compact`:
-This needs xapian 1.2
+ $ notmuch compact
-Backup your database:
+Note that this process can take a long time for large database and a write lock
+will be held for the duration to ensure data integrity. The process will create
+a new compacted database in a temporary directory and finally replace the
+original database if compaction is successful. For added safety you can provide
+`--backup=<directory>` option to preserve a copy of the original database. For
+more details on this process see `notmuch-compact` man page.
- $ cd
- $ notmuch dump --output=notmuch-dump-b4compact
-
-Compact it:
-
- $ cd $(notmuch config get database.path)/.notmuch
- $ xapian-compact --no-renumber xapian xapian-compacted
-
-Remove old and renamed compacted, if compacting went well:
+### More information about this in the email archives:
- $ rm -R xapian
- $ mv xapian-compacted xapian
+* [Carl Worth](https://notmuchmail.org/pipermail/notmuch/2010/001105.html)
+* [Sebastian Spaeth](https://notmuchmail.org/pipermail/notmuch/2011/004048.html)
-### More information about this in the email archives:
+* [Pieter Praet](https://notmuchmail.org/pipermail/notmuch/2011/004175.html)
-* [Carl Worth](http://notmuchmail.org/pipermail/notmuch/2010/001105.html)
+## Performance test suite
-* [Sebastian Spaeth](http://notmuchmail.org/pipermail/notmuch/2011/004048.html)
+Notmuch ships with a [[performance test suite|perf-test]] which uses a downloadable
+[[corpus]] of email.
-* [Pieter Praet](http://notmuchmail.org/pipermail/notmuch/2011/004175.html)
projects / notmuch-wiki / blobdiff