X-Git-Url: https://git.cworth.org/git?p=notmuch-wiki;a=blobdiff_plain;f=performance.mdwn;h=7f9f7dc7903cb6c7638328a4524d1c9e9596edcd;hp=8515e6a637bb196e6231c9bd115c117dbff1dd9d;hb=HEAD;hpb=058e62e9c21dc0e1c455ac9f50c7782848727430 diff --git a/performance.mdwn b/performance.mdwn index 8515e6a..664d67d 100644 --- a/performance.mdwn +++ b/performance.mdwn @@ -1,102 +1,101 @@ +[[!img notmuch-logo.png alt="Notmuch logo" class="left"]] # Optimize notmuch's performance [[!toc levels=2]] -## Xapian 1.2.x database format: chert vs. flint +## Use of ecryptfs slows notmuch down. -Xapian 1.2 (in contrast to 1.0.x) uses a new database format called -`chert` which is a lot faster than the old format `flint`. +I noticed that my `notmuch new` was really slow (several minutes) for +less then 20 new emails (around 20K in my database). -Check whether you are on chert or flint: +The performance tests in the notmuch source didn't show any real +issue, even on a non-encrypted ext4 fs. + +After investigation and running different tests, a switch to a +LUKS-btrfs based filesystem to replace the ext4-ecryptfs resolved the +issue. It now takes some seconds to index hundreds of messages. + +## Xapian database format: chert vs. flint vs. glass + +Depending on the Xapian version that was used during the initial notmuch +database creation, you might use another database format than the most recent +one, which is often faster. + +Check whether you are on chert, flint or glass: $ ls $(notmuch config get database.path)/.notmuch/xapian/iam* - iamchert + iamglass -If it does not say `iamchert`, but `iamflint`, than you should migrate -(read on). +If it does not say `iamglass`, but `iamchert` or `iamflint`, than you should +migrate (read on). -There might be a file named `flintlock`, it is no indicator for your -database format, but also used in case of `chert`. +There might be a file named `flintlock`, it is no indicator for your database +format, but also used in case of `chert`. -### Migrate from flint to chert +### Migrate from an older format to the most recent one (glass) -Make sure you are using xapian 1.2.x (depends on your distribution) +Make sure you are using xapian 1.4.x (depends on your distribution) XXX: add examples for different distributions Backup your notmuch tags: $ cd - $ notmuch dump notmuch-dump-b4chert + $ notmuch dump --output=notmuch-dump-oldformat Move away your old database: - $ mv $(notmuch config get database.path)/.notmuch/ notmuch-db-b4chert + $ mv $(notmuch config get database.path)/.notmuch/ notmuch-db.bak Find and import your messages, a new and fresh database will be created: - $ XAPIAN_PREFER_CHERT=1 notmuch new - -I did not need the `XAPIAN_PREFER_CHERT`, but am not sure whether -there are situations making it necessary (certain compile-time options -to xapian?). This note is just here to be extended or removed by -somebody more knowledgeable. - -After the initial import it is definitely not necessary to define -`XAPIAN_PREFER_CHERT` for subsequent calls of `notmuch new`. + $ notmuch new Restore your tags: - $ notmuch restore notmuch-dump-b4chert + $ notmuch restore --input=notmuch-dump-oldformat -Check whether you are on chert now: +Check whether you are on glass now: $ ls $(notmuch config get database.path)/.notmuch/xapian/iam* - iamchert + iamglass - If it says `iamflint`, most likely you are still using xapian 1.0.x. + If it says `iamflint` or `iamchert`, most likely you are still using an old + xapian version. Clean-up: if you are sure everything is fine, you can delete the old things. The dump of your tags you might want to keep anyway as a backup - on the other hand you should have some mechanism in place to make regular backups of your notmuch database: - $ rm notmuch-dump-b4chert - $ rm -R notmuch-db-b4chert + $ rm notmuch-dump-oldformat + $ rm -R notmuch-db.bak ### Compact your database -In order to greatly reduce the size of your database, you should -compact it. Carl says "This functionality is not yet available in the -Xapian library interface or else I would probably make notmuch call it -after building the database." and Pieter "[compacting] will cause -writes to be a bit more expensive in the short term, but will -significantly reduce the db's size, which your SSD will surely -appreciate." +In order to greatly reduce the size of your database, you should compact it. +This can be easily achieved with `notmuch compact`: -This needs xapian 1.2 + $ notmuch compact -Backup your database: +Note that this process can take a long time for large database and a write lock +will be held for the duration to ensure data integrity. The process will create +a new compacted database in a temporary directory and finally replace the +original database if compaction is successful. For added safety you can provide +`--backup=` option to preserve a copy of the original database. For +more details on this process see `notmuch-compact` man page. - $ cd - $ notmuch dump notmuch-dump-b4compact - -Compact it: - - $ cd $(notmuch config get database.path)/.notmuch - $ xapian-compact --no-renumber xapian xapian-compacted - -Remove old and renamed compacted, if compacting went well: +### More information about this in the email archives: - $ rm -R xapian - $ mv xapian-compacted xapian +* [Carl Worth](https://notmuchmail.org/pipermail/notmuch/2010/001105.html) +* [Sebastian Spaeth](https://notmuchmail.org/pipermail/notmuch/2011/004048.html) -### More information about this in the email archives: +* [Pieter Praet](https://notmuchmail.org/pipermail/notmuch/2011/004175.html) -* [Carl Worth](http://notmuchmail.org/pipermail/notmuch/2010/001105.html) +## Performance test suite -* [Sebastian Spaeth](http://notmuchmail.org/pipermail/notmuch/2011/004048.html) +Notmuch ships with a [[performance test suite|perf-test]] which uses a downloadable +[[corpus]] of email. -* [Pieter Praet](http://notmuchmail.org/pipermail/notmuch/2011/004175.html)