X-Git-Url: https://git.cworth.org/git?p=notmuch-wiki;a=blobdiff_plain;f=performance.mdwn;h=7f9f7dc7903cb6c7638328a4524d1c9e9596edcd;hp=aa1029bd3e3b8dadaf2e947b027bdb36cec54cc6;hb=HEAD;hpb=be39bee69bab9b08da8356cecf73931261aee803 diff --git a/performance.mdwn b/performance.mdwn index aa1029b..664d67d 100644 --- a/performance.mdwn +++ b/performance.mdwn @@ -1,103 +1,101 @@ +[[!img notmuch-logo.png alt="Notmuch logo" class="left"]] # Optimize notmuch's performance -## Xapian 1.2.x database format: chert vs. flint +[[!toc levels=2]] -Xapian 1.2 (in contrast to 1.0.x) uses a new database format called -`chert` which is a lot faster than the old format `flint`. +## Use of ecryptfs slows notmuch down. -Check whether you are on chert or flint: +I noticed that my `notmuch new` was really slow (several minutes) for +less then 20 new emails (around 20K in my database). - $ ls $(notmuch config get database.path)/.notmuch/xapian/iam* - iamchert - -If it does not say `iamchert`, but `iamflint`, than you should migrate -(read on). +The performance tests in the notmuch source didn't show any real +issue, even on a non-encrypted ext4 fs. -There might be a file named `flintlock`, it is no indicator for your -database format, but also used in case of `chert`. +After investigation and running different tests, a switch to a +LUKS-btrfs based filesystem to replace the ext4-ecryptfs resolved the +issue. It now takes some seconds to index hundreds of messages. -### Migrate from flint to chert +## Xapian database format: chert vs. flint vs. glass -1. make sure you are using xapian 1.2.x (depends on your distribution) +Depending on the Xapian version that was used during the initial notmuch +database creation, you might use another database format than the most recent +one, which is often faster. -XXX: add examples +Check whether you are on chert, flint or glass: -2. Backup your notmuch tags - - $ cd - $ notmuch dump notmuch-dump-b4chert + $ ls $(notmuch config get database.path)/.notmuch/xapian/iam* + iamglass -3. Move away your old database +If it does not say `iamglass`, but `iamchert` or `iamflint`, than you should +migrate (read on). - $ mv $(notmuch config get database.path)/.notmuch/ notmuch-db-b4chert +There might be a file named `flintlock`, it is no indicator for your database +format, but also used in case of `chert`. -4. Find and import your messages, a new and fresh database will be -created. +### Migrate from an older format to the most recent one (glass) - $ XAPIAN_PREFER_CHERT=1 notmuch new +Make sure you are using xapian 1.4.x (depends on your distribution) -I did not need the `XAPIAN_PREFER_CHERT`, but am not sure whether -there are situations making it necessary (certain compile-time options -to xapian?). This note is just here to be extended or removed by -somebody more knowledgeable. +XXX: add examples for different distributions -After the initial import it is definitely not necessary to define -`XAPIAN_PREFER_CHERT` for subsequent calls of `notmuch new`. +Backup your notmuch tags: -5. Restore your tags + $ cd + $ notmuch dump --output=notmuch-dump-oldformat - $ notmuch restore notmuch-dump-b4chert +Move away your old database: -6. Check whether you are on chert now. + $ mv $(notmuch config get database.path)/.notmuch/ notmuch-db.bak - $ ls $(notmuch config get database.path)/.notmuch/xapian/iam* - iamchert +Find and import your messages, a new and fresh database will be +created: -If it says `iamflint`, most likely you are still using xapian 1.0.x. + $ notmuch new -7. Clean-up +Restore your tags: -If you are sure everything is fine, you can delete the old things. The -dump of your tags you might want to keep anyway as a backup - on the -other hand you should have some mechanism in place to make regular -backups of your notmuch database. + $ notmuch restore --input=notmuch-dump-oldformat - $ rm notmuch-dump-b4chert - $ rm -R notmuch-db-b4chert +Check whether you are on glass now: -### Compact your database + $ ls $(notmuch config get database.path)/.notmuch/xapian/iam* + iamglass -In order to greatly reduce the size of your database, you should -compact it. Carl says "This functionality is not yet available in the -Xapian library interface or else I would probably make notmuch call it -after building the database." and Pieter "[compacting] will cause -writes to be a bit more expensive in the short term, but will -significantly reduce the db's size, which your SSD will surely -appreciate." + If it says `iamflint` or `iamchert`, most likely you are still using an old + xapian version. -This needs xapian 1.2 +Clean-up: if you are sure everything is fine, you can delete the old +things. The dump of your tags you might want to keep anyway as a +backup - on the other hand you should have some mechanism in place to +make regular backups of your notmuch database: -Backup your database: + $ rm notmuch-dump-oldformat + $ rm -R notmuch-db.bak - $ cd - $ notmuch dump notmuch-dump-b4compact +### Compact your database -Compact it: +In order to greatly reduce the size of your database, you should compact it. +This can be easily achieved with `notmuch compact`: - $ cd $(notmuch config get database.path)/.notmuch - $ xapian-compact --no-renumber xapian xapian-compacted + $ notmuch compact -Remove old and renamed compacted, if compacting went well: +Note that this process can take a long time for large database and a write lock +will be held for the duration to ensure data integrity. The process will create +a new compacted database in a temporary directory and finally replace the +original database if compaction is successful. For added safety you can provide +`--backup=` option to preserve a copy of the original database. For +more details on this process see `notmuch-compact` man page. - $ rm -R xapian - $ mv xapian-compacted xapian +### More information about this in the email archives: +* [Carl Worth](https://notmuchmail.org/pipermail/notmuch/2010/001105.html) -### More information about this in the email archives: +* [Sebastian Spaeth](https://notmuchmail.org/pipermail/notmuch/2011/004048.html) -* [Carl Worth](http://notmuchmail.org/pipermail/notmuch/2010/001105.html) +* [Pieter Praet](https://notmuchmail.org/pipermail/notmuch/2011/004175.html) -* [Sebastian Spaeth](http://notmuchmail.org/pipermail/notmuch/2011/004048.html) +## Performance test suite -* [Pieter Praet](http://notmuchmail.org/pipermail/notmuch/2011/004175.html) +Notmuch ships with a [[performance test suite|perf-test]] which uses a downloadable +[[corpus]] of email.