doc: Examples of notmuch-reindex use and crypto policy
Currently, notmuch has the levers needed to set coherent crypto policy
around how cleartext is indexed, which also has an impact on how
messages are rendered. But we don't have a lot of documentation about
how to do sensible things. This is an initial attempt to address
that.
The first example shows a way to selectively index specific messages.
The next two examples are about aligning the existing database with
crypto indexing policy
The default crypto policy is to not index cleartext, and to only
decrypt messages on display when explicitly requested.
The other sensible crypto policy is to index cleartext while stashing
session keys. messages indexed in this way will be searchable, and
will be decrypted on display automatically unless the user explicitly
asks for it to *not* be decrypted.
The policy for indexing *new* messages is stored in the database as
the config variable index.decrypt.
But setting policy for new messages doesn't retroactively affect
already indexed messages.
This patch attempts to document ways that someone can efficiently
align their pre-existing database with their new policy.
I'm not sure this is the right place to document these examples, but i
do want them to be user-facing and relatively easy to find. I'm happy
to entertain suggestions for where else we should put them.
In some cases (e.g. when building a publicly-visible e-mail archive)
it doesn't make any sense to restrict visibility of the message to the
current user account.
This adds a --world-readable boolean option for "notmuch insert", so
that those who want to archive their mail publicly can feed their
archiver with:
notmuch insert --world-readable
Other local delivery agents (postfix's local, and dovecot's lda) all
default to delivery in mode 0600 rather than relying on the user's
umask, so this fix doesn't change the default.
Also, this does not override the user's umask. if the umask is
already set tight, it will not become looser as the result of passing
--world-readable.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Matthew Lear [Tue, 6 Feb 2018 21:52:01 +0000 (21:52 +0000)]
Clarify the syntax required when searching using timestamps.
Need to be clearer about specifying time ranges using timestamps.
Legacy syntax which predates the date prefix is still supported, but
timestamps used in conjunction with the date prefix require additional
syntax.
No one knows why this invocation is here. it does weird things during
the build. If a developer really wants this behavior, they can
specify it in their own ~/.gbp.conf
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
Jani Nikula [Sat, 30 Dec 2017 17:16:11 +0000 (19:16 +0200)]
doc: unify definition list usage across man pages
Make all parameter descriptions etc. use reStructuredText definition
lists with uniform style and indentation. Remove redundant indentation
from around the lists. Remove blank lines between term lines and
definition blocks. Use four spaces for indentation.
This is almost completely whitespace and paragraph reflow changes.
David Bremner [Sat, 30 Dec 2017 15:57:44 +0000 (11:57 -0400)]
NEWS: Python bindings changes for 0.26
These are my own changes, plus those of Florian Klink and Lucas
Hoffmann. Gaute's change fixed a bug that was never in a released
version, so I left it out.
This brings the --decrypt argument to "notmuch reply" into line with
the other --decrypt arguments (in "show", "new", "insert", and
"reindex"). This patch is really just about bringing consistency to
the user interface.
We also use the recommended form in the emacs MUA when replying, and
update test T350 to match.
W. Trevor King [Thu, 28 Dec 2017 18:11:57 +0000 (10:11 -0800)]
nmbug: Bump to version 0.3
Changes since 0.2:
* Accept failures to unset core.worktree in clone (0a155847,
2017-10-10, unreleased).
* Use --no-renames in log (f9189a06, 2016-09-26, v0.24).
* Auto-checkout in clone if it wouldn't clobber (7ef3b653, 2017-10-10,
unreleased).
* Add a 'help' command for folks who don't like --help
(9d25c97d, 2014-10-03, v0.20).
* Setup a 'config' branch on clone to track origin/config (244f8739,
2015-03-22, v0.20). This branch may be consumed by
notmuch-report(1).
* Only error for invalid diff lines in tags/ (57225988, 2017-10-16,
unreleased).
* Ignore # comments in 'notmuch dump ...' output (9bbc54bd,
2016-03-27, v0.22).
* Respect 'expect' in _spawn(..., wait=True) (e263c5b1, 2017-10-10,
unreleased).
* Update URLs in documentation (554b90b5 and 6a833a6e8, 2016-06-02,
v0.23).
David Bremner [Sun, 24 Dec 2017 13:58:45 +0000 (09:58 -0400)]
python: remove obsolete debian directory
It is unlikely this still works since it has not been updated since
2010. The python packages for debian are now built by the top level
debian/ packaging.
David Bremner [Sun, 20 Aug 2017 21:32:40 +0000 (18:32 -0300)]
lib: add thread subqueries.
This change allows queries of the form
thread:{from:me} and thread:{from:jian} and not thread:{from:dave}
This is still somewhat brute-force, but it's a big improvement over
both the shell script solution and the previous proposal [1], because it
does not build the whole thread structure just generate a
query. A further potential optimization is to replace the calls to
notmuch with more specialized Xapian code; in particular it's not
likely that reading all of the message metadata is a win here.
Describe the introduction of the "reindex" subcommand. This blurb
acknowledges Subject: instability under reindexing when multiple
copies exist (suggesting that this is something that needn't
necessarily hold up a release).
David Bremner [Fri, 15 Dec 2017 02:29:57 +0000 (22:29 -0400)]
lib: return "" rather than NULL from notmuch_thread_get_authors
The current behaviour is at best under-documented. The modified test in
T470-missing-headers.sh previously relied on printf doing the right
thing with NULL, which seems icky.
The use of talloc_strdup here is probably overkill, but it avoids
having to enforce that thread->authors is never mutated outside
_resolve_thread_authors_string.
test: retrying lock tests are broken when built against older Xapian
If we're building against a version of Xapian that doesn't offer
retrying the lock, we should be honest and describe the tests as
broken, rather than marking them as missing a test prerequisite.
missing test prerequisites should be for specific components of the
test harness that are missing, not for the backend library notmuch
uses.
test: named query tests are broken when missing Xapian FieldProcessor
Named queries don't work without Xapian FieldProcessor. Rather than
silently skipping them, we should explictly mark them as broken when
building against an older version of Xapian.
test: searching by absolute date is broken without Xapian FieldProcessor
Previously, the test suite had simply silently skipped the absolute
date test if we're using an archaic version of Xapian. For
correctness, we should instead mark the test as broken.
This also changes from string to numeric comparison when checking
NOMTUCH_HAVE_XAPIAN_FIELD_PROCESSOR for consistency with other tests.
l-m-h@web.de [Thu, 7 Dec 2017 11:40:50 +0000 (12:40 +0100)]
python: Rename get_config_list to get_configs
The old name has a bit of a feeling of hungarian notation. Also many
generators in the core are named with the suffix "s" to indicate
iterables: dict.items, dict.keys for example.
l-m-h@web.de [Thu, 7 Dec 2017 11:40:46 +0000 (12:40 +0100)]
python: add bindings to access config
The C functions notmuch_database_get_config,
notmuch_database_get_config_list and notmuch_database_set_config are
part of the official C bindings. So there should also be some python
bindings for them.
Also they are the only way to access the named queries introduced in b9bf3f44.
The interface of the python functions is designed to be close to the C
functions.
W. Trevor King [Mon, 16 Oct 2017 18:01:47 +0000 (11:01 -0700)]
nmbug: Only error for invalid diff lines in tags/
Avoid:
Traceback (most recent call last):
File "/home/nmbug/bin/nmbug", line 834, in <module>
args.func(**kwargs)
File "/home/nmbug/bin/nmbug", line 385, in checkout
status = get_status()
File "/home/nmbug/bin/nmbug", line 580, in get_status
maybe_deleted = _diff_index(index=index, filter='D')
File "/home/nmbug/bin/nmbug", line 658, in _diff_index
for id, tag in _unpack_diff_lines(stream=p.stdout):
File "/home/nmbug/bin/nmbug", line 678, in _unpack_diff_lines
'Invalid line in diff: {!r}'.format(line.strip()))
ValueError: Invalid line in diff: u'.mailmap'
With this commit, folks can commit READMEs, .mailmap, etc. to their
nmbug repositories, and 'nmbug diff' and 'status' won't choke on them.
If you want to check for this sort of thing, you can set --log-level
to info or greater. nmbug will still error if the unrecognized path
is under tags/, since that's more likely to be a user error.
Jani Nikula [Sat, 14 Oct 2017 13:16:27 +0000 (16:16 +0300)]
cli/new: support /<regex>/ in new.ignore
Add support for using /<regex>/ style regular expressions in
new.ignore, mixed with the old style verbatim file and directory
basenames. The regex is matched against the relative path from the
database path.
Jani Nikula [Thu, 2 Nov 2017 20:01:17 +0000 (22:01 +0200)]
doc: arrange search prefix documentation in a definition list
Having first a list of prefixes followed by detailed descriptions was
viable when we didn't have all that many prefixes. Now, arranging the
prefix descriptions in a definition list makes more sense.
While at it, include all the supported prefix forms, especially some
missing regex ones.
Jani Nikula [Sat, 14 Oct 2017 13:16:06 +0000 (16:16 +0300)]
cli: add support for --no- prefixed boolean and keyword flag arguments
Add transparent support for negating boolean and keyword flag
arguments using --no-argument style on the command line. That is, if
the option description contains a boolean or a keyword flag argument
named "argument", --no-argument will match and negate it.
For boolean arguments this obviously means the logical NOT. For
keyword flag arguments this means bitwise AND of the bitwise NOT,
i.e. masking out the specified bits instead of OR'ing them in.
For example, you can use --no-exclude instead of --exclude=false in
notmuch show. If we had keyword flag arguments with some flags
defaulting to on, say --include=tags in notmuch dump/restore, this
would allow --no-include=tags to switch that off while not affecting
other flags.
As a curiosity, you should be able to warp your brain using
--no-exclude=true meaning false and --no-exclude=false meaning true if
you wish.
Specifying both "argument" and "no-argument" style arguments in the
same option description should be avoided. In this case, --no-argument
would match whichever is specified first, and --argument would only
match "argument".
W. Trevor King [Tue, 10 Oct 2017 22:49:51 +0000 (15:49 -0700)]
nmbug: Auto-checkout in clone if it wouldn't clobber
We currently auto-checkout after pull and merge to make those more
convenient. They're guarded against data-loss with a leading
_insist_committed(). This commit adds the same convenience to clone,
since in most cases users will have no NMBPREFIX-prefixed tags in
their database when they clone. Users that *do* have
NMBPREFIX-prefixed tags will get a warning (and I've bumped the
default log level to warning so folks who don't set --log-level will
see it) like:
$ nmbug clone http://nmbug.notmuchmail.org/git/nmbug-tags.git
Cloning into '/tmp/nmbug-clone.g9dvd0tv'...
Checking connectivity: 16674, done.
Branch config set up to track remote branch config from origin.
Not checking out to avoid clobbering existing tags: notmuch::0.25, ...
W. Trevor King [Tue, 10 Oct 2017 22:49:50 +0000 (15:49 -0700)]
nmbug: Accept failures to unset core.worktree in clone
Since 6311cfaf (init: do not set unnecessary core.worktree,
2016-09-25, 2.11.0 [1]), Git has no longer set core.worktree when
--separate-git-dir is used. This broke clone with:
The initial discussion that lead to the Git change is in [2], and
there is some more discussion around this specific change in [3].
There is some useful background on working trees in this 2009 message
[4]. There is also a git-worktree(1) since df0b6cfb (worktree: new
place for "git prune --worktrees", 2015-06-29, 2.5.0 [5]) which grew
the ability to add new worktrees in 799767cc (Merge branch
'es/worktree-add', 2015-07-13, 2.5.0 [6]). Folks relying on
core.worktree in the --separate-git-dir case fall into the "former
case" in [4], and as Junio pointed out in that message, Git
operations like 'add' don't really work there.
In nmbug we don't want core.worktree, because our effective working
tree is the notmuch database. By accepting failed core.worktree
unsets, clone will work with Gits older and younger than 2.11.0.
W. Trevor King [Tue, 10 Oct 2017 22:49:49 +0000 (15:49 -0700)]
nmbug: Respect 'expect' in _spawn(..., wait=True)
Fixing a bug from 7f2cb3be (nmbug: Translate to Python, 2014-10-03).
The bug had no direct impact though, because none of the wait=True
callers were setting expect.
Also add expected codes to the debug messages, to help log readers
understand why nonzero exits are occasionally accepted.
David Bremner [Wed, 6 Dec 2017 01:17:58 +0000 (21:17 -0400)]
test/emacs: add exploit mitigation test
This test will pass if either the notmuch show mitigation code is
working correctly, or upstream emacs mime handling code has it's own
fix for https://bugs.gnu.org/28350.
Tomi Ollila [Wed, 6 Dec 2017 01:17:57 +0000 (21:17 -0400)]
emacs: letf enriched-decode-display-prop for text/enriched display
Dynamically bind enriched-decode-display-prop when inserting
text/enriched part. This complements commit 9b0582383833 for
emacs versions before 24.4 which do not have advice-add
functionality.
crypto: signature verification reports valid User IDs
When i'm trying to understand a message signature, i care that i know
who it came from (the "validity" of the identity associated with the
key), *not* whether i'm willing to accept the keyholder's other
identity assertions (the "trust" associated with the certificate).
We've been reporting User ID information based on the "trust"
associated with the certificate, because GMime didn't clearly expose
the validity of the User IDs.
This change relies on fixes made in GMime 3.0.3 and later which
include https://github.com/jstedfast/gmime/pull/18.
docs: clean up documentation about decryption policies
Now that the range of sensible decryption policies has come into full
view, we take a bit of space to document the distinctions.
Most people will use either "auto" or "true" -- but we provide "false"
and "nostash" to handle use cases that might reasonably be requested.
Note also that these can be combined in sensible ways. Like, if your
mail comes in regularly to a service that doesn't have access to your
secret keys, but does have access to your index, and you feel
comfortable adding selected encrypted messages to the index after
you've read them, you could stay in "auto" normally, and then when you
find yourself reading an indexable message (e.g. one you want to be
able to search for in the future, and that you don't mind exposing to
whatever entities have access to your inde), you can do:
crypto: add --decrypt=nostash to avoid stashing session keys
Here's the configuration choice for people who want a cleartext index,
but don't want stashed session keys.
Interestingly, this "nostash" decryption policy is actually the same
policy that should be used by "notmuch show" and "notmuch reply",
since they never modify the index or database when they are invoked
with --decrypt.
We take advantage of this parallel to tune the behavior of those
programs so that we're not requesting session keys from GnuPG during
"show" and "reply" that we would then otherwise just throw away.
crypto: actually stash session keys when decrypt=true
If you're going to store the cleartext index of an encrypted message,
in most situations you might just as well store the session key.
Doing this storage has efficiency and recoverability advantages.
Combined with a schedule of regular OpenPGP subkey rotation and
destruction, this can also offer security benefits, like "deletable
e-mail", which is the store-and-forward analog to "forward secrecy".
But wait, i hear you saying, i have a special need to store cleartext
indexes but it's really bad for me to store session keys! Maybe
(let's imagine) i get lots of e-mails with incriminating photos
attached, and i want to be able to search for them by the text in the
e-mail, but i don't want someone with access to the index to be
actually able to see the photos themselves.
Fret not, the next patch in this series will support your wacky
uncommon use case.