doc: new notmuch show --sort and related emacs commands
New --sort CLI option documented in notmuch-show's man page, and
notmuch-search-toggle-order mentioned in doc/notmuch-emacs.rst and
devel/emacs-keybindings.org (in the latter, there's also some
whitespace changes in a table introduced by org-mode).
Add the command-line option --sort to the show command of the CLI
notmuch interface, with the same possible values as the same option in
notmuch search.
David Bremner [Sat, 26 Jun 2021 12:04:28 +0000 (09:04 -0300)]
emacs/tree use notmuch-show-single-message
This is more efficient than notmuch-show-only-matching-messages, since
we do not parse the potentially large thread structure to find a
single message.
This is only a partial fix for notmuch-tree view, because displaying
the thread structure in the tree-mode window still crashes on long
threads. It is however enough to make unthreaded view handle long
threads.
This dynamically bound variable can be set when the caller of
notmuch-show guarantees that exactly one message will match the
query. It avoids transporting and parsing the complete thread
structure.
edef [Sun, 13 Jun 2021 08:23:58 +0000 (08:23 +0000)]
emacs: remap send-message and send-message-and-exit
All three of C-c C-c, <menu-bar> <Message> <Send Message>,
and <tool-bar> <Send Message> are bound to message-send-and-exit by
message.el, but notmuch-mua.el only had an explicit override for the
keyboard binding. This mostly manifests as confusing Fcc behaviour for
GUI users.
Patching the bindings for specific keys is rather brittle, since it has
to be aware of every relevant binding. This change switches to instead
using a remap binding, which turns any binding for message-send or
message-send-and-exit into a binding for the corresponding notmuch-mua
command.
David Bremner [Tue, 18 May 2021 02:06:01 +0000 (23:06 -0300)]
lib: autocommit after some number of completed transactions
This change addresses two known issues with large sets of changes to
the database. The first is that as reported by Steven Allen [1],
notmuch commits are not "flushed" when they complete, which means that
if there is an open transaction when the database closes (or e.g. the
program crashes) then all changes since the last commit will be
discarded (nothing is irrecoverably lost for "notmuch new", as the
indexing process just restarts next time it is run). This does not
really "fix" the issue reported in [1]; that seems rather difficult
given how transactions work in Xapian. On the other hand, with the
default settings, this should mean one only loses less than a minutes
worth of work. The second issue is the occasionally reported "storm"
of disk writes when notmuch finishes. I don't yet have a test for
this, but I think committing as we go should reduce the amount of work
when finalizing the database.
David Bremner [Tue, 18 May 2021 11:03:22 +0000 (08:03 -0300)]
database/close: remove misleading code / comment
Unfortunately, it doesn't make a difference if we call
cancel_transaction or not, all uncommited changes are discarded if
there is an open (unflushed) transaction.
The minibuffer-prompt face that was used before made it impossible to
differentiate between two distinct UI elements: (i) the prompt's text
which itself cannot be acted upon, (ii) the actionable keys used to
jump to searches/tags.
The use of a named face, notmuch-jump-key, makes it possible for users
or theme developers to apply properties that are specific to each of
those two cases.
In the interest of backward compatibility, the new face inherits from
minibuffer-prompt.
emacs: Prefer email address over User ID when showing valid signature
Most concrete verification steps are likely only taken on the e-mail
address in the first place, and e-mail addresses render more
intelligibly than arbitrary User IDs in the first place.
When the certificate that signs a message is known to be valid, GMime
is capable of reporting on the e-mail address embedded in the
certificate.
We pass this information along to the caller of "notmuch show", as
often only the e-mail address of the certificate has actually been
checked/verified.
Furthermore, signature verification should probably at some point
compare the e-mail address of the caller against the sender address of
the message itself. Having to parse what gmime thinks is a "userid"
to extract an e-mail address seems clunky and unnecessary if gmime
already thinks it knows what the e-mail address is.
See id:878s41ax6t.fsf@fifthhorseman.net for more motivation and discussion.
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
David Bremner [Sat, 19 Jun 2021 00:59:43 +0000 (21:59 -0300)]
lib: write talloc report in notmuch_database_destroy
Since most memory allocation is (ultimately) in the talloc context
defined by a notmuch_database_t pointer, this gives a more complete
view of memory still allocated at program shutdown.
We also change the talloc report in notmuch.c to mode "a" to avoid
clobbering the newly reported log.
Ori [Thu, 24 Jun 2021 19:43:58 +0000 (15:43 -0400)]
emacs: Add compatibility for org-msg.el
Enable `notmuch-company` with `org-msg`, which does not derive from
`message-mode` in composing. This change was first proposed in the
discussion here:
https://github.com/jeremy-compostella/org-msg/issues/53
Tomi Ollila [Mon, 17 May 2021 08:11:09 +0000 (11:11 +0300)]
test: aggregate-results updates
notmuch-test will now call aggregate-results.sh with file list
that it compiles based on the test ran, and aggregate-results
will report failure is any of the test files are missing.
With this notmuch-test no longer has to exit in non-parallel
run if some test fail to write its report file -- so it works
as parallel tests in this sense.
Changed test_done() in test-lib.sh write report file in one write(2),
so there is (even) less chance it being partially written. Also,
now it writes 'total' last and aggregate-results.sh expects this
line to exist in all report files for reporting to be successful.
Added 'set -eu' to notmuch-test and modified code to work with
these settings. That makes it harder to get mistakes slipped
into committed code.
Jani Nikula [Fri, 21 May 2021 12:51:26 +0000 (15:51 +0300)]
emacs: update default mailing list archives for stashing links
Gmane web interface is long gone, remove it. Make MARC the new
default. Update LKML to Lore, where it already redirects anyway. Also
add Notmuch web archive.
Tomi Ollila [Sat, 16 Jan 2021 17:17:09 +0000 (19:17 +0200)]
emacs: fix declare-function definitions with related line breaks
- declare-function notmuch-unthreaded lacked file name
- declare-function notmuch-search had differently named last arg
- note: check-declare-directory did not complain about that
- declare-function notmuch-search-show-thread without nil
- some functions declared to be in different file than those
existed ("notmuch" -> "notmuch-lib")
- some related function/declare lines were (/are now) wider than
80-columns; added line breaks (and proper indentation) there
David Bremner [Fri, 4 Jun 2021 23:05:31 +0000 (20:05 -0300)]
emacs: drop setting mail-user-agent, and document how to set it
After some discussion [1], I decided it is better to make notmuch users
who rely on this behaviour customize mail-user-agent. This is
consistent with the behaviour of other emacs mail packages.
David Bremner [Sat, 24 Apr 2021 23:02:43 +0000 (20:02 -0300)]
test: add regression test for large number of deletions
Prior to 9ad19e4454f there was an unhandled Xapian exception when
reindexing after a large number of deletes. This test was used for
bisection, and will subsequently serve as a regression test.
David Bremner [Sat, 24 Apr 2021 17:46:18 +0000 (14:46 -0300)]
perf-test: add test for reindexing after removal/renaming msgs
At this point it is a bit tricky to measure the performance increase
from the new message deletion code, since the same commit (9ad19e4)
that improved the performance also seems to have fixed a bug with an
uncaught Xapian exception triggered by this test.
Tomi Ollila [Thu, 6 May 2021 14:12:32 +0000 (17:12 +0300)]
test: say_color() in one write(2)
say_color() used to call (builtin) printf (and tput(1) to stdout)
several times, which caused attempts to write messages with color
to have partial content (e.g. escape sequences) often intermixed
with other tests when parallel tests were run.
Now, with all output collected, then written out using one
printf, all strings with color print out correctly
((at least short) write(2)'s appear to write out "atomically").
While at it, used only one tput(1) execution to determine whether
color output works, and made bold/colors/sgr0 to tput(1) their
values once per test.
Jani Nikula [Fri, 21 May 2021 20:44:13 +0000 (23:44 +0300)]
doc: example command-line option reference
Example reference to a command-line option using the option role
reference. This creates a hyperlink in html, and the usual boldface
style in man page. This could be used throughout the man pages.
Jani Nikula [Fri, 21 May 2021 20:44:12 +0000 (23:44 +0300)]
doc: use program and option directives to document options
Use the program and option directives to document the subcommand
options. This unifies a lot of option documentation throughout.
This also makes it possible to reference options with :option:`--foo`
(within .. program::) or :option:`subcommand --foo` (globally). This
is left for later work.
See https://www.sphinx-doc.org/en/master/usage/restructuredtext/domains.html#directive-program
Note: There is a lot of indentation change, but intentionally there is
no reflow. Using 'git diff -w' or 'git show -w' to ignore white space
changes makes this a very easy change to review.
Jani Nikula [Fri, 21 May 2021 20:44:11 +0000 (23:44 +0300)]
doc: use envvar directive and role for environment variables
Make man1/notmuch.rst the single point of truth for describing notmuch
environment variables. Use the envvar directive for that, and
reference them with the envvar role.
Drive-by cleanup configuration file and hook directory search order
documentation.
Jani Nikula [Fri, 21 May 2021 20:44:10 +0000 (23:44 +0300)]
doc: cross-reference notmuch man pages with actual links
Add internal hyperlink targets for man pages and cross-reference them
using the any role reference. There are a number of alternatives to
accomplish this, but this seems like the combination that retains the
man page section number and the same boldface style in the man pages.
As a bonus, we get sanity checking on the links; for example
notmuch-search-terms.rst had a reference to notmuch-properties(1)
i.e. the wrong section.
The obvious semantic follow-up change would be to only have meaningful
"see also" references instead of having them all everywhere.
Tomi Ollila [Sat, 22 May 2021 13:39:57 +0000 (16:39 +0300)]
test: replace notmuch_passwd_sanitize() with _libconfig_sanitize()
notmuch_passwd_sanitize() in test-lib.sh is too generic, it cannot
work in many cases...
The more specific version _libconfig_sanitize() replaces it in
T590-libconfig.sh and the code that uses it is modified to output
the keys (ascending numbers printed in hex) so the sanitizer knows
what to sanitize in which lines...
"@" + fqdn -> "@FQDN" replacement is used as fqdn could
-- in theory -- be substring of 'USERNAME'.
'user -> 'USER_FULL_NAME replacement to work in cases where user
is empty -- as only first ' is replaced that works as expected.
In addition to ".(none)" now also ".localdomain" is filtered from
USERNAME@FQDN.
/dev/fd/{n} is not defined in posix, but it is portable enough
(if it weren't it is easy to fix -- now code looks clearer).
David Bremner [Sat, 15 May 2021 13:05:07 +0000 (10:05 -0300)]
test: add known broken test for duplicate thread-id terms
According to my bijection, this bug has been present since commit 411675a6ce in 2017. It is not completely clear what harm it causes in
regulary use, but it (at least) makes notmuch crash when compiled with
-DDEBUG_DATABASE_SANITY.
David Bremner [Sat, 8 May 2021 13:57:35 +0000 (10:57 -0300)]
perf-test/T00-new: fix backup creation.
"xargs tar cf backup.tar < $manifest" recreates the tar file with each
"batch" execed by xargs. In general this results in only a fraction of
the desired files being backed up.
Jani Nikula [Tue, 18 May 2021 21:40:44 +0000 (00:40 +0300)]
doc: fix man page build for Sphinx 4.x
Sphinx 4.0 changed the default value of man_make_section_directory
from False to True. We create the section directories and move the
files manually, so fix the immediate man build failure by disabling
the feature.
The Sphinx documentation on this [1] is confusing, and has the change
backwards. Git history says the default changed from False to True.
David Bremner [Sat, 8 May 2021 13:57:35 +0000 (10:57 -0300)]
perf-test/T00-new: fix backup creation.
"xargs tar cf backup.tar < $manifest" recreates the tar file with each
"batch" execed by xargs. In general this results in only a fraction of
the desired files being backed up.
Jani Nikula [Tue, 18 May 2021 21:40:44 +0000 (00:40 +0300)]
doc: fix man page build for Sphinx 4.x
Sphinx 4.0 changed the default value of man_make_section_directory
from False to True. We create the section directories and move the
files manually, so fix the immediate man build failure by disabling
the feature.
The Sphinx documentation on this [1] is confusing, and has the change
backwards. Git history says the default changed from False to True.
Felipe Contreras [Sat, 15 May 2021 20:47:43 +0000 (15:47 -0500)]
test: emacs: fix a couple of shellcheck complaints
In test-lib-emacs.sh line 20:
test_require_external_prereq ${TEST_EMACS} || ret=1
^-----------^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
test_require_external_prereq "${TEST_EMACS}" || ret=1
In test-lib-emacs.sh line 21:
test_require_external_prereq ${TEST_EMACSCLIENT} || ret=1
^-----------------^ SC2086: Double quote to prevent globbing and word splitting.
Did you mean:
test_require_external_prereq "${TEST_EMACSCLIENT}" || ret=1
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
Felipe Contreras [Sat, 15 May 2021 20:47:42 +0000 (15:47 -0500)]
test: emacs: check for configured emacs
Commit d59d9c81 (test: Make the emacsclient binary user-configurable,
2012-11-27) modified the prereq check for the configured emacsclient,
but we probably want to do the same for emacs itself.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
Felipe Contreras [Sat, 15 May 2021 21:21:03 +0000 (16:21 -0500)]
ruby: new notmuch_rb_object_destroy() helper
The struct used to store the types (rb_data_type_t) contains a "data"
field where we can store whatever we want. I use that field to store a
pointer to the corresponding destroy function. For example
notmuch_rb_database_type contains a pointer to notmuch_database_destroy.
I cast that pointer as a notmuch_status_t (func*)(void *) and call
that function passing the internal object (e.g. notmuch_database_t).
Using the rb_data_type_t data we can call the correct notmuch destroy
function.
Therefore this:
ret = ((notmuch_status_t (*)(void *)) type->data) (nm_object);
Is effectively the same as this:
ret = notmuch_database_destroy (database);
The advantage of doing it this way is that much less code is necesary
since each rb_data_type_t has the corresponding destroy function stored
in it.
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
Felipe Contreras [Sat, 15 May 2021 21:21:01 +0000 (16:21 -0500)]
ruby: move towards more modern RTypedData
Virtually the whole ruby core moved from RData to RTypeData, let's do so
ourselves too.
Basically the information typically passed through Data_Wrap_Struct is
now stored in a struct rb_data_type_t (mark and free functions). This
has the advantage that more information can be easily added, like the
name of the type, a custom data ponter, and more.
Data_Wrap_Struct is replaced with TypedData_Wrap_Struct, and the
information is stored in a struct rb_data_type_t, rather than passed
as arguments.
Check_Type is replaced with Check_TypedStruct, which is a wrapper for
rb_check_typeddata (with casts).
David Bremner [Mon, 10 May 2021 10:39:18 +0000 (07:39 -0300)]
lib/open: restore default database path of $HOME/mail
Although this default worked for "notmuch config get", it didn't work
most other places. Restore the previous functionality, with the
wrinkle that XDG locations will shadow $HOME/mail if they exist.
This fixes a bug reported by Jack Kamm in id:87eeefdc8b.fsf@gmail.com
Kyle Meyer [Sat, 8 May 2021 12:11:12 +0000 (09:11 -0300)]
emacs: restore tag-changes and query bindings for tag hooks
notmuch-before-tag-hook and notmuch-after-tag-hook are supposed to
have access to two dynamic variables, tag-changes and query, but these
were lost with the switch to lexical binding in fc4cda07 (emacs: use
lexical-bindings in all libraries, 2021-01-13).
Add a variant of Emacs's dlet (not available until Emacs 28) and use
it in notmuch-tag to expose tag-changes and query to the hooks.
David Bremner [Wed, 12 May 2021 23:22:18 +0000 (20:22 -0300)]
CLI: centralize initialization in notmuch_client_init
Initially make sure gmime_filter_reply initialization is
thread-safe (assuming notmuch_client_init is only called once).
For tidyness, also put talloc initialization in the new function.