]> git.cworth.org Git - notmuch/log
notmuch
9 months agodoc/lib: clarify ownership for notmuch_database_get_revision
David Bremner [Mon, 29 May 2023 11:01:40 +0000 (08:01 -0300)]
doc/lib: clarify ownership for notmuch_database_get_revision

The ownership is implicit in the const declaration (I think!), but
that does not show up in the doxygen generated API docs.

9 months agotest: add known broken test for message-id with embedded spaces.
David Bremner [Sat, 27 May 2023 17:20:51 +0000 (14:20 -0300)]
test: add known broken test for message-id with embedded spaces.

According to my reading of RFC5322, there is an obsolete syntax for
Message-Id which permits folding whitespace (i.e. to be removed /
ignored by parsers). In [1] Paul Wise observed that notmuch removed
whitespace on indexing, but does not do any corresponding
normalization of queries. Mark the latter as a bug by adding a failing
test.

[1]: id:20230409044143.4041560-1-pabs3@bonedaddy.net

9 months agopython: adjust legacy bindings to py 3.12
Michael J Gruber [Fri, 16 Jun 2023 11:19:26 +0000 (13:19 +0200)]
python: adjust legacy bindings to py 3.12

Py 3.12 finally pulled the plug on the `SafeConfigParser` class which
has been deprecated since py 3.2.

We use it in the legacy bindings only, so take the easy route of
importing `ConfigParser` as `SafeConfigParser` and monkey-patching so
that the class has the expected interface.

10 months agoruby: db: reorganize initializer
Felipe Contreras [Fri, 31 Mar 2023 20:40:50 +0000 (14:40 -0600)]
ruby: db: reorganize initializer

In order to make it more extensible.

No functional changes.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
10 months agonotmuch-mutt: check that the search cache Maildir is not a real Maildir
Paul Wise [Sun, 28 May 2023 02:29:46 +0000 (10:29 +0800)]
notmuch-mutt: check that the search cache Maildir is not a real Maildir

This prevents data loss when users configure the search cache Maildir to be a
real Maildir containing their real mail data, since the search cache Maildir
is expected to contain only symlinks to the real mail data.

Prevents: <ZCsQBNmbzwkvbpHA@localhost.localdomain>

10 months agonotmuch-mutt: do not clear search cache Maildir when nothing is found
Paul Wise [Sun, 28 May 2023 02:29:45 +0000 (10:29 +0800)]
notmuch-mutt: do not clear search cache Maildir when nothing is found

The previous results might be useful to the user but
an empty directory definitely isn't useful.

10 months agonotmuch-mutt: clarify the empty Maildir function operates on search caches
Paul Wise [Sun, 28 May 2023 02:29:44 +0000 (10:29 +0800)]
notmuch-mutt: clarify the empty Maildir function operates on search caches

Rename the function and adjust the documentation comment.

10 months agonotmuch-mutt: replace extra command with notmuch thread search feature
Paul Wise [Sun, 9 Apr 2023 04:41:43 +0000 (12:41 +0800)]
notmuch-mutt: replace extra command with notmuch thread search feature

This should be be slightly faster since it avoids forking a shell
and is less code in and less dependencies for the script.

Since String::ShellQuote isn't used elsewhere, drop mention of it.

10 months agonotmuch-mutt: fix Xapian query construction
Paul Wise [Sun, 9 Apr 2023 04:41:42 +0000 (12:41 +0800)]
notmuch-mutt: fix Xapian query construction

Spaces need to be stripped when querying the Message-Id,
because notmuch stores them in Xapian with spaces stripped.

All double-quote characters need to be doubled to escape them,
otherwise they will be added as extra query terms outside the id.

10 months agonotmuch-mutt: convert ISO-8859-15 copyright statement to UTF-8
Paul Wise [Tue, 23 May 2023 22:30:21 +0000 (06:30 +0800)]
notmuch-mutt: convert ISO-8859-15 copyright statement to UTF-8

Suggested-by: isutf8 & check-all-the-things
10 months agoruby: query: fix get sort
Felipe Contreras [Mon, 3 Apr 2023 20:27:21 +0000 (14:27 -0600)]
ruby: query: fix get sort

The order was wrong, right now `query.sort` doesn't return a number.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
11 months agovim: doc: minor spelling fix
John Gliksberg [Wed, 3 May 2023 22:12:20 +0000 (16:12 -0600)]
vim: doc: minor spelling fix

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
11 months agovim: doc: small fixes
Felipe Contreras [Wed, 3 May 2023 22:12:19 +0000 (16:12 -0600)]
vim: doc: small fixes

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
12 months agodoc: fix typos
Jakub Wilk [Thu, 13 Apr 2023 13:52:11 +0000 (15:52 +0200)]
doc: fix typos

12 months agoruby: remove FileNames object
Felipe Contreras [Mon, 27 Mar 2023 21:59:42 +0000 (15:59 -0600)]
ruby: remove FileNames object

Not used anymore now that we return an array of strings directly.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
12 months agoruby: filenames: return string array directly
Felipe Contreras [Mon, 27 Mar 2023 21:59:41 +0000 (15:59 -0600)]
ruby: filenames: return string array directly

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
12 months agoruby: add filenames helper
Felipe Contreras [Mon, 27 Mar 2023 21:59:40 +0000 (15:59 -0600)]
ruby: add filenames helper

Right now it doesn't do much, but it will help for further
reorganization.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
12 months agolib: index attachments with mime types matching index.as_text
David Bremner [Fri, 6 Jan 2023 00:02:06 +0000 (20:02 -0400)]
lib: index attachments with mime types matching index.as_text

Instead of skipping indexing all attachments, we check of a (user
configured) mime type that is indexable as text.

12 months agolib: parse index.as_text
David Bremner [Fri, 6 Jan 2023 00:02:05 +0000 (20:02 -0400)]
lib: parse index.as_text

We pre-parse into a list of compiled regular expressions to avoid
calling regexc on the hot (indexing) path.  As explained in the code
comment, this cannot be done lazily with reasonable error reporting,
at least not without touching a lot of the code in index.cc.

12 months agolib: add config key INDEX_AS_TEXT
David Bremner [Fri, 6 Jan 2023 00:02:04 +0000 (20:02 -0400)]
lib: add config key INDEX_AS_TEXT

Higher level processing as a list of regular expressions and
documentation will follow.

12 months agoruby: remove Tags object
Felipe Contreras [Wed, 22 Mar 2023 23:43:46 +0000 (17:43 -0600)]
ruby: remove Tags object

Not used anymore now that we return an array of strings directly.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
12 months agoruby: tags: return string array directly
Felipe Contreras [Wed, 22 Mar 2023 23:43:45 +0000 (17:43 -0600)]
ruby: tags: return string array directly

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
12 months agoruby: add tags helper
Felipe Contreras [Wed, 22 Mar 2023 23:43:44 +0000 (17:43 -0600)]
ruby: add tags helper

Right now it doesn't do much, but it will help for further
reorganization.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
12 months agolib: replace some uses of Query::MatchAll with a thread-safe alternative
Kevin Boulain [Thu, 2 Mar 2023 17:59:15 +0000 (18:59 +0100)]
lib: replace some uses of Query::MatchAll with a thread-safe alternative

This replaces two instances of Xapian::Query::MatchAll with the
equivalent but thread-safe alternative Xapian::Query(std::string()).
Xapian::Query::MatchAll maintains an internal pointer to a refcounted
Xapian::Internal::QueryTerm.

None of this is thread-safe but that wouldn't be an issue if
Xapian::Query::MatchAll wasn't static. Because it's static, the
refcounting goes awry when Notmuch is called from multiple threads.
This is actually documented by Xapian:
https://github.com/xapian/xapian/blob/4715de3a9fcee741587439dc3cc1d2ff01ffeaf2/xapian-core/include/xapian/query.h#L65

While static, Xapian::Query::MatchNothing is safe because it doesn't
maintain an internal object and as such, doesn't use references.

Two best-effort tests making use of TSan were added to showcase the
issue (I couldn't figure out a way to deterministically reproduce it
without making an unmaintainable mess).

First, when two databases are created in parallel, a query that uses
Xapian::Query::MatchAll is made (lib/query.cc), resulting in the
following backtrace on a segfault:
  #0  0x00007ffff76822af in Xapian::Query::get_terms_begin (this=0x7fffe80137f0) at api/query.cc:141
  #1  0x00007ffff7f933f5 in _notmuch_query_cache_terms (query=0x7fffe80137c0) at lib/query.cc:176
  #2  0x00007ffff7f93784 in _notmuch_query_ensure_parsed_xapian (query=0x7fffe80137c0) at lib/query.cc:225
  #3  0x00007ffff7f9381a in _notmuch_query_ensure_parsed (query=0x7fffe80137c0) at lib/query.cc:260
  #4  0x00007ffff7f93bfe in _notmuch_query_search_documents (query=0x7fffe80137c0, type=0x7ffff7fa9b1e "mail", out=0x7ffff666da18) at lib/query.cc:361
  #5  0x00007ffff7f93ba4 in notmuch_query_search_messages (query=0x7fffe80137c0, out=0x7ffff666da18) at lib/query.cc:349
  #6  0x00007ffff7f83d98 in notmuch_database_upgrade (notmuch=0x7fffe8000bd0, progress_notify=0x0, closure=0x0) at lib/database.cc:934
  #7  0x00007ffff7fa110f in notmuch_database_create_with_config (database_path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", config_path=0x7ffff7faab3c "", profile=0x0, database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:754
  #8  0x00007ffff7fa0d6f in notmuch_database_create_verbose (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0, status_string=0x7ffff666dc90) at lib/open.cc:653
  #9  0x00007ffff7fa0ceb in notmuch_database_create (path=0x7ffff666dcb0 "/tmp/notmuch.MZ2AGr", database=0x0) at lib/open.cc:637
  ...

Second, some queries would make use of Xapian::Query::MatchAll
(lib/regexp-fields.cc), resulting in the following backtrace on a
segfault:
  #0  0x00007f629828b690 in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800def0, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
  #1  0x00007f629828c260 in Xapian::Internal::QueryScaleWeight::gather_terms (this=0x7f628800df70, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1434
  #2  0x00007f629828b69f in Xapian::Internal::QueryBranch::gather_terms (this=0x7f628800dd90, void_terms=0x7f629726d5a0) at api/queryinternal.cc:1245
  #3  0x00007f6298282571 in Xapian::Query::get_unique_terms_begin (this=0x7f628800dcd8) at api/query.cc:166
  #4  0x00007f629841a59b in Xapian::Weight::Internal::accumulate_stats (this=0x7f628800dca0, subdb=..., rset=...) at weight/weightinternal.cc:86
  #5  0x00007f62983c15ba in LocalSubMatch::prepare_match (this=0x7f628800df20, nowait=true, total_stats=...) at matcher/localsubmatch.cc:172
  #6  0x00007f62983c8fcc in prepare_sub_matches (leaves=std::vector of length 1, capacity 1 = {...}, stats=...) at matcher/multimatch.cc:237
  #7  0x00007f62983c98a3 in MultiMatch::MultiMatch (this=0x7f629726d9a0, db_=..., query_=..., qlen=3, omrset=0x0, collapse_max_=0, collapse_key_=4294967295, percent_cutoff_=0, weight_cutoff_=0, order_=Xapian::Enquire::ASCENDING, sort_key_=0, sort_by_=Xapian::Enquire::Internal::VAL, sort_value_forward_=true, time_limit_=0, stats=..., weight_=0x7f6288008d50, matchspies_=std::vector of length 0, capacity 0, have_sorter=false, have_mdecider=false) at matcher/multimatch.cc:353
  #8  0x00007f629826fcba in Xapian::Enquire::Internal::get_mset (this=0x7f628800e0b0, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:569
  #9  0x00007f629827181c in Xapian::Enquire::get_mset (this=0x7f629726db80, first=0, maxitems=0, check_at_least=0, rset=0x0, mdecider=0x0) at api/omenquire.cc:937
  #10 0x00007f6298be529a in _notmuch_query_search_documents (query=0x7f6288009750, type=0x7f6298bfaafe "mail", out=0x7f629726dcc0) at lib/query.cc:447
  #11 0x00007f6298be4ae8 in notmuch_query_search_messages (query=0x7f6288009750, out=0x7f629726dcc0) at lib/query.cc:349
  ...

Printing Xapian::Query::MatchAll->internal.px->_refs in these
circumstances can help quickly identifying this scenario.

This is motivated by some test frameworks (like Rust's Cargo) that
runs unit tests in parallel and would easily encounter this issue,
unless client code gates every call to Notmuch behind a lock.

This is what can be expected from the tests when they fail:
   == stderr ==
  +==================
  +WARNING: ThreadSanitizer: data race (pid=207931)
  +  Read of size 1 at 0x7b10000001a0 by thread T2:
  +    #0 memcpy <null> (libtsan.so.2+0x62506)
  +    #1 void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) [clone .isra.0] <null> (libxapian.so.30+0x872b3)
  +
  +  Previous write of size 8 at 0x7b10000001a0 by thread T1:
  +    #0 operator new(unsigned long) <null> (libtsan.so.2+0x8ba83)
  +    #1 Xapian::Query::Query(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned int, unsigned int) <null> (libxapian.so.30+0x855cd)
  ...

12 months agoruby: database: make path arg optional
Felipe Contreras [Thu, 23 Mar 2023 02:05:23 +0000 (20:05 -0600)]
ruby: database: make path arg optional

It can be automatically loaded from the configuration now.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
12 months agoruby: use database_open_with_config
Felipe Contreras [Thu, 23 Mar 2023 02:05:22 +0000 (20:05 -0600)]
ruby: use database_open_with_config

Fixes warning:

  warning: ‘notmuch_database_open’ is deprecated: function deprecated as of libnotmuch 5.4

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
12 months agotest: add test for notmuch_message_remove_all_properties_with_prefix
Kevin Boulain [Wed, 29 Mar 2023 16:13:33 +0000 (18:13 +0200)]
test: add test for notmuch_message_remove_all_properties_with_prefix

It wasn't covered, though it shares most of its implementation with
notmuch_message_remove_all_properties.

12 months agolib/message-property: sync removed properties to the database
Kevin Boulain [Wed, 29 Mar 2023 16:13:32 +0000 (18:13 +0200)]
lib/message-property: sync removed properties to the database

_notmuch_message_remove_all_properties wasn't syncing the message back
to the database but was still invalidating the metadata, giving the
impression the properties had actually been removed.

Also move the metadata invalidation to _notmuch_message_remove_terms
to be closer to what's done in _notmuch_message_modify_property and
_notmuch_message_remove_term.

12 months agotest: reveal notmuch_message_remove_all_properties as broken
David Bremner [Thu, 30 Mar 2023 10:56:17 +0000 (07:56 -0300)]
test: reveal notmuch_message_remove_all_properties as broken

Close and re-open the database to show that the removal is not
committed to the database.

12 months agotest: reorganize tests and mark a few of them as broken
Kevin Boulain [Wed, 29 Mar 2023 16:13:31 +0000 (18:13 +0200)]
test: reorganize tests and mark a few of them as broken

notmuch_message_remove_all_properties should have removed the
  testkey1 = testvalue1
property but hasn't. Delay the execution of the corresponding test
to avoid updating a few tests that actually relied on the broken
behavior.

12 months agotest: remove unnecessary sorting
Kevin Boulain [Wed, 29 Mar 2023 16:13:30 +0000 (18:13 +0200)]
test: remove unnecessary sorting

The other tests rely on a stable output.

12 months agotest: display key name in property tests
Kevin Boulain [Wed, 29 Mar 2023 16:13:29 +0000 (18:13 +0200)]
test: display key name in property tests

To make the tests a bit easier to understand.

12 months agolib/message-property: catch xapian exceptions
Kevin Boulain [Wed, 29 Mar 2023 16:19:58 +0000 (18:19 +0200)]
lib/message-property: catch xapian exceptions

Since libnotmuch exposes a C interface there's no way for clients to
catch this.
Inspired by what's done for tags (see notmuch_message_remove_tag).

12 months agotest: uncaught exception when editing properties of a removed message
Kevin Boulain [Wed, 29 Mar 2023 16:19:57 +0000 (18:19 +0200)]
test: uncaught exception when editing properties of a removed message

These two functions don't fail gracefully when editing a removed
message:
 BROKEN edit property on removed message without uncaught exception
        --- T610-message-property.20.EXPECTED   2023-02-27 11:33:25.792764376 +0000
        +++ T610-message-property.20.OUTPUT     2023-02-27 11:33:25.793764381 +0000
        @@ -1,2 +1,3 @@
         == stdout ==
         == stderr ==
        +terminate called after throwing an instance of 'Xapian::DocNotFoundError'

The other functions appear to be safe.

13 months agolib/notmuch: update example
Kevin Boulain [Mon, 27 Feb 2023 12:06:29 +0000 (13:06 +0100)]
lib/notmuch: update example

Likely missed in 86cbd215e, when notmuch_query_search_messages_st was
renamed to notmuch_query_search_messages.

13 months agoemacs/tree: use two argument form of setq-local
David Bremner [Tue, 21 Feb 2023 11:49:15 +0000 (07:49 -0400)]
emacs/tree: use two argument form of setq-local

Apparently the macro setq-local only takes two arguments in Emacs 26.1

13 months agolib: add better diagnostics for over long filenames.
David Bremner [Tue, 27 Dec 2022 17:08:47 +0000 (13:08 -0400)]
lib: add better diagnostics for over long filenames.

Previously we just crashed with an internal error. With this change,
the caller can handle it better. Update notmuch-new so that it doesn't
crash with "unknown error code" because of this change.

13 months agotest: add known broken test for diagnostics from over long filenames.
David Bremner [Tue, 27 Dec 2022 17:08:46 +0000 (13:08 -0400)]
test: add known broken test for diagnostics from over long filenames.

Previously we tested over long directory names, add similar testing
for over long filenames.

13 months agoemacs: notmuch-tree-outline-mode
jao [Tue, 13 Dec 2022 02:15:42 +0000 (02:15 +0000)]
emacs: notmuch-tree-outline-mode

With this mode, one can fold trees in the notmuch-tree buffer as if
they were outlines, using all the commands provided by
outline-minor-mode.  We also define a couple of movement commands
that, optional, will ensure that only the thread around point is
unfolded.

The implementation is based on registering a :level property in the
messages p-list, that is then used by outline-minor-mode to to
recognise headers.

Amended by db: Copy docstring to manual and edit for presentation. Add
two tests. Fix typo "wether".

15 months agoconfigure: use pkg-config for gpgme
Daniel Kahn Gillmor [Wed, 2 Nov 2022 16:15:12 +0000 (12:15 -0400)]
configure: use pkg-config for gpgme

GnuPG upstream has supported pkg-config since gpgme version 1.13 and
gpg-error 1.33, and now prefers the use of pkg-config by default,
instead of relying on gpg-error-config and gpgme-config.

As of libgpg-error 1.46, upstream deliberately does not ship
gpg-error-config by default.  As of gpgme 1.18.0, upstream does not
ship gpgme-config if gpg-error-config is also not present.

Both of these versions of upstream libraries are in debian unstable
now.  To the extent that notmuch is dependent on GnuPG, it should
follow GnuPG upstream's lead.

Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
15 months agotest: mark some tests as broken when run as root.
David Bremner [Wed, 4 Jan 2023 12:41:42 +0000 (08:41 -0400)]
test: mark some tests as broken when run as root.

File permission errors e.g., are hard to trigger as root.

15 months agolib/message: move xapian call inside try/catch block in _n_m_delete
David Bremner [Sun, 4 Dec 2022 02:28:57 +0000 (22:28 -0400)]
lib/message: move xapian call inside try/catch block in _n_m_delete

The call to delete_document can throw exceptions (and can happen in
practice [1]), so catch the exception and extract the error
message. As a side effect, also move the call to _n_m_has_term inside
the try/catch. This should not change anything as that function
already traps any Xapian exceptions.

[1]: id:wwuk039sk2p.fsf@chaotikum.eu

15 months agotest: add known broken test for exception handling in _n_m_delete
David Bremner [Sun, 4 Dec 2022 02:28:56 +0000 (22:28 -0400)]
test: add known broken test for exception handling in _n_m_delete

In [1], Thomas Schneider reported an uncaught Xapian exception when
running out of disk space. We generate the same exception via database
corruption.

[1]: id:wwuk039sk2p.fsf@chaotikum.eu

15 months agolib/database: propagate status code from _notmuch_message_delete
David Bremner [Sun, 4 Dec 2022 02:28:55 +0000 (22:28 -0400)]
lib/database: propagate status code from _notmuch_message_delete

_notmuch_message_delete can return (at least)
NOTMUCH_STATUS_XAPIAN_EXCEPTION, which we should not ignore.

15 months agoCLI/git: use --exclude=false when calling notmuch-search
David Bremner [Sun, 16 Oct 2022 12:49:00 +0000 (09:49 -0300)]
CLI/git: use --exclude=false when calling notmuch-search

We use notmuch search in two places in notmuch-git.py: to find which
tags have a given prefix, and to see if message with given id exists
locally. In both cases we do not want the presence of exclude tags
(e.g. deleted) to change the results.

16 months agodoc: mark `--output=summary` as default
Michael J Gruber [Wed, 30 Nov 2022 13:58:34 +0000 (14:58 +0100)]
doc: mark `--output=summary` as default

`notmuch search` behaves differently depending on the output option: It
either outputs information pertaining to all threads with matching
messages (summary, threads) or to all matching messages (messages,
files, tags). The man page refres solely to the former in the main
description.

Help the user by clearly marking `summary` as the default output option.

Signed-off-by: Michael J Gruber <git@grubix.eu>
17 months agoemacs/show: use n-s-i-headerline to update tags
David Bremner [Fri, 11 Nov 2022 21:48:31 +0000 (16:48 -0500)]
emacs/show: use n-s-i-headerline to update tags

Although this has more steps than the previous regular expression
search and replace, it should be more robust against changes in the
headerline format, such as the inclusion of duplicate numbers (which
broke the previous version).

17 months agoemacs/show: add optional orig-tags argument to n-s-i-headerline
David Bremner [Fri, 11 Nov 2022 21:48:30 +0000 (16:48 -0500)]
emacs/show: add optional orig-tags argument to n-s-i-headerline

This will support use of this function in notmuch-show-update-tags.

17 months agoemacs/show: use plist to pass message info to n-s-insert-headerline
David Bremner [Fri, 11 Nov 2022 21:48:29 +0000 (16:48 -0500)]
emacs/show: use plist to pass message info to n-s-insert-headerline

This should allow calling notmuch-show-insert-headerline from other
places without duplicating the set of plist accesses.

17 months agoemacs: move From header addition to after header intern
Russell Sim [Tue, 20 Sep 2022 20:18:03 +0000 (22:18 +0200)]
emacs: move From header addition to after header intern

OTHER-HEADERS are expected to be passed as strings, to match the
implementation of `compose-mail'.  But the "From" header is currently
expected to be passed as a symbol.  Instead the "From" header can be
safely added after converting all the headers to symbols.

17 months agocli: add options --offset and --limit to notmuch show
Robin Jarry [Tue, 18 Oct 2022 19:41:58 +0000 (21:41 +0200)]
cli: add options --offset and --limit to notmuch show

notmuch search does not output header values. However, when browsing
through a large email corpus, it can be time saving to be able to
paginate without running notmuch show for each message/thread.

Add --offset and --limit options to notmuch show. This is inspired from
commit 796b629c3b82 ("cli: add options --offset and --limit to notmuch
search").

Update man page, shell completion and add a test case to ensure it works
as expected.

Cc: Tim Culverhouse <tim@timculverhouse.com>
Cc: Tomi Ollila <tomi.ollila@iki.fi>
Signed-off-by: Robin Jarry <robin@jarry.cc>
17 months agoemacs: fix dangling overlays in notmuch-search
Matt Armstrong [Thu, 13 Oct 2022 03:20:38 +0000 (20:20 -0700)]
emacs: fix dangling overlays in notmuch-search

notmuch-search-insert-authors now sets the evaporate property on the
ellipsis overlays.  Emacs will delete them when the buffer contents
are zeroed out, which happens with `notmuch-refresh-buffer`.  This
prevents them from being collapsed to zero-width overlays in position
1.  See Emacs bug#58479.  An upcoming change in Emacs will make these
dangling overlays visible to the user.

18 months agonmweb: Fix spacing around punctuation in headers
Jakub Wilk [Tue, 11 Oct 2022 08:19:44 +0000 (10:19 +0200)]
nmweb: Fix spacing around punctuation in headers

18 months agoemacs: add notmuch-search-edit-search and notmuch-tree-edit-search
Tomi Ollila [Sun, 18 Sep 2022 13:05:50 +0000 (16:05 +0300)]
emacs: add notmuch-search-edit-search and notmuch-tree-edit-search

...and bind these to "E" in their respective keymaps.

Expected to be called interactively, then using read-from-minibuffer
with current search string as initial contents for editing.
(Noninteractive use makes little sense, but is supported.)

With this one can expand (as an opposite to limit) their
query and have e.g. (some of their) saved searches as search
"templates".

While at it, removed `(defvar notmuch-search-query-string)` from
notmuch-tree.el; it is unused (`notmuch-tree-basic-query` is used
instead).

Thanks to Jose Antonio Ortega Ruiz for his example for notmuch-tree
code, and better interactive use.

18 months agonmweb: escape subject in search view
David Bremner [Mon, 5 Sep 2022 11:03:39 +0000 (08:03 -0300)]
nmweb: escape subject in search view

Fix a bug reported by Jakub Wilk [1].

[1]: id:20220822064717.qftn4tr7cs4r2ian@jwilk.net

18 months agotest: replace aging OpenPGP key used in the test suite
Justus Winter [Thu, 22 Sep 2022 08:46:06 +0000 (10:46 +0200)]
test: replace aging OpenPGP key used in the test suite

This replaces the old OpenPGPv4 key that is used in the test suite
with a more modern OpenPGPv4 key.  All cryptographic artifacts in the
test suite are updated accordingly.

Having old cryptographic artifacts in the test suite presents a
problem once the old algorithms are rejected by contemporary
implementations.

For reference, this is the old key.

  sec   rsa1024 2011-02-05 [SC]
        5AEAB11F5E33DCE875DDB75B6D92612D94E46381
  uid           [ unknown] Notmuch Test Suite <test_suite@notmuchmail.org> (INSECURE!)
  ssb   rsa1024 2011-02-05 [E]

And this is the new key.  Note that is has the same shape, but uses
Ed25519 and Cv25519 instead of 1024-bit RSA.

  sec   ed25519 2022-09-07 [SC]
        9A3AFE6C60065A148FD4B58A7E6ABE924645CC60
  uid           [ultimate] Notmuch Test Suite (INSECURE!) <test_suite@notmuchmail.org>
  ssb   cv25519 2022-09-07 [E]

18 months agotest: compute expected keyid from fingerprint
Justus Winter [Fri, 9 Sep 2022 16:12:50 +0000 (18:12 +0200)]
test: compute expected keyid from fingerprint

19 months agotest: add known broken test for indexing text/* attachments
David Bremner [Sat, 20 Aug 2022 18:50:06 +0000 (11:50 -0700)]
test: add known broken test for indexing text/* attachments

The general problem of indexing attachments requires some help to turn
things into text, but (most?) text/* should be doable internally,
possibly with optimizations as for the text/html case.

19 months agotest: rename indexing corpus
David Bremner [Sat, 20 Aug 2022 18:50:05 +0000 (11:50 -0700)]
test: rename indexing corpus

The corpus is not really suitable for general indexing test since the
sole message is ignored (and will most likely continue to be ignored)
by notmuch-new.

19 months agolib: add field processor for lastmod: prefix
David Bremner [Sun, 14 Aug 2022 15:02:59 +0000 (12:02 -0300)]
lib: add field processor for lastmod: prefix

By sharing the existing logic used by the sexp query parser, this
allows negative lastmod revisions to be interpreted as relative to the
most recent revision.

19 months agolib: factor out lastmod range handling from sexp parser.
David Bremner [Sun, 14 Aug 2022 15:02:58 +0000 (12:02 -0300)]
lib: factor out lastmod range handling from sexp parser.

This will permit the re-use of the same logic in the infix query
parser. The location of the shared code in the infix side is for
consistency with the other shared parsing logic. It will make more
sense when a Xapian field processor is added for the lastmod prefix.

19 months agolib/sexp: provide relative lastmod queries
David Bremner [Sun, 14 Aug 2022 15:02:57 +0000 (12:02 -0300)]
lib/sexp: provide relative lastmod queries

Test the relatively trivial logic changes for the sexp query parser
first before refactoring that logic to share with the infix query
parser.

19 months agolib/open: create database path in some cases
David Bremner [Fri, 29 Jul 2022 11:31:37 +0000 (08:31 -0300)]
lib/open: create database path in some cases

There is some duplication of code here, but not all of the locations
valid to find a database make sense to create. Furthermore we nead two
passes, so the control flow in _choose_database_path would get a bit
convoluted.

19 months agolib/open: return non-SUCCESS on missing database path
David Bremner [Fri, 29 Jul 2022 11:23:11 +0000 (08:23 -0300)]
lib/open: return non-SUCCESS on missing database path

This simplifies the logic of creating the directory path when it doesn't
exist.

19 months agolib/open: refactor call to mkdir into function
David Bremner [Thu, 28 Jul 2022 10:41:02 +0000 (06:41 -0400)]
lib/open: refactor call to mkdir into function

This makes the error handling available for re-use. Using
g_mkdir_with_parents also handles the case of a pre-existing
directory. This introduces new functionality, namely creating the
parent directories, which will be useful for creating directories like
'.local/share/notmuch/default'.

19 months agotest/path-config: set database.mail_root but not database.path
David Bremner [Tue, 26 Jul 2022 01:04:21 +0000 (21:04 -0400)]
test/path-config: set database.mail_root but not database.path

The failing "create database" test replicates a bug reported by Sean
Whitton [1]. The other two failures also look related to the database
being (re)created in the wrong place.

[1]: id:87y1wqkw13.fsf@athena.silentflame.com.

19 months agotest: add test for creating database in various configurations.
David Bremner [Tue, 26 Jul 2022 00:37:41 +0000 (20:37 -0400)]
test: add test for creating database in various configurations.

The existing database creation (via add_email_corpus) was always done
in the traditional configuration. The use of xapian-metadata is just
to portably ensure that there is a database created where we expect
there to be.

19 months agoemacs/show: use read-shell-command instead of read-string
Antoine Beaupré [Tue, 30 Aug 2022 14:17:45 +0000 (10:17 -0400)]
emacs/show: use read-shell-command instead of read-string

This enables auto-completion of commands, something which plain
read-string does not do. It's otherwise a drop-in
replacement. According to `C-h f`, read-shell-command was introduced
in Emacs 23.1 or earlier.

19 months agoemacs: new notmuch-tree-process-exit-functions
jao [Sun, 21 Aug 2022 03:06:18 +0000 (04:06 +0100)]
emacs: new notmuch-tree-process-exit-functions

Hook run when the tree insertion process finishes its job.

--
This patch supersedes <id:20220816214023.1523322-1-jao@gnu.org>, but
changing the new variable name.

Right now, it can be used for silly things like removing or changing
the the "End of search." hardcoded message in the tree buffer.  But
also for more sophisticated things like folding all threads in add-ons
like my outline mode for tree buffers (to be submitted).

Signed-off-by: jao <jao@gnu.org>
19 months agodebian: update changelog for new build-depends archive/debian/0.37-1 debian/0.37-1
David Bremner [Wed, 24 Aug 2022 16:12:57 +0000 (09:12 -0700)]
debian: update changelog for new build-depends

19 months agodebian: build-depend on emacs-el
David Bremner [Wed, 24 Aug 2022 16:10:02 +0000 (09:10 -0700)]
debian: build-depend on emacs-el

This is hopefully temporary, to work around some bugs related to emacs
28 in Debian.

19 months agoNEWS: set release date for 0.37 0.37
David Bremner [Sun, 21 Aug 2022 20:17:37 +0000 (13:17 -0700)]
NEWS: set release date for 0.37

19 months agodebian: changelog for 0.37-1
David Bremner [Sun, 21 Aug 2022 20:09:25 +0000 (13:09 -0700)]
debian: changelog for 0.37-1

19 months agoversion: update to 0.37
David Bremner [Sun, 21 Aug 2022 20:05:34 +0000 (13:05 -0700)]
version: update to 0.37

19 months agoNEWS: minimal entries for two emacs changes.
David Bremner [Sun, 21 Aug 2022 20:02:08 +0000 (13:02 -0700)]
NEWS: minimal entries for two emacs changes.

20 months agotest: increase cffi timeout
Michael J Gruber [Sun, 14 Aug 2022 15:22:17 +0000 (17:22 +0200)]
test: increase cffi timeout

By default, the test suite uses 2min for other tests and 5s for cffi
tests. Sporadically, this leads to test failures caused by the timeout
on slower or loaded test infrastructure (as seen on ppc64le in Fedora's
infrastructure during branch time).

Increase the cffi timeout to the same 2m=120s.

Signed-off-by: Michael J Gruber <git@grubix.eu>
20 months agochangelog for 0.37~rc0-3 archive/debian/0.37_rc0-3 debian/0.37_rc0-3
David Bremner [Sun, 14 Aug 2022 14:49:54 +0000 (11:49 -0300)]
changelog for 0.37~rc0-3

20 months agodebian: changelog for 0.37~rc0-2 archive/debian/0.37_rc0-2 debian/0.37_rc0-2
David Bremner [Sun, 14 Aug 2022 13:56:15 +0000 (10:56 -0300)]
debian: changelog for 0.37~rc0-2

20 months agodebian: upload release candidate to experimental, not unstable archive/debian/0.37_rc0-1 debian/0.37_rc0-1
David Bremner [Sun, 14 Aug 2022 10:28:40 +0000 (07:28 -0300)]
debian: upload release candidate to experimental, not unstable

Oops, tagged "upstream" version is incorrect, but it's a minor issue.

20 months agodebian: changelog stanza for 0.37~rc0-1 0.37_rc0
David Bremner [Sun, 14 Aug 2022 09:55:40 +0000 (06:55 -0300)]
debian: changelog stanza for 0.37~rc0-1

20 months agoversion: update to 0.37~rc0
David Bremner [Sun, 14 Aug 2022 09:53:41 +0000 (06:53 -0300)]
version: update to 0.37~rc0

20 months agodoc/emacs: use define-key instead of hard-coding markup
David Bremner [Sun, 31 Jul 2022 11:33:23 +0000 (08:33 -0300)]
doc/emacs: use define-key instead of hard-coding markup

Add keys to index. Allow indexing of both keys and commands. Use
standard typesetting for multiple keybindings.

20 months agodoc/emacs: replace use of emacsvar with el:defcustom
David Bremner [Sun, 31 Jul 2022 11:33:22 +0000 (08:33 -0300)]
doc/emacs: replace use of emacsvar with el:defcustom

Mainly this allows dropping the ad-hoc directive emacsvar

20 months agodoc/emacs: replace emacscmd with el:define-key
David Bremner [Sun, 31 Jul 2022 11:33:21 +0000 (08:33 -0300)]
doc/emacs: replace emacscmd with el:define-key

This allows us to eliminate the ad-hoc emacscmd directive / role, and
improve indexing.

doc/emacs: convert remaining uses of emacscmd to define-key

This allows us to remove the no-longer used object type.

20 months agodoc/emacs: split index category from label
David Bremner [Sun, 31 Jul 2022 11:33:20 +0000 (08:33 -0300)]
doc/emacs: split index category from label

A more verbose category is needed for the index, but not the
description in the manual.

20 months agodoc/emacs: drop domain label
David Bremner [Sun, 31 Jul 2022 11:33:19 +0000 (08:33 -0300)]
doc/emacs: drop domain label

This makes the generate info files somewhat less verbose, so that they
say e.g. "Key Binding" instead of "Emacs Key Binding"

20 months agodoc/elisp: replace label for Emacs commands and key bindings.
David Bremner [Sun, 31 Jul 2022 11:33:18 +0000 (08:33 -0300)]
doc/elisp: replace label for Emacs commands and key bindings.

In the notmuch doc index it is important to be clear which parts are
Emacs related.

20 months agodoc: copy elisp.py from flycheck, make available
David Bremner [Sun, 31 Jul 2022 11:33:17 +0000 (08:33 -0300)]
doc: copy elisp.py from flycheck, make available

This corresponds to flycheck commit
1702d2db3d8ba9bcb9b2bf810e791e907fcc3adc, which is apparently the last
time this file was modified (2018).

Embedding seems defensible since the file is evidently not changing
that much, and we need to change some labels to make it fit with the
not-just-emacs nature of notmuch docs.

20 months agodoc: add some documentation about duplicate files.
David Bremner [Sat, 30 Jul 2022 23:45:51 +0000 (20:45 -0300)]
doc: add some documentation about duplicate files.

This is mainly motivated by the new emacs-UI features to select
duplicates, but hopefully it will help demystify things for users of
the CLI as well.

20 months agoemacs: add docstring for notmuch-show-choose-duplicate
David Bremner [Sun, 31 Jul 2022 12:46:51 +0000 (09:46 -0300)]
emacs: add docstring for notmuch-show-choose-duplicate

It should have one in any case, but in particular it is needed for the
manual.

20 months agoemacs/show: restrict inlined mimetypes on refresh.
David Bremner [Mon, 1 Aug 2022 11:39:18 +0000 (08:39 -0300)]
emacs/show: restrict inlined mimetypes on refresh.

This fixes the bug reported by Al [1]. Essentially apply the same fix
as [2] in a different place.

[1]: id:877d41nmr1.fsf@gmail.com
[2]: 90a7c1af368a527700dcde9b0dcbd760afc7bd92

20 months agotest/emacs: add known broken test for inlining on refresh.
David Bremner [Mon, 1 Aug 2022 11:39:17 +0000 (08:39 -0300)]
test/emacs: add known broken test for inlining on refresh.

This duplicates the bug reported by Al [1]

[1]: id:877d41nmr1.fsf@gmail.com

20 months agoemacs: add duplicate support to notmuch-show-view-raw-message
David Bremner [Thu, 4 Aug 2022 11:48:08 +0000 (08:48 -0300)]
emacs: add duplicate support to notmuch-show-view-raw-message

This fixes one of the bugs reported by Gregor in [1]

[1]: id:87edxw8jp4.fsf@no.workgroup

20 months agoemacs/show: provide a more friendly function to get duplicate num
David Bremner [Thu, 4 Aug 2022 11:48:07 +0000 (08:48 -0300)]
emacs/show: provide a more friendly function to get duplicate num

There is not much code here, but at least we can fix the indexing as 1
based.

20 months agotest/emacs: add known broken test for viewing raw duplicate
David Bremner [Thu, 4 Aug 2022 11:48:06 +0000 (08:48 -0300)]
test/emacs: add known broken test for viewing raw duplicate

This duplicates one of the bugs reported by Gregor in [1].

[1]: id:87edxw8jp4.fsf@no.workgroup

20 months agoemacs: update notmuch-show-get-filename to handle duplicates
David Bremner [Thu, 4 Aug 2022 11:08:33 +0000 (08:08 -0300)]
emacs: update notmuch-show-get-filename to handle duplicates

This fixes one of the bugs reported by Gregor in [1].

[1]: id:87edxw8jp4.fsf@no.workgroup

20 months agotest/emacs: add known broken test for stashing duplicate filename
David Bremner [Thu, 4 Aug 2022 11:08:32 +0000 (08:08 -0300)]
test/emacs: add known broken test for stashing duplicate filename

This duplicates (no pun intended) one of the bugs reported by Gregor
in id:87edxw8jp4.fsf@no.workgroup

20 months agoNEWS: initial set of NEWS for 0.37.
David Bremner [Mon, 1 Aug 2022 13:14:15 +0000 (10:14 -0300)]
NEWS: initial set of NEWS for 0.37.

There are one or two more changes to apply (and potentially document)
before release, but this covers most of my commits.

20 months agodoc/notmuch: merge two sections on external commands.
David Bremner [Sat, 30 Jul 2022 23:45:49 +0000 (20:45 -0300)]
doc/notmuch: merge two sections on external commands.

When adding the description of the propagation of NOTMUCH_CONFIG, I
missed that there was already a section on external commands, with a
different title.

20 months agotests: remove dead code from T055-path-config.sh
David Bremner [Fri, 29 Jul 2022 12:19:56 +0000 (09:19 -0300)]
tests: remove dead code from T055-path-config.sh

This case statement does nothing.