David Bremner [Fri, 16 Apr 2021 12:00:38 +0000 (09:00 -0300)]
perf-tests: add test for removing files.
Without the proposed fix for deletion performance [1], this test is
probably unreasonably slow compared to others at the same
small/medium/large scale.
David Bremner [Sat, 13 Mar 2021 12:43:37 +0000 (08:43 -0400)]
test: put shim at end of LD_PRELOAD path
Certain tools like the address-sanitizer fail if they are not the
first LD_PRELOADed library. It does not seem to matter for our shims,
as long as they are loaded before libnotmuch.
David Bremner [Sat, 27 Feb 2021 14:11:23 +0000 (10:11 -0400)]
CLI: rename notmuch_config_t to notmuch_conffile_t
The name is a compromise between keeping function names from getting
too long, and avoiding confusion with the newer notmuch_config library
API. Try to make clear that the residual functionality is only about
updating configuration files.
David Bremner [Thu, 25 Feb 2021 12:00:26 +0000 (08:00 -0400)]
CLI/config: support set/get with split configuration
There are two small code changes. The main one is to retrieve the
possibly updated config file name found during the database opening
call. The second change is to allow empty config file names, as
a (currently broken) way of specifying that configuration should only
be taken from the database.
David Bremner [Sun, 21 Feb 2021 11:34:52 +0000 (07:34 -0400)]
lib: provide notmuch_config_path
Since the library searches in several locations for a config file, the
caller does not know which of these is chosen in the usual case of
passing NULL as a config file. This changes provides an API for the
caller to retrieve the name of the config file chosen. It will be
tested in a following commit.
David Bremner [Tue, 23 Feb 2021 20:19:13 +0000 (16:19 -0400)]
CLI+lib: detect missing database in split configurations.
Eventually we want to do all opening of databases in the top
level (main function). This means that detection of missing databases
needs to move out of subcommands. It also requires updating the
library to use the new NO_DATABASE status code.
David Bremner [Fri, 19 Feb 2021 12:41:36 +0000 (08:41 -0400)]
CLI/config: default to storing all config in external files
Previously the fact that some configuration options were only stored
in the database (and thus editing the config file had no effect) was a
source of user confusion. This was fixed with the series ending at d9af0af1646.
On the other hand, the underlying partition of config options into
those stored by default in the database and those stored in the config
file remained. This is also confusing, since now some invocations of
"notmuch config set" modify the config file, and others silently
modify the database instead.
With this commit, it is up to the user to decide which configuration
to modify. A new "--database" option is provided for notmuch config to
enable modifying the configuration information in the database;
otherwise the default is to update an external config file.
David Bremner [Tue, 23 Feb 2021 13:22:37 +0000 (09:22 -0400)]
CLI/notmuch: switch notmuch_command to notmuch_config_get
The goal at this point is to remove the dependence on
notmuch_config_get_* without breaking any existing functionality. This
is a step towards removing notmuch_config_get_* in a future commit.
David Bremner [Sat, 13 Feb 2021 17:49:32 +0000 (13:49 -0400)]
CLI/setup: switch to new configuration framework
Most of the changes are the elimination of notmuch_config_t accessor
use. We also migrate some of the diagnostics to the top level where we
are opening the files in question.
David Bremner [Sat, 13 Feb 2021 13:01:50 +0000 (09:01 -0400)]
CLI/config: use merged config for "config get"
This commit starts the conversion of notmuch-config.c
functionality (as opposed to just interface) to the new config
framework.
The change to T030-config is because of the move of the
canonicalization database paths from the notmuch_config_t accessor to
the internal function _choose_database_path.
David Bremner [Wed, 3 Mar 2021 12:31:44 +0000 (08:31 -0400)]
lib/open: canonicalize relative path read from config file
This matches functionality in the the CLI function
notmuch_config_get_database_path, which was previously used in the CLI
code for all calls to open a database.
David Bremner [Sun, 14 Feb 2021 11:42:58 +0000 (07:42 -0400)]
lib/config: add config_pairs iterators
The layer of shims here seems a bit wasteful compared to just calling
the corresponding string map functions directly, but it allows control
over the API (calling with notmuch_database_t *) and flexibility for
future changes.
David Bremner [Sat, 27 Mar 2021 12:12:28 +0000 (09:12 -0300)]
lib/open: fix leaks calling _trial_open
_trial_open can't know if the PATH_ERROR return value will cause the
error message to be returned from the library, so it's up the caller
to clean up if not.
David Bremner [Wed, 17 Feb 2021 15:26:15 +0000 (11:26 -0400)]
CLI/new: use configuration variable for backup directory
The stat is essentially replaced by the mkdir for error detection
purposes. This changes the default location for backups to make
things tidier, even in non-split configurations. Hopefully there is
not too many user scripts relying on the previous location.
Because the default location may not exist, replace the use of stat
for error detection with a call to mkdir.
David Bremner [Wed, 17 Feb 2021 15:04:54 +0000 (11:04 -0400)]
lib/config: add configuration variable for backup directory
Like the hook directory, we primarily need a way to communicate this
directory between various components, but we may as well let the user
configure it.
Most of the diff is generalizing choose_hook_dir to work for both
backup and hook directories.
David Bremner [Sat, 9 Jan 2021 11:29:07 +0000 (07:29 -0400)]
CLI/insert: support split database and mail root
The new test is in T055-path-config because it uses the helper
function split_config, and because it seems easier to put the
database path related tests in one place.
David Bremner [Mon, 4 Jan 2021 00:04:46 +0000 (20:04 -0400)]
lib/open: support XDG_DATA_HOME as a fallback database location.
This changes some error reporting, either intentionally by reporting
the highest level missing directory, or by side effect from looking in
XDG locations when given null database location.
David Bremner [Sat, 9 Jan 2021 11:25:25 +0000 (07:25 -0400)]
CLI/new: support split database and mail location
This adds new state variable for the mail root, and uses it most
places db_path was used. The notable exception is dumps from
backups. The latter will be dealt with properly in a future commit.
David Bremner [Wed, 17 Feb 2021 00:08:23 +0000 (20:08 -0400)]
lib/open: check for split configuration when creating database.
The main functionality will be tested when notmuch-new is converted to
support split configuration. Here only the somewhat odd case of split
mail root which is actually symlinked to the database path is tested.
David Bremner [Thu, 31 Dec 2020 22:20:31 +0000 (18:20 -0400)]
lib: support splitting mail from database location.
Introduce a new configuration value for the mail root, and use it to
locate mail messages in preference to the database.path (which
previously implied the mail messages were also in this location.
Initially only a subset of the CLI is tested in a split
configuration. Further changes will be needed for the remainder of the
CLI to work in split configurations.
David Bremner [Sat, 30 Jan 2021 17:57:32 +0000 (13:57 -0400)]
lib/open: factor out the second half of n_d_open_with_config
The idea is to allow reuse in n_d_create_with_config. This is
primarily code movement, with some changes in error messages to reduce
the number of input parameters.
David Bremner [Fri, 29 Jan 2021 11:44:18 +0000 (07:44 -0400)]
lib/open: factor out library initialization
This is slightly more tidy, but more importantly it allows for re-use
of this code in n_d_create_with_config. That re-use will be crucial
when we no longer call n_d_open_with_config from
n_d_create_with_config.
David Bremner [Tue, 2 Feb 2021 20:43:58 +0000 (16:43 -0400)]
lib: remove "path" from notmuch struct
This removes duplication between the struct element and the
configuration string_map entry. Create a simple wrapper for setting
the database path that makes sure the trailing / is stripped.
David Bremner [Sat, 6 Feb 2021 22:50:20 +0000 (18:50 -0400)]
CLI/show: complete conversion to new configuration framework.
In order to open the database in main() for this command, we may need
to re-open it in the (possibly less common) case where crypto options
require write access.
David Bremner [Fri, 19 Mar 2021 01:58:02 +0000 (22:58 -0300)]
CLI/new: drop the write lock to run the pre-new hook.
This fixes a bug reported in [1]. In principle it could be possible
avoid one of these reopens, but it complicates the logic in main with
respect to creating new databases.
David Bremner [Fri, 19 Mar 2021 01:54:13 +0000 (22:54 -0300)]
test: Add tests for write access to database from hooks.
Recent changes to configuration handling meant the pre-new hook was
run while the database was open read only, limiting what could be done
in the hook. Add some known broken tests for this problem, as well as
a regression test for write access from the post-new hook.
David Bremner [Sun, 14 Mar 2021 14:01:58 +0000 (11:01 -0300)]
lib: save path of xapian database in notmuch struct.
This will allow re-opening in a different mode (read/write
vs. read-only) with current Xapian API. It will also prove useful when
updating the compact functions to support more flexible database
location.
David Bremner [Sun, 14 Mar 2021 14:01:57 +0000 (11:01 -0300)]
lib: publish API for notmuch_database_reopen
Include the (currently unused) mode argument which will specify which
mode to re-open the database in. Functionality and docs to be
finalized in a followup commit.
Tomi Ollila [Thu, 25 Feb 2021 20:04:30 +0000 (22:04 +0200)]
test: T020-compact.sh: fix work directory location
Fix use of $TEST_DIRECTORY ($NOTMUCH_BUILDDIR/test/) with use of
$TMP_DIRECTORY ($NOTMUCH_BUILDDIR/test/tmp.T020-compact/ in case
of T020-compact.sh) as root directory where to write test files
and directories.
David Bremner [Mon, 15 Feb 2021 00:55:55 +0000 (20:55 -0400)]
test: Fix race condition in T568-lib-thread.sh
The assignment of thread-ids is (apparently) non-deterministic in a
way that mostly seems to show up on multicore machines. In my tests
the number is different from that previously assumed by this test
about 15% of the time on a 50 thread (25 core) Xeon.
Since message id's are fixed, use a message known to be in the thread
of interest to pick out the correct thread-id.
David Bremner [Mon, 15 Feb 2021 20:43:50 +0000 (16:43 -0400)]
fix build failure with glib 2.67
Based on a patch from Michael J Gruber [1]. As of glib 2.67 (more
specifically [2]), including "gmime-extra.h" inside an extern "C"
block causes build failures, because glib is using C++ features.
Observing that "gmime-extra.h" is no longer needed in
notmuch-private.h, which can simply delete that include, but
we have to correspondingly move the includes which might include
it (in particular crypto.h) out of the extern "C" block also.
This seems less fragile than only moving gmime-extra, and relying on
preprocessor sentinels to keep the deeper includes from happening.
Move to the include to the outside of the extern block.
This is the last bit of "python" left in the notmuch codebase.
https://www.python.org/dev/peps/pep-0394/#recommendation encourages
"third-party distributors" to use more-specific shebang lines. I'm
not certain that the notmuch project itself is a "third-party
contributor" but I think this is a safe way to encourage people to use
python3 when they're developing notmuch.
We already have python3 explicitly elsewhere in the codebase for
developers (in nmbug).
Signed-off-by: Daniel Kahn Gillmor <dkg@fifthhorseman.net>
David Bremner [Sun, 10 Jan 2021 15:54:35 +0000 (11:54 -0400)]
doc: describe new config framework
Remove STORED IN DATABASE discussion, describe search rules.
Currently profiles seem a bit pointless. They will make more sense
when they apply to databases as well.
David Bremner [Sun, 10 Jan 2021 11:50:14 +0000 (07:50 -0400)]
CLI: use configured hook directory
This enables support for hooks outside the database directory.
It relies strongly on configuration information being usable between
closing the database and destroying it.
David Bremner [Sat, 9 Jan 2021 13:19:50 +0000 (09:19 -0400)]
lib/config: add HOOK_DIR
The hook directory configuration needs to be kept in synch with the
other configuration information, so add scaffolding to support this at
database opening time.
David Bremner [Sat, 2 Jan 2021 17:10:39 +0000 (13:10 -0400)]
cli/new: convert to new config framework
In addition to the same type of changes as converting other
subcommands, add the possibility of creating a database at the top
level. It would probably make sense to use this for insert as well.
David Bremner [Sun, 3 Jan 2021 12:30:46 +0000 (08:30 -0400)]
lib: add NOTMUCH_STATUS_NO_CONFIG
This will allow client code to provide more meaningful diagnostics. In
particular it will enable "notmuch new" to continue suggsting the user
run "notmuch setup" to create a config after "notmuch new" is
transitioned to the new configuration framework.
David Bremner [Sat, 8 Aug 2020 14:16:53 +0000 (11:16 -0300)]
CLI/show: mostly switch show to new config framework
This will need some cleanup when the transition completes, and we stop
passing notmuch_config_t structs to the subcommands.
Unlike the general case, we open the database in the subcommand, since
we don't know whether it should be opened read/write until we parse
the command line arguments.
Add a test to make sure passing config file on the command line is not
broken by these or future config related changes.
David Bremner [Sat, 8 Aug 2020 14:16:51 +0000 (11:16 -0300)]
cli/config: add accessor for config file name
This is intended for use in temporary code transitioning to the new
configuration system. The name is chosen to avoid cluttering the
notmuch_config_* namespace further with non-library functions.