Library Changes
---------------
-Work around Xapian bug with `get_mset(0,0, x)`.
+Work around Xapian bug with `get_mset(0,0, x)`
This causes aborts in `_notmuch_query_count_documents` on
e.g. Fedora 28. The underlying bug is fixed in Xapian commit
Transition `notmuch_database_add_message` to `notmuch_database_index_file`
- When indexing an e-mail message, the new
- `notmuch_database_index_file` function is the preferred form, and
- the old `notmuch_database_add_message` is deprecated. The new form
- allows passing a set of options to the indexing engine, which the
- operator may decide to change from message to message.
+ When indexing an e-mail message, the new
+ `notmuch_database_index_file` function is the preferred form, and
+ the old `notmuch_database_add_message` is deprecated. The new form
+ allows passing a set of options to the indexing engine, which the
+ operator may decide to change from message to message.
Test Suite
----------
-#!/usr/bin/python
+#!/usr/bin/env python
#
# Copyright (c) 2011-2012 David Bremner <david@tethera.net>
#
# If you want to number the parts, i suggest piping the output through
# something like "cat -n"
+from __future__ import print_function
+
import email
import sys
-def test(z, prefix=''):
+def print_part(z, prefix):
fname = '' if z.get_filename() is None else ' [' + z.get_filename() + ']'
cset = '' if z.get_charset() is None else ' (' + z.get_charset() + ')'
disp = z.get_params(None, header='Content-Disposition')
for d in disp:
if d[0] in [ 'attachment', 'inline' ]:
disposition = ' ' + d[0]
+ if z.is_multipart():
+ nbytes = len(z.as_string())
+ else:
+ nbytes = len(z.get_payload())
+
+ print('{}{}{}{}{} {:d} bytes'.format(
+ prefix,
+ z.get_content_type(),
+ cset,
+ disposition,
+ fname,
+ nbytes,
+ ))
+
+def test(z, prefix=''):
if (z.is_multipart()):
- print prefix + '┬╴' + z.get_content_type() + cset + disposition + fname, z.as_string().__len__().__str__() + ' bytes'
+ print_part(z, prefix+'┬╴')
if prefix.endswith('└'):
prefix = prefix.rpartition('└')[0] + ' '
if prefix.endswith('├'):
test(parts[i], prefix + '└')
# FIXME: show epilogue?
else:
- print prefix + '─╴'+ z.get_content_type() + cset + disposition + fname, z.get_payload().__len__().__str__(), 'bytes'
+ print_part(z, prefix+'─╴')
test(email.message_from_file(sys.stdin), '└')
``--output=(sender|recipients|count|address)``
Controls which information appears in the output. This option can
be given multiple times to combine different outputs. When
- neither --output=sender nor --output=recipients is
- given, --output=sender is implied.
+ neither ``--output=sender`` nor ``--output=recipients`` is
+ given, ``--output=sender`` is implied.
**sender**
Output all addresses from the *From* header.
**no**
Output all occurrences of addresses in the matching
- messages. This is not applicable with --output=count.
+ messages. This is not applicable with ``--output=count``.
**mailbox**
Deduplicate addresses based on the full, case sensitive name
and email address, or mailbox. This is effectively the same as
- piping the --deduplicate=no output to **sort | uniq**, except
+ piping the ``--deduplicate=no`` output to **sort | uniq**, except
for the order of results. This is the default.
**address**
Deduplicate addresses based on the case insensitive address
part of the mailbox. Of all the variants (with different name
or case), print the one occurring most frequently among the
- matching messages. If --output=count is specified, include all
+ matching messages. If ``--output=count`` is specified, include all
variants in the count.
``--sort=``\ (**newest-first**\ \|\ **oldest-first**)
By default, results will be displayed in reverse chronological
order, (that is, the newest results will be displayed first).
- However, if either --output=count or --deduplicate=address is
+ However, if either ``--output=count`` or ``--deduplicate=address`` is
specified, this option is ignored and the order of the results is
unspecified.
See **notmuch-search-terms(7)** for details of the supported syntax
for <search-terms>. With no search terms, a dump of all messages in
-the database will be generated. A "--" argument instructs notmuch that
+the database will be generated. A ``--`` argument instructs notmuch that
the remaining arguments are search terms.
Supported options for **dump** include
If ``true``, decrypt any MIME encrypted parts found in the
selected content (i.e., "multipart/encrypted" parts). Status
of the decryption will be reported (currently only supported
- with --format=json and --format=sexp), and on successful
+ with ``--format=json`` and ``--format=sexp``), and on successful
decryption the multipart/encrypted part will be replaced by
the decrypted content.
**threads**
Output the thread IDs of all threads with any message matching
- the search terms, either one per line (--format=text),
- separated by null characters (--format=text0), as a JSON array
- (--format=json), or an S-Expression list (--format=sexp).
+ the search terms, either one per line (``--format=text``),
+ separated by null characters (``--format=text0``), as a JSON array
+ (``--format=json``), or an S-Expression list (``--format=sexp``).
**messages**
Output the message IDs of all messages matching the search
- terms, either one per line (--format=text), separated by null
- characters (--format=text0), as a JSON array (--format=json),
- or as an S-Expression list (--format=sexp).
+ terms, either one per line (``--format=text``), separated by null
+ characters (``--format=text0``), as a JSON array (``--format=json``),
+ or as an S-Expression list (``--format=sexp``).
**files**
Output the filenames of all messages matching the search
- terms, either one per line (--format=text), separated by null
- characters (--format=text0), as a JSON array (--format=json),
- or as an S-Expression list (--format=sexp).
+ terms, either one per line (``--format=text``), separated by null
+ characters (``--format=text0``), as a JSON array (``--format=json``),
+ or as an S-Expression list (``--format=sexp``).
Note that each message may have multiple filenames associated
with it. All of them are included in the output (unless
- limited with the --duplicate=N option). This may be
+ limited with the ``--duplicate=N`` option). This may be
particularly confusing for **folder:** or **path:** searches
in a specified directory, as the messages may have duplicates
in other directories that are included in the output, although
**tags**
Output all tags that appear on any message matching the search
- terms, either one per line (--format=text), separated by null
- characters (--format=text0), as a JSON array (--format=json),
- or as an S-Expression list (--format=sexp).
+ terms, either one per line (``--format=text``), separated by null
+ characters (``--format=text0``), as a JSON array (``--format=json``),
+ or as an S-Expression list (``--format=sexp``).
``--sort=``\ (**newest-first**\ \|\ **oldest-first**)
This option can be used to present results in either chronological
http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/mail-mbox-formats.html
- **raw** (default if --part is given)
+ **raw** (default if ``--part`` is given)
Write the raw bytes of the given MIME part of a message to
standard out. For this format, it is an error to specify a
query that matches more than one message.
``--verify``
Compute and report the validity of any MIME cryptographic
- signatures found in the selected content (ie. "multipart/signed"
+ signatures found in the selected content (e.g., "multipart/signed"
parts). Status of the signature will be reported (currently only
- supported with --format=json and --format=sexp), and the
+ supported with ``--format=json`` and ``--format=sexp``), and the
multipart/signed part will be replaced by the signed data.
``--decrypt=(false|auto|true|stash)``
If ``true``, decrypt any MIME encrypted parts found in the
- selected content (i.e. "multipart/encrypted" parts). Status of
+ selected content (e.g., "multipart/encrypted" parts). Status of
the decryption will be reported (currently only supported
- with --format=json and --format=sexp) and on successful
+ with ``--format=json`` and ``--format=sexp``) and on successful
decryption the multipart/encrypted part will be replaced by
the decrypted content.
excluded message will be marked with the exclude flag (except when
output=mbox when there is nowhere to put the flag).
- If --entire-thread is specified then complete threads are returned
+ If ``--entire-thread`` is specified then complete threads are returned
regardless (with the excluded flag being set when appropriate) but
threads that only match in an excluded message are not returned
when ``--exclude=true.``
``--include-html``
Include "text/html" parts as part of the output (currently only
- supported with --format=json and --format=sexp). By default,
+ supported with ``--format=json`` and ``--format=sexp``). By default,
unless ``--part=N`` is used to select a specific part or
``--include-html`` is used to include all "text/html" parts, no
part with content type "text/html" is included in the output.
**notmuch** **count** [option ...] <*search-term*> ...
-**notmuch** **dump** [--format=(batch-tag|sup)] [--] [--output=<*file*>] [--] [<*search-term*> ...]
+**notmuch** **dump** [--gzip] [--format=(batch-tag|sup)] [--output=<*file*>] [--] [<*search-term*> ...]
**notmuch** **reindex** [option ...] <*search-term*> ...
The **lastmod:** prefix can be used to restrict the result by the
database revision number of when messages were last modified (tags
were added/removed or filenames changed). This is usually used in
- conjunction with the **--uuid** argument to **notmuch search** to
+ conjunction with the ``--uuid`` argument to **notmuch search** to
find messages that have changed since an earlier query.
query:<name>
as that will interfere with the handling of stderr and the exit
status."
- (let (err-file err-buffer proc
+ (let (err-file err-buffer proc err-proc
;; Find notmuch using Emacs' `exec-path'
(command (or (executable-find notmuch-command)
(error "Command not found: %s" notmuch-command))))
:buffer buffer
:command (cons command args)
:connection-type 'pipe
- :stderr err-buffer))
+ :stderr err-buffer)
+ err-proc (get-buffer-process err-buffer))
(process-put proc 'err-buffer err-buffer)
- ;; Silence "Process NAME stderr finished" in stderr by adding a
- ;; no-op sentinel to the fake stderr process object
- (set-process-sentinel (get-buffer-process err-buffer) #'ignore))
+
+ (process-put err-proc 'err-file err-file)
+ (process-put err-proc 'err-buffer err-buffer)
+ (set-process-sentinel err-proc #'notmuch-start-notmuch-error-sentinel))
;; On Emacs versions before 25, there is no way to capture
;; stdout and stderr separately for asynchronous processes, or
;; Emacs behaves strangely if an error escapes from a sentinel,
;; so turn errors into messages.
(message "%s" (error-message-string err))))
- (when err-buffer (kill-buffer err-buffer))
(when err-file (ignore-errors (delete-file err-file)))))
+(defun notmuch-start-notmuch-error-sentinel (proc event)
+ (let* ((err-file (process-get proc 'err-file))
+ ;; When `make-process' is available, use the error buffer
+ ;; associated with the process, otherwise the error file.
+ (err-buffer (or (process-get proc 'err-buffer)
+ (find-file-noselect err-file))))
+ (when err-buffer (kill-buffer err-buffer))))
+
;; This variable is used only buffer local, but it needs to be
;; declared globally first to avoid compiler warnings.
(defvar notmuch-show-process-crypto nil)
const char **thread_id)
{
GHashTable *parents = NULL;
- const char *refs, *in_reply_to, *in_reply_to_message_id;
+ const char *refs, *in_reply_to, *in_reply_to_message_id, *strict_message_id = NULL;
const char *last_ref_message_id, *this_message_id;
GList *l, *keys = NULL;
notmuch_status_t ret = NOTMUCH_STATUS_SUCCESS;
parents, refs);
in_reply_to = _notmuch_message_file_get_header (message_file, "in-reply-to");
+ if (in_reply_to)
+ strict_message_id = _notmuch_message_id_parse_strict (message,
+ in_reply_to);
+
in_reply_to_message_id = parse_references (message,
this_message_id,
parents, in_reply_to);
- /* For the parent of this message, use the last message ID of the
- * References header, if available. If not, fall back to the
- * first message ID in the In-Reply-To header. */
- if (last_ref_message_id) {
+ /* For the parent of this message, use
+ * 1) the In-Reply-To header, if it looks sane, otherwise
+ * 2) the last message ID of the References header, if available.
+ * 3) Otherwise, fall back to the first message ID in
+ * the In-Reply-To header.
+ */
+
+ if (strict_message_id) {
+ _notmuch_message_add_term (message, "replyto", strict_message_id);
+ } else if (last_ref_message_id) {
_notmuch_message_add_term (message, "replyto",
last_ref_message_id);
} else if (in_reply_to_message_id) {
#include "notmuch-private.h"
+#include "string-util.h"
/* Advance 'str' past any whitespace or RFC 822 comments. A comment is
* a (potentially nested) parenthesized sequence with '\' used to
return result;
}
+
+char *
+_notmuch_message_id_parse_strict (void *ctx, const char *message_id)
+{
+ const char *s, *end;
+
+ if (message_id == NULL || *message_id == '\0')
+ return NULL;
+
+ s = skip_space (message_id);
+ if (*s == '<')
+ s++;
+ else
+ return NULL;
+
+ for (end = s; *end && *end != '>'; end++)
+ if (isspace (*end))
+ return NULL;
+
+ if (*end != '>')
+ return NULL;
+ else {
+ const char *last = skip_space (end + 1);
+ if (*last != '\0')
+ return NULL;
+ }
+
+ return talloc_strndup (ctx, s, end - s);
+}
int frozen;
char *message_id;
char *thread_id;
+ size_t thread_depth;
char *in_reply_to;
notmuch_string_list_t *tag_list;
notmuch_string_list_t *filename_term_list;
notmuch_message_file_t *message_file;
notmuch_string_list_t *property_term_list;
notmuch_string_map_t *property_map;
+ notmuch_string_list_t *reference_list;
notmuch_message_list_t *replies;
unsigned long flags;
/* For flags that are initialized on-demand, lazy_flags indicates
/* the message is initially not synchronized with Xapian */
message->last_view = 0;
+ /* Calculated after the thread structure is computed */
+ message->thread_depth = 0;
+
/* Each of these will be lazily created as needed. */
message->message_id = NULL;
message->thread_id = NULL;
message->author = NULL;
message->property_term_list = NULL;
message->property_map = NULL;
+ message->reference_list = NULL;
message->replies = _notmuch_message_list_create (message);
if (unlikely (message->replies == NULL)) {
*type_prefix = _find_prefix ("type"),
*filename_prefix = _find_prefix ("file-direntry"),
*property_prefix = _find_prefix ("property"),
+ *reference_prefix = _find_prefix ("reference"),
*replyto_prefix = _find_prefix ("replyto");
/* We do this all in a single pass because Xapian decompresses the
_notmuch_database_get_terms_with_prefix (message, i, end,
property_prefix);
+ /* get references */
+ assert (strcmp (property_prefix, reference_prefix) < 0);
+ if (!message->reference_list) {
+ message->reference_list =
+ _notmuch_database_get_terms_with_prefix (message, i, end,
+ reference_prefix);
+ }
+
/* Get reply to */
assert (strcmp (property_prefix, replyto_prefix) < 0);
if (!message->in_reply_to)
_notmuch_message_list_add_message (message->replies, reply);
}
+size_t
+_notmuch_message_get_thread_depth (notmuch_message_t *message) {
+ return message->thread_depth;
+}
+
+void
+_notmuch_message_label_depths (notmuch_message_t *message,
+ size_t depth)
+{
+ message->thread_depth = depth;
+
+ for (notmuch_messages_t *messages = _notmuch_messages_create (message->replies);
+ notmuch_messages_valid (messages);
+ notmuch_messages_move_to_next (messages)) {
+ notmuch_message_t *child = notmuch_messages_get (messages);
+ _notmuch_message_label_depths (child, depth+1);
+ }
+}
+
+const notmuch_string_list_t *
+_notmuch_message_get_references (notmuch_message_t *message)
+{
+ _notmuch_message_ensure_metadata (message, message->reference_list);
+ return message->reference_list;
+}
+
+static int
+_cmpmsg (const void *pa, const void *pb)
+{
+ notmuch_message_t **a = (notmuch_message_t **) pa;
+ notmuch_message_t **b = (notmuch_message_t **) pb;
+ time_t time_a = notmuch_message_get_date (*a);
+ time_t time_b = notmuch_message_get_date (*b);
+
+ if (time_a == time_b)
+ return 0;
+ else if (time_a < time_b)
+ return -1;
+ else
+ return 1;
+}
+
+notmuch_message_list_t *
+_notmuch_message_sort_subtrees (void *ctx, notmuch_message_list_t *list)
+{
+
+ size_t count = 0;
+ size_t capacity = 16;
+
+ if (! list)
+ return list;
+
+ void *local = talloc_new (NULL);
+ notmuch_message_list_t *new_list = _notmuch_message_list_create (ctx);
+ notmuch_message_t **message_array = talloc_zero_array (local, notmuch_message_t *, capacity);
+
+ for (notmuch_messages_t *messages = _notmuch_messages_create (list);
+ notmuch_messages_valid (messages);
+ notmuch_messages_move_to_next (messages)) {
+ notmuch_message_t *root = notmuch_messages_get (messages);
+ if (count >= capacity) {
+ capacity *= 2;
+ message_array = talloc_realloc (local, message_array, notmuch_message_t *, capacity);
+ }
+ message_array[count++] = root;
+ root->replies = _notmuch_message_sort_subtrees (root, root->replies);
+ }
+
+ qsort (message_array, count, sizeof (notmuch_message_t *), _cmpmsg);
+ for (size_t i = 0; i < count; i++) {
+ _notmuch_message_list_add_message (new_list, message_array[i]);
+ }
+
+ talloc_free (local);
+ talloc_free (list);
+ return new_list;
+}
+
notmuch_messages_t *
notmuch_message_get_replies (notmuch_message_t *message)
{
list->tail = &node->next;
}
+bool
+_notmuch_message_list_empty (notmuch_message_list_t *list)
+{
+ if (list == NULL)
+ return TRUE;
+
+ return (list->head == NULL);
+}
+
notmuch_messages_t *
_notmuch_messages_create (notmuch_message_list_t *list)
{
return (messages->iterator != NULL);
}
+bool
+_notmuch_messages_has_next (notmuch_messages_t *messages)
+{
+ if (! notmuch_messages_valid (messages))
+ return false;
+
+ if (! messages->is_of_list_type)
+ INTERNAL_ERROR("_notmuch_messages_has_next not implimented for msets");
+
+ return (messages->iterator->next != NULL);
+}
+
notmuch_message_t *
notmuch_messages_get (notmuch_messages_t *messages)
{
#ifdef DEBUG
# define DEBUG_DATABASE_SANITY 1
+# define DEBUG_THREADING 1
# define DEBUG_QUERY 1
#endif
notmuch_message_list_t *
_notmuch_message_list_create (const void *ctx);
+bool
+_notmuch_message_list_empty (notmuch_message_list_t *list);
+
void
_notmuch_message_list_add_message (notmuch_message_list_t *list,
notmuch_message_t *message);
notmuch_messages_t *
_notmuch_messages_create (notmuch_message_list_t *list);
+bool
+_notmuch_messages_has_next (notmuch_messages_t *messages);
+
/* query.cc */
bool
char *
_notmuch_message_id_parse (void *ctx, const char *message_id, const char **next);
+/* Parse a message-id, discarding leading and trailing whitespace, and
+ * '<' and '>' delimiters.
+ *
+ * Apply a probably-stricter-than RFC definition of what is allowed in
+ * a message-id. In particular, forbid whitespace.
+ *
+ * Returns a newly talloc'ed string belonging to 'ctx'.
+ *
+ * Returns NULL if there is any error parsing the message-id.
+ */
+
+char *
+_notmuch_message_id_parse_strict (void *ctx, const char *message_id);
+
/* message.cc */
const char *
_notmuch_message_get_thread_id_only(notmuch_message_t *message);
+size_t _notmuch_message_get_thread_depth (notmuch_message_t *message);
+
+void
+_notmuch_message_label_depths (notmuch_message_t *message,
+ size_t depth);
+
+notmuch_message_list_t *
+_notmuch_message_sort_subtrees (void *ctx, notmuch_message_list_t *list);
+
/* sha1.c */
char *
void
_notmuch_string_list_sort (notmuch_string_list_t *list);
+const notmuch_string_list_t *
+_notmuch_message_get_references(notmuch_message_t *message);
+
/* string-map.c */
typedef struct _notmuch_string_map notmuch_string_map_t;
typedef struct _notmuch_string_map_iterator notmuch_string_map_iterator_t;
#include <gmime/gmime.h>
#include <glib.h> /* GHashTable */
+#ifdef DEBUG_THREADING
+#define THREAD_DEBUG(format, ...) fprintf(stderr, format " (%s).\n", ##__VA_ARGS__, __location__)
+#else
+#define THREAD_DEBUG(format, ...) do {} while (0) /* ignored */
+#endif
+
#define EMPTY_STRING(s) ((s)[0] == '\0')
struct _notmuch_thread {
_thread_add_matched_author (thread, _notmuch_message_get_author (hashed_message));
}
+static bool
+_parent_via_in_reply_to (notmuch_thread_t *thread, notmuch_message_t *message) {
+ notmuch_message_t *parent;
+ const char *in_reply_to;
+
+ in_reply_to = _notmuch_message_get_in_reply_to (message);
+ THREAD_DEBUG("checking message = %s in_reply_to=%s\n",
+ notmuch_message_get_message_id (message), in_reply_to);
+
+ if (in_reply_to && (! EMPTY_STRING(in_reply_to)) &&
+ g_hash_table_lookup_extended (thread->message_hash,
+ in_reply_to, NULL,
+ (void **) &parent)) {
+ _notmuch_message_add_reply (parent, message);
+ return true;
+ } else {
+ return false;
+ }
+}
+
+static void
+_parent_or_toplevel (notmuch_thread_t *thread, notmuch_message_t *message)
+{
+ size_t max_depth = 0;
+ notmuch_message_t *new_parent;
+ notmuch_message_t *parent = NULL;
+ const notmuch_string_list_t *references =
+ _notmuch_message_get_references (message);
+
+ THREAD_DEBUG("trying to reparent via references: %s\n",
+ notmuch_message_get_message_id (message));
+
+ for (notmuch_string_node_t *ref_node = references->head;
+ ref_node; ref_node = ref_node->next) {
+ THREAD_DEBUG("checking reference=%s\n", ref_node->string);
+ if ((g_hash_table_lookup_extended (thread->message_hash,
+ ref_node->string, NULL,
+ (void **) &new_parent))) {
+ size_t new_depth = _notmuch_message_get_thread_depth (new_parent);
+ THREAD_DEBUG("got depth %lu\n", new_depth);
+ if (new_depth > max_depth || !parent) {
+ THREAD_DEBUG("adding at depth %lu parent=%s\n", new_depth, ref_node->string);
+ max_depth = new_depth;
+ parent = new_parent;
+ }
+ }
+ }
+ if (parent) {
+ THREAD_DEBUG("adding reply %s to parent=%s\n",
+ notmuch_message_get_message_id (message),
+ notmuch_message_get_message_id (parent));
+ _notmuch_message_add_reply (parent, message);
+ } else {
+ THREAD_DEBUG("adding as toplevel %s\n",
+ notmuch_message_get_message_id (message));
+ _notmuch_message_list_add_message (thread->toplevel_list, message);
+ }
+}
+
static void
_resolve_thread_relationships (notmuch_thread_t *thread)
{
notmuch_message_node_t *node, *first_node;
- notmuch_message_t *message, *parent;
- const char *in_reply_to;
+ notmuch_message_t *message;
+ void *local;
+ notmuch_message_list_t *maybe_toplevel_list;
first_node = thread->message_list->head;
if (! first_node)
return;
+ local = talloc_new (thread);
+ maybe_toplevel_list = _notmuch_message_list_create (local);
+
for (node = first_node->next; node; node = node->next) {
message = node->message;
- in_reply_to = _notmuch_message_get_in_reply_to (message);
- if (in_reply_to && strlen (in_reply_to) &&
- g_hash_table_lookup_extended (thread->message_hash,
- in_reply_to, NULL,
- (void **) &parent))
- _notmuch_message_add_reply (parent, message);
- else
- _notmuch_message_list_add_message (thread->toplevel_list, message);
+ if (! _parent_via_in_reply_to (thread, message))
+ _notmuch_message_list_add_message (maybe_toplevel_list, message);
}
/*
*/
if (first_node) {
message = first_node->message;
- in_reply_to = _notmuch_message_get_in_reply_to (message);
- if (thread->toplevel_list->head &&
- in_reply_to && strlen (in_reply_to) &&
- g_hash_table_lookup_extended (thread->message_hash,
- in_reply_to, NULL,
- (void **) &parent))
- _notmuch_message_add_reply (parent, message);
+ THREAD_DEBUG("checking first message %s\n",
+ notmuch_message_get_message_id (message));
+
+ if (_notmuch_message_list_empty (maybe_toplevel_list) ||
+ ! _parent_via_in_reply_to (thread, message)) {
+
+ THREAD_DEBUG("adding first message as toplevel = %s\n",
+ notmuch_message_get_message_id (message));
+ _notmuch_message_list_add_message (maybe_toplevel_list, message);
+ }
+ }
+
+ for (notmuch_messages_t *messages = _notmuch_messages_create (maybe_toplevel_list);
+ notmuch_messages_valid (messages);
+ notmuch_messages_move_to_next (messages))
+ {
+ notmuch_message_t *message = notmuch_messages_get (messages);
+ _notmuch_message_label_depths (message, 0);
+ }
+
+ for (notmuch_messages_t *roots = _notmuch_messages_create (maybe_toplevel_list);
+ notmuch_messages_valid (roots);
+ notmuch_messages_move_to_next (roots)) {
+ notmuch_message_t *message = notmuch_messages_get (roots);
+ if (_notmuch_messages_has_next (roots) || ! _notmuch_message_list_empty (thread->toplevel_list))
+ _parent_or_toplevel (thread, message);
else
_notmuch_message_list_add_message (thread->toplevel_list, message);
}
- /* XXX: After scanning through the entire list looking for parents
- * via "In-Reply-To", we should do a second pass that looks at the
- * list of messages IDs in the "References" header instead. (And
- * for this the parent would be the "deepest" message of all the
- * messages found in the "References" list.)
- *
- * Doing this will allow messages and sub-threads to be positioned
- * correctly in the thread even when an intermediate message is
- * missing from the thread.
+ /* XXX this could be made conditional on messages being inserted
+ * (out of order) in later passes
*/
+ thread->toplevel_list = _notmuch_message_sort_subtrees (thread, thread->toplevel_list);
+
+ talloc_free (local);
}
/* Create a new notmuch_thread_t object by finding the thread
const char *
notmuch_config_get_database_path (notmuch_config_t *config)
{
- return _config_get (config, &config->database_path, "database", "path");
+ char *db_path = (char *)_config_get (config, &config->database_path, "database", "path");
+
+ if (db_path && *db_path != '/') {
+ /* If the path in the configuration file begins with any
+ * character other than /, presume that it is relative to
+ * $HOME and update as appropriate.
+ */
+ char *abs_path = talloc_asprintf (config, "%s/%s", getenv ("HOME"), db_path);
+ talloc_free (db_path);
+ db_path = config->database_path = abs_path;
+ }
+
+ return db_path;
}
void
GMimeContentType *content_type = g_mime_object_get_content_type (GMIME_OBJECT (part));
GMimeStream *stream_filter = NULL;
GMimeFilter *crlf_filter = NULL;
+ GMimeFilter *windows_filter = NULL;
GMimeDataWrapper *wrapper;
const char *charset;
if (stream_out == NULL)
return;
+ charset = g_mime_object_get_content_type_parameter (part, "charset");
+ charset = charset ? g_mime_charset_canon_name (charset) : NULL;
+ wrapper = g_mime_part_get_content_object (GMIME_PART (part));
+ if (wrapper && charset && !g_ascii_strncasecmp (charset, "iso-8859-", 9)) {
+ GMimeStream *null_stream = NULL;
+ GMimeStream *null_stream_filter = NULL;
+
+ /* Check for mislabeled Windows encoding */
+ null_stream = g_mime_stream_null_new ();
+ null_stream_filter = g_mime_stream_filter_new (null_stream);
+ windows_filter = g_mime_filter_windows_new (charset);
+ g_mime_stream_filter_add(GMIME_STREAM_FILTER (null_stream_filter),
+ windows_filter);
+ g_mime_data_wrapper_write_to_stream (wrapper, null_stream_filter);
+ charset = g_mime_filter_windows_real_charset(
+ (GMimeFilterWindows *) windows_filter);
+
+ if (null_stream_filter)
+ g_object_unref (null_stream_filter);
+ if (null_stream)
+ g_object_unref (null_stream);
+ /* Keep a reference to windows_filter in order to prevent the
+ * charset string from deallocation. */
+ }
+
stream_filter = g_mime_stream_filter_new (stream_out);
crlf_filter = g_mime_filter_crlf_new (false, false);
g_mime_stream_filter_add(GMIME_STREAM_FILTER (stream_filter),
crlf_filter);
g_object_unref (crlf_filter);
- charset = g_mime_object_get_content_type_parameter (part, "charset");
if (charset) {
GMimeFilter *charset_filter;
charset_filter = g_mime_filter_charset_new (charset, "UTF-8");
}
}
- wrapper = g_mime_part_get_content_object (GMIME_PART (part));
if (wrapper && stream_filter)
g_mime_data_wrapper_write_to_stream (wrapper, stream_filter);
if (stream_filter)
g_object_unref(stream_filter);
+ if (windows_filter)
+ g_object_unref (windows_filter);
}
static const char*
$(dir)/arg-test: $(dir)/arg-test.o command-line-arguments.o util/libnotmuch_util.a
$(call quiet,CC) $^ -o $@ $(LDFLAGS)
+$(dir)/message-id-parse: $(dir)/message-id-parse.o lib/libnotmuch.a util/libnotmuch_util.a
+ $(call quiet,CC) $^ -o $@ $(LDFLAGS) $(TALLOC_LDFLAGS)
+
$(dir)/hex-xcode: $(dir)/hex-xcode.o command-line-arguments.o util/libnotmuch_util.a
$(call quiet,CC) $^ -o $@ $(LDFLAGS) $(TALLOC_LDFLAGS)
$(dir)/smtp-dummy.c \
$(dir)/symbol-test.cc \
$(dir)/make-db-version.cc \
- $(dir)/ghost-report.cc
+ $(dir)/ghost-report.cc \
+ $(dir)/message-id-parse.c
test_srcs=$(test_main_srcs) $(dir)/database-test.c
test_begin_subtest "Writing config file through symlink follows symlink"
test_expect_equal "$(readlink alt-config-link)" "alt-config"
+test_begin_subtest "Absolute database path returned"
+notmuch config set database.path ${HOME}/Maildir
+test_expect_equal "$(notmuch config get database.path)" \
+ "${HOME}/Maildir"
+
+test_begin_subtest "Relative database path properly expanded"
+notmuch config set database.path Maildir
+test_expect_equal "$(notmuch config get database.path)" \
+ "${HOME}/Maildir"
+
test_done
test_expect_equal_file batch_removeall.expected OUTPUT
rm batch_removeall.expected
+test_begin_subtest "--batch, dependence on previous line"
+notmuch dump --format=batch-tag > backup.tags
+notmuch tag --batch<<EOF
++trigger -- One
++second_tag -- tag:trigger
+EOF
+NOTMUCH_DUMP_TAGS tag:second_tag > OUTPUT
+notmuch restore --format=batch-tag < backup.tags
+cat <<EOF >EXPECTED
++inbox +second_tag +tag5 +trigger +unread -- id:msg-001@notmuch-test-suite
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
test_begin_subtest "--batch, blank lines and comments"
notmuch dump | sort > EXPECTED
notmuch tag --batch <<EOF
output=$(notmuch search id:${gen_msg_id} 2>&1 | notmuch_show_sanitize)
test_expect_equal "$output" "thread:0000000000000005 2001-01-05 [1/1] Notmuch Test Suite; encodedword withoutspace (inbox unread)"
+test_begin_subtest "Mislabeled Windows-1252 encoding"
+add_message '[content-type]="text/plain; charset=iso-8859-1"' \
+ "[body]=$'This text contains \x93Windows-1252\x94 character codes.'"
+cat <<EOF > EXPECTED
+\fmessage{ id:XXXXX depth:0 match:1 excluded:0 filename:XXXXX
+\fheader{
+Notmuch Test Suite <test_suite@notmuchmail.org> (2001-01-05) (inbox unread)
+Subject: Mislabeled Windows-1252 encoding
+From: Notmuch Test Suite <test_suite@notmuchmail.org>
+To: Notmuch Test Suite <test_suite@notmuchmail.org>
+Date: GENERATED_DATE
+\fheader}
+\fbody{
+\fpart{ ID: 1, Content-type: text/plain
+This text contains “Windows-1252” character codes.
+\fpart}
+\fbody}
+\fmessage}
+EOF
+notmuch show id:${gen_msg_id} 2>&1 | notmuch_show_sanitize_all > OUTPUT
+test_expect_equal_file EXPECTED OUTPUT
+
test_done
expected=`echo "$expected" | notmuch_json_show_sanitize`
test_expect_equal_json "$output" "$expected"
-test_begin_subtest "Prefer References to In-Reply-To"
+test_begin_subtest "Prefer References to dodgy In-Reply-To"
add_message '[id]="foo@two.com"' \
'[subject]=two'
-add_message '[in-reply-to]="<bar@baz.com>"' \
+add_message '[in-reply-to]="Your message of December 31 1999 <bar@baz.com>"' \
'[references]="<foo@two.com>"' \
'[subject]="Re: two"'
output=$(notmuch show --format=json 'subject:two' | notmuch_json_show_sanitize)
expected=`echo "$expected" | notmuch_json_show_sanitize`
test_expect_equal_json "$output" "$expected"
-test_begin_subtest "Use last Reference"
+test_begin_subtest "Use last Reference when In-Reply-To is dodgy"
add_message '[id]="foo@four.com"' \
'[subject]="four"'
add_message '[id]="bar@four.com"' \
'[subject]="not-four"'
-add_message '[in-reply-to]="<baz@four.com>"' \
+add_message '[in-reply-to]="<baz@four.com> (RFC822 4lyfe)"' \
'[references]="<baz@four.com> <foo@four.com>"' \
'[subject]="neither"'
output=$(notmuch show --format=json 'subject:four' | notmuch_json_show_sanitize)
expected=`echo "$expected" | notmuch_json_show_sanitize`
test_expect_equal_json "$output" "$expected"
+add_email_corpus threading
+
+test_begin_subtest "reply to ghost"
+notmuch show --entire-thread=true id:000-real-root@example.org | grep ^Subject: | head -1 > OUTPUT
+cat <<EOF > EXPECTED
+Subject: root message
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "reply to ghost (tree view)"
+test_emacs '(notmuch-tree "id:000-real-root@example.org")
+ (notmuch-test-wait)
+ (test-output)
+ (delete-other-windows)'
+cat <<EOF > EXPECTED
+ 2016-06-17 Alice ┬►root message (inbox unread)
+ 2016-06-18 Alice ╰┬►child message (inbox unread)
+ 2016-06-17 Mallory ├─►fake root message (inbox unread)
+ 2016-06-18 Alice ├┬►grand-child message (inbox unread)
+ 2016-06-18 Alice │╰─►great grand-child message (inbox unread)
+ 2016-06-18 Daniel ╰─►grand-child message 2 (inbox unread)
+End of search results.
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "reply to ghost (RT)"
+notmuch show --entire-thread=true id:87bmc6lp3h.fsf@len.workgroup | grep ^Subject: | head -1 > OUTPUT
+cat <<EOF > EXPECTED
+Subject: FYI: xxxx xxxxxxx xxxxxxxxxxxx xxx
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "reply to ghost (RT/tree view)"
+test_emacs '(notmuch-tree "id:87bmc6lp3h.fsf@len.workgroup")
+ (notmuch-test-wait)
+ (test-output)
+ (delete-other-windows)'
+cat <<EOF > EXPECTED
+ 2016-06-19 Gregor Zattler ┬┬►FYI: xxxx xxxxxxx xxxxxxxxxxxx xxx (inbox unread)
+ 2016-06-19 via RT │╰─►[support.xxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de #33575] AutoReply: FYI: xxxx xxxxxxx xxxxxxxxxxxx xxx (inbox unread)
+ 2016-06-26 via RT ╰─►[support.xxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de #33575] Resolved: FYI: xxxx xxxxxxx xxxxxxxxxxxx xxx (inbox unread)
+End of search results.
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "trusting reply-to (tree view)"
+test_emacs '(notmuch-tree "id:B00-root@example.org")
+ (notmuch-test-wait)
+ (test-output)
+ (delete-other-windows)'
+cat <<EOF > EXPECTED
+ 2016-06-17 Alice ┬►root message (inbox unread)
+ 2016-06-18 Alice ╰┬►child message (inbox unread)
+ 2016-06-18 Alice ╰─►grand-child message (inbox unread)
+End of search results.
+EOF
+test_expect_equal_file EXPECTED OUTPUT
test_done
--- /dev/null
+#!/usr/bin/env bash
+test_description="message id parsing"
+
+. $(dirname "$0")/test-lib.sh || exit 1
+
+test_begin_subtest "good message ids"
+${TEST_DIRECTORY}/message-id-parse <<EOF >OUTPUT
+<018b1a8f2d1df62e804ce88b65401304832dfbbf.1346614915.git.jani@nikula.org>
+<1530507300.raoomurnbf.astroid@strange.none>
+<1258787708-21121-2-git-send-email-keithp@keithp.com>
+EOF
+cat <<EOF >EXPECTED
+GOOD: 018b1a8f2d1df62e804ce88b65401304832dfbbf.1346614915.git.jani@nikula.org
+GOOD: 1530507300.raoomurnbf.astroid@strange.none
+GOOD: 1258787708-21121-2-git-send-email-keithp@keithp.com
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "leading and trailing space is OK"
+${TEST_DIRECTORY}/message-id-parse <<EOF >OUTPUT
+ <018b1a8f2d1df62e804ce88b65401304832dfbbf.1346614915.git.jani@nikula.org>
+<1530507300.raoomurnbf.astroid@strange.none>
+ <1258787708-21121-2-git-send-email-keithp@keithp.com>
+EOF
+cat <<EOF >EXPECTED
+GOOD: 018b1a8f2d1df62e804ce88b65401304832dfbbf.1346614915.git.jani@nikula.org
+GOOD: 1530507300.raoomurnbf.astroid@strange.none
+GOOD: 1258787708-21121-2-git-send-email-keithp@keithp.com
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "<> delimeters are required"
+${TEST_DIRECTORY}/message-id-parse <<EOF >OUTPUT
+018b1a8f2d1df62e804ce88b65401304832dfbbf.1346614915.git.jani@nikula.org>
+<1530507300.raoomurnbf.astroid@strange.none
+1258787708-21121-2-git-send-email-keithp@keithp.com
+EOF
+cat <<EOF >EXPECTED
+BAD: 018b1a8f2d1df62e804ce88b65401304832dfbbf.1346614915.git.jani@nikula.org>
+BAD: <1530507300.raoomurnbf.astroid@strange.none
+BAD: 1258787708-21121-2-git-send-email-keithp@keithp.com
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+test_begin_subtest "embedded whitespace is forbidden"
+${TEST_DIRECTORY}/message-id-parse <<EOF >OUTPUT
+<018b1a8f2d1df62e804ce88b65401304832dfbbf.1346614915 .git.jani@nikula.org>
+<1530507300.raoomurnbf.astroid @strange.none>
+<1258787708-21121-\f2-git-send-email-keithp@keithp.com>
+EOF
+cat <<EOF >EXPECTED
+BAD: <018b1a8f2d1df62e804ce88b65401304832dfbbf.1346614915 .git.jani@nikula.org>
+BAD: <1530507300.raoomurnbf.astroid @strange.none>
+BAD: <1258787708-21121-\f2-git-send-email-keithp@keithp.com>
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+
+test_begin_subtest "folded real life bad In-Reply-To values"
+${TEST_DIRECTORY}/message-id-parse <<EOF >OUTPUT
+<22597.31869.380767.339702@chiark.greenend.org.uk> (Ian Jackson's message of "Mon, 5 Dec 2016 14:41:01 +0000")
+<20170625141242.loaalhis2eodo66n@gaara.hadrons.org> <149719990964.27883.13021127452105787770.reportbug@seneca.home.org>
+Your message of Tue, 09 Dec 2014 13:21:11 +0100. <1900758.CgLNVPbY9N@liber>
+EOF
+cat <<EOF >EXPECTED
+BAD: <22597.31869.380767.339702@chiark.greenend.org.uk> (Ian Jackson's message of "Mon, 5 Dec 2016 14:41:01 +0000")
+BAD: <20170625141242.loaalhis2eodo66n@gaara.hadrons.org> <149719990964.27883.13021127452105787770.reportbug@seneca.home.org>
+BAD: Your message of Tue, 09 Dec 2014 13:21:11 +0100. <1900758.CgLNVPbY9N@liber>
+EOF
+test_expect_equal_file EXPECTED OUTPUT
+
+
+test_done
--- /dev/null
+From: Gregor Zattler <g.zattler@xxxxxxx-xxxxxxxxx.de>
+To: xxx request tracker <rt-xxx@xxxxxxx-xxxxxxxxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de>
+Subject: FYI: xxxx xxxxxxx xxxxxxxxxxxx xxx
+Date: Tue, 19 Jun 2016 18:26:26 +0200
+Message-ID: <87bmc6lp3h.fsf@len.workgroup>
+MIME-Version: 1.0
+Content-Type: text/plain; charset=utf-8
+Content-Transfer-Encoding: quoted-printable
+
--- /dev/null
+Return-Path: <prvs=701fd58e1=www-data@support.xxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de>
+Subject: [support.xxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de #33575] AutoReply: FYI: xxxx xxxxxxx xxxxxxxxxxxx xxx
+From: " via RT" <rt-xxx@xxxxxxx-xxxxxxxxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de>
+Reply-To: rt-xxx@xxxxxxx-xxxxxxxxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de
+In-Reply-To: <87bmc6lp3h.fsf@len.workgroup>
+References: <RT-Ticket-33575@xxxxxxx-xxxxxxxxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de>
+ <87bmc6lp3h.fsf@len.workgroup>
+Message-ID: <rt-4.2.8-22046-1529425595-591.33575-211-0@xxxxxxx-xxxxxxxxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de>
+To: g.zattler@xxxxxxx-xxxxxxxxx.de
+Content-Type: text/plain; charset="utf-8"
+Date: Tue, 19 Jun 2016 18:26:36 +0200
+MIME-Version: 1.0
+Content-Transfer-Encoding: 8bit
+
+
+
+
--- /dev/null
+Return-Path: <prvs=708ebe06b=www-data@support.xxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de>
+Subject: [support.xxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de #33575] Resolved: FYI: xxxx xxxxxxx xxxxxxxxxxxx xxx
+From: " via RT" <rt-xxx@xxxxxxx-xxxxxxxxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de>
+Reply-To: rt-xxx@xxxxxxx-xxxxxxxxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de
+References: <RT-Ticket-33575@xxxxxxx-xxxxxxxxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de>
+Message-ID: <rt-4.2.8-6644-1530017064-1465.33575-215-0@xxxxxxx-xxxxxxxxxxxxxxxxx-xxxxxxxxx-xxxxxxxxx.de>
+To: g.zattler@xxxxxxx-xxxxxxxxx.de
+Content-Type: text/plain; charset="utf-8"
+Date: Tue, 26 Jun 2016 14:44:24 +0200
+MIME-Version: 1.0
+Content-Transfer-Encoding: 8bit
+
+
+
+
+According to our records, your request has been resolved. If you have any
+further questions or concerns, please respond to this message.
+
--- /dev/null
+From: Alice <alice@example.org>
+To: Daniel <daniel@example.org>
+Subject: child message
+Message-ID: <001-child@example.org>
+In-Reply-To: <000-real-root@example.org>
+References: <000-real-root@example.org>
+Date: Fri, 17 Jun 2016 22:14:41 -0400
+
+
--- /dev/null
+From: Mallory <mallory@example.org>
+To: Daniel <daniel@example.org>
+Subject: fake root message
+Message-ID: <001-fake-message-root@example.org>
+In-Reply-to: <nonexistent-message@example.org>
+References: <000-real-root@example.org> <001-child@example.org> <nonexistent-message@example.org>
+Date: Thu, 16 Jun 2016 22:14:41 -0400
+
+This message has an in-reply-to pointing to a non-existent message
--- /dev/null
+From: Alice <alice@example.org>
+To: Daniel <daniel@example.org>
+Subject: grand-child message
+Message-ID: <001-grand-child@example.org>
+In-Reply-To: <001-child@example.org>
+References: <000-real-root@example.org> <001-child@example.org>
+Date: Fri, 17 Jun 2016 22:24:41 -0400
+
+
--- /dev/null
+From: Daniel <daniel@example.org>
+To: Alice <alice@example.org>
+Subject: grand-child message 2
+Message-ID: <001-grand-child2@example.org>
+In-Reply-To: <001-child@example.org>
+References: <000-real-root@example.org> <001-child@example.org>
+Date: Fri, 17 Jun 2016 22:34:41 -0400
+
+
--- /dev/null
+From: Alice <alice@example.org>
+To: Daniel <daniel@example.org>
+Subject: great grand-child message
+Message-ID: <001-great-grand-child@example.org>
+In-Reply-To: <001-grand-child@example.org>
+References: <000-real-root@example.org> <001-grand-child@example.org>
+Date: Fri, 17 Jun 2016 22:44:41 -0400
+
+
--- /dev/null
+From: Alice <alice@example.org>
+To: Daniel <daniel@example.org>
+Subject: root message
+Message-ID: <000-real-root@example.org>
+Date: Thu, 16 Jun 2016 22:14:41 -0400
+
+This message has no in-reply-to
--- /dev/null
+From: Alice <alice@example.org>
+To: Daniel <daniel@example.org>
+Subject: child message
+Message-ID: <B01-child@example.org>
+In-Reply-To: <B00-root@example.org>
+References: <B00--root@example.org>
+Date: Fri, 17 Jun 2016 22:14:41 -0400
+
+This is a normal-ish reply, and has both a references header and an
+in-reply-to header.
+
--- /dev/null
+From: Alice <alice@example.org>
+To: Daniel <daniel@example.org>
+Subject: grand-child message
+Message-ID: <B01-grand-child@example.org>
+In-Reply-To: <B01-child@example.org>
+References: <B01-child@example.org> <B00-root@example.org>
+Date: Fri, 17 Jun 2016 22:24:41 -0400
+
+This has the references headers in the wrong order, with oldest first.
+Debbugs does this.
--- /dev/null
+From: Alice <alice@example.org>
+To: Daniel <daniel@example.org>
+Subject: root message
+Message-ID: <B00-root@example.org>
+Date: Thu, 16 Jun 2016 22:14:41 -0400
+
+This message has no reply-to
--- /dev/null
+#include <stdio.h>
+#include <talloc.h>
+#include "notmuch-private.h"
+
+int
+main (unused (int argc), unused (char **argv))
+{
+ char *line = NULL;
+ size_t len = 0;
+ ssize_t nread;
+ void *local = talloc_new (NULL);
+
+ while ((nread = getline (&line, &len, stdin)) != -1) {
+ int last = strlen (line) - 1;
+ if (line[last] == '\n')
+ line[last] = '\0';
+
+ char *mid = _notmuch_message_id_parse_strict (local, line);
+ if (mid)
+ printf ("GOOD: %s\n", mid);
+ else
+ printf ("BAD: %s\n", line);
+ }
+
+ talloc_free (local);
+}
return 0;
}
-static const char*
+const char*
skip_space (const char *str)
{
while (*str && isspace ((unsigned char) *str))
void strip_trailing (char *str, char ch);
+const char* skip_space (const char *str);
+
#ifdef __cplusplus
}
#endif
projects / notmuch / commitdiff