Currently we're looking up all parents (based on In-reply-to and
References header) and using the list of all thread_id values
from those as our thread_id value. We're missing one step which
sup does which is to also look up any children in the database
that have reference our message ID. So we'll need to do that next.
Carl Worth [Tue, 13 Oct 2009 23:09:54 +0000 (16:09 -0700)]
notmuch-index-message: Don't index quoted lines and signatures.
At this point, we're achieving a result that is *very* close to
what sup does. The only difference is that we are still indexing
the "excerpts from message ..." line, and we are not yet indexing
references.
Carl Worth [Tue, 13 Oct 2009 22:56:16 +0000 (15:56 -0700)]
notmuch-index-message: Add code to actually create a Xapian index
Most of this code is fairly clean and works well. One part is
fairly painful---namely extracting the body of an email message
from libgmime. Currently, I'm just extracting the offset to
the end of the headers, and then separately opening the message.
Surely there's a better way.
Anyway, with that the results are looking very similar to sup-sync
now, (as verified by xapian-dump). The only substantial difference
I'm seeing now is that sup does not seem to index quoted portions
of messages nor signatures. I'm not actually sure whether I want
to follow sup's lead in that or not.
Carl Worth [Tue, 13 Oct 2009 20:18:32 +0000 (13:18 -0700)]
Include document data in the dump.
At the same time, I've started hacking up sup with a new NotmuchIndex
class in the place of the previous XapianIndex class. The new class
stores only the source_info field in the document data, (rather than
a serialized ruby hash with a bunch of data that can be found in the
original message).
Eventually, I plan to replace source_info with a relative filename for
the message, (or even a list of filenames for when multiple messages
in the database share a common message ID).
Carl Worth [Tue, 13 Oct 2009 16:36:25 +0000 (09:36 -0700)]
xapian-dump: Add support to unserialize values.
The interface for this is cheesy, (bare integer value numbers on the
command line indicating that unserialization is desired for those
value numbers). But this at least lets us print sup databases with
human-readable output for the date values.
Carl Worth [Tue, 13 Oct 2009 14:50:04 +0000 (07:50 -0700)]
Add the beginnings of a xapian-dump program.
This will (when it is finished) make a much more reliable way to
ensure that notmuch's sync program behaves identically to sup-sync.
It doesn't actually do anything yet.
Carl Worth [Tue, 13 Oct 2009 14:23:14 +0000 (07:23 -0700)]
Add sup-compatible prefixes and achieve sup-compatible print output
What I've done here is to instrument sup-sync to print the text
and terms objects it constructs just before indexing a message.
Then I've made my g_mime_test program achieve (nearly) identical
output for an example email message, (just missing the body
text). Next we can start shoving this data into a Xapian index.