--- /dev/null
+[[!meta title="Sup - a mail client for geeks"]]
+[[!tag sup tools]]
+
+For quite some time I've been struggling with hundreds of email
+messages a day, and hundreds-of-thousands of email messages in
+archives to search. I've used several different email programs to try
+to handle my mail, (pine, vm, mh-e, wanderlust, evolution, mutt), and
+experimented with several others, but not until now did I find one
+that really helps me process mail the way I want to.
+
+Before I introduce [sup](http://sup.rubyforge.org/), though, I want to
+at least mention in passing a mail indexer named
+[mairix](http://www.rpcurnow.force9.co.uk/mairix/) since I've been
+meaning to write a blog entry about it for several months. Anthony
+Towns first let me know about mairix earlier this year when I was
+complaining about email near him. He described his approach to mail as
+"mutt for reading mail and mairix for searching".
+
+The way mairix works is that it creates an index of a mail archive
+then provides a command-line-based search that creates results by
+creating a new maildir, (with just symlinks to original messages if
+the original archive is also maildir-based).
+
+So mairix can give you a simple way to add fast, full-archive
+searching to a program like evolution when its builtin search
+capability is hopelessly inadequate for your large mail archive. The
+mail client can be configured to look at a single search-results
+maildir folder, and mairix can be used to create the results there.
+Do note that mairix is somewhat limited as a mail search engine since
+it has no proximity or phrase-based searching, (the best you get is
+whether messages contain given words). But the maildir-in and
+maildir-out approach it has makes it easy to integrate, and is the
+kind of concept that can perhaps let us avoid some of the disasters
+out there in the form of monolithic email clients.
+
+But I've got a much more interesting story in sup. I was just a few
+days into reinventing my email workflow around mutt when I got into an
+interesting conversation with Jamey Sharp about what the ideal
+mail-handling system would look like. Here are some of the ideas we
+came up with:
+
+ * Program presents me with messages that need my attention, (a
+ chronological sort of the archive with unread items in bold does
+ not count).
+
+ * When I decide a message does not need my attention I can make it
+ "go away" into the archive, (and "mark as read" does not
+ count---I want to file away messages but still track whether
+ they are read or not in the archive).
+
+ * Threads must be first-level objects, (in particular I want a
+ kill-thread command that archives not only the current messages,
+ but all future messages received in a given thread).
+
+ * System should support arbitrary tags on messages. Tags can be
+ applied automatically, (such as applying categories to mail
+ received on lists), and applied ad-hoc by the user.
+
+ * System should allow searching over the entire archive with most
+ recent results appearing instantly. Search terms can include
+ tags, message headers, or full-body search (including phrases).
+
+ * In addition to full-archive search, incremental refinement
+ should be possible, (a new search based on the results of a
+ previous search).
+
+ * There's no need for folders in this system. Tags and
+ incrementally refined, tag-based searching provide all the
+ benefits, (without the bug in folder-based systems where a user
+ has to hunt among many folders to find one where a search
+ returns non-empty results). In particular, the "inbox" view
+ should just be a search for unread and non-archived messages.
+
+That was basically our list of core features. Beyond that, we also
+discussed some further ideas for improving the prioritization of email
+threads to the user. One primary job of the system is to present the
+user with the most important thread to be read next, (based on tags,
+etc.). The user supplies feedback by either reading the thread or
+archiving it, so there's opportunity to apply machine learning to
+improve the prioritization.
+
+During the course of this conversation, Jamey mentioned that I should
+go look at [sup](http://sup.rubyforge.org/) which he thought might
+implement some of these features, but he hadn't tried it out yet. I
+was delighted to find that each one of our core features already
+exists in sup, (and sup would make a fine platform for research into
+the learning-based approaches).
+
+It also does a few things I hadn't specified, such as displaying email
+in a full-thread nested view, (rather than one message at a time),
+with quoted elements and signatures folded out of view until the user
+asks to see them.
+
+[Note: I have been told that several of the above features are also
+implemented in gmail. I've never tried gmail myself, since it fails to
+provide some even more fundamental features: 1. offline usage,
+2. personal ownership of email storage, 3. free-software
+implementation for customization.
+
+In the few days I've been using sup, it's definitely transformed the
+way I process mail. Keeping my inbox empty is simple now, and I now
+easily tag messages for specific, future actions without fearing that
+I will forget, (and without leaving them cluttering up my
+inbox). Searching through my entire archive is fast---the first
+results come back instantly and they're generally what I want, (the
+index is optimized to return the most-recent messages first).
+
+Sup is implemented in ruby, and the author and maintainers on the
+[sup-talk](http://rubyforge.org/mailman/listinfo/sup-talk) mailing
+list have been friendly and receptive to me, (in spite of my lack of
+ruby-clue), so I've already got a patch or two in the upstream git
+repository for sup. The indexing in sup has been performed by
+[ferret](http://www.davebalmain.com/), but is currently switching to
+[xapian](http://xapian.org/).
+
+Sup is not entirely mature yet, (I discovered fairly easy ways to make
+the 0.8.1 version in Debian crash), but I'm happily running the latest
+code from git now, and trusting my mail to it. I did find that a bug
+in a dependent package causes sup to crash when running sup from git
+while sup is also installed from the Debian package. So I recommend
+doing "apt-get install sup-mail" to get the dependencies installed,
+then "apt-get remove sup-mail" (but don't autoremove the
+dependencies), then run sup from:
+
+ git clone git://gitorious.org/sup/mainline.git
+
+I've also posted a [short
+list](http://rubyforge.org/pipermail/sup-talk/2009-August/002815.html)
+of some changes I'd like to see in sup, (mostly in the client
+stuff---the indexing and searching stuff seems just fine).
+
+There's a nice New User's Guide included with sup, but no reference
+manual yet, so the [sup wiki](http://sup.rubyforge.org/wiki/wiki.pl)
+is an essential (and short) substitute for a reference manual for now.
+
+One thing sup does not do yet is that it doesn't separate the
+mail-indexing/tagging/searching system from the mail client. So if
+you're not happy with a curses client written in ruby, (and perhaps
+more significantly, a *different* client than what you're currently
+using for composing messages, making attachments, viewing mime stuff,
+etc.) then you're currently out of luck. Fortunately, William Morgan
+has [plans](http://all-thing.net/old13) to reinvent the most
+interesting aspects of sup as a server process and he recently
+reported making progress on those plans over the last year. (This
+isn't too surprising, since why would William want to maintain all that
+code to do things like dealing with mbox, maildir, attachments,
+etc. when those are already solved problems for users everywhere.)
+
+And hey, I would *love* to have a mailing list archiver based on sup's
+threading. So if anyone wants to cook something like that up, please
+feel free, (sup is licensed under the GNU GPLv2).
projects / cworth.org / commitdiff