8.2 nnmairix

This paragraph describes how to set up mairix and the back end nnmairix for indexing and searching your mail from within Gnus. Additionally, you can create permanent “smart” groups which are bound to mairix searches and are automatically updated.

8.2.1 About mairix

Mairix is a tool for indexing and searching words in locally stored mail. It was written by Richard Curnow and is licensed under the GPL. Mairix comes with most popular GNU/Linux distributions, but it also runs under Windows (with cygwin), Mac OS X and Solaris. The homepage can be found at http://www.rpcurnow.force9.co.uk/mairix/index.html

Though mairix might not be as flexible as other search tools like swish++ or namazu, which you can use via the nnir back end, it has the prime advantage of being incredibly fast. On current systems, it can easily search through headers and message bodies of thousands and thousands of mails in well under a second. Building the database necessary for searching might take a minute or two, but only has to be done once fully. Afterwards, the updates are done incrementally and therefore are really fast, too. Additionally, mairix is very easy to set up.

For maximum speed though, mairix should be used with mails stored in Maildir or MH format (this includes the nnml back end), although it also works with mbox. Mairix presents the search results by populating a virtual maildir/MH folder with symlinks which point to the “real” message files (if mbox is used, copies are made). Since mairix already presents search results in such a virtual mail folder, it is very well suited for using it as an external program for creating smart mail folders, which represent certain mail searches.

8.2.2 nnmairix requirements

Mairix searches local mail—that means, mairix absolutely must have direct access to your mail folders. If your mail resides on another server (e.g., an IMAP server) and you happen to have shell access, nnmairix supports running mairix remotely, e.g., via ssh.

Additionally, nnmairix only supports the following Gnus back ends: nnml, nnmaildir, and nnimap. You must use one of these back ends for using nnmairix. Other back ends, like nnmbox, nnfolder or nnmh, won’t work.

If you absolutely must use mbox and still want to use nnmairix, you can set up a local IMAP server, which you then access via nnimap. This is a rather massive setup for accessing some mbox files, so just change to MH or Maildir already... However, if you’re really, really passionate about using mbox, you might want to look into the package ‘mairix.el’, which comes with Emacs 23.

8.2.3 What nnmairix does

The back end nnmairix enables you to call mairix from within Gnus, either to query mairix with a search term or to update the database. While visiting a message in the summary buffer, you can use several pre-defined shortcuts for calling mairix, e.g., to quickly search for all mails from the sender of the current message or to display the whole thread associated with the message, even if the mails are in different folders.

Additionally, you can create permanent nnmairix groups which are bound to certain mairix searches. This way, you can easily create a group containing mails from a certain sender, with a certain subject line or even for one specific thread based on the Message-ID. If you check for new mail in these folders (e.g., by pressing g or M-g), they automatically update themselves by calling mairix.

You might ask why you need nnmairix at all, since mairix already creates the group, populates it with links to the mails so that you can then access it with Gnus, right? Well, this might work, but often does not—at least not without problems. Most probably you will get strange article counts, and sometimes you might see mails which Gnus claims have already been canceled and are inaccessible. This is due to the fact that Gnus isn’t really amused when things are happening behind its back. Another problem can be the mail back end itself, e.g., if you use mairix with an IMAP server (I had Dovecot complaining about corrupt index files when mairix changed the contents of the search group). Using nnmairix should circumvent these problems.

nnmairix is not really a mail back end—it’s actually more like a wrapper, sitting between a “real” mail back end where mairix stores the searches and the Gnus front end. You can choose between three different mail back ends for the mairix folders: nnml, nnmaildir or nnimap. nnmairix will call the mairix binary so that the search results are stored in folders named zz_mairix-<NAME>-<NUMBER> on this mail back end, but it will present these folders in the Gnus front end only with <NAME>. You can use an existing mail back end where you already store your mail, but if you’re uncomfortable with nnmairix creating new mail groups alongside your other mail, you can also create, e.g., a new nnmaildir or nnml server exclusively for mairix, but then make sure those servers do not accidentally receive your new mail (see section nnmairix caveats). A special case exists if you want to use mairix remotely on an IMAP server with nnimap—here the mairix folders and your other mail must be on the same nnimap back end.

8.2.4 Setting up mairix

First: create a backup of your mail folders (see section nnmairix caveats).

Setting up mairix is easy: simply create a ‘.mairixrc’ file with (at least) the following entries:

# Your Maildir/MH base folder
base=~/Maildir

This is the base folder for your mails. All the following directories are relative to this base folder. If you want to use nnmairix with nnimap, this base directory has to point to the mail directory where the IMAP server stores the mail folders!

maildir= ... your maildir folders which should be indexed ...
mh= ... your nnml/mh folders which should be indexed ...
mbox = ... your mbox files which should be indexed ...

This specifies all your mail folders and mbox files (relative to the base directory!) you want to index with mairix. Note that the nnml back end saves mails in MH format, so you have to put those directories in the mh line. See the example at the end of this section and mairixrc’s man-page for further details.

omit=zz_mairix-*

This should make sure that you don’t accidentally index the mairix search results. You can change the prefix of these folders with the variable nnmairix-group-prefix.

mformat= ... 'maildir' or 'mh' ...
database= ... location of database file ...

The format setting specifies the output format for the mairix search folder. Set this to mh if you want to access search results with nnml. Otherwise choose maildir.

To summarize, here is my shortened ‘.mairixrc’ file as an example:

base=~/Maildir
maildir=.personal:.work:.logcheck:.sent
mh=../Mail/nnml/*...
mbox=../mboxmail/mailarchive_year*
mformat=maildir
omit=zz_mairix-*
database=~/.mairixdatabase

In this case, the base directory is ‘~/Maildir’, where all my Maildir folders are stored. As you can see, the folders are separated by colons. If you wonder why every folder begins with a dot: this is because I use Dovecot as IMAP server, which again uses Maildir++ folders. For testing nnmairix, I also have some nnml mail, which is saved in ‘~/Mail/nnml’. Since this has to be specified relative to the base directory, the ../Mail notation is needed. Note that the line ends in *..., which means to recursively scan all files under this directory. Without the three dots, the wildcard * will not work recursively. I also have some old mbox files with archived mail lying around in ‘~/mboxmail’. The other lines should be obvious.

See the man page for mairixrc for details and further options, especially regarding wildcard usage, which may be a little different than you are used to.

Now simply call mairix to create the index for the first time. Note that this may take a few minutes, but every following index will do the updates incrementally and hence is very fast.

8.2.5 Configuring nnmairix

In group mode, type G b c (nnmairix-create-server-and-default-group). This will ask you for all necessary information and create a nnmairix server as a foreign server. You will have to specify the following:

• The name of the nnmairix server—choose whatever you want.
• The name of the back end server where mairix should store its searches. This must be a full server name, like nnml:mymail. Just hit TAB to see the available servers. Currently, servers which are accessed through nnmaildir, nnimap and nnml are supported. As explained above, for locally stored mails, this can be an existing server where you store your mails. However, you can also create, e.g., a new nnmaildir or nnml server exclusively for nnmairix in your secondary select methods (see section Finding the News). If you use a secondary nnml server just for mairix, make sure that you explicitly set the server variable nnml-get-new-mail to nil, or you might lose mail (see section nnmairix caveats). If you want to use mairix remotely on an IMAP server, you have to choose the corresponding nnimap server here.
• The command to call the mairix binary. This will usually just be mairix, but you can also choose something like ssh SERVER mairix if you want to call mairix remotely, e.g., on your IMAP server. If you want to add some default options to mairix, you could do this here, but better use the variable nnmairix-mairix-search-options instead.
• The name of the default search group. This will be the group where all temporary mairix searches are stored, i.e., all searches which are not bound to permanent nnmairix groups. Choose whatever you like.
• If the mail back end is nnimap or nnmaildir, you will be asked if you work with Maildir++, i.e., with hidden maildir folders (=beginning with a dot). For example, you have to answer ‘yes’ here if you work with the Dovecot IMAP server. Otherwise, you should answer ‘no’ here.

8.2.6 nnmairix keyboard shortcuts

In group mode:

G b c

Creates nnmairix server and default search group for this server (nnmairix-create-server-and-default-group). You should have done this by now (see section Configuring nnmairix).

G b s

Prompts for query which is then sent to the mairix binary. Search results are put into the default search group which is automatically displayed (nnmairix-search).

G b m

Allows you to create a mairix search or a permanent group more comfortably using graphical widgets, similar to a customization group. Just try it to see how it works (nnmairix-widget-search).

G b i

Another command for creating a mairix query more comfortably, but uses only the minibuffer (nnmairix-search-interactive).

G b g

Creates a permanent group which is associated with a search query (nnmairix-create-search-group). The nnmairix back end automatically calls mairix when you update this group with g or M-g.

G b q

Changes the search query for the nnmairix group under cursor (nnmairix-group-change-query-this-group).

G b t

Toggles the ’threads’ parameter for the nnmairix group under cursor, i.e., if you want see the whole threads of the found messages (nnmairix-group-toggle-threads-this-group).

G b u

Calls mairix binary for updating the database (nnmairix-update-database). The default parameters are -F and -Q for making this as fast as possible (see variable nnmairix-mairix-update-options for defining these default options).

G b r

Keep articles in this nnmairix group always read or unread, or leave the marks unchanged (nnmairix-group-toggle-readmarks-this-group).

G b d

Recreate nnmairix group on the “real” mail back end (nnmairix-group-delete-recreate-this-group). You can do this if you always get wrong article counts with a nnmairix group.

G b a

Toggles the allow-fast parameters for group under cursor (nnmairix-group-toggle-allowfast-this-group). The default behavior of nnmairix is to do a mairix search every time you update or enter the group. With the allow-fast parameter set, mairix will only be called when you explicitly update the group, but not upon entering. This makes entering the group faster, but it may also lead to dangling symlinks if something changed between updating and entering the group which is not yet in the mairix database.

G b p

Toggle marks propagation for this group (nnmairix-group-toggle-propmarks-this-group). (see section Propagating marks).

G b o

Manually propagate marks (nnmairix-propagate-marks); needed only when nnmairix-propagate-marks-upon-close is set to nil.

In summary mode:

$ m

Allows you to create a mairix query or group based on the current message using graphical widgets (same as nnmairix-widget-search) (nnmairix-widget-search-from-this-article).

$ g

Interactively creates a new search group with query based on the current message, but uses the minibuffer instead of graphical widgets (nnmairix-create-search-group-from-message).

$ t

Searches thread for the current article (nnmairix-search-thread-this-article). This is effectively a shortcut for calling nnmairix-search with ‘m:msgid’ of the current article and enabled threads.

$ f

Searches all messages from sender of the current article (nnmairix-search-from-this-article). This is a shortcut for calling nnmairix-search with ‘f:From’.

$ o

(Only in nnmairix groups!) Tries determine the group this article originally came from and displays the article in this group, so that, e.g., replying to this article the correct posting styles/group parameters are applied (nnmairix-goto-original-article). This function will use the registry if available, but can also parse the article file name as a fallback method.

$ u

Remove possibly existing tick mark from original article (nnmairix-remove-tick-mark-original-article). (see section nnmairix tips and tricks).

8.2.7 Propagating marks

First of: you really need a patched mairix binary for using the marks propagation feature efficiently. Otherwise, you would have to update the mairix database all the time. You can get the patch at

http://www.randomsample.de/mairix-maildir-patch.tar

You need the mairix v0.21 source code for this patch; everything else is explained in the accompanied readme file. If you don’t want to use marks propagation, you don’t have to apply these patches, but they also fix some annoyances regarding changing maildir flags, so it might still be useful to you.

With the patched mairix binary, you can use nnmairix as an alternative to mail splitting (see section Fancy Mail Splitting). For example, instead of splitting all mails from ‘david@foobar.com’ into a group, you can simply create a search group with the query ‘f:david@foobar.com’. This is actually what “smart folders” are all about: simply put everything in one mail folder and dynamically create searches instead of splitting. This is more flexible, since you can dynamically change your folders any time you want to. This also implies that you will usually read your mails in the nnmairix groups instead of your “real” mail groups.

There is one problem, though: say you got a new mail from ‘david@foobar.com’; it will now show up in two groups, the “real” group (your INBOX, for example) and in the nnmairix search group (provided you have updated the mairix database). Now you enter the nnmairix group and read the mail. The mail will be marked as read, but only in the nnmairix group—in the “real” mail group it will be still shown as unread.

You could now catch up the mail group (see section Group Data), but this is tedious and error prone, since you may overlook mails you don’t have created nnmairix groups for. Of course, you could first use nnmairix-goto-original-article (see section nnmairix keyboard shortcuts) and then read the mail in the original group, but that’s even more cumbersome.

Clearly, the easiest way would be if marks could somehow be automatically set for the original article. This is exactly what marks propagation is about.

Marks propagation is inactive by default. You can activate it for a certain nnmairix group with nnmairix-group-toggle-propmarks-this-group (bound to G b p). This function will warn you if you try to use it with your default search group; the reason is that the default search group is used for temporary searches, and it’s easy to accidentally propagate marks from this group. However, you can ignore this warning if you really want to.

With marks propagation enabled, all the marks you set in a nnmairix group should now be propagated to the original article. For example, you can now tick an article (by default with !) and this mark should magically be set for the original article, too.

A few more remarks which you may or may not want to know:

Marks will not be set immediately, but only upon closing a group. This not only makes marks propagation faster, it also avoids problems with dangling symlinks when dealing with maildir files (since changing flags will change the file name). You can also control when to propagate marks via nnmairix-propagate-marks-upon-close (see the doc-string for details).

Obviously, nnmairix will have to look up the original group for every article you want to set marks for. If available, nnmairix will first use the registry for determining the original group. The registry is very fast, hence you should really, really enable the registry when using marks propagation. If you don’t have to worry about RAM and disc space, set gnus-registry-max-entries to a large enough value; to be on the safe side, choose roughly the amount of mails you index with mairix.

If you don’t want to use the registry or the registry hasn’t seen the original article yet, nnmairix will use an additional mairix search for determining the file name of the article. This, of course, is way slower than the registry—if you set hundreds or even thousands of marks this way, it might take some time. You can avoid this situation by setting nnmairix-only-use-registry to t.

Maybe you also want to propagate marks the other way round, i.e., if you tick an article in a "real" mail group, you’d like to have the same article in a nnmairix group ticked, too. For several good reasons, this can only be done efficiently if you use maildir. To immediately contradict myself, let me mention that it WON’T work with nnmaildir, since nnmaildir stores the marks externally and not in the file name. Therefore, propagating marks to nnmairix groups will usually only work if you use an IMAP server which uses maildir as its file format.

If you work with this setup, just set nnmairix-propagate-marks-to-nnmairix-groups to t and see what happens. If you don’t like what you see, just set it to nil again. One problem might be that you get a wrong number of unread articles; this usually happens when you delete or expire articles in the original groups. When this happens, you can recreate the nnmairix group on the back end using G b d.

8.2.8 nnmairix tips and tricks

• Checking Mail

  I put all my important mail groups at group level 1. The mairix groups have group level 5, so they do not get checked at start up (see section Group Levels).

  I use the following to check for mails:

  (defun my-check-mail-mairix-update (level) (interactive "P") ;; if no prefix given, set level=1 (gnus-group-get-new-news (or level 1)) (nnmairix-update-groups "mairixsearch" t t) (gnus-group-list-groups)) (define-key gnus-group-mode-map "g" 'my-check-mail-mairix-update)

  Instead of ‘"mairixsearch"’ use the name of your nnmairix server. See the doc string for nnmairix-update-groups for details.

• Example: search group for ticked articles

  For example, you can create a group for all ticked articles, where the articles always stay unread:

  Hit G b g, enter group name (e.g., ‘important’), use ‘F:f’ as query and do not include threads.

  Now activate marks propagation for this group by using G b p. Then activate the always-unread feature by using G b r twice.

  So far so good—but how do you remove the tick marks in the nnmairix group? There are two options: You may simply use nnmairix-remove-tick-mark-original-article (bound to $ u) to remove tick marks from the original article. The other possibility is to set nnmairix-propagate-marks-to-nnmairix-groups to t, but see the above comments about this option. If it works for you, the tick marks should also exist in the nnmairix group and you can remove them as usual, e.g., by marking an article as read.

  When you have removed a tick mark from the original article, this article should vanish from the nnmairix group after you have updated the mairix database and updated the group. Fortunately, there is a function for doing exactly that: nnmairix-update-groups. See the previous code snippet and the doc string for details.

• Dealing with auto-subscription of mail groups

  As described before, all nnmairix groups are in fact stored on the mail back end in the form ‘zz_mairix-<NAME>-<NUMBER>’. You can see them when you enter the back end server in the server buffer. You should not subscribe these groups! Unfortunately, these groups will usually get auto-subscribed when you use nnmaildir or nnml, i.e., you will suddenly see groups of the form ‘zz_mairix*’ pop up in your group buffer. If this happens to you, simply kill these groups with C-k. For avoiding this, turn off auto-subscription completely by setting the variable gnus-auto-subscribed-groups to nil (see section Filtering New Groups), or if you like to keep this feature use the following kludge for turning it off for all groups beginning with ‘zz_’:

  (setq gnus-auto-subscribed-groups "^\\(nnml\\|nnfolder\\|nnmbox\\|nnmh\\|nnbabyl\\|nnmaildir\\).*:\\([^z]\\|z$\\|\\z[^z]\\|zz$\\|zz[^_]\\|zz_$\\).*")

8.2.9 nnmairix caveats

• You can create a secondary nnml server just for nnmairix, but then you have to explicitly set the corresponding server variable nnml-get-new-mail to nil. Otherwise, new mail might get put into this secondary server (and would never show up again). Here’s an example server definition:

  (nnml "mairix" (nnml-directory "mairix") (nnml-get-new-mail nil))

  (The nnmaildir back end also has a server variable get-new-mail, but its default value is nil, so you don’t have to explicitly set it if you use a nnmaildir server just for mairix.)

• If you use the Gnus registry: don’t use the registry with nnmairix groups (put them in gnus-registry-unfollowed-groups; this is the default). Be extra careful if you use gnus-registry-split-fancy-with-parent; mails which are split into nnmairix groups are usually gone for good as soon as you check the group for new mail (yes, it has happened to me...).
• Therefore: Never ever put “real” mails into nnmairix groups (you shouldn’t be able to, anyway).
• If you use the Gnus agent (see section Gnus Unplugged): don’t agentize nnmairix groups (though I have no idea what happens if you do).
• mairix does only support us-ascii characters.
• nnmairix uses a rather brute force method to force Gnus to completely reread the group on the mail back end after mairix was called—it simply deletes and re-creates the group on the mail back end. So far, this has worked for me without any problems, and I don’t see how nnmairix could delete other mail groups than its own, but anyway: you really should have a backup of your mail folders.
• All necessary information is stored in the group parameters (see section Group Parameters). This has the advantage that no active file is needed, but also implies that when you kill a nnmairix group, it is gone for good.
• If you create and kill a lot of nnmairix groups, the “zz_mairix-*” groups will accumulate on the mail back end server. To delete old groups which are no longer needed, call nnmairix-purge-old-groups. Note that this assumes that you don’t save any “real” mail in folders of the form zz_mairix-<NAME>-<NUMBER>. You can change the prefix of nnmairix groups by changing the variable nnmairix-group-prefix.
• The following only applies if you don’t use the mentioned patch for mairix (see section Propagating marks):

  A problem can occur when using nnmairix with maildir folders and comes with the fact that maildir stores mail flags like ‘Seen’ or ‘Replied’ by appending chars ‘S’ and ‘R’ to the message file name, respectively. This implies that currently you would have to update the mairix database not only when new mail arrives, but also when mail flags are changing. The same applies to new mails which are indexed while they are still in the ‘new’ folder but then get moved to ‘cur’ when Gnus has seen the mail. If you don’t update the database after this has happened, a mairix query can lead to symlinks pointing to non-existing files. In Gnus, these messages will usually appear with “(none)” entries in the header and can’t be accessed. If this happens to you, using G b u and updating the group will usually fix this.

This document was generated on January 25, 2015 using texi2html 1.82.