Fetching Mail with Movemail
The movemail program in GNU Mailutils was initially designed as a better replacement for the similar utility in Emacs, whose main purpose was to move email messages from one UNIX mailbox to another. As many other Mailutils implementations, movemail is able to do much more than its predecessor. In fact, it offers a great power and flexibility for a number of tasks related to electronic mail handling.
This article explains how to use movemail as a mail retrieval agent. Mail retrieval agents (MRA for short), are utilities that retrieve email from a remote mail server and deliver it to a local or remote email mailbox. One of the most popular MRAs is fetchmail.
The movemail invocation scheme is:
movemail [options] source-mailbox dest-mailbox
where options are movemail-specific command line options, source-mailbox is the mailbox to retrieve messages from, and dest-mailbox is the mailbox to move those messages to. Both mailboxes can be specified as URLs, for example:
movemail pop://user:firstname.lastname@example.org mh://Mail/inbox
This command will connect to pop.example.net using the POP protocol, as user user with password pass, and move messages from there to the local MH folder in Mail/inbox.
If the dest-mailbox URL begins with
smtp://, movemail will re-send the retrieved messages to the SMTP server specified in the URL. For example, to get messages from the same POP server as in the example above and to feed them to the SMTP server at localhost, one should write:
movemail pop://user:email@example.com smtp://localhost
This invocation is equivalent to the following fetchmail configuration:
poll pop.example.net proto pop3 user "user" password "pass" smtphost localhost
The SMTP URL syntax is:
The only mandatory part, beside the smtp:// scheme designator, is hostname -- a hostname or IP address of the server to connect to. The rest of elements (shown enclosed in square brackets) is optional:
- The user part supplies the user name for authentication.
- Password for this user is supplied by the path component.
- The auth= parameter is used to specify a list of mechanisms acceptable for authentication. Each mech is a name of a mechanism.
- hostname is the name or IP address of the host to connect to.
- port is the port number or symbolic name (from /etc/services.
- Additional parameters (param) are a semicolon-separated list of key-value pairs.
The port element, if supplied, specifies the port number to use instead of the default 25. For example, to inject mail using the mail submission port (587), we can write:
movemail pop://user:firstname.lastname@example.org smtp://localhost:587
A service number from /etc/services can be used instead of the numeric value, as in:
movemail pop://user:email@example.com smtp://localhost:submission
The user, pass, and auth elements supply credentials for ESMTP authentication, if the server supports it. See TLS and ESMTP Authentication, for a detailed description.
A number of additional data can be supplied using parameters, shown as param in the syntax above. These are a semicolon-separated list of words. Each word is either a single keyword which, when present, enables a certain feature (this is called a boolean parameter), or a compound statement, consisting of a keyword and a value separated with an equals sign, e.g. firstname.lastname@example.org. In the discussion below we will describe each of the available parameters where the relevant functionality is covered. For an exhaustive list of parameters, see SMTP Parameters.
If no additional parameters are supplied, movemail tries to deduce the envelope recipient addresses, i.e. the addresses originally used in the RCPT TO commands during the SMTP transaction. Unfortunately, there is no standard way of preserving this information in the message Movemail uses the following algorithm to deduce it. First, the following headers are examined:
The first of them which is present, is parsed and its value is used as a list of recipients.
If neither of these headers is present, movemail analyzes the value of the topmost Received header. If it contains a by/for clause, the email following the for keyword is used as a recipient.
Finally, if everything else fails, movemail will use the values of To, Cc and Bcc headers from the message.
Several parameters can be used to control this process. First of all, the to parameter, if present, supplies the recipient email explicitly. Its presence turns off the above algorithm. For example, to move all mail from the POP account of user smith to user email@example.com using the local SMTP server, one would write:
movemail pop://smith:firstname.lastname@example.org 'smtp://localhost;to=smith%40foo.com'
Notice the use of %40 instead of the @ character in the URL. The at sign has a special meaning in URLs: it delimits authentication credentials from the user name, therefore it must be escaped if you want it to be treated literally. To escape a special character in a URL, replace it with a percent sign, followed by the ASCII code of that chara cter in hex. Thus, the escape for : (colon) is %3A, for ; is %3B, and for / is %2F. To insert a % character verbatim, use %25.
Notice also the quotes around the last argument. URL parameters are delimited by semicolons, which are special to the shell and therefore must be quoted or escaped.
To modify the list of headers searched for recipient addresses, use the recipient-headers parameter. Its value is a comma-separated list of headers, which will be used instead of the default list. For example:
When used as a boolean (i.e. without value), recipient-headers disables header scanning:
For any recipient address that lacks domain part, the domain part of the hostname of the local machine is appended to it. To supply another default domain, use the domain parameter. For example, given the following destination mailbox:
recipient addresses without domain part will get @example.com apended to them. Thus smith becomes email@example.com and so on.
You can also request removing any original domain part from recipient addresses by specifying the strip-domain parameter. When used together with domain it causes replacing of the original domain name with the new one:
To determine the message sender movemail scans the headers of the message looking for one of the following headers (listed in the order of their precedence):
The first header found is parsed and its value is used as sender address. To supply a fixed sender address for all messages, use the from parameter.
TLS and ESMTP Authentication
Unless told otherwise, movemail tries to make use of all ESMTP capabilities offered by the remote server. In particular, it the SMTP server supports TLS encryption, movemail will try to establish encrypted channel. To disable this behavior the notls parameter is provided. For example, the following destination URL forces movemail to use plaintext channel even if the server offers a TLS capability:
If the server supports ESMTP authentication, movemail will use it if you supply the needed user credentials. For example, to authenticate to the localhost server as user smith with password guessme, use:
Movemail will select the best authentication mechanism from the list offered by the server. To force it to use a particular authentication mechanism, use the auth authentication parameter. Its value is a comma-separated list of authentication mechanisms, in the order from the most preferred to the least preferred one, e.g.:
When given this URL movemail will try to use either cram-md5 or digest-md5, whichever is supported by the server. If neither of them is supported, movemail will continue without authentication.
The auth keyword can also be used as a usual parameter, as shown in the example below:
Putting authentication credentials in the URL itself can be unsecure, because they become visible for other users in the ps output. To avoid compromising your authentication credentials, store them in the Mailutils ticket file. See Mailutils Ticket File, for a detailed description of this feature.
To turn off the ESMTP authentication, use the noauth keyword:
This subsection summarizes the parameters available for use in SMTP URLs. For each parameter we give a short description and a reference to the section which contains an in-depth discussion.
- If ESMTP server supports authentication, negotiate one of the supplied
- authentication mechanisms. See #TLS and ESMTP Authentication.
- Use string as a domain part for those recipients that lack it.
See Mail Recipients.
- Use addr as sender address. See Sender Address.
- Do not use ESMTP authentication. See #TLS and ESMTP Authentication.
- Do not use TLS, even if the SMTP server supports it. See #TLS and ESMTP Authentication.
- Use the supplied header names to determine recipient addresses.
- See Mail Recipients. When no values are supplied, disables header scanning.
- Strip domain part from all recipient addresses. See Mail Recipients.
- Deliver messages to the supplied addresses. See Mail Recipients.
Mailutils Ticket File
If, when trying to authenticate to a remote server, movemail sees that the corresponding URL does not provide required credentials, it will try to obtain them from a ticket file.
Ticket file is a plain text file, located in the home directory of the current user, and named .mu-tickets. It is formatted as a usual UNIX line-oriented configuration file, i.e. comments are introduced by a # sign and extend to the end of the line on which this sign appears, emtpy lines and comments are ignored. Each remaining line must be formatted as a URL, which contains at least one of username and password parts. We call such an URL a ticket.
When scanning this file, movemail looks for a ticket that matches the actual URL most closely. The username and password from that ticket are then used for authentication. The following URL parts are considered when matching: scheme, user, host and port. If a part consists of a single wildcard (*), it matches any value in the corresponding part of the actual URL. Otherwise the parts are compared as follows:
- scheme and host use case-insensitve comparison.
- user uses case-sensitive comparison.
- port is compared for numeric equality.
A missing part in the ticket entry is treated exactly as if a wildcard were present instead.
For example, the URL smtp://gnu.org matches the following ticket entries:
smtp://foo:bar@* smtp://bar:firstname.lastname@example.org *://baz:qux@* *://quux:email@example.com
If several tickets match the URL, movemail selects the best match by using the following algorithm. Each matching entry is assigned a penalty value computed as a sum of individual penalties for each part of the URL. The individual penalty is 0 if the ticket part equals the corresponding URL part. Otherwise, if this part is missing either from the ticket or from the URL, or if the ticket part is represented by a wildcard, the penalty value is 4 for user part, 3 for scheme part, 2 for host part and 1 for port part. Once penalty values are computed, movemail selects the matching ticket with the lowest penalty value.
In the example above, the best match is smtp://bar:firstname.lastname@example.org.
If some of the credentials is still missing after consulting the ticket file, movemail will ask the user for the missing credential (this is possible, of course, only if the controlling terminal is a tty).
Consider this example:
$ movemail pop://example.net smtp://gray@localhost Pop user: gray # The user types in his username Pop password: # The user types the password for his POP account at example.net SMTP password: # The user types the password for SMTP at localhost
The syntax for POP3 URLs is similar to SMTP:
The hostname gives the name or IP address of the host running a POP3 server. Optional port is used to connect to a port other than the default 110.
The user and pass supply authentication credentials. If any of them is missing, movemail first tries to obtain it from the ticket file. It that fails, its behavior depends on the type of the controlling terminal. If the terminal is a tty device (i.e. the program was started from the command line), it will ask the user to supply the missing credentials. Otherwise it will issue an appropriate error message and terminate.
By default, the usual POP3 authentication is used. The auth=+APOP authentication parameter instructs movemail to use APOP authentication instead.
If the server offers the STLS capability, movemail attempts to establish encrypted TLS connection. To disable this behavior, use the notls parameter.
The deprecated POP3S protocol is also supported. It is enabled by an URL beginning with pops:// instead of pop://. The URL syntax is similar to the above:
Unless port is given, the default value 995 is used. The notls parameter is not supported, since POP3S goes over an already encrypted channel.
For example, to fetch mail from your gmail account and submit it to the local MSA, use:
movemail pops://rname:email@example.com 'smtp://localhost:587;to=lname'
Replace rname and pass with your user name and password on gmail and lname with your local name.
Various errors can occur during retrieval of each particular messages as well as during injecting it into the MTA queue. When an error occurs, movemail normally stops any further processing and terminates, without deleting already moved messages. Although such a behavior is a safest way, there may be cases when it is not desired. Therefore, a special configuration option is provided, which controls the movemail actions when an error occurs while trying to append the message to the destination mailbox:
Each keyword is one of the following:
- Abort any further processing, reset source mailbox to its original state, and terminate with code 1. This is the default behavior. No other keyword can be given together with abort.
- Skip this message and try to move the next one. None of the messages is deleted before closing the mailbox. Movemail terminates with code 1.
- Count this message as successfully processed, but do not mark it for deletion. Movemail expunges the successfully processed messages from the source mailbox and terminates with code 0.
- Mark the message for deletion, and continue processing. Before exiting, movemail will expunge both successfully copied messages and failed ones. It will, however, terminate with code 1, unless count is also given.
These keywords can also be given from the command line, using the --onerror command line option.