FAQ

Duplicate delivery supression is a feature of the Cyrus IMAP server that allows multiple copies of an identical messages delivered to one user to be supressed, so that the user only recieves one copy. This can be conveinent if, say, one user is on multiple mailing lists that commonly recieve the same message.

The way that a message is determined to be a duplicate is a lookup is done in the duplicate delivery database for a message-id/mailbox pair. If a match is found, the message is supressed. If a match is not found, the pair is added to the database and the message is delivered.

Duplicate Delivery Supression can also affect sieve redirects. In this case, supression is done on a message-id/redirect-target basis.

Duplicate Delivery Supression can also affect vacation messages. In this case, supression is done based on a hash of the sender's address and the vacation message.

Here's a brief summary of the three major database types:

• Berkeley DB: Slow enumeration, fast random access, fast write, binary support. However, it has proved to be somewhat unstable and prone to locking problems.
• Berkeley DB (no sync): Slow enumeration, fast random access, very fast write, binary support, but no guarantee of database durability; recent writes can be lost on crashes.
• Skiplist: Proprietary Cyrus Format, fast enumeration, moderately fast write, moderately fast random access, binary support
• Flat: Easy to maintain format, fast enumeration, very slow write, moderate random access, no binary support

The default database backend for each database is the type currently recommended by the Cyrus developers. Please reference the imapd.conf(5) manpage for your version of Cyrus imapd to see what the defaults are.

When trying to compile Cyrus IMAPd on Red Hat Linux or Fedora, you may be encountering errors like these:

In file included from /usr/include/openssl/ssl.h:179,
                from prot.h:56,
                from prot.c:72:
/usr/include/openssl/kssl.h:72:18: krb5.h: No such file or directory
In file included from /usr/include/openssl/ssl.h:179,
                from prot.h:56,
                 from prot.c:72:
/usr/include/openssl/kssl.h:134: syntax error before "krb5_enctype"
/usr/include/openssl/kssl.h:136: syntax error before '*' token
/usr/include/openssl/kssl.h:137: syntax error before '}' token
/usr/include/openssl/kssl.h:149: syntax error before "kssl_ctx_setstring"
/usr/include/openssl/kssl.h:149: syntax error before '*' token
/usr/include/openssl/kssl.h:150: syntax error before '*' token
/usr/include/openssl/kssl.h:151: syntax error before '*' token
/usr/include/openssl/kssl.h:151: syntax error before '*' token
/usr/include/openssl/kssl.h:152: syntax error before '*' token
/usr/include/openssl/kssl.h:153: syntax error before "kssl_ctx_setprinc"
/usr/include/openssl/kssl.h:153: syntax error before '*' token
/usr/include/openssl/kssl.h:155: syntax error before "kssl_cget_tkt"
/usr/include/openssl/kssl.h:155: syntax error before '*' token
/usr/include/openssl/kssl.h:157: syntax error before "kssl_sget_tkt"
/usr/include/openssl/kssl.h:157: syntax error before '*' token
/usr/include/openssl/kssl.h:159: syntax error before "kssl_ctx_setkey"
/usr/include/openssl/kssl.h:159: syntax error before '*' token
/usr/include/openssl/kssl.h:161: syntax error before "context"
/usr/include/openssl/kssl.h:162: syntax error before
"kssl_build_principal_2"
/usr/include/openssl/kssl.h:162: syntax error before "context"
/usr/include/openssl/kssl.h:165: syntax error before "kssl_validate_times"
/usr/include/openssl/kssl.h:165: syntax error before "atime"
/usr/include/openssl/kssl.h:167: syntax error before "kssl_check_authent"
/usr/include/openssl/kssl.h:167: syntax error before '*' token
/usr/include/openssl/kssl.h:169: syntax error before "enctype"
In file included from prot.h:56,
                from prot.c:72:
/usr/include/openssl/ssl.h:909: syntax error before "KSSL_CTX"
/usr/include/openssl/ssl.h:931: syntax error before '}' token
make[1]: *** [prot.o] Error 1
make[1]: Leaving directory `/opt/cyrus-imapd-2.2.3/lib'
make: *** [all] Error 1

The key bit of this error is:

/usr/include/openssl/kssl.h:72:18: krb5.h: No such file or directory

Essentially, what is happening is that Cyrus is trying to use OpenSSL, and OpenSSL's headers include some Kerberos header files. Red Hat, in their infinite wisdom, chose to place the Kerberos headers outside the normal header search path, despite having built OpenSSL to require the Kerberos headers. As a result, even if your program is not using Kerberos, it may fail to compile if gcc can't find the Kerberos headers.

The solution is to either add the Kerberos headers to gcc's header search path, or prevent OpenSSL from trying to use the Kerberos includes in the first place. To tell OpenSSL you really don't want kerberos, just run

export LOCALDEFS="-DOPENSSL_NO_KRB5"

(as suggested by Ken Murchison on info-cyrus) before you run ./configure. Alternately, you can tell gcc where to find the Kerberos includes so that it'll stop complaining:

export C_INCLUDE_PATH="/usr/kerberos/include"

If neither of these work, make sure you have the Kerberos development libraries installed ( you should have if you have openssl-devel, but one never does know ...). If you run rpm -q openssl-devel krb5-devel you should get a result like:

openssl-devel-0.9.7a-23
krb5-devel-1.3.1-6

Modern Redhat Linux ships with a Berkeley DB compiled --with-pthreads, which cyrus is unable to directly link against.

You can either recompile Berkeley DB without pthread support, or you can add "-lpthread" to your LDFLAGS enviornment variable before you configure/build.

Currently, the Cyrus project only distributes Cyrus IMAPd and Cyrus SASL as source code. If you prefer not to work directly with the source distribution - you don't have the time to learn it, find it too difficult, or it's otherwise unsuitable - there are a few other options.

There are a number of projects outside the main Cyrus project that provide easy-to-install packages of the Cyrus software. Additionally, there are a fair few commercial software packages that include the Cyrus software as a component. A partial list of these can be found on the ExternalLinks page.

Most people active on the info-cyrus mailing list appear to use the source distribution, so more people are likely to be familiar with that than any vendor-specific custom versions. Additionally, it can be easier to upgrade to the latest version if you're using the source packages. Finally - you learn more! In the end, the best way to get Cyrus depends on what your specific needs are.

Given that you have a username john@domain.tld, unless you escape the '@' as explained below, cyrus deliver will claim that the mailbox does not exist.

instead of

• deliver john@domain.tld

you should be using

• deliver john\\\@domain.tld

and instead of

• deliver -a john@domain.tld -m user/john@domain.tld

you should be using

• deliver -a john\@domain.tld -m user/john\\\@domain.tld

As far as the '@' is concerned, I believe the same applies while not using the unixhierarchysep.

keywords: cyrdeliver master.cf

If you have more than one imapd-service configured in cyrus.conf then they must have distinct names (the first argument in each line). Otherweise a deadlock can occur because the services will try to use the same lockfile, whose filename is derived from the name you gave to the service. Certain characters, while not strictly forbidden, should not be used in these names, because they will cause the name to be truncated. For example, "imap" and "imap_remote" will result in the same lockfile (/var/imapd/socket/imap-0.lock). It is best to only use letters and digits.

Cyrus Sieve

The Cyrus Murder clustering solution drasticly increases system-wide reliability as compared to a traditional standalone IMAP server. However, this configuration is not a true High Availability/Redundancy solution (nor is such a beast trivial to implement given the constraints on traditional Email systems).

The frontends of the CyrusMurder are, in every sense, 100% redundant. No server-specific data is maintained on them, and one is as good as any other from the point of view of a user.

The mupdate server is indeed a single point of failure, however a failure at this point only prohibits message delivery (via the LMTP proxies that are querying it for up-to-date mailbox data) and certain mailbox operations (create, delete, setacl, and so on). Mail can still be accessed by all users of the system, and incoming mail will recieve temporary failures, so the sending MTA should continue to queue the mail.

The failure of a backend will result in a partial systemwide failure -- the mailboxes that were stored on that backend will no longer be available for use by any users. Delivery of new mail will again be tempfailed, and queued by the sending MTA.

Some people have suggested shared storage and the use of hot spare backends to combat this solution. The Cyrus authors do not generally recommend the use of shared storage in the Cyrus enviornment. However people have reported success in various configurations. The shared filesystem still needs to support file locking, and it is highly recommended that it have an efficient mmap implementation.

The short answer is no, there isn't any logging based on the size of message retrieved or when a specific message is retrieved. Patches are welcome.

The altnamespace setting in /etc/imapd.conf changes the way the mailbox list is presented to users (including fileinto directives in Sieve scripts). Instead of appearing as subfolders of INBOX, altnamespace makes subfolders appear at the same level as INBOX. It does not affect the internal storage nor the way the folders are presented through cyradm. See the man page for imapd.conf(5) for more information.

Use cyradm

Having a large deliver.db (on the order of tens of megabytes) is not uncommon, since a record is kept of every message that goes through lmtpd (so that duplicate supression can do its work), so there's a lot of data there. Note also that it is common for databases to have unused space that may not be immediately recovered for performance reasons (this is the case with Berkeley DB and CyrusSkiplist) You can control the size by increasing the frequency at which it gets pruned.

Berkeley DB maintains a log of all the transactions since the last checkpoint of the database. In order to ensure the database is in a consistent state, you must recover the log after any outage (thus the recommendation to run these processes when cyrus starts). They can take a long time for a few reasons.

The most common one is that you need to checkpoint the cyrusdb more often. This can be done with a simple ctl_cyrusdb -c If you do this very often, the amount of log that needs to be recovered will be significantly shorter. We recommend doing this at least once every half hour, and more often on busy sites.

The other reason is that your deliver.db may be very large. This is solvable by increasing the pruning interval (the -E parameter to ctl_deliver, which you should run on a regular basis), or (in a pinch) by just removing the database (since the effects of losing it do not prevent operation, they just cause vacation messages to be resent, and duplicate delivery supression to possibly deliver duplicates).

• "by increasing the pruning interval": My understanding is that the number after "-E" is the number of days after which entries are discarded. Is there a way to reduce it to a number of hours? Since most of our mail is internal mail should rarely be delayed by more then an hour or two.
• In case it's useful to anyone we discovered that moving /var/lib/imap from an ext2 to an ext3 journaled filesystem made a vast difference for the worse. While recovering the database Berkeley DB does a vast quantity of small writes and that combined with the updates to the journal absolutely kills disk performance (with journalling it was taking about 40 minutes to start Cyrus on a mail server with about 500 users and 200G of mail). On the flip side moving /var/lib/imap to a hardware RAID system with a decent amount of onboard cache reduced this time to under 30 seconds. I think Berkeley DB could probably be optimized to deal with this better, but in the mean time avoid journaling filesystems, or at least be prepared to experiment to find something that works for you.

This error is caused if you build Berkley DB 4 with threading support, but do not then link the pthreads library to Cyrus. This is commonly caused by a Linux distribution which supplies the pre-threaded library version.

One workaround is to rebuild the db4 library without thread support. Another is provided by Scott Adkins (adkinss@ohio.edu):

• I talked to SleepyCat about it and they suggested making sure that the threads library was being linked into the IMAP server. It used to be that when compiling db3 that the IMAP server would automatically link in the threads library, probably because of an rpath listed in the db3 shared library. They must have changed something with db4, since the threads library is no longer linked into applications if they link against the db4 library.
  As it turns out, linking with threads (I believe I simply included the -pthread CC command line option to have that happen) caused the join errors to go away. I was still having some flakiness with db4 though...

Starting with Cyrus 2.2.1, it is possible to build without linking Berkeley DB at all, which may be a viable option for some sites.

Because many of the Sieve features are autoresponders, Cyrus tries to be a good network citizen by adhering to the appropriate RFCs:

From RFC 2821:

This notification message must be from the SMTP server at the relay host or the host that first determines that delivery cannot be accomplished. Of course, SMTP servers MUST NOT send notification messages about problems transporting notification messages. One way to prevent loops in error reporting is to specify a null reverse-path in the MAIL command of a notification message. When such a message is transmitted the reverse-path MUST be set to null (see section 1.5.5 for additional discussion).

A MAIL command with a null reverse-path appears as follows:

• MAIL FROM:<>

Also from draft-moore-auto-email-response-02.txt:

The primary purpose of the MAIL FROM address is to serve as the destination for delivery status messages and other automatic responses. Since in most cases it is not appropriate to respond to an automatic response, and the responder is not interested in delivery status messages, a MAIL FROM address of <> MAY be used for this purpose.

This is of course slightly more applicable to reject than it is to vacation, but there isn't a very strong argument why they should be treated differently.

Note that draft-moore-auto-email-response-02.txt also states:

A MAIL FROM address which is specifically chosen for the purpose of sending automatic responses, and which will not automatically respond to any message sent to it, MAY be used instead of <>.

Therefore, it may be reasonable to have this be a configurable fixed address in the future.

This message is generally not harmful unless it is increasing at an alarming rate or you are seeing other performance problems. It indicates contention within a Berkeley DB database page, and especially on small databases will occur quite frequently.

If there are performance problems, often times the best solution is to convert your mailbox list to a skiplist format.

Paul M Fleming (pfleming@siumed.edu) also notes:

• We found limiting the number of lmtp processes to 10 helped with locker problems. Let sendmail/postfix handle the queuing and lmtp the delivery.. When we tried to blast 10-20 messages/sec at our test server locking became a problem with the duplicate delivery db. Across 3 servers and 600 concurrent logins (lmtpd maxchild=10 on each server) we never see any locking problems and usually have <1 second delivery times..

Andreas (andreas@conectiva.com.br) writes:

• Take a look at this text: http://www.openldap.org/faq/index.cgi?_highlightWords=locks&file=893
  It's written for openldap, but explains several important Berkeley DB configuration parameters. In particular:
  • On a very busy system you might see error messages talking about running out of locks, lockers, or lock objects. Usually the default values are plenty, and in older versions of the BDB library the errors were more likely due to library bugs than actual system load. However, it is possible that you have actually run out of lock resources due to heavy system usage. If this happens, you should read about the set_lk_max_lockers[1], set_lk_max_locks[2], and set_lk_max_objects[3] keywords.

• http://www.sleepycat.com/docs/api_c/env_set_lk_max_lockers.html
• http://www.sleepycat.com/docs/api_c/env_set_lk_max_locks.html
• http://www.sleepycat.com/docs/api_c/env_set_lk_max_objects.html

Update, 2007/07/09: They've moved since sleepycat got bought by Oracle (did they really I wonder, I heard something about this, and their site seems to have moved....) Anyway, I found the pages listed above:

• http://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/C/envset_lk_max_lockers.html
• http://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/C/envset_lk_max_locks.html
• http://www.oracle.com/technology/documentation/berkeley-db/db/api_reference/C/envset_lk_max_objects.html

The links above talk about the C api, but don't get alarmed, these paremeters can be set with a DB_CONFIG configuration file located in the DB environment home.

I also suppose the db_stat utility can be used to diagnose this.

By default, the permissions on newly created mailboxes do not allow access to any other user. Therefore even root user can not delete mailbox.

To delete a mailbox, you need to give appropriate rights with the command "sam user.mailbox username all"

after that you can delete the mailbox. Even just giving the user the "c" right (Create and Delete mailbox) is enough.

From the reconstruct man page:

−m     NOTE: CURRENTLY UNAVAILABLE
             Rebuild  the  mailboxes file.  Use whatever data in the existing
             mailboxes file it can scavenge, then scans all partitions listed
             in the imapd.conf(5) file for additional mailboxes.

Reconstruct is currently unable to rebuild the mailboxes db, and comments on the mailing list indicate this ability will not be added - at least, any time soon.

The solution here is to make sure that you back up your mailboxes db, preferably including a plain-text copy of it. Please see Backups for more information.

If you do find yourself with a corrupted mailboxes.db, there are a few things you can try. The first is to see if db_recover can recover your database. If that doesnt' work, there should be backups in $CONFIGDIRECTORY/db.backup1 and $CONFIGDIRECTORY/db.backup2 that may be OK.

If you're using Simon Matter's RPMs, plain-text copies of the mailboxes database should be being generated and saved in /var/lib/imap/backup . Try rebuilding the db from one of those using ctl_mboxlist. Alternately, try asking the mailing list for help.

> I'm installing Cyrus on a ssytem that will have access to an IBM
> FAStT SAN with GPFS (a parallel filesystem allowing multiple servers
> to share a filesystem on a SAN).
...
> Has anyone done this before?

2 x p630 [backends]
 4 x 1.4GHz Power4+ CPU
 8GB Real Memory

4 x p615 [frontends]
 2 x 1.2GHz Pwer4+ CPU
 4GB Real Memory

[1] GPFS: using CVSD then RVSD in our SP
[2] crit sit: Critical Situation: IBM's tool for managing
   barely-tenable customer relationship situations
[3] Something like 90% of our PMRs resulted in code changes
[4] In typical IBM fasion, they implemented *exactly* the POSIX
   specification, and not an penny more. I'm not convinced that this
   is bad, but it bites me a lot.
[5] High Availability Cluster Multi-Processing(tm), IBM

In short, we don't recommend it. If you want to do it, it may possibly work but you may also lose your email or have corrupted cyrus.* files. You can look at the mailing list archives for more information.

There are several critical things that must be supported for this to work: * file locking * mmap() * writing to an mmap()'d file with the write() syscall In general, this is bad news. Don't do it. Really. About the closest you can currently get is having a "warm spare" that takes over the IP of the affected backend, and have a SAN between the two systems.

Cyrus Murder may be a "good enough" solution in some enviornments, as a way to partition failures and spread load across many machines. Combined with a warm-spare approach this can be a good way to achieve highly available systems (though not ones which are fully redundant)

Using a database as the main message store for Cyrus is not currently supported. According to discussions on the mailing list, there is little benefit in adding such support (given typical IMAP access patterns, optimizations in the current mail store that make it fast, and the amount of effort involved to retrofit a different mailstore into the backend), so it's unlikely to ever be written.

The Cyrus mail store is a normal directory tree, with mailboxes stored as directories and messages stored as individual files. Some additional information, eg index data, is kept in the "cyrus.*" files in the mailboxes.

Cyrus DOES have the option of using databases of various types for storing some other information, such as authentication data, mailbox lists, etc. There may be reasons to add a SQL backend to these databases in the future (in addition to flat, skiplist, and berkeley).

If you are using a newer and standard compliant MTA to deliver mail to Cyrus IMAP's LMTP agent you may run into a problem where mail will be bounced and not delivered with errors like this in your maillog..

Feb 29 10:01:02 myhost lmtpunix[12345]: append_check() of 'MyDomain.tld!user.MyUser' failed (Mailbox does not exist)

This is due to LMTP adhering to the RFC standards and the MTA adhering to the RFC standards of preserving the case in the e-mail address. so User@MyDomain.tld is not the same as user@mydomain.tld. And this causes issues if with users who alter the case of their e-mail address when signing up for mailing lists, or sending e-mail.

However, all is not lost there is a solution. Cyrus IMAP has an option that can be added into the imapd.conf..

lmtp_downcase_rcpt: true

Inserting that into your imapd.conf and restarting cyrus imap will cause cyrus to "downcase" the e-mail address up to the recipient delimiter of +. The result will be this..

• User@MyDomain.tld -> user@mydomain.tld
• user@mydomain.tld -> user@mydomain.tld
• User+PostFolder@MyDomain.tld -> user+PostFolder@MyDomain.tld
  • someone verify that the domain will actually be downcased when a + is used

Note that this option first appeared in Cyrus IMAP 2.1.14.

In some configurations POP3 needs to generate random data for use by APOP authentications. This can lead to delays if /dev/random does not have enough entropy immediately available.

If you don't need or want support for APOP, you can either compile SASL with --disable-checkapop, or you can disable APOP support in pop3[proxy]d at runtime by setting allowapop:0 in imapd.conf.

Otherwise, since there isn't a strong need for high-quality random numbers with SASL, there are two options for dealing with this -- link /dev/urandom to /dev/random. Alternatively, recompile SASL with --with-devrandom=/dev/urandom. It's prefered to do the later, as it avoids affecting other applications.

Cyrus annotations are based on a draft (http://tools.ietf.org/html/draft-daboo-imap-annotatemore-08) version of RFC 5464.

/admin - Sets the administrator email address for the server. (See cyradm(1))

/check - Boolean value "true" or "false" that indicates whether this mailbox should be checked at regular intervals by the client. The interval in minutes is specified by the

/checkperiod entry. (Draft RFC)

/checkperiod - Numeric value indicating a period of minutes that the client uses to determine the interval of regular 'new mail' checks for the corresponding mailbox. (Draft RFC)

/comment - Sets a comment or description associated with the mailbox. (cyradm(1)) /motd - Sets a "message of the day". The message gets displayed as an ALERT after authentication.

/sort - Defines the default sort criteria [I-D.ietf-imapext-sort] to use when first displaying the mailbox contents to the user, or NIL if sorting is not required. (Draft RFC)

/thread - Defines the default thread criteria [I-D.ietf-imapext-sort] to use when first displaying the mailbox contents to the user, or NIL if threading is not required. If both sort and thread are not NIL, then threading should take precedence over sorting. (Draft RFC)

/vendor/cmu/cyrus-imapd/condstore - Enables the IMAP CONDSTORE extension (modification sequences) on the mailbox. (cyradm(1))

/vendor/cmu/cyrus-imapd/duplicatedeliver - Flag signalling that we're allowing duplicate delivery of messages to the mailbox, overriding system-wide duplicate suppression. (http://cyrusimap.web.cmu.edu/imapd/internal/mailbox-format.html)

/vendor/cmu/cyrus-imapd/expire - Sets the number of days after which messages will be expired from the mailbox. (cyradm(1))

/vendor/cmu/cyrus-imapd/freespace - Undocumented.

/vendor/cmu/cyrus-imapd/lastpop - (time_t) of the last pop3 login to this INBOX, used to enforce the "poptimeout" imapd.conf option. (http://cyrusimap.web.cmu.edu/imapd/internal/mailbox-format.html)

/vendor/cmu/cyrus-imapd/lastupdate - (time_t) of the last time a message was appended (http://cyrusimap.web.cmu.edu/imapd/internal/mailbox-format.html)

/vendor/cmu/cyrus-imapd/news2mail - Sets an email address to which messages injected into the server via NNTP will be sent. (cyradm(1))

/vendor/cmu/cyrus-imapd/partition - Undocumented.

/vendor/cmu/cyrus-imapd/pop3newuidl - Flag signalling that we're using "uidvalidity.uid" instead of just "uid" for the output of the POP3 UIDL command. (http://cyrusimap.web.cmu.edu/imapd/internal/mailbox-format.html)

/vendor/cmu/cyrus-imapd/server - Undocumented.

/vendor/cmu/cyrus-imapd/sharedseen - Enables the use of a shared \Seen flag on messages rather than a per-user \Seen flag. The ’s’ right in the mailbox ACL still controls whether a user can set the shared \Seen flag. (cyradm(1))

/vendor/cmu/cyrus-imapd/shutdown - Sets a shutdown message. The message gets displayed as an ALERT and all users are disconnected from the server (subsequent logins are disallowed). (cyradm(1))

/vendor/cmu/cyrus-imapd/sieve - Indicates the name of the global sieve script that should be run when a message is delivered to the shared mailbox (not used for personal mailboxes). (cyradm(1))

/vendor/cmu/cyrus-imapd/size - Undocumented.

/vendor/cmu/cyrus-imapd/squat - Indicates that the mailbox should have a squat index created for it. (squatter(8))

It's safe to put both of these in tmpfs, and since they're fairly busy it's a good idea for performance reasons. Further, nothing ever removes files from <configdirectory>/lock/, so it's probably a good idea in general to mount that on tmpfs such that a reboot will purge it. The Cyrus code will create all the necessary directory structure under <configdirectory>/lock on-demand.

8-bit characters are not allowed in the headers of an RFC 822 message.

We're not about to consider a patch to "fix" the problem of replacing 8-bit characters with 'X's that doesn't atleast supply a default character set and properly QP-encode the nonconforming header.

Another possiblity is suggested by Chris Newman:

• The correct long term thing to do is to interpret unlabelled 8-bit as UTF-8 if it meets the UTF-8 syntax, and otherwise give it the "unknown" charset label and downconvert to 7-bit using RFC 2047. If you want to do something really fancy, you might allow a mapping from the hostname in the envelope from address to a default 8-bit charset (Innosoft's MTA includes an equivalent facility) so the administrator can set up private agreements with specific hosts.

Please first read about Cyrus Interoperability.

Bare newlines is a nono in an RFC822 message. You should first try to fix the software that is causing the problem.

As it happens, the bare newlines-rejection has never been a problem for us.

-- RobSiemborski - 24 Jul 2003

I have encountered this issue when migrating mail from other mail systems, such as MDaemon for WinNT? , that may not be so fussy about RFC822 compliance. It hasn't been an issue for us post-migration however.

-- Craig Ringer, 5 Nov 2003

Well, the mail client of an American software vendor, infamous because of its operating system and its behaviour against open standards, seems quite often to use bare newlines, especially for binary attachments. Unfortunately they won't give me the sourcecode so I can't fix it. My question for you: Is there a patch, or even better a setting to suppress the bare newlines check? Any information here is very appreciated!

-- Karsten Gresch, 25 Oct 2005

Please first read about Cyrus Interoperability.

Note that this does not apply to Cyrus versions starting with 2.1.14 (the interoperability problem was solved). Note that due to the unique nature of this problem the later versions have a rfc2046_strict ImapdConf? option which will restore the old behavior.

RFC 2046 has some text which, if interpreted literally, does not allow outer mime boundrys to be substrings of inner mime boundries. Eudora, as it happens, creates messages which have just that format. The problem is that earlier versions of the MIME spec did not contain this restriction, so 2046 is not backwards compatible (in a literal interpretation).

After discussions with other implementors within the IETF, it was decided that Eudora's behavior was clearly broken, Cyrus's behavior was unfriendly to interoperability but technically correct, and RFC 2046 probably needs a clarification. Thus, we corrected the behavior in Cyrus with an option to revert to the strict behavior if a site so desires.

This problem results from Cyrus using CRLF sequences to denote line breaks in its pipe to the sendmail process, which normal Sendmail handles just fine. However, exim does not handle these sequences ok. Exim 4.20 and later has a drop_cr option which you can use, the following wrapper script was suggested by Bernhard Erdmann (be@berdmann.de):

- /etc/imapd.conf:
sendmail: /opt/exim/exim_dropcr

- /opt/exim/exim_dropcr:
#!/bin/sh
/opt/exim/exim -dropcr $@

There are a number of things that can affect mail delivery. Here are some common causes. 1. Murder Master is overloaded. In a normal setup the murder master is queried for each incoming messages 2. Delivery.db corruption. A backend will/may reject mail if the delivery.db is corrupt 3. Slow dns. Some MTAs (like sendmail) talk to dns a lot. If dns is slow, mail delivery appears slow. We run named on each Cyrus machine 4. Lmtpproxyd is having trouble. Sometimes lmtpproxyd will die and dissapear. Sometimes it just hangs. Not sure why, but restarting may help.

If you have a large volume of mail then Cyrus Murder Mail Delivery has some tips.

We experienced some filesystem locking problems when allowing to many lmtpd-processes to be started. Performance was seriously degraded, resulting in timeouts and endless retries. We use a postfix instance between to limit parallel deliveries to lmtpd resulting in much higher throughput. With berkeley-nosync as duplicate suppression db, 50-100 Messages/sec are doable.

Saslauthd is only capabile of verifying plaintext passwords (it takes a plaintext password and a username and responds with "yes" or "no", essentially). Therefore, since the plaintext password isn't passed from client to server in DIGEST-MD5 and CRAM-MD5, Saslauthd can't verify the password.

Authentication in a CyrusSaslauthd? -only enviornment will not only fail with these mechanisms, it doesn't really make a lot of sense. You'll want to use an AuxpropPlugin? instead (for example, sasldb).

To operate with the CRAM-MD5 and DIGEST-MD5 mechanisms, Cyrus SASL stores plaintext versions of the passwords in its secret database (an AuxpropPlugin? ).

This is typically regarded as insecure practice, however the alternative is not much better. For CRAM-MD5 and DIGEST-MD5 to function, they must have a plaintext equivalent locally in order to confirm the hash that actually goes across a wire. This, if these equivalents were compromised, it is trivially easy for an attacker to have access to any account on the system.

Note that for DIGEST-MD5 this isn't strictly true: the hash that DIGEST can use limits the attack to only the realm for which the password applies, but this is a questionable security gain for the increased management hassles (you can't share them between mechanisms) that the plaintext equivalents cause.

Authentication is the act of proving who you are. "Hello, I'm Dave. To prove it, here's my password: Foo." Authorization is the act of deciding whether to grant access to resources.

• "I'd like to read Kellie's mail for her."

In the example, I'm trying to read my wife's mail. I supply my own username as the *"authentication identifier", my own password (Or biometric scan, or whatever else is required to prove I'm really me with whichever mechanism is in use), and my wife's username as the "authorization identifier".

At no point need I know my wife's password - instead, either Kellie or an administrator needs to explicitly state that I am allowed in "as Kellie". Once I've logged in, all the access checks are done against Kellie, not against Dave, because I'm acting for her. To all intents and purposes, after the authentication exchange itself, the server can simply forget about who authenticated - it's not important anymore - and concentrate on who needs to be authorized.

Another, more common example of the use of differing authentication identifiers and authorization identifiers is in the design of many proxy systems. You authenticate perfectly normally to the proxy, authorizing as yourself. The proxy then authenticates to the master as itself, but supplies you as the authorization identifier, thus getting all the right access checks done at source, but not having to have access to your authentication credentials. Finally, some mechanisms don't support passing a distinct authorization identifier, and for most its optional, and defaults to the case that most people are familiar with, where authorization and authentication identifiers are the same.

The migration is documented in the official documentation.

When migrating the /etc/sasldb database using the utils/dbconverter-2 utility, you may encounter the error message "Error opening password file". This is usually due to the fact your SASL V1 library was compiled using a different version of Berkeley DB than the SASL V2 library. You can work around this by using Berkeley DB's db_upgrade utility (possibly chaining the DB3 and DB4 upgrade utilities) to upgrade a copy of sasldb prior to conversion using dbconverter-2.

Here is the script we use at our installation, where SASL has to coexist with SASL2:

#!/bin/sh
cp /etc/sasldb /tmp/sasldb.$$
/usr/local/BerkeleyDB.4/bin/db_upgrade /etc/sasldb
echo ""|/usr/local/sasl/sbin/dbconverter-2
cp /tmp/sasldb.$$ /etc/sasldb

This article assumes that you have read and followed the SASL chapter of the OpenLDAP Administrator's Guide [1]. You should have a Kerberos server installed (such as Heimdal or MIT), and created all the appropriate principals (client and service) necessary.

To verify that you have the Cyrus GSSAPI mechanism properly installed, use the pluginviewer command. For instance:

server:~# pluginviewer  | grep -i gssapi
 CRAM-MD5 PLAIN NTLM GSSAPI OTP DIGEST-MD5 ANONYMOUS LOGIN EXTERNAL 
Plugin "gssapiv2" [loaded],     API version: 4         
       SASL mechanism: GSSAPI, best SSF: 56, supports setpass: no 
CRAM-MD5 PLAIN NTLM GSSAPI OTP DIGEST-MD5 ANONYMOUS LOGIN EXTERNAL 
Plugin "gssapiv2" [loaded],     API version: 4         
       SASL mechanism: GSSAPI, best SSF: 56 

Both your server and client systems will need to have this mechanism installed. If not, you may find the mechanism located in a binary package that you do not yet have installed, or you may need to recompile your Cyrus SASL installation.

On your client system, search the Root DSE of the server to view advertised mechanisms:

client:~# ldapsearch -LLL -x -H ldap://ldap.example.org -s "base" -b "" supportedSASLMechanisms 
dn: 
supportedSASLMechanisms: DIGEST-MD5 
supportedSASLMechanisms: NTLM 
supportedSASLMechanisms: GSSAPI 
supportedSASLMechanisms: OTP 
supportedSASLMechanisms: CRAM-MD5

If you received a No Such Object error, you may have an ACL misconfiguration on your server. See [3].

If you do not see GSSAPI listed, verify that the server can read the appropriate keytab. The default is /etc/krb5.keytab and is typically only readable by the root user. If your Kerberos library supports it, you can create a keytab in an alternate location, such as /etc/krb5.keytab-ldap, with only your LDAP service principal, and read permissions for your OpenLDAP user.

If your OpenLDAP server is looking for an unexpected principal within your keytab, use sasl-host and sasl-realm to influence which principal it will use (see the slapd.conf man page).

For more control over how the SASL library operates within the OpenLDAP? server, you can create a slapd.conf SASL configuration. This is not the same configuration file as the OpenLDAP configuration (slapd.conf). The location of this file is dependent how on Cyrus SASL was compiled (via the --with-plugindir option during configure). It is the directory where your plugins are installed.

For instance, if you create /usr/lib/sasl2/slapd.conf (assuming that is the correct location on your system) with the following contents:

keytab: /etc/krb5.keytab-ldap 
mech_list: CRAM-MD5 DIGEST-MD5 GSSAPI 

then the server will search within /etc/krb5.keytab-ldap when initializing the GSSAPI plugin. The server will only offer the mechanisms listed in mech_list. If mech_list is not specified, the server will offer all the mechanisms available, and that it can initialize.

Once you have verified that the server is advertising GSSAPI support, then try:

client:~# ldapsearch -LLL -Y GSSAPI -H ldap://ldap.example.org -s "base" -b "" supportedSASLMechanisms 
SASL/GSSAPI authentication started 
SASL username: host/client.example.org@EXAMPLE.ORG 
SASL SSF: 56 SASL data security layer installed. 
dn: 
supportedSASLMechanisms: DIGEST-MD5 
supportedSASLMechanisms: NTLM 
supportedSASLMechanisms: GSSAPI 
supportedSASLMechanisms: OTP 
supportedSASLMechanisms: CRAM-MD5 

If you receive a list of mechanisms, then congratulations, you're done.

If instead you receive an error, then it's possible that your client is requesting a service principal that mismatches what your server is using.

Verify that you have received a TGT ticket from your kerberos server, and that you have also received a service principal ticket for the server (e.g. ldap/ldap.example.org@EXAMPLE.ORG). Use klist to verify:

client:~# klist
Credentials cache: FILE:/tmp/krb5cc_0
       Principal: host/client.example.org@EXAMPLE.ORG

 Issued           Expires          Principal
Feb 22 11:00:02  Feb 22 21:00:01  krbtgt/EXAMPLE.ORG@EXAMPLE.ORG
Feb 22 11:24:39  Feb 22 21:00:01  ldap/ldap.example.org@EXAMPLE.ORG

If not, then a network capture utility, such as wireshark, offers a good way to show you which service principal your client is requesting, and what errors are being returned by the server.

If you wish to also view the interaction with the LDAP server with a capture utility, you will need to negotiate a SASL layer without encryption. You can specify '-O maxssf=1' in your client side command (ldapwhoami, ldapsearch etc.), e.g.:

client:~# ldapsearch -LLL -Y GSSAPI -H ldap://ldap.example.org -O maxssf=1 -s "base" -b "" supportedSASLMechanisms 

You might find it easier to specify defaults on your client system, to keep your commands shorter. See the man page for ldap.conf for details. You can specify the default server in your ldap.conf file:

URI    ldap://ldap.example.org 

And you can specify the default mechanism in ~/.ldaprc (in your home directory):

ASL_MECH GSSAPI 

References:

1. http://www.openldap.org/doc/admin24/sasl.html
2. https://bugzilla.andrew.cmu.edu/cgi-bin/cvsweb.cgi/~checkout~/src/sasl/doc/gssapi.html?rev=1.4;content-type=text%2Fhtml
3. http://www.openldap.org/doc/admin24/appendix-common-errors.html#ldap_sasl_interactive_bind_s: ..