Cyrus IMAP 2.5.0 Release Notes

Important

Make sure to read the Upgrading to Cyrus IMAP 2.5.0 notes (all of them).

Download from GitHub:

Upgrading to Cyrus IMAP 2.5.0

It is strongly recommended to shut down the Cyrus IMAP services before performing the upgrade, as the newer binaries will end up writing to mailboxes.db in a way that is not compatible with the older binaries (that would otherwise still be running).

You can start the server immediately after upgrading, however.

Underscores in cmd Names in cyrus.conf(5)

Underscores (the _ character) are no longer allowed in the START, SERVICES and EVENTS sections of cyrus.conf(5), as they interfere with configuration options in imapd.conf(5) being prefixed by service names and an underscore (_) character.

Database Format Upgrade for Mailboxes

Upgrading to Cyrus IMAP 2.5.0 recommends the upgrade of database formats, for which the reconstruct utility has been modified allowing administrators to run:

# /usr/lib/cyrus-imapd/reconstruct -V max [options]

This command can be run at the administrator's convenience, and while running the database format upgrade process is not strictly required, it is certainly strongly recommended.

Releases prior to 2.5.0, namely the 2.4 series, upgraded cyrus.index database formats in-line (i.e. as the mailbox in question as being used). This may have caused an I/O storm in some situations, which 2.5.0 aims to address.

The minor version of the cyrus.index file in 2.5.0 is 13. While mailboxes with an older format will continue to work "forever", newer features such as annotation quotas will fail, as there is no space in the database formats prior to version 13 to store the new feature's information.

Until you upgrade the database format, you may experience slightly inconsistent search results, due to the cyrus.cache format in Cyrus IMAP versions prior to 2.4.0 running words together and not obeying word boundaries (as well as versions in the 2.4 and 2.5 series).

If you were running Cyrus 2.3.x with expunge_mode: delayed, then the contents of the cyrus.expunge file will be removed the first time the mailbox is opened, and the mailbox will act as if expunge_mode was set to immediate -- until it is upgraded.

This is because version 2.3 mailboxes used the cyrus.expunge file, and it's not worth the complexity to try to recreate that file.

This upgrade path supports upgrades from Cyrus IMAP version 2.2.12 and later versions.

Important

The above command can run while the Cyrus IMAP services are in operation, and does not interfere with the service's operations.

Note

Additional options can be specified to reconstruct(8) to more granularly specify which mailboxes' cyrus.index is to be upgraded.

Quota Fixes and Enhancements

After all mailboxes are upgraded, you should run quota -f to make all message and folder counts are correct. The new quota types that Cyrus IMAP 2.5.0 supports require the counts to need to be accurate.

Cyrus IMAP Murder Topologies

Environments that run a Cyrus IMAP Murder topology will want to upgrade their backends before they upgrade their frontends.

Greater Memory Footprint

For those upgrading from 2.3.X; newer releases of Cyrus IMAP will use significantly more memory per selected mailbox. This is not an error or bug; it's a feature. The newer code is holding more data and metadata in memory for purposes of faster access to more of the mailbox. This is not a memory leak.

New Features

Index Namelock Release

Long-running (idling) connections may have previously intervened with mailbox deletions and general cleanup work, resulting in one connection blocking another from, for example, deleting and recreating a mailbox:

1
2
3
C1: SELECT "Foo"
C2: DELETE "Foo"
C2: CREATE "Foo"

The CREATE command on line 3 (by client 2, while client 1 still has the Foo mailbox selected) would have failed before, but does now succeed.

Extended Quota Types

New ways to restrict resource usage:

  • Number of Folders,

  • Number of Messages,

  • Number of Annotations

CalDAV and CardDAV Support

CalDAV and CardDAV support no longer live out-of-tree in the same GIT repository, but are now mainstream and included in Cyrus IMAP 2.5.0.

This has been a major effort by Carnegie Mellon University and FastMail, and continues to be, enriching the experience of Cyrus IMAP users globally.

If upgrading to Cyrus IMAP 2.5 from one of the 2.4.17-caldav-beta releases, you MUST run the dav_reconstruct utility for each of your CalDAV users.

Support for RFC 5464: IMAP METADATA

Cyrus IMAP now fully supports RFC 5464, The IMAP METADATA Extension.

This also means the support for the ANNOTATEMORE draft for IMAP will ultimately be dropped.

Aside from the trusted folder metadata, this also introduces message annotations. Users will need to be given the n right to allow them to set message annotations.

Event Notifications

Various events occuring in Cyrus IMAP, among which mailbox, message and access events, can now be pushed out through a side-channel and notify client applications or provide other infrastructure with detailed information.

Mailbox Distribution Enhancements: Backend and Partition Selection

Thanks to the work of Julien Coloos and colleagues, a new mode is available for server and partition selection upon mailbox creation.

Prior to Cyrus IMAP 2.5.0, the server and/or partition on which to create a new mailbox was selected by detecting the largest amount of absolute free disk space on all servers and partitions. The mailbox distribution feature allows for more intelligent and flexible routines to be used in the selection. Please see our Administrator Guide for more details.

New Database Format for mailboxes.db

The database format for mailboxes.db has been upgraded, adding;

  • A new mailbox type for deleted mailboxes.

    In versions of Cyrus IMAP prior to 2.5.0, mailboxes that were deleted may have become unavailable for actual cleanup expecting another session on the same mailbox to clean up the directories and files. See also Index Namelock Release.

  • A key-value storage format is used, allowing for faster and better parsing of mailboxes.db, more granular updates to runtime environments, and more sustainable future upgrades.

New Database Format twoskip

A new database format has been added, called twoskip 1.

This new database format is reputedly better, faster, safer and 64-bit capable, as opposed to our former favorite skiplist.

twoskip can be used for the following databases:

  • annotation_db

  • duplicate_db

  • mboxkey_db

  • mboxlist_db

  • ptscache_db

  • quota_db

  • seenstate_db

  • subscription_db

  • statuscache_db

  • tls_sessions_db

  • user_deny_db

Miscellaneous

Allowing Undefined Annotations

Cyrus IMAP 2.5.0 allows administrators to configure that undefined annotations should be allowed, using a new annotation_allow_undefined setting in imapd.conf(5).

Catchall Mailbox for LMTP

Thanks to the work by Carsten Hoeger and Ralf Haferkamp, this new feature enables administrators to configure a target mailbox for mail delivered through LMTP to targetted mailboxes that do not exist.

For example, a mail that LMTP would deliver to user/bovik, which for the sake of argument does not exist in this example, setting lmtp_catchall_mailbox to admin will instead deliver the mail to user/admin.

Note

Mailbox name, not Email Address

Note that lmtp_catchall_mailbox must be a user mailbox name, not an email address. Also note that the user/ namespace indicator as well as the hierarchy separator are to be omitted.

Does this impact lmtp_fuzzy_mailbox_match?

Environments that have lmtp_fuzzy_mailbox_match enabled, in order to have LMTP seek from the targetted, non-existent mailbox sub-folder (example: user/bovik/spam/probably) all the way to the toplevel mailbox folder (i.e. user/bovik) until it finds a mailbox (sub-)folder that does exist (example: user/bovik/spam), are not impacted by this setting.

Can the lmtp_catchall_mailbox include the path to a sub-folder of a target mailbox?

UNCONFIRMED

Can the lmtp_catchall_mailbox be a shared folder?

UNCONFIRMED

Callout for SETMETADATA

A callout program can be called when annotations are set, configured through annotation_callout.

Host & User Login Restrictions

Logins can now be restricted on a per host (source IP address) or per user basis, using the settings maxlogins_per_host and maxlogins_per_user.

Configuration Option Changes and Enhancements

Important

While it is not mandatory to update your configuration file with these new settings, not doing so may have undesired side-effects, including but not limited to deprecation warnings in log messages.

Option Name Changes for autocreate

The options related to automatic creation of user mailboxes and sub-folders (aka. autocreate) have been changed to hold a prefix of autocreate_.

The following autocreate options are now available:

autocreate_inbox_folders (was: autocreateinboxfolders)

autocreate_inbox_folders controls which folders to create in addition to the INBOX folder.

Separate the folder names by |.

autocreate_post (was: createonpost)

Controls whether or not to create a folder when a message is first posted to it (by LTMP).

autocreate_quota (was: autocreatequota)

When creating a user mailbox, set the quota for that mailbox to the value of this configuration option.

autocreate_quota_messages (not available)

When creating a user mailbox, set the message quota (maximum number of messages allowed in the folder hierarchy) to the value of this configuration option.

autocreate_sieve_folders (was: autosievefolders)

Limit the folders that can be created automatically by a Sieve script performing a "fileinto" action, to the folders listed in this configuration option.

Separate the folder names by |.

autocreate_sieve_script (unchanged)

When creating a user mailbox, associate the Sieve script configured here.

autocreate_sieve_script_compile (was: generate_compiled_sieve_script)

Whether or not to compile the Sieve script configured by autocreate_sieve_script.

autocreate_sieve_script_compiled (was: autocreate_sieve_compiled_script)

When creating a user mailbox, associate the already compiled Sieve script configured here.

autocreate_subscribe_folders (was: autosubscribeinboxfolders)

List the folder names to which the user for which a mailbox is being created should be subscribed.

Note

All folders listed here are considered to reside in the personal namespace.

Separate the folder names by |.

autocreate_subscribe_sharedfolders (was: autosubscribesharedfolders)

List the folder names of shared folders to which the user for which a mailbox is being automatically created should be subscribed.

Separate the folder names by |.

autocreate_subscribe_sharedfolders_all (was: autosubscribe_all_sharedfolders)

Rather than subscribe the user for which a mailbox is being automatically created to some shared folders, simply subscribe the user to all shared folders.

autocreate_users (unchanged)

Limit the users for which mailboxes may be created to the list configured here.

Default Change: delete_mode

The default for the imapd.conf(5) configuration option delete_mode has changed from immediate to delayed.

This causes mail folders that are deleted by a client to not immediately dissappear from the filesystem. Instead, they are renamed to a deleted namespace that is visible only to administrators.

A separate job cyr_expire -D $x is to be included in the master service configuration file cyrus.conf(5), specifically in the EVENTS section. $x is a number of days to keep already deleted folders.

Example section of :cyrusman:`cyrus.conf(5)`

EVENTS {
    deleteprune cmd="cyr_expire -D 69" at=0430
}

In the aforementioned example, folders are purged from the filesystem only after 2 times 31 plus 7 days, corresponding with 2 cycles of a monthly (full, virtual) backup of which one might fail.

Default Change: expunge_mode

The default for the imapd.conf(5) configuration option expunge_mode has changed from default to delayed.

This causes the mail message files associated with messages that are flagged as Deleted in a folder that is subsequently expunged, or individual messages that are expunged, to not be removed from the filesystem directly.

A separate job cyr_expire -X $x is to be included in the master service configuration file cyrus.conf(5), specifically in the EVENTS section. $x is a number of days to keep the message files on the filesystem.

Example section of :cyrusman:`cyrus.conf(5)`

EVENTS {
    expungeprune cmd="cyr_expire -X 69" at=0430
}

In the aforementioned example, message files are purged from the filesystem only after 2 times 31 plus 7 days, corresponding with 2 cycles of a monthly (full, virtual) backup of which one might fail.

Option Name Changes for ldap_tls_*

Configuration option names for LDAP SSL/TLS configuration in imapd.conf(5) have been changed:

ldap_ca_dir (was: ldap_tls_cacert_dir)

ldap_ca_file (was: ldap_tls_cacert_file)

ldap_client_cert (was: ldap_tls_cert)

ldap_verify_peer (was: ldap_tls_check_peer)

ldap_ciphers (was: ldap_tls_ciphers)

ldap_client_key (was: ldap_tls_key)

Option Name Changes for tls_*

Configuration option names for SSL/TLS configuration in imapd.conf(5) have been changed to better reflect how they are used, as enhancements would otherwise create great confusion.

tls_client_ca_dir (was: tls_ca_path)

tls_client_ca_file (was: tls_ca_file)

The former tls_ca_* configuration options specified one or more SSL Certificate Authority certificates against which SSL certificates offered by clients could be verified.

In a Cyrus IMAP Murder topology however, Cyrus IMAP servers themselves become clients of other Cyrus IMAP servers, but may not have been issued certificates under the same verification chain.

With the (too) generic names for tls_ca_* configuration options out of the way, Cyrus IMAP 2.5.0 adds the following configuration options:

tls_server_cert (was: tls_cert_file)

tls_server_key (was: tls_key_file)

Server SSL certificate and key to use for connections from clients.

tls_sessions_db (was: tlscache_db)

tls_sessions_db_path (was: tlscache_db_path)

New Options for tls_*

tls_client_cert (<none>)

tls_client_key (<none>)

Client SSL certificate and key to use when cyrus-imapd behaves as a client (to other cyrus-imapd server (instances)).

tls_client_ca_file (<none>)

tls_client_ca_dir (<none>)

Certificate Authority file or directory used to verify client SSL certificates.

tls_client_certs (optional)

Disable (off), allow (optional) or require (require) clients authenticate with an SSL certificate.

tls_server_ca_file (<none>)

tls_server_ca_dir (<none>)

Certificate Authority file or directory used to verify SSL certificates offered by other servers.

tls_compression (0)

Enable TLS compression. Disabled by default.

tls_eccurve (prime256v1)

Select the elliptic curve used for ECDHE. See openssl ecparams -list_curves for supported values on your platform.

tls_prefer_server_ciphers (0)

Prefer the cipher order configured on the server-side.

tls_versions (ssl2 ssl3 tls1_0 tls1_1 tls1_2)

Disable SSL/TLS protocols not in this list.

Development: Switch to autoconf and libtool

With the release of Cyrus IMAP 2.5.0, the Cyrus IMAP project has switched to using autoconf and libtool.

Development: Switch to Phabricator

An instance of Phabricator is going to be replacing our old Bugzilla.

We believe this better facilitates our processes, and will make it easier to contribute code and collaborate.

Please see https://git.cyrus.foundation/.

Note

The Phabricator installation suffered a disk failure. We now use GitHub for code and bug management. https://github.com/cyrusimap/cyrus-imapd

Development: Sphinx for Documentation

While a work in progress still, you're looking at the new and improved documentation effort for the Cyrus project as a whole.

This documentation is written in reStructuredText, and rendered by Sphinx.

The GIT repository for the documentation is at

Footnotes

1

http://opera.brong.fastmail.fm.user.fm/talks/twoskip/twoskip-yapc12.pdf