imap/mailbox.h

Mailboxes: the in-memory API and the on-disk format.

A mailbox is a directory of message files plus a few metadata files. Nothing but Cyrus itself should touch them: the supported ways in are the protocols and the command-line tools.

Where the files live

A mailbox lives on a partition. Unless its mbtype has MBTYPE_LEGACY_DIRS, its directory is named for its uniqueid, as <partition>/uuid/<u1>/<u2>/<uniqueid>, where u1 and u2 are the first two characters of the uniqueid; a legacy mailbox's directory is named for the hashed mailbox name. The metapartition_files option can move each metadata file to the same relative path on a metapartition. An archived message (FLAG_INTERNAL_ARCHIVED) lives on the archive partition, and its cache record is in that partition's own cyrus.cache.

A message file is named for its UID followed by a dot, so UID 423 is 423.. It holds the message as delivered, with CRLF line endings.

The metadata files, named by the FNAME_* macros:

cyrus.header: mailbox-wide data that rarely changes
cyrus.index: a fixed-size header, then a fixed-size record per message; the only copy of flags, modseqs and the like
cyrus.cache: data parsed from the message files, including some header fields, so FETCH, SEARCH and SORT needn't reparse them
cyrus.annotations: the annotations on the mailbox's messages; the mailbox's own annotations are in the global annotations.db, keyed by its uniqueid
cyrus.dav: the DAV database, only for a mailbox outside any user's tree; a user's DAV database is per-user (see dav_getpath())
cyrus.squat: the search index, only when search_engine is squat
cyrus.expunge: obsolete; read only to recover UIDs during reconstruct

cyrus.header

MAILBOX_HEADER_MAGIC, then a single line holding a DLIST kvlist, with these keys:

T: the mbtype, as mboxlist_mbtype_to_string() names it
N: the mailbox name
I: the uniqueid
J: the JMAP id
Q: the quota root, if any
A: the ACL, as a kvlist of identifier and rights
U: the user flag names, as a list; a flag's position is its bit in index_record::user_flags, so a removed flag leaves a NIL behind

An older format, still read, has three lines after the magic: the quota root and uniqueid separated by a tab, the user flag names separated by spaces, and the ACL as tab-separated identifier and rights pairs.

The file is only ever replaced whole, by writing a new copy and renaming it into place. Its CRC32 is stored in index_header::header_file_crc, so changing it needs the exclusive index lock.

cyrus.index

A header of index_header::start_offset bytes, then index_header::num_records records of index_header::record_size bytes, in UID order. Record number N (counting from 1) is at start_offset + (N-1) * record_size. Every field is in network byte order. The header and each record end with a CRC32 of all the bytes before it. In the current version, times are 64-bit counts of nanoseconds since the epoch, and every 64-bit field is 8-byte aligned.

The OFFSET_* macros give the current version's layout. The layout of every version that can still be read, 6 through MAILBOX_MINOR_VERSION, is in the templates in imap/index_file.c; mailbox_setversion() converts between them.

Expunging a message doesn't remove its record, only sets FLAG_INTERNAL_EXPUNGED on it. When the message is going to be removed for good (which is "right now" an immediate expunge and otherwise later) by cyr_expire, the record also gets FLAG_INTERNAL_UNLINKED and the message file is deleted. A repack rewrites the index without the unlinked records, which is the only time a record changes position.

Two record fields encode more than their names say. The high 16 bits of the stored system_flags are the internal flags (MsgInternalFlags), and the high 16 bits of the stored cache_version are bits 32-47 of cache_offset.

cyrus.cache

The first four bytes are the index_header::generation_no of the index the file belongs to. Then come the cache records, each appended at the end of the file and found through index_record::cache_offset. A record is NUM_CACHE_FIELDS items, in the order of the CACHE_* enum, each a 32-bit length followed by that many bytes, padded to a multiple of four (see CACHE_ITEM_NEXT()).

index_record::cache_crc is the CRC32 of the whole cache record, and index_record::cache_version says what's in it, notably which header fields CACHE_HEADERS holds: those imap/mailbox_header_cache.gperf lists with a minimum version no higher than it, plus any field it doesn't list whose name doesn't start with "X-" (see mailbox_cached_header()). Replacing a record leaves the old one behind, counted in index_header::leaked_cache_records, until the next repack. Everything here can be rebuilt from the message files by reconstruct.

A repack writes cyrus.index.NEW, and a cyrus.cache.NEW stamped with the next generation, then renames the index into place and then the cache.

Locking and commit order

Opening a mailbox takes a shared namelock on it, held until the struct mailbox is closed. A repack needs the exclusive namelock, so a record doesn't move under anyone who has the mailbox open. Reading the index needs at least the shared index lock, so the CRCs can be checked, and changing any of the metadata files needs the exclusive one.

Changes to cyrus.header and cyrus.index are held in memory until mailbox_commit(), so an error or crash before then leaves both untouched. New cache records are the exception: they're appended to cyrus.cache as they're made, but nothing refers to them until the index records naming them are written.

mailbox_commit() syncs the cache, writes cyrus.header, then the index records, and last the index header. cyrus.header is synced before the index is written, and the index after. The index header is what makes a change visible: readers see only num_records records, so records appended before a crash but not yet published by the header are ignored, and later overwritten.

Defines

MAILBOX_MINOR_VERSION

The cyrus.index format version that new and repacked mailboxes get.

NOTE: the mailbox minor version must be changed whenever any on-disk format changes are made to any mailbox files. It is also important to make sure all the mailbox upgrade and downgrade code in mailbox.c is changed to be able to convert both backwards and forwards between the new version and all supported previous versions. If you change MAILBOX_MINOR_VERSION you MUST also make corresponding changes to backend_version() in backend.c.

MAILBOX_CACHE_MINOR_VERSION: The index_record::cache_version that new cache records get

OFFSET_LAST_APPENDDATE: grew to 64-bit in v20

OFFSET_QUOTA_MAILBOX_USED: offset for 64bit quotas

OFFSET_POP3_LAST_LOGIN: grew to 64-bit in v20

OFFSET_DELETED: added for ACAP

OFFSET_EXISTS: Non-expunged records

OFFSET_LEAKED_CACHE: Number of leaked records in cache file

OFFSET_HIGHESTMODSEQ: CONDSTORE (64-bit modseq)

OFFSET_DELETEDMODSEQ: CONDSTORE (64-bit modseq)

OFFSET_HEADER_FILE_CRC: CRC32 of cyrus.header

OFFSET_SYNCCRCS_BASIC: XOR of SYNC CRCs of unexpunged records

OFFSET_RECENTTIME: last timestamp for seen data (grew to 64-bit in v20)

OFFSET_POP3_SHOW_AFTER: time after which to show messages to POP3 (grew to 64-bit in v20)

OFFSET_SYNCCRCS_ANNOT: SYNC_CRC of the annotations

OFFSET_UNSEEN: total number of UNSEEN messages (owner)

OFFSET_MAILBOX_CREATEDMODSEQ: MODSEQ at creation time

OFFSET_QUOTA_DELETED_USED: bytes of \Deleted messages for this mailbox (64-bit)

OFFSET_QUOTA_EXPUNGED_USED: bytes of \Expunged messages for this mailbox (64-bit)

OFFSET_QUOTA_ANNOT_USED: bytes of per-mailbox and per-message annotations for this mailbox

OFFSET_CHANGES_EPOCH: time from which we can calculate changes (grew to 64-bit in v20)

OFFSET_FIRST_EXPUNGED: last_updated of oldest expunged message (grew to 64-bit in v20)

OFFSET_LAST_REPACK_TIME: time of last expunged cleanup (grew to 64-bit in v20)

OFFSET_RECENTUID: last UID the owner was told about

OFFSET_INTERNALDATE: grew to 64-bit in v20 (nsec since epoch)

OFFSET_SENTDATE: grew to 64-bit in v20

OFFSET_SIZE: grew to 64-bit in v20

OFFSET_MODSEQ: CONDSTORE (64-bit modseq)

OFFSET_CID: conversation id, added in v13

OFFSET_CREATEDMODSEQ: modseq of creation time, added in v16

OFFSET_GMTIME: grew to 64-bit in v20

OFFSET_LAST_UPDATED: grew to 64-bit in v20

OFFSET_SAVEDATE: added in v15

OFFSET_BASECID: base conversation id, added in v20

OFFSET_CACHE_CRC: CRC32 of cache record

OPT_POP3_NEW_UIDL: added for Outlook stupidity

OPT_IMAP_CONDSTORE: added for CONDSTORE extension

OPT_IMAP_SHAREDSEEN: added for shared \Seen flag

OPT_IMAP_DUPDELIVER: added to allow duplicate delivery

OPT_IMAP_HAS_ALARMS: messages in mailbox have alarms

OPT_MAILBOX_NEEDS_UNLINK: files to be unlinked

OPT_MAILBOX_NEEDS_REPACK: repacking to do

OPT_MAILBOX_DELETED: mailbox is deleted an awaiting cleanup

Typedefs

typedef enum _MsgFlags MsgFlags: The IMAP system flags, in index_record::system_flags

typedef enum _MsgInternalFlags MsgInternalFlags: Cyrus's own per-message state, in index_record::internal_flags

Enums

enum _MsgFlags

The IMAP system flags, in index_record::system_flags

Values:

enumerator FLAG_ANSWERED

enumerator FLAG_FLAGGED

enumerator FLAG_DELETED

enumerator FLAG_DRAFT

enumerator FLAG_SEEN: the owner's \Seen, or everyone's with OPT_IMAP_SHAREDSEEN

enum _MsgInternalFlags

Cyrus's own per-message state, in index_record::internal_flags

Values:

enumerator FLAG_INTERNAL_SNOOZED: snoozed, by JMAP

enumerator FLAG_INTERNAL_SPLITCONVERSATION: index_record::basecid is meaningful

enumerator FLAG_INTERNAL_NEEDS_CLEANUP: the message file must be moved to or from the archive partition, or deleted

enumerator FLAG_INTERNAL_ARCHIVED: the message file is on the archive partition

enumerator FLAG_INTERNAL_UNLINKED: the message file is gone, or about to be

enumerator FLAG_INTERNAL_EXPUNGED: expunged

enum cache_item_index

Cache item positions: the items of a cyrus.cache record, in order.

CACHE_SECTION is binary, built from 32-bit values in network byte order by message_write_section(), and it nests. A section for a part with no subparts is a single 0. Otherwise it is N+1, for N subparts; a description of part 0, the header, which a multipart lacks; a description of each subpart; and then a section for each subpart. A description is the offset and size of the part's header and of its content in the message file, with a size of -1 for a part that doesn't exist; a word holding the length of the charset name (high 16 bits) and the encoding (low 8 bits, 0xff for none); the charset name, NUL-padded to that length; the content GUID (20 bytes); and the decoded size and line count of the content.

Values:

enumerator CACHE_ENVELOPE: the IMAP ENVELOPE

enumerator CACHE_BODYSTRUCTURE: the IMAP BODYSTRUCTURE

enumerator CACHE_BODY: the IMAP BODY

enumerator CACHE_SECTION: where each MIME part is; see above

enumerator CACHE_HEADERS: the cached header fields, as in the message; see mailbox_cached_header()

enumerator CACHE_FROM: the From addresses, for SEARCH

enumerator CACHE_TO: the To addresses, for SEARCH

enumerator CACHE_CC: the Cc addresses, for SEARCH

enumerator CACHE_BCC: the Bcc addresses, for SEARCH

enumerator CACHE_SUBJECT: the decoded Subject, for SEARCH and SORT

struct cacheitem

struct cacherecord

struct statusdata

struct index_record

#include <mailbox.h>

One message's cyrus.index record, as it is in memory.

The fields down to guid are stored on disk, at the OFFSET_* positions; the rest are working state.

Public Members

uint32_t uid: the message's UID

uint32_t header_size: octets of the message's header

uint32_t system_flags: MsgFlags

uint32_t internal_flags: MsgInternalFlags; stored in the high bits of system_flags

uint32_t cache_crc: CRC32 of the cyrus.cache record

uint32_t cache_version: format of the cyrus.cache record

uint64_t cache_offset: where the cyrus.cache record starts

uint64_t size: octets of the whole message

uint64_t modseq: modseq of the last change (CONDSTORE)

uint64_t createdmodseq: modseq when the record was created

uint64_t cid: conversation id

uint64_t basecid: the conversation id before a split by conversations_max_thread, if any

struct timespec internaldate: IMAP INTERNALDATE; where possible, also the message file's mtime

struct timespec sentdate: the Date header, at day resolution and with no zone, for SEARCH SENTON etc.

struct timespec gmtime: the Date header, in UTC, for SORT

struct timespec last_updated: when the record last changed

struct timespec savedate: when the message was saved to this mailbox (RFC 8514)

uint32_t user_flags[(16 * 8) / 32]: a bit per user flag, in the cyrus.header order

struct message_guid guid: SHA-1 of the message file

struct synccrcs

#include <mailbox.h>

The sync CRCs replication compares to find mailboxes that differ

Public Members

uint32_t basic: XOR of the CRCs of the unexpunged records

uint32_t annot: XOR of the CRCs of the annotations

struct index_header

#include <mailbox.h>

The cyrus.index header, as it is in memory.

Everything but dirty is stored on disk, at the OFFSET_* positions.

Public Members

bit32 generation_no: bumped by each repack; must match the first four bytes of cyrus.cache

int format: obsolete

int minor_version: the index format version

uint32_t start_offset: octets of header before the first record

uint32_t record_size: octets in each record

uint32_t num_records: records in the file, expunged or not

struct timespec last_appenddate: time of the last append

uint32_t last_uid: the highest UID used, so UIDNEXT - 1

quota_t quota_mailbox_used: octets of the unexpunged messages

struct timespec pop3_last_login: the owner's last POP3 login, for the poptimeout option

uint32_t uidvalidity: the IMAP UIDVALIDITY

uint32_t deleted: unexpunged messages with \Deleted

uint32_t answered: unexpunged messages with \Answered

uint32_t flagged: unexpunged messages with \Flagged

uint32_t unseen: unexpunged messages without FLAG_SEEN: the owner's \Seen, or everyone's with OPT_IMAP_SHAREDSEEN

uint32_t options: OPT_* flags

uint32_t leaked_cache_records: dead records in cyrus.cache

modseq_t highestmodseq: the IMAP HIGHESTMODSEQ

modseq_t deletedmodseq: modseq below which expunges may have been forgotten, for QRESYNC and replication

uint32_t exists: unexpunged records: the IMAP EXISTS

struct timespec first_expunged: last_updated of the oldest expunged record, to decide when to repack

struct timespec last_repack_time: time of the last repack

struct timespec changes_epoch: time from which changes can be calculated; see deletedmodseq

modseq_t createdmodseq: the modseq when the mailbox was created; a new mailbox's JMAP id is made from it (see mailbox_create())

bit32 header_file_crc: CRC32 of cyrus.header

struct synccrcs synccrcs: for replication

uint32_t recentuid: the highest UID the owner has been shown, for \Recent

struct timespec recenttime: when recentuid last changed

struct timespec pop3_show_after: POP3 hides messages whose internaldate is no later

quota_t quota_annot_used: octets of annotations

quota_t quota_deleted_used: octets of messages with \Deleted

quota_t quota_expunged_used: octets of expunged, not unlinked, messages

struct index_change

struct mailbox_header

struct mailbox

union indexbuffer_t

struct MsgFlagMap