POSIX Message Queues

A POSIX message queue is a kernel-resident, named queue of discrete, fixed-maximum-size messages that processes exchange by name, each message carrying an explicit priority so the highest-priority message is always delivered first. It is the modern redesign of the System V message queue, and it improves on a plain pipe in three ways that matter: messages are framed (a mq_receive returns exactly one whole message, never a half-read byte stream), they are priority-ordered rather than strictly FIFO, and the queue descriptor is a real file descriptor so it slots into an epoll loop and even supports asynchronous arrival notification via mq_notify (per mq_overview(7)). On Linux the implementation is a small virtual filesystem in ipc/mqueue.c, with messages stored in a per-queue red-black tree keyed by priority. The API — mq_open, mq_send, mq_receive, mq_getattr/mq_setattr, and mq_notify — is part of the POSIX real-time extensions and links against librt. This note traces the full mechanism end to end. All version-sensitive facts are pinned to Linux 6.12 LTS (November 2024).

Mental Model: a Priority Inbox With a Doorbell

Think of a POSIX message queue as a fixed-capacity inbox. Each letter (message) is a self-contained blob up to a maximum size set when the inbox was created, and each letter is stamped with a priority number. Senders drop letters in; the inbox keeps them sorted so the receiver always picks up the highest-priority letter first, and among equal priorities the oldest first. Two extra properties make it more than a pipe: the inbox has a fill light (it is a pollable fd, so an epoll loop can watch “is there mail”), and it has a doorbell (mq_notify) that can ring exactly once — by sending a signal or spawning a thread — the first time mail lands in an empty inbox, so a process need not sit blocked waiting.

flowchart TB
  P1["sender A<br/>mq_send(prio=5)"] --> Q
  P2["sender B<br/>mq_send(prio=9)"] --> Q
  P3["sender C<br/>mq_send(prio=5)"] --> Q
  subgraph Q["red-black tree in ipc/mqueue.c<br/>(per priority, FIFO within priority)"]
    H["prio 9: [B]"]
    M["prio 5: [A, C]  (A older)"]
  end
  Q -->|"mq_receive returns<br/>highest prio, oldest first"| R["receiver:<br/>B, then A, then C"]
  Q -.->|"first arrival on<br/>empty queue"| N["mq_notify doorbell:<br/>SIGEV_SIGNAL / SIGEV_THREAD"]

The delivery order of a POSIX message queue. What it shows: three sends with priorities 9, 5, 5 (A before C) produce the receive order B, A, C — priority dominates, and FIFO breaks ties within a priority. The dashed edge is the one-shot mq_notify doorbell that fires only on the first arrival into an empty queue. The insight to take: the queue is not a simple FIFO; it is a priority queue with FIFO tie-breaking, implemented as a red-black tree where the kernel walks to the rightmost (highest-priority) node on every receive (ipc/mqueue.c).

Opening a Queue: mq_open

A queue is created or opened with:

mqd_t mq_open(const char *name, int oflag);
mqd_t mq_open(const char *name, int oflag, mode_t mode, struct mq_attr *attr);

The name is a path-like string of up to NAME_MAX (255) characters that must begin with a slash and contain no other slash — /sensor-data, not sensor/data (mq_open(3)). The glibc wrapper requires the leading slash but strips it before handing the name to the kernel syscall, which is why the on-disk presentation under a mounted /dev/mqueue shows the name without the slash. The shared naming/refcount design is covered in POSIX IPC Overview and not re-derived here.

The oflag argument carries exactly one access mode — O_RDONLY, O_WRONLY, or O_RDWR — OR-ed with optional bits: O_CREAT to create the queue if absent, O_EXCL (with O_CREAT) to fail with EEXIST if it already exists, O_NONBLOCK to make sends/receives return EAGAIN rather than block, and O_CLOEXEC to set close-on-exec (mq_open(3)). When O_CREAT is present, the two-extra-argument form supplies a mode (file permission bits, masked by umask) and an optional attr pointer.

The attr is a struct mq_attr, defined in the v6.12 uapi header as:

struct mq_attr {
    long mq_flags;    /* queue flags: 0 or O_NONBLOCK; ignored by mq_open */
    long mq_maxmsg;   /* max number of messages the queue can hold */
    long mq_msgsize;  /* max size of a single message, in bytes */
    long mq_curmsgs;  /* messages currently queued; ignored by mq_open */
};

At creation time only mq_maxmsg and mq_msgsize are honoured; mq_flags and mq_curmsgs are ignored (mq_open(3)). If attr is NULL, the queue gets implementation-defined defaults — on Linux these come from /proc/sys/fs/mqueue/msg_default and msgsize_default, which are DFLT_MSG = 10 messages and DFLT_MSGSIZE = 8192 bytes per the v6.12 include/linux/ipc_namespace.h. These two *_default knobs exist “since Linux 3.5” specifically so a NULL-attr creation gets sane sizes rather than a system-wide maximum (mq_overview(7)).

mq_open returns a message queue descriptor of type mqd_t. The man page is explicit that on Linux “a message queue descriptor is actually a file descriptor” (mq_overview(7)) — this is the property the rest of the note leans on. The error returns are the familiar file-creation set: EACCES (permission denied, or a name with an embedded slash), EEXIST (O_CREAT | O_EXCL but the queue exists), EINVAL (bad name, or mq_maxmsg/mq_msgsize out of bounds), EMFILE/ENFILE (per-process / system fd limits), ENOENT (no such queue and no O_CREAT), ENOSPC (no room to create — e.g. queues_max reached), and ENOMEM (mq_open(3)).

Sending and Receiving: mq_send / mq_receive and Priority

int     mq_send(mqd_t mqdes, const char *msg_ptr, size_t msg_len, unsigned int msg_prio);
ssize_t mq_receive(mqd_t mqdes, char *msg_ptr, size_t msg_len, unsigned int *msg_prio);

mq_send copies msg_len bytes from msg_ptr into the queue with priority msg_prio. The length must be ≤ the queue’s mq_msgsize, or the call fails with EMSGSIZE; zero-length messages are explicitly allowed (mq_send(3)). The msg_prio is an unsigned int in the range 0 (low) to sysconf(_SC_MQ_PRIO_MAX) − 1 (high). On Linux that ceiling is large — _SC_MQ_PRIO_MAX returns 32768, matching MQ_PRIO_MAX = 32768 in the v6.12 include/uapi/linux/mqueue.h — although POSIX only requires implementations to support priorities 0–31 (mq_overview(7)). The ordering rule is the heart of the API: “Messages are placed on the queue in decreasing order of priority, with newer messages of the same priority being placed after older messages with the same priority” (mq_send(3)). That is, a strict priority queue with FIFO tie-breaking — the single feature that distinguishes a message queue from a plain pipe, where an urgent message would queue behind everything already buffered.

mq_receive removes and returns one message. Its rule is the dual: it “removes the oldest message with the highest priority” from the queue (mq_receive(3)). The caller’s buffer msg_len must be ≥ the queue’s mq_msgsize, otherwise mq_receive fails with EMSGSIZE — note the asymmetry with mq_send: you must always offer a buffer big enough for the largest possible message, because the receiver cannot know in advance how big the next message is. If msg_prio is non-NULL, the message’s priority is written there. The return value is the number of bytes in the received message.

Blocking behaviour is symmetric and governed by O_NONBLOCK. By default, mq_send on a full queue blocks until space frees up, and mq_receive on an empty queue blocks until a message arrives. With O_NONBLOCK set (at mq_open or later via mq_setattr), both instead fail immediately with EAGAIN (mq_send(3), mq_receive(3)). The timed variants mq_timedsend and mq_timedreceive take an absolute struct timespec deadline (measured against CLOCK_REALTIME) and return ETIMEDOUT if it passes first. A blocking call interrupted by a signal handler returns EINTR, the usual interaction described in EINTR and Interrupted System Calls.

Inspecting and Tuning at Runtime: mq_getattr / mq_setattr

int mq_getattr(mqd_t mqdes, struct mq_attr *attr);
int mq_setattr(mqd_t mqdes, const struct mq_attr *newattr, struct mq_attr *oldattr);

mq_getattr fills in the live mq_attr: the current mq_flags (0 or O_NONBLOCK), the fixed mq_maxmsg and mq_msgsize, and mq_curmsgs — how many messages are queued right now (mq_getattr(3)). mq_curmsgs is how you check fill level without consuming anything.

The asymmetry to internalise is that mq_setattr can change almost nothing. The man page is blunt: “The only attribute that can be modified is the setting of the O_NONBLOCK flag in mq_flags” and “The other fields in newattr are ignored” (mq_getattr(3)). mq_maxmsg and mq_msgsize are frozen at mq_open and cannot be resized — to change a queue’s capacity you must unlink it and recreate it. So mq_setattr’s real-world job is exactly one thing: flip a descriptor between blocking and non-blocking mode at runtime, the message-queue analogue of fcntl(fd, F_SETFL, O_NONBLOCK).

The Doorbell: mq_notify and the glibc SIGEV_THREAD Fake

int mq_notify(mqd_t mqdes, const struct sigevent *sevp);

mq_notify registers the calling process to receive a one-shot asynchronous notification the next time a message arrives. The struct sigevent selects the delivery method through its sigev_notify field (mq_notify(3), timer_create(2)):

• SIGEV_NONE — register but deliver nothing (a way to “claim” the queue’s single notification slot).
• SIGEV_SIGNAL — deliver the real-time signal in sigev_signo; the handler’s siginfo_t has si_code == SI_MESGQ, with si_pid/si_uid identifying the sender (mq_notify(3)). Pair this with Realtime Signals for queued, data-carrying delivery.
• SIGEV_THREAD — invoke sigev_notify_function “as if it were the start function of a new thread.”

Four rules govern the doorbell, and all four are easy to get wrong:

1. It fires only on arrival into a previously empty queue. The man page: “Message notification occurs only when a new message arrives and the queue was previously empty” (mq_notify(3)). If messages are already queued when you register, you get no notification until the queue drains to empty and refills.
2. A blocked receiver suppresses it. If any thread is already blocked in mq_receive on the empty queue, that thread takes the arriving message and the registration is simply ignored — the notification is for idle waiters, not competing ones.
3. It is one-shot. “After a notification is delivered, the notification registration is removed” — you must call mq_notify again to re-arm. The idiomatic pattern is: in the notification handler, first re-register, then drain all currently available messages with non-blocking mq_receive until EAGAIN. Re-registering first avoids a race where messages arriving during the drain are missed.
4. Only one process at a time. “Only one process can be registered to receive notification from a message queue.” A second mq_notify while another process is registered fails with EBUSY. Passing sevp == NULL deregisters the caller if it currently holds the slot.

The critical implementation detail — and the one most worth flagging — is how SIGEV_THREAD is realised. SIGEV_THREAD is not a kernel feature; it is faked by glibc with a helper thread. The mq_notify(3) man page states it directly: “For SIGEV_THREAD, much of the implementation resides within the library, rather than the kernel. … The implementation involves the use of a raw netlink(7) socket and creates a new thread for each notification that is delivered” (mq_notify(3)). The mechanism, confirmed against the v6.12 kernel source, is: glibc opens a raw netlink socket and registers it with the kernel; the kernel side in ipc/mqueue.c __do_notify() has a switch (info->notify.sigev_notify) whose SIGEV_THREAD arm does not know about C functions or threads at all — it merely netlink_sendskb()s a pre-prepared cookie skb to that socket. glibc’s internal manager thread is blocked reading the netlink socket; on receiving the cookie it spawns a fresh thread that runs the user’s sigev_notify_function. So from the kernel’s view, SIGEV_THREAD and SIGEV_SIGNAL collapse to “deliver a thing” — a signal in one case, a netlink message in the other; the entire “run a callback in a new thread” abstraction lives in userspace. The kernel only ever knows three primitives: do nothing (SIGEV_NONE), send a signal (SIGEV_SIGNAL), or send a netlink cookie (the path glibc drives for SIGEV_THREAD).

Uncertain

Verify: the precise division of labour — specifically that the kernel SIGEV_THREAD arm sends a netlink skb cookie and glibc’s manager thread spawns the callback thread per notification. Reason: the SIGEV_THREAD-is-faked-by-glibc claim is stated verbatim in mq_notify(3) and the kernel netlink_sendskb/alloc_skb(NOTIFY_COOKIE_LEN)/__do_notify path is confirmed in the v6.12 ipc/mqueue.c source I fetched — so the core claim is well grounded; what I did not fetch is the glibc-side sysdeps/.../mq_notify.c to confirm the “one new thread per notification” and the manager-thread detail directly from glibc. To resolve: read glibc’s mq_notify.c (helper_thread, init_mq_netlink). The headline claim (SIGEV_THREAD faked by glibc, kernel uses netlink) is verified; only the glibc-internal threading minutiae carry residual uncertainty. uncertain

Pollable: the Queue fd in an epoll Loop

Because mqd_t is a file descriptor, a POSIX message queue can be watched with select, poll, or epoll exactly like a socket: the fd is readable when at least one message is queued and writable when there is room to send (mq_overview(7)). This is the preferred alternative to mq_notify for an event-driven server: rather than arm a one-shot signal/thread doorbell with its empty-queue subtlety, you register the queue fd in an existing epoll set and treat “message arrived” identically to “socket readable.” That folds message queues into the epoll readiness model alongside timerfd, signalfd, and ordinary sockets — one loop, many sources. The choice between mq_notify and epoll is essentially: use mq_notify when the process is otherwise idle and you want it woken (it can even wake a process that is not running an event loop at all), and use epoll when the queue is one of many fds a running event loop already multiplexes.

The Kernel Side: the mqueue Filesystem, Limits, and RLIMIT_MSGQUEUE

Linux implements POSIX message queues as a small virtual filesystem of type "mqueue" (magic 0x19800202) defined in ipc/mqueue.c, authored originally by Krzysztof Benedyczak and Michał Wroński, with the lockless send/receive and fd-based notification added by Manfred Spraul (per the file’s copyright header). Each queue is an inode on this filesystem; the queues of an IPC namespace are isolated from those of another, so CLONE_NEWIPC gives a container its own private set of POSIX message queues. The filesystem is not mounted by default — mount -t mqueue none /dev/mqueue exposes each queue as a readable file reporting QSIZE, NOTIFY_PID, and the notification method.

Messages are stored not as a linked list but as a per-queue red-black tree keyed by priority. The v6.12 source comment captures the design: “During insert, low priorities go to the left and high to the right. On receive, we want the highest priorities first, so walk all the way to the right.” The queue struct caches the rightmost node (msg_tree_rightmost) so mq_receive reaches the highest priority in O(1) amortised rather than re-walking the tree, and a node_cache reuses freed tree nodes to avoid repeated allocation on the hot path. This is why priority lookup is cheap even with a 32768-wide priority space: the tree has at most one node per distinct priority present, and messages of equal priority chain within a node in FIFO order.

System-wide and per-queue limits are exposed under /proc/sys/fs/mqueue/ (registered by ipc/mq_sysctl.c):

sysctl	Default	Meaning	Bound
queues_max	DFLT_QUEUESMAX = 256	system-wide max number of queues	—
msg_max	DFLT_MSGMAX = 10	ceiling on a queue’s mq_maxmsg	MIN_MSGMAX = 1 .. HARD_MSGMAX = 65536
msgsize_max	DFLT_MSGSIZEMAX = 8192	ceiling on a queue’s mq_msgsize	MIN_MSGSIZEMAX = 128 .. HARD_MSGSIZEMAX = 16 MiB
msg_default	DFLT_MSG = 10	mq_maxmsg when attr == NULL (since 3.5)	—
msgsize_default	DFLT_MSGSIZE = 8192	mq_msgsize when attr == NULL (since 3.5)	—

(Default and bound constants verified in the v6.12 include/linux/ipc_namespace.h.) A process with CAP_SYS_RESOURCE may exceed msg_max and queues_max up to the HARD_* caps (mq_overview(7)).

Independently of those sysctls, each unprivileged process is bounded by RLIMIT_MSGQUEUE, which limits “the amount of space that can be consumed by all of the message queues belonging to a process’s real user ID” (mq_overview(7)). The v6.12 source charges, at queue creation, roughly mq_maxmsg × mq_msgsize plus per-message struct msg_msg overhead plus the tree-node overhead against UCOUNT_RLIMIT_MSGQUEUE; if the total would exceed rlimit(RLIMIT_MSGQUEUE) the creation fails. The practical consequence: a queue’s worst-case memory is reserved up front at mq_open, not pay-as-you-go, so creating a queue with mq_maxmsg = 10, mq_msgsize = 8192 immediately charges ~80 KB plus overhead against your RLIMIT_MSGQUEUE budget (default 819200 bytes = MQ_BYTES_MAX, per the uapi header) regardless of whether you ever send a message.

Configuration Walk-through: a Priority Producer and an epoll Consumer

/* producer.c — link with -lrt */
#include <mqueue.h>
#include <fcntl.h>        /* O_* */
#include <sys/stat.h>     /* mode constants */
 
int main(void) {
    struct mq_attr attr = {
        .mq_maxmsg  = 10,        /* up to msg_max (default 10) */
        .mq_msgsize = 256,       /* up to msgsize_max (default 8192) */
    };
    /* Create the queue if absent, writable, mode 0660. attr sizes it. */
    mqd_t q = mq_open("/sensor", O_CREAT | O_WRONLY, 0660, &attr);
    if (q == (mqd_t)-1) { /* handle EACCES/EEXIST/EINVAL/ENOSPC */ }
 
    const char *urgent = "OVERHEAT";
    const char *normal = "tick";
    mq_send(q, normal, 4, 1);     /* priority 1  */
    mq_send(q, urgent, 8, 9);     /* priority 9 — jumps ahead of 'tick' */
    mq_close(q);                  /* drop our fd; queue persists (kernel persistence) */
    return 0;
}

Line by line: the mq_attr initialiser sets only the two fields mq_open honours; mq_flags/mq_curmsgs are left zero and ignored. mq_open with O_CREAT | O_WRONLY creates a send-only descriptor; 0660 is masked by umask. The two mq_send calls show priority in action — although tick is sent first, the later OVERHEAT at priority 9 is dequeued first by any receiver. mq_close drops this process’s descriptor but the queue survives (it has kernel persistence; only mq_unlink("/sensor") plus last-close reclaims it — see POSIX IPC Overview).

/* consumer.c — epoll-driven, link with -lrt */
#include <mqueue.h>
#include <sys/epoll.h>
#include <fcntl.h>
#include <unistd.h>
 
int main(void) {
    mqd_t q = mq_open("/sensor", O_RDONLY | O_NONBLOCK);   /* non-blocking for epoll */
    struct mq_attr a;
    mq_getattr(q, &a);                                     /* learn mq_msgsize */
    char *buf = malloc(a.mq_msgsize);                      /* buffer >= mq_msgsize, else EMSGSIZE */
 
    int ep = epoll_create1(0);
    struct epoll_event ev = { .events = EPOLLIN, .data.fd = q };
    epoll_ctl(ep, EPOLL_CTL_ADD, q, &ev);                  /* mqd_t is a real fd */
 
    for (;;) {
        struct epoll_event out;
        epoll_wait(ep, &out, 1, -1);                       /* block until a message is queued */
        unsigned int prio;
        ssize_t n;
        while ((n = mq_receive(q, buf, a.mq_msgsize, &prio)) >= 0)
            handle(buf, n, prio);                          /* highest prio, oldest first */
        /* loop drained: n == -1, errno == EAGAIN */
    }
}

The consumer opens the queue O_NONBLOCK and registers its mqd_t directly in an epoll set — possible only because the descriptor is a genuine fd. The buffer is sized to mq_msgsize (anything smaller would fail mq_receive with EMSGSIZE). On each EPOLLIN the inner loop drains every available message until mq_receive returns EAGAIN, then re-blocks in epoll_wait. This is the level-triggered idiom; for edge-triggered (EPOLLET) the drain-to-EAGAIN is mandatory, not optional.

Failure Modes and Common Misunderstandings

“My receiver buffer is the message size and I get EMSGSIZE.” The receive buffer must be ≥ the queue’s mq_msgsize, not the size of the message you expect. The kernel rejects an undersized buffer before looking at the actual message. Always mq_getattr and size the buffer to mq_msgsize.

“mq_notify never fires even though messages arrive.” Almost always rule 1 or 2: messages were already queued when you registered (so the queue was not empty), or a thread is blocked in mq_receive (so it consumes arrivals first). The notification is for the first message into an idle, empty queue with no blocked reader.

“My notification fired once, then silence.” It is one-shot; you forgot to re-register. Re-arm with mq_notify inside the handler, before draining, every time.

“I can’t resize my queue.” mq_maxmsg/mq_msgsize are immutable after mq_open; mq_setattr only toggles O_NONBLOCK. Unlink and recreate to resize.

“mq_open fails with ENOSPC or EMFILE under load.” ENOSPC means queues_max (default 256) is exhausted system-wide or RLIMIT_MSGQUEUE is hit for the user; EMFILE means the process fd table is full — because each queue is an fd, queues count against RLIMIT_NOFILE too.

Signal-handler safety with SIGEV_SIGNAL. The handler runs in async-signal context, so it must use only async-signal-safe functions. mq_receive is among the AS-safe functions, but most logging and allocation is not — the safe pattern is to set a flag (or write to a self-pipe) and do the real work in the main loop. SIGEV_THREAD sidesteps this because the callback runs in an ordinary thread, not signal context — at the cost of glibc spawning a thread per notification.

Alternatives and When to Choose POSIX Message Queues

Against a pipe or Unix datagram socket, POSIX message queues win when you specifically need priority ordering or async notification to an idle process — a pipe is strictly FIFO and has no doorbell. Against the System V message queue, the POSIX version wins on every ergonomic axis (pollable fd, named, refcounted cleanup, async mq_notify); the only reason to use System V is its typed messages (mtype field allowing msgrcv to select by type, a feature POSIX replaces with the cruder priority axis) or legacy interop. Against a raw shared-memory ring buffer plus a futex, message queues are far simpler and framed but slower (every message copies through the kernel twice), so high-throughput, low-latency pipelines that can amortise the complexity prefer shared memory. For most “deliver bounded, prioritised, framed messages between a few processes” needs — especially in embedded and real-time Linux — POSIX message queues are the right default.

Production Notes

POSIX message queues are heavily used in real-time and embedded Linux, where their bounded, fixed-size, priority-ordered semantics map cleanly onto deterministic task communication, and far less in general server software, which overwhelmingly reaches for Unix sockets (bidirectional, fd-passing) or higher-level brokers instead. A recurring operational surprise is the up-front memory reservation interacting with RLIMIT_MSGQUEUE: a process that creates many queues at default sizes can hit the limit and get mq_open failures that look like permission or capacity bugs but are really the rlimit; raising it (or lowering per-queue mq_maxmsg) is the fix. Another is the SIGEV_THREAD cost model — because glibc spawns a new thread per notification, a high message rate driving SIGEV_THREAD notifications can become a thread-churn bottleneck, which is one more reason the epoll-the-fd approach is generally preferred for high-rate consumers. Finally, because queues are IPC-namespaced, a queue created on the host is invisible inside a container with its own IPC namespace and vice versa — a common “the other side can’t see my queue” confusion in containerised deployments.

See Also

• POSIX IPC Overview — the shared naming/fd/refcount design these calls inherit
• System V Message Queues — the legacy message queue with typed (mtype) messages
• The epoll Readiness Model and IPC — why a pollable mqd_t matters and how it composes
• Realtime Signals — the queued, data-carrying signals SIGEV_SIGNAL delivers
• Async-Signal-Safety and Reentrant Handlers — constraints on a SIGEV_SIGNAL handler
• Shared Memory via mmap — the faster, unframed alternative for high-throughput streams
• The Futex System Call — the synchronisation primitive a shared-memory queue would need
• Linux IPC MOC — the parent map