vault

System V Message Queues

16 min read

System V Message Queues

A System V message queue is a kernel-resident, persistent linked list of discrete, typed messages that unrelated processes can write to and read from. Unlike a pipe, which carries an undifferentiated byte stream, a message queue preserves message boundaries — each msgsnd(2) deposits one self-contained message and each msgrcv(2) removes exactly one — and stamps every message with a caller-chosen positive integer type, the queue’s most distinctive feature: a receiver can ask for any message, for a message of a specific type, or for the message with the lowest type at or below a bound, turning one physical queue into a set of logical priority lanes or per-recipient mailboxes (msgrcv(2)). The object is created or opened with msgget(2) against an integer key_t (see System V IPC Overview and Keys), lives in a global kernel namespace, and — like all System V IPC — persists until explicitly destroyed with msgctl(2)’s IPC_RMID, not when its creator exits. This note pins its mechanism to Linux 6.12 LTS (released 2024-11-17), verified against ipc/msg.c and include/uapi/linux/msg.h at the v6.12 tag.

Mental Model — A Shared Mailbox With Typed Slots

Think of a message queue as a single mailbox sitting in the kernel that any process with the right key and permissions can reach. Senders drop envelopes into it; receivers take envelopes out. Three properties separate it from a pipe and make it worth the clumsy API.

First, messages are discrete. A pipe hands you whatever bytes happen to be buffered, with no record of where one write() ended and the next began; reassembling message framing is the application’s problem. A message queue stores each msgsnd as one indivisible unit — msgrcv returns exactly one message’s payload, never a fragment and never two glued together.

Second, every message carries a type, a long that the sender chooses and that must be strictly positive. The type is not metadata the kernel ignores — it is the selector the receiver uses to pick which message to dequeue. This is the queue’s superpower: a server can multiplex many logical conversations over one queue by giving each its own type number, and a receiver can pull “the next message for me” out of a queue full of messages for others.

Third, the queue is kernel-persistent and globally named. It does not belong to a process; it belongs to the kernel, identified by a numeric key and an integer ID. Close all your file descriptors, exit every process — the queue and its undrained messages remain until someone runs IPC_RMID (or the IPC namespace is torn down). This is the classic System V leak hazard.

flowchart LR
  S1["Sender A<br/>msgsnd type=10"] --> Q
  S2["Sender B<br/>msgsnd type=20"] --> Q
  S3["Sender C<br/>msgsnd type=10"] --> Q
  subgraph Q["msg_queue (kernel)"]
    direction TB
    M1["mtype=10"] --> M2["mtype=20"] --> M3["mtype=10"]
  end
  Q -->|"msgrcv msgtyp=20<br/>(SEARCH_EQUAL)"| R1["Receiver X<br/>gets the type-20 msg"]
  Q -->|"msgrcv msgtyp=0<br/>(SEARCH_ANY)"| R2["Receiver Y<br/>gets first msg = type 10"]

A message queue as a typed mailbox. What it shows: several senders deposit typed messages into one FIFO-ordered in-kernel list; receivers pull selectively by type rather than strictly in arrival order — a msgtyp of 20 skips past the type-10 message at the head to grab the matching one, while msgtyp of 0 takes whatever is first. The insight to take: the type field turns a single physical queue into many logical channels, which is why the queue, despite its awkward API, has no clean replacement for typed routing among the POSIX successors.

Mechanical Walk-through — Creating, Sending, Receiving, Controlling

Opening the queue: msgget

int msgget(key_t key, int msgflg) (msgget(2)) returns the integer message queue identifier (msqid) associated with key. The semantics mirror every other System V *get call (see System V IPC Overview and Keys): with the special key IPC_PRIVATE the kernel always creates a brand-new queue; with IPC_CREAT in msgflg it creates the queue if no queue exists for that key and otherwise returns the existing one; adding IPC_EXCL makes creation fail with EEXIST if the queue already exists, giving an atomic “create exclusively” primitive. The least significant 9 bits of msgflg are the permission bits, with the same owner/group/other read-write layout as a file’s mode passed to open(2).

On creation the kernel zeroes the bookkeeping in the associated struct msqid_ds — msg_qnum (message count), msg_lspid/msg_lrpid (last sender/receiver PID), and msg_stime/msg_rtime (last send/receive time) are all set to 0, msg_ctime is set to the current time, and crucially msg_qbytes — the maximum number of bytes the queue may hold — is initialized to the system limit MSGMNB, which is 16384 bytes by default (msg.h, v6.12; readable and writable at runtime via /proc/sys/kernel/msgmnb, msgsnd(2)). The system-wide cap on the number of queues, MSGMNI, defaults to 32000 since Linux 3.19 (/proc/sys/kernel/msgmni), and the largest single message, MSGMAX, is 8192 bytes (/proc/sys/kernel/msgmax).

The message format: struct msgbuf

Every message a process sends or receives begins with a type header followed by the payload (msgsnd(2)):

struct msgbuf {
    long mtype;       /* message type, must be > 0 */
    char mtext[1];    /* message data (flexible-length) */
};

The mtype is the selector discussed above and must be strictly positive — a zero or negative mtype makes msgsnd fail with EINVAL. The mtext[1] is a placeholder; the real payload length is whatever you pass as msgsz to the syscall, and applications routinely define their own struct with a larger mtext array (or a typed payload). Messages of zero length (no mtext at all) are permitted.

Sending: do_msgsnd and the pipelined fast path

int msgsnd(int msqid, const void *msgp, size_t msgsz, int msgflg) copies the message into the kernel and links it onto the queue. Internally the kernel (ipc/msg.c, v6.12) runs do_msgsnd: it validates msqid and the positive mtype, copies the payload in with load_msg() (allocating a struct msg_msg plus segment pages), takes the queue lock, and checks msg_fits_inqueue() — which enforces both byte and count limits, succeeding only when msgsz + msq->q_cbytes <= msq->q_qbytes. Here q_cbytes is the current byte occupancy and q_qbytes is the capacity (the msg_qbytes field).

If the queue has room, the kernel does not blindly append. It first calls pipelined_send(), which walks the list of blocked receivers (q_receivers) and uses testmsg() to find one whose selection criteria the new message satisfies. If a waiting receiver matches, the message is handed directly to that receiver — smp_store_release(&msr->r_msg, msg) publishes the message pointer, q_lrpid is updated, and wake_q_add() schedules the receiver to wake — and the message never touches the queue’s message list at all. This pipelining avoids an enqueue-then-immediately-dequeue round trip and is the reason a sender can wake a blocked receiver in one step. Only if no blocked receiver matches does the message get list_add_tail(&msg->m_list, &msq->q_messages), with q_qnum and q_cbytes incremented, msg_lspid set to the sender’s PID, and msg_stime updated.

If the queue is full — adding the message would push byte count past msg_qbytes or message count past its limit — the default behavior is to block until space appears (msgsnd(2)). The sender enrolls itself via ss_add() onto the q_senders list in state TASK_INTERRUPTIBLE and calls schedule(). When a receiver later drains a message it calls ss_wakeup(), which re-examines blocked senders and wakes those whose message now fits. If IPC_NOWAIT is set in msgflg, a full queue instead causes msgsnd to fail immediately with EAGAIN.

Receiving: do_msgrcv, convert_mode, and type selection

ssize_t msgrcv(int msqid, void *msgp, size_t msgsz, long msgtyp, int msgflg) removes one message and returns the number of payload bytes copied. The msgtyp argument plus the MSG_EXCEPT flag determine which message (msgrcv(2)). The kernel’s do_msgrcv first calls convert_mode() to translate (msgtyp, msgflg) into one of the internal search modes, then find_msg() walks q_messages applying testmsg():

  • msgtyp == 0 → SEARCH_ANY — the first message in the queue (head of the list, i.e. oldest) is read, regardless of type.
  • msgtyp > 0 → SEARCH_EQUAL — the first message whose type equals msgtyp is read. This is the per-channel selector.
  • msgtyp > 0 with MSG_EXCEPT → SEARCH_NOTEQUAL — the first message whose type is not msgtyp is read. Useful for “everything except the control channel.”
  • msgtyp < 0 → SEARCH_LESSEQUAL — the first message with the lowest type that is ≤ |msgtyp| is read. Because lower type numbers are served first, this implements a priority queue: assign urgent messages low type numbers and call msgrcv with a negative msgtyp to always drain the highest-priority (lowest-numbered) pending message first.

When a matching message is found, the kernel unlinks it (list_del(&msg->m_list)), decrements q_qnum and q_cbytes, sets q_lrpid and q_rtime, and crucially calls ss_wakeup() to wake any senders that were blocked on a full queue. The payload is then copied out to userspace.

If no matching message exists and IPC_NOWAIT is not set, the receiver blocks. It builds a struct msg_receiver (msr_d) recording its desired type (r_msgtype), its buffer ceiling (r_maxsize), and a slot for the delivered message (r_msg), links itself onto q_receivers, primes r_msg to ERR_PTR(-EAGAIN), and sleeps in schedule(). It is woken either by pipelined_send() handing it a message directly (the fast path above) or by expunge_all() if the queue is destroyed. With IPC_NOWAIT set, the absence of a matching message instead returns ENOMSG immediately.

Two more receive flags matter. MSG_NOERROR tells the kernel to truncate a too-large message to msgsz bytes rather than failing; without it, a message whose payload exceeds msgsz causes msgrcv to fail with E2BIG and leave the message on the queue. MSG_COPY (since Linux 3.8, and only if the kernel was built with CONFIG_CHECKPOINT_RESTORE, msgrcv(2)) performs a non-destructive read: msgtyp is reinterpreted as an ordinal position and the kernel returns a copy of the message at that index (prepare_copy() / copy_msg(), search mode SEARCH_NUMBER) without unlinking it — used by checkpoint/restore to snapshot a queue. MSG_COPY requires IPC_NOWAIT and cannot be combined with MSG_EXCEPT.

Controlling: msgctl

int msgctl(int msqid, int op, struct msqid_ds *buf) manages the queue (msgctl(2)). The three workhorse operations:

  • IPC_STAT copies the kernel’s struct msqid_ds for the queue into buf — permissions (msg_perm), the last-send/last-receive/last-change times, current byte count, message count (msg_qnum), capacity (msg_qbytes), and last sender/receiver PIDs.
  • IPC_SET writes back a subset: msg_qbytes, and msg_perm.uid, msg_perm.gid, and the low 9 mode bits. Raising msg_qbytes beyond the MSGMNB system limit requires the CAP_SYS_RESOURCE capability.
  • IPC_RMID immediately removes the queue, awakening all blocked readers and writers, each of which returns -1 with errno set to EIDRM. There is no reference counting and no “remove on last close” — removal is instant and global.

Linux adds the introspection commands IPC_INFO/MSG_INFO (system-wide limits and usage in a struct msginfo) and MSG_STAT/MSG_STAT_ANY (treat the argument as a kernel array index rather than a msqid, used by ipcs to enumerate; MSG_STAT_ANY, since Linux 4.17, skips the read-permission check).

Worked Example — A Dispatcher Routing Typed Messages to Worker Classes

Consider a job server where one dispatcher receives requests and routes them to three classes of worker by request kind: type 1 = fast/interactive, type 2 = batch, type 3 = bulk export. We use a single queue and exploit message types both for routing and for replies. Define the protocol:

#include <sys/msg.h>
#include <sys/ipc.h>
#include <string.h>
#include <stdio.h>
#include <stdlib.h>
 
#define MTYPE_FAST   1L
#define MTYPE_BATCH  2L
#define MTYPE_BULK   3L
 
struct request {
    long mtype;            /* 1, 2, or 3 = which worker class */
    long reply_type;       /* worker replies with this type (e.g. client PID + 1000) */
    char payload[256];
};

The dispatcher creates the queue and injects requests with the routing type as mtype:

int q = msgget(IPC_PRIVATE, IPC_CREAT | 0660);          /* private queue, rw for owner+group */
struct request r = { .mtype = MTYPE_BATCH,
                     .reply_type = 1000 + getpid(),
                     .payload = "export Q3 ledger" };
/* send only the trailing bytes after mtype: */
msgsnd(q, &r, sizeof(r) - sizeof(long), 0);             /* msgsz excludes the mtype field */

The critical detail on line msgsnd(...): msgsz is sizeof(r) - sizeof(long) because the mtype long is the header, not part of the payload the kernel sizes. Getting this wrong (passing sizeof(r)) over-counts by 8 bytes and is a perennial bug.

Each batch worker pulls only its class by passing the routing type as a positive msgtyp:

struct request r;
for (;;) {
    /* SEARCH_EQUAL: take the next message whose mtype == MTYPE_BATCH, skipping fast/bulk */
    ssize_t n = msgrcv(q, &r, sizeof(r) - sizeof(long), MTYPE_BATCH, 0);
    if (n == -1) { if (errno == EIDRM) break; perror("msgrcv"); continue; }
    process_batch(r.payload);
    /* reply on the same queue using the client-specific reply_type */
    struct request resp = { .mtype = r.reply_type };
    strcpy(resp.payload, "done");
    msgsnd(q, &resp, sizeof(resp) - sizeof(long), 0);
}

Because every batch worker calls msgrcv with msgtyp = MTYPE_BATCH, fast and bulk requests sitting at the head of the list are skipped — the type acts as a routing filter, and the kernel hands each waiting worker the next message of its class via the pipelined fast path. The reply uses a per-client reply_type (here the client PID offset into a private range) so the original requester can msgrcv(q, ..., 1000 + getpid(), 0) and pick up its own reply out of a queue full of other clients’ replies — the same type-as-address trick that underlies the classic SysV “client/server over one queue” pattern.

Priority variant: if instead we wanted strict priority (fast before batch before bulk) at a single generic worker, the worker calls msgrcv(q, &r, ..., -3L, 0) (SEARCH_LESSEQUAL, |msgtyp| = 3): the kernel returns the lowest-typed pending message — so any waiting type-1 (fast) request is delivered before any type-2, which is delivered before any type-3, automatically.

Failure Modes and Common Misunderstandings

Leaked queues survive process death. The most common operational surprise: a process creates a queue, crashes, and the queue — plus any messages — lingers forever, counting against MSGMNI and consuming kernel memory. There is no O_CLOEXEC-style auto-cleanup. You must msgctl(q, IPC_RMID, NULL) or run ipcrm (see ipcs ipcrm and System V IPC Limits). A long-running system that repeatedly creates IPC_PRIVATE queues without removing them will eventually hit ENOSPC on msgget.

msgsz off-by-sizeof(long). As shown above, msgsz is the size of mtext only, never including mtype. Passing the full struct size sends 8 (or 4 on 32-bit) extra garbage bytes and, on receive, an E2BIG if the receiver’s buffer was sized correctly. This is the single most frequent message-queue bug.

EINTR on blocking calls. A msgsnd/msgrcv blocked on a full/empty queue is interruptible — a delivered signal aborts the call with EINTR, and System V message queue syscalls are not restarted by SA_RESTART (they are among the syscalls that always return EINTR). Robust code loops on EINTR. Compare EINTR and Interrupted System Calls.

EIDRM mid-operation. If another process runs IPC_RMID while you are blocked in msgrcv/msgsnd, your call returns with EIDRM (“identifier removed”). Distinguish it from EINVAL (the ID was never valid): EIDRM means it was valid and got destroyed under you.

Full-queue deadlock between two queues. Two processes that each block in msgsnd on a queue the other should be draining can deadlock if message flow is mutual and both queues fill. IPC_NOWAIT plus application-level flow control, or generously sized msg_qbytes, avoids it.

Type 0 confusion. mtype of 0 is illegal to send (EINVAL) but msgtyp of 0 is the legal “any” selector on receive. The same value means “forbidden” on one side and “wildcard” on the other.

Alternatives and When to Choose Them

The closest sibling is the POSIX message queue (mq_open/mq_send/mq_receive), the cleaner redesign of the same idea. POSIX queues are named with a path-like string (/myqueue), are referenced by a file descriptor so they fit an epoll loop and can be passed across exec, support per-message priorities natively (a numeric priority distinct from a routing type), and offer async notification via mq_notify. What POSIX queues lack is System V’s flexible type-selective receive: a POSIX receiver always gets the highest-priority message, never “the next message of type 17” or “anything except type 3.” If your design depends on typed routing — one queue serving many logical channels addressed by type — System V message queues remain the only kernel primitive that does it directly; otherwise POSIX queues are almost always the better choice.

For a pure byte stream between processes, a pipe or FIFO is simpler and faster, but you give up message framing. For bidirectional local IPC with the ability to pass file descriptors, a Unix-domain socket is the modern default. For sharing bulk data with no copy, a message queue is the wrong tool entirely — use shared memory coordinated by a semaphore or futex. Message queues shine specifically when you want discrete, typed, kernel-buffered messages with selective receive and you can tolerate the copy through kernel memory.

Production Notes

System V message queues are legacy infrastructure — heavily present in long-lived enterprise code (Oracle and other databases historically used SysV IPC, and many older Unix middleware stacks route control messages over them) but rarely chosen for new designs, which reach for POSIX queues, Unix sockets, or a userspace broker. Their defaults are generous on modern kernels: since Linux 3.19 a host can hold 32000 queues, far above the cramped historical limits, so the old ipcs-tuning rituals on Linux 2.x are mostly obsolete (verify your distro’s /proc/sys/kernel/{msgmni,msgmnb,msgmax} if you push high message volumes).

The operational discipline that matters most is cleanup and namespacing. Inside a container, System V IPC objects are isolated by the IPC namespace — a queue created in one container’s namespace is invisible to another, and the namespace’s teardown reaps its queues, which sidesteps the classic leak for containerized workloads. On a bare host, leaked queues are a real monitoring concern: tools watch ipcs -q for queues with growing msg_qnum (a stuck consumer) or orphaned ownership. Because the kernel copies each message in and out, message queues are not a high-throughput data path; using them to shovel megabytes is an anti-pattern that will saturate msg_qbytes and block senders. Keep messages small (control/command granularity) and move bulk payloads through shared memory, passing only a reference (offset/length) over the queue.

See Also

Graph View

Backlinks