vault

inotify File System Notification

17 min read

inotify File System Notification

inotify (“inode notify”) is the Linux kernel subsystem that lets a userspace process watch files and directories and be told, asynchronously, when something happens to them — a file is created, written, deleted, opened, closed, renamed, or has its metadata changed. The interface is deliberately tiny: a process calls inotify_init() to get a single file descriptor, calls inotify_add_watch() to register paths and the event mask it cares about, and then read()s a stream of fixed-header struct inotify_event records off that one descriptor (per inotify(7)). It replaced the older, clumsier dnotify and is the engine behind file-manager refresh, editor “file changed on disk” prompts, systemd path units, and build-tool watch modes. Its defining limitations — watches are not recursive and the event identifies files only by name — shape every program that uses it. inotify’s userspace interface is byte-for-byte identical between kernels 6.12 LTS and 6.18 LTS (verified by diffing include/uapi/linux/inotify.h at both tags, 2026-06-04): it is a mature, frozen API, and the richer events live in its sibling fanotify and Filesystem Monitoring.

Mental Model

The right way to picture inotify is as a subscription to a stream of inode events, multiplexed over a single file descriptor. You do not get a callback per file; instead one inotify instance (one fd) holds a list of watches, each watch is a (inode, event-mask) subscription, and the kernel funnels every matching event into one ordered queue that you drain by reading the fd. A watch descriptor (wd) — a small nonnegative integer — is the handle that tells you which watch a given event came from. This is the inverse of a file descriptor: an fd names an open file, a wd names a subscription to changes on an inode.

The subscription is attached to the inode, not the path string. If you watch /var/log/app.log and someone hard-links it to /tmp/x and writes through /tmp/x, you still get the modify event, because both names point at the same inode (this is the “inode-based monitoring” property, per inotify(7)). Conversely, when you watch a directory, the watch is on the directory’s inode, and you are told about changes to its entries (a child created, deleted, moved) — but you are not automatically subscribed to the child inodes themselves. That single fact is the origin of the “not recursive” limitation that dominates real-world use.

flowchart TB
  subgraph K["Kernel"]
    VFS["VFS operation<br/>(write, unlink, rename...)"]
    FSN["fsnotify hook<br/>fsnotify_create / fsnotify_modify / ..."]
    M1["watch / mark on inode A<br/>(wd=1, mask=IN_MODIFY or IN_CREATE)"]
    M2["watch / mark on inode B<br/>(wd=2, mask=IN_CREATE or IN_DELETE)"]
    Q["per-instance event queue<br/>(ordered, coalesced)"]
  end
  subgraph U["Userspace process"]
    FD["inotify fd<br/>(from inotify_init1)"]
    APP["read(fd) loop<br/>parse struct inotify_event"]
  end
  VFS --> FSN
  FSN -->|"event matches mask?"| M1
  FSN -->|"event matches mask?"| M2
  M1 --> Q
  M2 --> Q
  Q --> FD --> APP

How an inotify event travels from a VFS operation to userspace. What it shows: every file mutation passes through an in-kernel fsnotify hook, which checks the affected inode’s attached watches (marks); a matching watch enqueues an event onto the per-instance queue, which the process drains by reading the single inotify fd. The insight to take: there is exactly one queue and one fd per instance no matter how many watches you add — read() returns a packed stream of variable-length records, and the wd field is the only thing telling you which watched object each record refers to.

The fsnotify Framework Underneath

inotify is not a standalone subsystem; it is one backend of a shared in-kernel notification framework called fsnotify, living in fs/notify/. fanotify and the legacy dnotify are the other backends. fsnotify provides the common machinery — the marks (the kernel’s name for a subscription attached to an inode, mount, or superblock), the hook points sprinkled through the VFS (fsnotify_create(), fsnotify_modify(), fsnotify_unlink(), etc.), and the event-dispatch path — and each backend layers its own queue format and userspace API on top. Understanding this matters because inotify and fanotify and Filesystem Monitoring share the same event-detection plumbing and therefore the same blind spots (described in Failure Modes).

A neat implementation detail makes the relationship concrete: inotify’s public IN_* event bits line up exactly, bit for bit, with the kernel-internal FS_* bits. The header include/linux/fsnotify_backend.h (v6.12) says so directly: “IN_ from inotify.h lines up EXACTLY with FS_, this is so we can easily convert between them.” So IN_ACCESS (0x1) equals FS_ACCESS, IN_MODIFY (0x2) equals FS_MODIFY, and so on — the kernel even compiles in BUILD_BUG_ON(IN_ACCESS != FS_ACCESS) assertions in inotify_user_setup() to guarantee the two never drift apart (verified in fs/notify/inotify/inotify_user.c, v6.12). The practical consequence: when a write happens, the VFS raises FS_MODIFY, and the inotify backend can hand that bit straight to userspace as IN_MODIFY with zero translation.

The API, Walked Through

The whole interface is four system calls plus read() and close().

inotify_init1(int flags) creates an inotify instance and returns a file descriptor referring to a new, empty watch list (the older inotify_init() takes no flags; inotify_init1() was added in Linux 2.6.27, per inotify_init(2)). Two flags matter: IN_NONBLOCK sets O_NONBLOCK on the fd so that read() returns EAGAIN instead of blocking when the queue is empty, and IN_CLOEXEC sets the close-on-exec flag so the fd is not inherited across execve(). In the UAPI header these are literally defined as #define IN_CLOEXEC O_CLOEXEC and #define IN_NONBLOCK O_NONBLOCK (v6.12). Each instance you create counts against the per-user max_user_instances limit; hitting it yields EMFILE — note that EMFILE is overloaded, also meaning “the per-process open-fd limit was hit,” so the two causes must be distinguished by context.

inotify_add_watch(int fd, const char *pathname, uint32_t mask) registers a watch and returns its watch descriptor, a nonnegative integer that uniquely identifies the watch within that instance (inotify_add_watch(2)). The crucial subtlety: if you call it again for a path that is already watched in this instance, you do not get a new watch — you get the same wd back, and (by default) the new mask replaces the old one. To accumulate events instead of replacing, set IN_MASK_ADD, which ORs the new event bits into the existing mask. To refuse to touch an existing watch at all — fail with EEXIST rather than modify it — set IN_MASK_CREATE (Linux 4.18+). Each successful new watch consumes one slot against the per-user max_user_watches limit; exhausting it gives ENOSPC.

inotify_rm_watch(int fd, int wd) removes a watch. The kernel then emits one final IN_IGNORED event for that wd so the reader knows the subscription is gone. A watch is also removed automatically — again with IN_IGNORED — when the watched file is deleted or its filesystem is unmounted.

read(fd, buf, len) drains the queue. It returns one or more packed struct inotify_event records. The header is fixed (wd, mask, cookie, len), followed by an optional, len-byte, null-padded name — present only for events on a directory’s children (e.g. IN_CREATE of foo.txt inside a watched dir gives name = "foo.txt"). You must size your buffer to hold at least one full event including its name, or read() returns EINVAL; robust code uses a buffer of at least sizeof(struct inotify_event) + NAME_MAX + 1. The fd works with select(2), poll(2), and epoll(7), which is how every real program waits on it.

Worked example: a minimal directory watcher

#include <sys/inotify.h>
#include <limits.h>      /* NAME_MAX */
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
 
int main(int argc, char *argv[]) {
    /* 1. Create the instance. IN_CLOEXEC so children don't inherit it. */
    int fd = inotify_init1(IN_CLOEXEC);
    if (fd < 0) { perror("inotify_init1"); exit(1); }
 
    /* 2. Watch argv[1] for creates, deletes, writes-then-close, and moves.
     *    IN_CREATE/IN_DELETE/IN_MOVED_* fire for the directory's *entries*;
     *    IN_CLOSE_WRITE is the classic "a writer just finished" signal. */
    uint32_t mask = IN_CREATE | IN_DELETE | IN_CLOSE_WRITE
                  | IN_MOVED_FROM | IN_MOVED_TO;
    int wd = inotify_add_watch(fd, argv[1], mask);
    if (wd < 0) { perror("inotify_add_watch"); exit(1); }
 
    /* 3. Buffer must fit at least one event + its longest possible name. */
    char buf[4096] __attribute__((aligned(__alignof__(struct inotify_event))));
 
    for (;;) {
        ssize_t n = read(fd, buf, sizeof buf);   /* blocks until events arrive */
        if (n <= 0) { perror("read"); exit(1); }
 
        /* 4. Walk the packed records: each is header + event->len name bytes. */
        for (char *p = buf; p < buf + n; ) {
            struct inotify_event *e = (struct inotify_event *) p;
            const char *name = e->len ? e->name : "(self)";
            printf("wd=%d mask=0x%08x cookie=%u name=%s%s\n",
                   e->wd, e->mask, e->cookie, name,
                   (e->mask & IN_ISDIR) ? "/" : "");
            p += sizeof(struct inotify_event) + e->len;   /* advance by len, not strlen */
        }
    }
}

Line-by-line: step 1 creates the single fd. Step 2 builds a mask combining several events with bitwise OR — the mask is just the union of the IN_* constants you care about. Step 3’s aligned attribute matters because the kernel writes naturally-aligned __u32/__s32 fields and the buffer must be aligned for the struct inotify_event cast to be well-defined. Step 4 is the load-bearing part: you must advance the cursor by sizeof(header) + e->len, not by strlen(name), because len includes the null-padding the kernel adds to keep the next record aligned — getting this wrong is the single most common inotify bug. The IN_ISDIR bit in the mask tells you the event subject was a directory.

The Events, and What Each Means

The full event vocabulary (from inotify.h, v6.12) divides into events you ask for and events the kernel volunteers. The ones you watch for:

  • IN_ACCESS / IN_MODIFY — file read / file written or truncated.
  • IN_ATTRIB — metadata changed: permissions, owner, timestamps, extended attributes, or link count.
  • IN_OPEN — file or directory opened.
  • IN_CLOSE_WRITE / IN_CLOSE_NOWRITE — a writable / non-writable open was closed. IN_CLOSE_WRITE is the workhorse “the file is now fully written, safe to process” signal that build tools and uploaders rely on, because it fires after the writer is done, unlike IN_MODIFY which fires on every write(2).
  • IN_CREATE / IN_DELETE — an entry was created / deleted inside a watched directory (these carry a name).
  • IN_DELETE_SELF / IN_MOVE_SELF — the watched object itself was deleted / moved.
  • IN_MOVED_FROM / IN_MOVED_TO — the two halves of a rename, carrying a shared cookie (see below).

Convenience unions exist: IN_CLOSE = IN_CLOSE_WRITE | IN_CLOSE_NOWRITE, IN_MOVE = IN_MOVED_FROM | IN_MOVED_TO, and IN_ALL_EVENTS for everything.

The events the kernel volunteers regardless of your mask:

  • IN_IGNORED — this wd is gone (you removed it, or the object was deleted/unmounted). Stop using the wd.
  • IN_Q_OVERFLOW — the queue overflowed and events were lost; wd is −1. You must treat this as “rescan from scratch,” because you cannot know what you missed.
  • IN_UNMOUNT — the watched object’s filesystem was unmounted.
  • IN_ISDIR — OR’d into a mask to say “the subject was a directory.”

A rename within or between watched directories produces two events: IN_MOVED_FROM on the source directory (with the old name) and IN_MOVED_TO on the destination directory (with the new name). To pair them, the kernel stamps both with the same nonzero cookie value, so an application can recognize “this file that vanished from A is the same file that appeared in B” and treat it as a move rather than a delete-plus-create. The pairing is inherently racy though: inotify(7) warns the two events may not be adjacent in the stream, and a process must buffer an unmatched IN_MOVED_FROM for a short while in case its partner is still coming — or never comes, if the file was moved out of all watched directories.

Watch-creation modifier flags

Beyond the events, several flags shape how a watch is created (from inotify.h, v6.12): IN_ONLYDIR makes inotify_add_watch() fail with ENOTDIR unless the path is a directory (closes a TOCTOU race); IN_DONT_FOLLOW watches a symlink itself rather than its target; IN_EXCL_UNLINK stops events for children once they have been unlinked from the watched directory; IN_ONESHOT delivers exactly one event then auto-removes the watch; and the IN_MASK_ADD / IN_MASK_CREATE pair discussed above.

Limits: the max_user_watches Myth, Corrected

Every inotify resource is capped per real user ID, exposed under /proc/sys/fs/inotify/. There are three knobs: max_queued_events (queue depth per instance), max_user_instances (instances per user), and max_user_watches (watches per user). The internet is saturated with the claim that max_user_watches defaults to 8192. That is wrong on a modern kernel, and the source proves it.

In inotify_user_setup() (fs/notify/inotify/inotify_user.c, v6.12), the default is computed at boot from physical RAM, not hard-coded:

/* Allow up to 1% of addressable memory to be allocated for inotify
 * watches (per user) limited to the range [8192, 1048576]. */
watches_max = (((si.totalram - si.totalhigh) / 100) << PAGE_SHIFT) / INOTIFY_WATCH_COST;
watches_max = clamp(watches_max, 8192UL, 1048576UL);
...
inotify_max_queued_events = 16384;
init_user_ns.ucount_max[UCOUNT_INOTIFY_INSTANCES] = 128;
init_user_ns.ucount_max[UCOUNT_INOTIFY_WATCHES] = watches_max;

Walking it symbol by symbol: si.totalram - si.totalhigh is low-memory pages; dividing by 100 takes 1%; << PAGE_SHIFT converts pages to bytes; dividing by INOTIFY_WATCH_COST (the kernel’s estimate of bytes pinned per watch) yields a watch count; and clamp(x, 8192, 1048576) forces the result into the range [8192, 1,048,576]. So 8192 is the floor, not the default — on any machine with more than a few gigabytes of RAM the real default is far higher (up to ~1 million). The defaults that are fixed constants are max_queued_events = 16384 and max_user_instances = 128. The raised ~1-million ceiling for max_user_watches dates to v5.10 (per the comment in fanotify_user.c, which notes the fanotify mark limit was bumped to “match the increased limit of inotify max_user_watches since v5.10”); pin any “default is N” claim to the kernel you measured on.

Uncertain

Verify: the exact numeric default of max_user_watches on a given machine, and the value of INOTIFY_WATCH_COST. Reason: the default is a runtime function of totalram, so it varies per host and cannot be stated as one number; INOTIFY_WATCH_COST is a private kernel constant not exported to userspace. To resolve: read /proc/sys/fs/inotify/max_user_watches on the target box, and grep inotify_user.c for the INOTIFY_WATCH_COST definition at the running kernel’s tag. The formula and clamp range above are verified against v6.12 source. uncertain

Why these limits exist at all: each watch pins the watched inode in memory (it cannot be reclaimed while watched), so a process watching a huge tree consumes non-trivial unswappable kernel memory — hence the 1%-of-RAM budget. This is the mechanism that makes the “watch a million files” workloads (see below) bump into the ceiling.

Event Coalescing — Why You Cannot Count Events

A property that surprises people: inotify coalesces identical consecutive events. Per inotify(7): “If successive output events produced on the inotify file descriptor are identical (same wd, mask, cookie, and name), they are coalesced into a single event if the older event has not yet been read.” So ten back-to-back write()s to the same file, read in one go, may surface as a single IN_MODIFY. This makes inotify excellent for “something changed, go re-read it” semantics and useless for “exactly how many times did it change” semantics. Treat every event as a level-triggered hint (“re-stat the object”), never as an edge you must count.

Failure Modes

Not recursive — the big one. A watch on a directory tells you about that directory’s immediate entries only; it says nothing about events inside a subdirectory. To watch a whole tree you must walk it and inotify_add_watch() every subdirectory yourself, and — critically — when IN_CREATE | IN_ISDIR tells you a new subdirectory appeared, you must immediately add a watch on it, racing against files being created inside it before your watch lands. For large trees this is slow (one syscall per directory) and resource-heavy (one watch per directory against max_user_watches). This single limitation is the main reason projects reach for fanotify and Filesystem Monitoring’s mount-wide and filesystem-wide marks instead.

Name-based identification is racy. Events name files by string, but by the time you process the event the name may already be gone or reused. There is no stable file identity in an inotify event (no inode number, no file handle). fanotify’s FAN_REPORT_FID exists precisely to fix this.

Blind spots inherited from fsnotify. inotify does not see: changes made via mmap(2) writes, msync(2), or munmap(2) (memory-mapped writes bypass the VFS write hook); remote changes on network filesystems like NFS (you only see local operations, so network filesystems must be polled); and pseudo-filesystems such as /proc, /sys, and /dev/pts are not watchable at all. Before Linux 3.19, fallocate(2) also produced no events.

Queue overflow loses data. If you read too slowly, the queue fills and you get one IN_Q_OVERFLOW (wd = -1); everything after the last good event until you catch up is gone. The only correct recovery is a full rescan. Slow readers watching busy trees hit this routinely.

No actor identity. An inotify event tells you what happened but not who did it — no PID, no UID. A process cannot even distinguish events it caused itself from events others caused. fanotify’s FAN_REPORT_PIDFD addresses this; inotify cannot.

Watch-descriptor reuse. A removed wd number can be reallocated to a new watch even while stale unread events bearing that number sit in the queue — a long-standing, low-probability bug noted in inotify(7). Defensive code that maps wd → path should refresh the mapping on IN_IGNORED.

Alternatives and When to Choose Them

  • fanotify and Filesystem Monitoring — the more powerful sibling. Choose it when you need to watch an entire mount or filesystem without a watch-per-directory (FAN_MARK_MOUNT / FAN_MARK_FILESYSTEM), when you need an open fd or a file handle to the affected object delivered with the event, when you need to deny an operation (permission events, used by antivirus), or when you need the actor’s PID. inotify’s advantages over fanotify are that it needs no privilege (fanotify’s powerful modes require CAP_SYS_ADMIN) and that it can name the affected directory entry simply, so it remains the right tool for unprivileged “watch this directory for changes” use cases.
  • dnotify — the obsolete predecessor (signal-based, per-directory only). Never use it in new code; inotify exists to replace it.
  • Polling stat(2) — the only fallback on network filesystems and pseudo-filesystems inotify cannot watch. Crude, but unavoidable for NFS where local notification mechanisms see nothing.
  • io_uring and the File Path — orthogonal: io_uring is about issuing I/O asynchronously, not about being notified of others’ changes. You would not replace inotify with it, though an io_uring-based loop can read() an inotify fd without a blocking thread.

Production Notes

inotify is everywhere in the Linux userspace stack. systemd .path units use it to start a service when a watched file or directory appears or changes. GUI file managers (GNOME Files, Dolphin) use it to refresh views live. Editors use IN_CLOSE_WRITE / IN_MODIFY to show the “file changed on disk, reload?” prompt. Build and dev tools — inotifywait from inotify-tools, webpack/Vite “watch mode,” entr, tilt — use it to trigger rebuilds.

The classic operational pitfall is the ENOSPC / “watch limit reached” failure: a single process trying to recursively watch a large source tree (think a node_modules with hundreds of thousands of directories, or a deep monorepo) exhausts max_user_watches. The symptom is an error like “unable to watch … ENOSPC: System limit for number of file watchers reached” from editors and bundlers. The fix is to raise the sysctl, e.g. sysctl fs.inotify.max_user_watches=524288 (persisted in /etc/sysctl.d/). This is the most-Googled inotify problem, and it is a direct consequence of the not-recursive design forcing one watch per directory against a memory-pinned budget. The better architectural fix — used by tools that watch enormous trees — is to switch to fanotify and Filesystem Monitoring’s single filesystem-wide mark, which avoids the per-directory explosion entirely.

See Also

Graph View

Backlinks