The Socket Layer

The socket layer is the thin, protocol-agnostic slab of kernel code — almost all of it in net/socket.c — that sits between the userspace Berkeley socket syscalls and the per-protocol implementations (TCP/IPv4, UDP/IPv6, AF_UNIX, AF_PACKET, …). Its job is dispatch and bookkeeping: when userspace calls socket(AF_INET, SOCK_STREAM, 0), the socket layer does not know one byte about TCP. Instead it (1) looks up the requested protocol family in a small net_families[] table and calls that family’s ->create function, (2) wires the resulting object up to two layers of function-pointer vtables so that every later bind/listen/sendmsg call is forwarded to the right protocol code, and (3) installs the socket as a file descriptor backed by a struct file whose f_op is socket_file_ops, so the socket plugs into the VFS and can be read/write/close/epoll’d like any other file. This note traces that machinery in net/socket.c against the Linux 6.12 LTS kernel (released 2024-11-17), line by line.

The two paired data structures the socket layer manipulates — the VFS-facing struct socket and the protocol-facing struct sock — have their own anatomy note; this note treats them as the operands of the dispatch and points to struct socket and struct sock for the field-by-field tour. Likewise the AF_/PF_ family taxonomy itself lives in Protocol Families and Address Families; here we care only about how a family is dispatched, not the catalogue of which families exist.

Mental Model: A Two-Level Vtable Indirection

The socket layer is a textbook case of object-oriented C via function-pointer tables. There are two such tables, and keeping them straight is the whole game:

• struct net_proto_family — one per address family. It has essentially a single interesting member, ->create, plus the family number and owning module. The kernel keeps a global array net_families[NPROTO] indexed by family number (AF_INET == 2, AF_INET6 == 10, …). Registering a family (via sock_register()) drops a pointer into this slot. This is the entry point — “given a family, how do I make a socket of it?”
• struct proto_ops — one per (family, socket-type) combination. It is a fat vtable of ~30 function pointers: ->bind, ->connect, ->accept, ->listen, ->sendmsg, ->recvmsg, ->poll, ->shutdown, and so on (include/linux/net.h). Each socket’s struct socket carries a const struct proto_ops *ops pointer. After creation, every socket syscall is dispatched through this pointer: listen() becomes sock->ops->listen(sock, backlog). For IPv4 there are distinct instances — inet_stream_ops (TCP), inet_dgram_ops (UDP), inet_sockraw_ops (raw) — and which one a socket points at is decided at creation by its type.

There is a third layer below proto_ops — struct proto (e.g. tcp_prot, udp_prot), reached through struct sock’s sk_prot pointer — but that belongs to the per-protocol implementation, not the generic socket layer. The socket layer’s responsibility ends at setting up proto_ops and the fd; proto_ops methods then call into proto methods.

flowchart TB
    SYS["socket(AF_INET, SOCK_STREAM, 0)<br/>SYSCALL_DEFINE3(socket, ...)"]
    SYS --> SC["__sys_socket -> sock_create<br/>-> __sock_create()"]
    SC --> NF["net_families[AF_INET]<br/>= &inet_family_ops"]
    NF -->|"pf->create()"| IC["inet_create()<br/>walk inetsw[SOCK_STREAM]"]
    IC -->|"found protosw {TCP}"| ASSIGN["sock->ops = &inet_stream_ops<br/>sk->sk_prot = &tcp_prot"]
    ASSIGN --> MAP["sock_map_fd()<br/>sock_alloc_file(): struct file<br/>f_op = socket_file_ops"]
    MAP --> FD["return fd (e.g. 3)"]

    LATER["later: listen(3, 128)"] -.->|"sock->ops->listen"| INETL["inet_listen() (TCP)"]
    LATERU["UDP socket: listen(fd)"] -.->|"sock->ops->listen"| NOLISTEN["sock_no_listen()<br/>returns -EOPNOTSUPP"]

The dispatch chain for socket() plus two later-call examples. What it shows: socket() resolves a family through net_families[] to inet_create, which resolves a type through inetsw[] to a proto_ops vtable, then assigns that vtable to sock->ops; the fd is installed last. The insight to take: there is no if (family == AF_INET) ... else if (family == AF_INET6) switch anywhere in the hot path — it is all table lookups and indirect calls, which is why adding a new protocol family is a matter of sock_register() and a proto_ops struct, not editing the socket layer. The dashed edges show the same sock->ops->listen indirection landing on real TCP code for a stream socket and on a no-op stub for a datagram socket.

Step 1: The socket() Syscall and __sock_create

Userspace socket(2) enters the kernel at SYSCALL_DEFINE3(socket, ...) in net/socket.c, which immediately calls __sys_socket() (net/socket.c L1728). That helper strips the SOCK_NONBLOCK/SOCK_CLOEXEC flags out of the type argument (they are OR-ed into the same int), calls sock_create() to build the struct socket, and finally sock_map_fd() to turn it into a descriptor. sock_create() is a one-line wrapper that fills in the current task’s network namespace and calls the real workhorse, __sock_create():

/* net/socket.c (Linux 6.12), abridged */
int __sock_create(struct net *net, int family, int type, int protocol,
                  struct socket **res, int kern)
{
    const struct net_proto_family *pf;
    struct socket *sock;
 
    if (family < 0 || family >= NPROTO)   return -EAFNOSUPPORT;  /* range check */
    if (type < 0 || type >= SOCK_MAX)     return -EINVAL;
 
    err = security_socket_create(family, type, protocol, kern);  /* LSM hook */
    if (err) return err;
 
    sock = sock_alloc();                  /* allocate struct socket (+ inode) */
    sock->type = type;
 
    rcu_read_lock();
    pf = rcu_dereference(net_families[family]);   /* the per-family vtable */
    if (!pf) goto out_release;                     /* -EAFNOSUPPORT */
    if (!try_module_get(pf->owner)) goto out_release; /* pin the module */
    rcu_read_unlock();
 
    err = pf->create(net, sock, protocol, kern);   /* <-- the dispatch */
    ...
    *res = sock;
    return 0;
}

The shape is worth dwelling on. The family number is range-checked (family >= NPROTO → -EAFNOSUPPORT), an LSM hook (security_socket_create) lets SELinux/AppArmor veto creation, and a bare struct socket is allocated by sock_alloc(). Then the single indirection that defines the socket layer: pf = rcu_dereference(net_families[family]) reads the family’s vtable out of the global array under RCU (the table is read on every socket() call but written only when modules register/unregister, the canonical read-mostly RCU pattern), and pf->create(...) calls into the family. For AF_INET that is inet_create. Note the try_module_get(pf->owner) — because a protocol family can live in a loadable module (af_packet, af_unix, SCTP), the layer pins the module across the call so it cannot be unloaded mid-creation; if the family slot is NULL, a request_module("net-pf-%d", family) (just above, under CONFIG_MODULES) tries to demand-load it first.

Step 2: The Family’s ->create Picks a proto_ops Vtable

AF_INET registers itself at boot with a static net_proto_family:

/* net/ipv4/af_inet.c (Linux 6.12) */
static const struct net_proto_family inet_family_ops = {
    .family = PF_INET,
    .create = inet_create,
    .owner  = THIS_MODULE,
};
/* ... at init: sock_register(&inet_family_ops); */

So pf->create resolves to inet_create(). Its job is the second table lookup — mapping the socket type (and protocol) to the right proto_ops and proto. IPv4 keeps a per-type list, inetsw[SOCK_MAX], populated from a static array inetsw_array[]:

/* net/ipv4/af_inet.c (Linux 6.12) */
static struct inet_protosw inetsw_array[] = {
  { .type = SOCK_STREAM, .protocol = IPPROTO_TCP, .prot = &tcp_prot,
    .ops = &inet_stream_ops, ... },
  { .type = SOCK_DGRAM,  .protocol = IPPROTO_UDP, .prot = &udp_prot,
    .ops = &inet_dgram_ops, ... },
  { .type = SOCK_DGRAM,  .protocol = IPPROTO_ICMP,.prot = &ping_prot,
    .ops = &inet_sockraw_ops, ... },
  { .type = SOCK_RAW,    .protocol = IPPROTO_IP,  .prot = &raw_prot,
    .ops = &inet_sockraw_ops, ... },
};

Each entry binds a (type, protocol) pair to a proto_ops vtable (.ops) and a proto (.prot). inet_create() walks inetsw[sock->type] to find the entry matching the requested protocol (with protocol == 0 meaning “the default for this type” via the IPPROTO_IP wildcard), then performs the two assignments that make the socket be a TCP or UDP socket:

/* net/ipv4/af_inet.c, inet_create(), abridged */
sock->ops = answer->ops;                 /* e.g. &inet_stream_ops for TCP */
answer_prot = answer->prot;              /* e.g. &tcp_prot */
sk = sk_alloc(net, PF_INET, GFP_KERNEL, answer_prot, kern); /* sets sk->sk_prot */

The first line is the whole point: sock->ops now points at inet_stream_ops, so every future sock->ops->bind/listen/sendmsg lands in TCP’s inet glue. The second pair allocates the struct sock and stamps it with tcp_prot as its sk_prot. (This is also where the CAP_NET_RAW check for SOCK_RAW lives — inet_create rejects an unprivileged raw socket with -EPERM, which is why creating a raw socket needs the capability that The Berkeley Socket API mentioned.) Here is inet_stream_ops itself — the TCP stream vtable:

/* net/ipv4/af_inet.c (Linux 6.12) */
const struct proto_ops inet_stream_ops = {
    .family   = PF_INET,
    .bind     = inet_bind,
    .connect  = inet_stream_connect,
    .accept   = inet_accept,
    .listen   = inet_listen,
    .poll     = tcp_poll,
    .sendmsg  = inet_sendmsg,
    .recvmsg  = inet_recvmsg,
    .shutdown = inet_shutdown,
    ...
};

Step 3: The Socket Becomes a File Descriptor

A struct socket is not yet usable from userspace — userspace only ever holds integer fds. Back in __sys_socket, sock_map_fd() bridges that gap:

/* net/socket.c (Linux 6.12) */
static int sock_map_fd(struct socket *sock, int flags)
{
    struct file *newfile;
    int fd = get_unused_fd_flags(flags);        /* reserve an fd slot */
    if (unlikely(fd < 0)) { sock_release(sock); return fd; }
 
    newfile = sock_alloc_file(sock, flags, NULL); /* wrap socket in a struct file */
    if (!IS_ERR(newfile)) {
        fd_install(fd, newfile);                /* publish fd -> file mapping */
        return fd;
    }
    put_unused_fd(fd);
    return PTR_ERR(newfile);
}

sock_alloc_file() allocates a struct file, sets its f_op to the socket layer’s file-operations table socket_file_ops, and stores the struct socket * in file->private_data. That f_op table is the bridge to the VFS:

/* net/socket.c (Linux 6.12) */
static const struct file_operations socket_file_ops = {
    .read_iter      = sock_read_iter,   /* read()  on a socket fd */
    .write_iter     = sock_write_iter,  /* write() on a socket fd */
    .poll           = sock_poll,        /* select/poll/epoll readiness */
    .unlocked_ioctl = sock_ioctl,
    .uring_cmd      = io_uring_cmd_sock, /* io_uring opcodes */
    .release        = sock_close,       /* close() */
    .splice_read    = sock_splice_read,
    ...
};

This is why a socket fd behaves like a file: when userspace calls read(3, ...) on socket fd 3, the VFS finds file->f_op->read_iter and dispatches to sock_read_iter, which recovers the struct socket and calls sock_recvmsg. The same indirection makes epoll_wait work — epoll calls f_op->poll, which is sock_poll, which calls sock->ops->poll (e.g. tcp_poll). The reverse lookup, “given a struct file, get its socket,” is sock_from_file(), which simply checks file->f_op == &socket_file_ops and returns file->private_data — every internal syscall path (bind, listen, accept) calls this (via sockfd_lookup_light) to translate the userspace fd back into the kernel struct socket before dispatching through ops.

Step 4: How a Later Call (listen, sendmsg) Is Dispatched

Once the socket exists, every other syscall follows the same pattern: fd → file → socket → ops->method. listen(2) is the clearest small example:

/* net/socket.c (Linux 6.12) */
int __sys_listen_socket(struct socket *sock, int backlog)
{
    int somaxconn = READ_ONCE(sock_net(sock->sk)->core.sysctl_somaxconn);
    if ((unsigned int)backlog > somaxconn)   /* cap to net.core.somaxconn */
        backlog = somaxconn;
    err = security_socket_listen(sock, backlog);   /* LSM hook */
    if (!err)
        err = READ_ONCE(sock->ops)->listen(sock, backlog);  /* <-- vtable dispatch */
    return err;
}

Userspace listen(fd, backlog) → __sys_listen does sockfd_lookup_light(fd) to get the struct socket, then __sys_listen_socket caps the backlog to net.core.somaxconn (the kernel side of the 4096-default cap documented in listen(2)) and calls sock->ops->listen. For a TCP socket sock->ops is inet_stream_ops, so this is inet_listen. accept() is the one call that mints a second socket and fd, and the socket layer does that minting generically before ever touching protocol code. __sys_accept4 reserves a fresh fd with get_unused_fd_flags(), then do_accept() allocates a brand-new struct socket via sock_alloc(), copies the listening socket’s vtable onto it (newsock->ops = ops; __module_get(ops->owner) to pin the protocol module), wraps it in a struct file with sock_alloc_file(), and only then calls ops->accept(sock, newsock, arg) to let TCP fill in the connected struct sock; finally fd_install(newfd, newfile) publishes the new fd (net/socket.c L1915, L2014). This is the kernel-side mechanism behind the userspace observation that “accept() returns a new fd while the listening fd stays open” — the listening struct socket is never reused; a separate one is allocated per connection, inheriting the listener’s proto_ops so the connected socket speaks the same protocol.

The data-transfer calls are simpler: sock_sendmsg() is a wrapper around sock->ops->sendmsg (with an LSM hook and a trace_sock_send_length tracepoint), and sock_recvmsg() wraps sock->ops->recvmsg. The kernel uses INDIRECT_CALL_INET() macros around these so that for the overwhelmingly common IPv4/IPv6 case the indirect branch can be turned into a direct call to inet_sendmsg/inet6_sendmsg — a Spectre-era retpoline-avoidance optimization, but semantically still “dispatch through the proto_ops vtable.”

A Worked Example: Why listen() on a UDP Socket Fails

The vtable design makes a subtle behaviour fall out for free. Recall that a UDP socket gets sock->ops = &inet_dgram_ops. Look at how that table wires the connection-oriented methods:

/* net/ipv4/af_inet.c (Linux 6.12) */
const struct proto_ops inet_dgram_ops = {
    .family  = PF_INET,
    .bind    = inet_bind,
    .connect = inet_dgram_connect,
    .accept  = sock_no_accept,    /* <-- stub */
    .listen  = sock_no_listen,    /* <-- stub */
    .sendmsg = inet_sendmsg,
    .recvmsg = inet_recvmsg,
    ...
};

inet_dgram_ops.listen is sock_no_listen and inet_dgram_ops.accept is sock_no_accept — generic stubs in net/core/sock.c that simply return -EOPNOTSUPP (verified: net/core/sock.c L3359). So when userspace calls listen() on a UDP fd, the socket layer dispatches sock->ops->listen(sock, backlog) exactly as it would for TCP, but the vtable slot points at the stub, and the syscall returns EOPNOTSUPP. There is no special-case check for “is this a stream socket?” anywhere in __sys_listen — the per-type proto_ops table is the policy. This is the elegance of the design: connection semantics are expressed by which functions populate each type’s vtable, not by conditional logic in the generic layer. The same mechanism explains why accept() on a datagram socket fails and why send() on an unconnected datagram socket without a destination errors out.

The Generic/Per-Protocol Boundary

It is worth naming precisely where the socket layer’s responsibility ends. The socket layer (net/socket.c) owns: the syscall entry points, fd↔socket translation, the net_families[] family dispatch, the socket_file_ops VFS integration, LSM hooks, address copy-in/out (move_addr_to_kernel/move_addr_to_user), and the sock_sendmsg/sock_recvmsg wrappers. Everything behind the proto_ops and proto vtables — TCP’s state machine, UDP’s hash tables, the sk_buff handling, checksums, the actual packet I/O — is per-protocol code (net/ipv4/, net/ipv6/, net/unix/, …) that the socket layer knows nothing about. The contract between them is exactly the proto_ops interface: as long as a protocol fills in those function pointers and registers a net_proto_family, the generic layer can drive it without a single protocol-specific line. This separation is what lets the same socket()/bind()/send() syscalls serve AF_INET, AF_UNIX, AF_PACKET, AF_NETLINK, AF_XDP, and dozens more.

Failure Modes and Diagnosis

• EAFNOSUPPORT from socket() — net_families[family] is NULL: the protocol family is not built in and its module did not load. Common with AF_PACKET (af_packet module) or exotic families in a stripped kernel. Diagnose by checking lsmod/CONFIG_PACKET and watching for the request_module("net-pf-%d") attempt.
• EPROTONOSUPPORT / ESOCKTNOSUPPORT — the (type, protocol) pair has no entry in inetsw[]; e.g. SOCK_STREAM with IPPROTO_UDP. Raised inside inet_create’s lookup loop, not the generic layer.
• EOPNOTSUPP from listen/accept — the proto_ops slot is a sock_no_* stub, i.e. you called a stream-only operation on a datagram/raw socket (the worked example above).
• EPERM from socket(.., SOCK_RAW, ..) — inet_create rejected an unprivileged raw socket lacking CAP_NET_RAW.
• ENFILE/EMFILE around sock_map_fd — get_unused_fd_flags failed (per-process or system-wide fd limit). The socket was allocated then released; the syscall returns the fd-allocation error.

See Also

• The Berkeley Socket API — the userspace contract this layer services; “the socket fd” that other notes refer to is established here in sock_map_fd/socket_file_ops
• struct socket and struct sock — field-by-field anatomy of the two structures this layer wires together
• Protocol Families and Address Families — the AF_/PF_ catalogue dispatched through net_families[]
• Socket Buffers and Memory Accounting — what sock_sendmsg ultimately charges against sk->sk_sndbuf
• The TCP Protocol in Linux — the tcp_prot/inet_stream_ops code behind a stream socket
• The UDP Protocol in Linux — the udp_prot/inet_dgram_ops code behind a datagram socket
• epoll and Scalable Readiness Notification — reaches the socket via socket_file_ops.poll → sock_poll → sock->ops->poll
• Linux System Call Interface MOC — how SYSCALL_DEFINE3(socket, ...) is reached from userspace
• Linux Networking Stack MOC — the parent map (§1, The Socket Layer and the Socket API)