UNIX domain socket
Q: How do you find the process listening on an UNIX domain socket?
UNIX domain sockets - like TCP sockets - provide a bi-directional for communication between processes. But in contrast to TCP sockets, they only exist locally on the host. As such they cannot be used for inter-host-communication.
UNIX domain sockets do not use port numbers to distinguish between multiple instances. Traditionally they use paths and are represented in the file system as socket Inodes. But there also exists a second variant, which is described in below.
There also is a limit on the maximum path length, with is
UNIX_PATH_MAX = 108 characters, so be careful to not use deeply nested directories.
Some operating systems other then Linux do not check the file permissions on the socket Inode itself. Therefor it is good practice for portable applications to put the socket in a separate directory with only read- and search-permission for the desired user or group.
More details are described in the manual page <man:unix(7)>, which also includes an example C program. Python also provides support and includes server code in socketserver.
If we starts two processes listening on the same path, which one will get new connections?
$ nc -l -N -U /tmp/socket & $ ls -i /tmp/socket 1569897 /tmp/socket ^^^^^^^ $ nc -l -N -U /tmp/socket & $ ls -i /tmp/socket 1569946 /tmp/socket ^^^^^^^
As you see, both listening processes create a new UNIX socket Inode in the file system. Opening a connection on that path will thus reach only one process: the latest:
$ nc -N -U /tmp/socket </dev/null + Fertig nc -l -N -U /tmp/socket $ nc -N -U /tmp/socket </dev/null nc: unix connect failed: Connection refused
So given the path we want to find the process serving that Inode!
The duplicate issue
Multiple processes serving the same socket often happens by accident as
bind() fails if the socket still exists.
If the server process crashes or does not cleanup the path itself, the server will fail to start next time.
The following Python program shows this nicely:
import socket s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) s.bind('/tmp/socket') Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/socket.py", line 228, in meth return getattr(self._sock,name)(*args) socket.error: [Errno 98] Address already in use
Therefore many services unconditionally unlink the socket file before starting. If the old service is still running, its socket will no longer be reachable.
Also be aware that
close()ing the socket will not remove the socket path, so your daemon should
unlink() it itself when shutting down.
The naive try
Do you know
They are useful tools for finding processes having opened files and sockets:
$ fuser /tmp/socket /tmp/socket: 12550 12554 $ lsof /tmp/socket COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME nc 12550 phahn 3u unix 0x000000005149f1ab 0t0 13381347 /tmp/socket type=STREAM nc 12554 phahn 3u unix 0x00000000ac502341 0t0 13383220 /tmp/socket type=STREAM ^^^^^^^^
They both find both processes, but we know that only the latest will get all the requests. Even worse they might provide no output at all:
$ cd /tmp $ fuser ./socket /tmp/socket: 12550 12554 $ lsof ./socket
Yes, the output of
lsof remains empty!
It also does if your path contains symbolic links and you are using the wrong path to the socket Inode.
So the tools are less ideal for UNIX sockets as the problem there is more difficult:
For a file you have the file system ID and the Inode within that file system, which uniquely identifies the file. The tools just have to walk
/proc/$pid/fd/$fdand compare this to the 2-tuple of the requested files.
Same for UDP and TCP sockets, where the kernel guarantees that only one process per host can uniquely use that socket (
But for UNIX domain sockets we already have seen that multiple processes can open the same path for serving, each one gets a new UNIX socket Inode and thus only the latest process gets all new requests. All previous serving processes remain until they terminate themselves or are killed by someone.
Linux provides some information on UNIX domain sockets in its
/proc file system in
$ grep /tmp/socket /proc/net/unix # Num RefCount Protocol Flags Type St Inode Path # 000000005149f1ab: 00000002 00000000 00010000 0001 01 13381347 /tmp/socket # 00000000ac502341: 00000002 00000000 00010000 0001 01 13383220 /tmp/socket ^^^^^^^^
lsof use that information to find the processes, which opened UNIX sockets.
Pathmatches exactly the string, which was used by the process when it called
bind(). If the sockets gets moved or renamed, both
lsofcan no longer match the path to a process.
lsofeven will fail, when the path is relative or is different because it contains symbolic links.
You might assume, that the socket Inode in the file system is not relevant, but it is. If you manually rename the socket afterwards, a program can still connect your daemons on the renamed path:
$ mv /tmp/socket /tmp/socket.old $ nc -N -U /tmp/socket.old </dev/null
Inodein is different from the Inode in the file system: To handle sockets like files the Linux kernel implements a virtual file system called
$ grep sock /proc/filesystems nodev sockf
For each opened socket it contains an Inode, which then the processes reference:
$ readlink /proc/12554/fd/3 /proc/12550/fd/3 socket: socket: ^^^^^^^^
The numbers in the square brackets ‘socket:[…]
match the Inode numbers from/proc/net/unix’.
But we are still missing the link from the Inode in the file system path to the socket Inode of the process.
The question is quiet popular and on StackExchange you will find interesting solutions like using
Num column really is a kernel address, which can be linked to a struct unix_sock, which then can be used to find the process.
The Linux kernel implements the <man:sock_diag(7)> extension since 4.2, which provides additional information to diagnose socket issues.
ss from iproute2 starting with version v4.19.0~55 does show the VFS information:
$ ss --processes --unix --all --extended 'sport = /tmp/socket' Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port u_str LISTEN 0 5 /tmp/socket 13381347 * 0 users:(("nc",pid=12550,fd=3)) <-> ino:1569897 dev:0/65025 peers: u_str LISTEN 0 5 /tmp/socket 13383220 * 0 users:(("nc",pid=12554,fd=3)) <-> ino:1569946 dev:0/65025 peers: ^^^^^^^^ ^^^^^^^
This both lists the socket Inode as Port and the path Inode in the last column as ino:.
dev corresponds to the device number given by
stat -c 'ino:%i dev:0/%d' /tmp/socket ino:1569946 dev:0/65025
So finally we have a solution:
statthe socket to get the device ID and Inode
ss --processes --unix --all --extenedto look-up that tuple and match it to the process ID.
There are some more nice-to-know details about UNIX domain sockets, which are mostly irrelevant for this problem so far. But I want to mention them anyway for reference:
There exist a second class called abstract sockets:
- their path name starts with the
NULcharacters, making their path length 0.
- then they can use the remaining 107 characters to define a unique identifier, which other programs can use to connect.
- they are not represented in the file system.
As the communicating processes are both local to the host, the listening process can get additional information from the sender.
This includes the process, user and group IDs (PID, UID, GID).
SCM_CREDENTIALS in the manual page.
File descriptor passing
UNIX domain sockets can be used to pass file descriptors between processes.
This can be used to have a privileged process, which opens a file on request of another process.
Usually it performs additional checks before it passes its opened file on to the requesting process.
SCM_RIGHTS in the manual page.
Python only supports this natively since Python-3.3.