Comment 17 for bug 1669578

Here's an updated version of the patch and a comment explaining in a little more detail what is happening:

Subject: [PATCH] screen: handle pts devices in different namespaces

Various programs that deal with namespaces will use pty devices that exist in
another namespace. One obvious candiate are containers. So far ttyname() was
incorrectly handling this case because the pts device stems from the host and
thus cannot be found amongst the current namespace's /dev/pts/<n> entries.
Serge Hallyn and I recently upstreamed patches to glibc that allow
ttyname{_r}() to correctly handle this case. At a minimum, ttyname{_r}() will
set errno to ENODEV in case it finds that the /dev/pts/<n> device that the
symlink points to exists in another namespace.

(The next comment is a little longer but tries to ensure that one can still
understand what is going on after some time has passed.)
In case we detect that ttyname{_r}() returns NULL and sets errno to ENODEV we
have ample reason to assume that the pts device exists in a different
namespace. In this case, the code will set a global flag indicating this case
to true. Furthermore, all operations (e.g. chmod(), chown(), etc.) will now
need to operate on the symbolic link /proc/self/fd/0 directly. While this
sounds straightforward, it becomes difficult to handle this case correctly when
we reattach to an already existing screen session from a different pts device
than the original one. Let's look at the general reattach logic a little
closer:

Assume we are running a shell that uses a pts device from a different
namespace:

 root@zest1:~# ls -al /proc/self/fd/
 total 0
 dr-x------ 2 root root 0 Apr 2 20:22 .
 dr-xr-xr-x 9 root root 0 Apr 2 20:22 ..
 lrwx------ 1 root root 64 Apr 2 20:22 0 -> /dev/pts/6
 lrwx------ 1 root root 64 Apr 2 20:22 1 -> /dev/pts/6
 lrwx------ 1 root root 64 Apr 2 20:22 2 -> /dev/pts/6
 l-wx------ 1 root root 64 Apr 2 20:22 3 -> pipe:[3067913]
 lr-x------ 1 root root 64 Apr 2 20:22 4 -> /proc/27413/fd
 lrwx------ 1 root root 64 Apr 2 20:22 9 -> socket:[32944]

 root@zest1:~# ls -al /dev/pts/
 total 0
 drwxr-xr-x 2 root root 0 Mar 30 17:55 .
 drwxr-xr-x 8 root root 580 Mar 30 17:55 ..
 crw--w---- 1 root tty 136, 0 Mar 30 17:55 0
 crw--w---- 1 root tty 136, 1 Mar 30 17:55 1
 crw--w---- 1 root tty 136, 2 Mar 30 17:55 2
 crw--w---- 1 root tty 136, 3 Mar 30 17:55 3
 crw--w---- 1 root tty 136, 4 Mar 30 17:55 4
 crw-rw-rw- 1 root root 5, 2 Apr 2 20:22 ptmx

(As one can see /dev/pts/6 does not exist in the current namespace.)
Now, start a screen session in this shell. In this case this patch will have
screen directly operate on /proc/self/fd/0.
Let's look at the attach case. When we attach to an existing screen session
where the associated pts device lives in another namespace we need a way to
uniquely identify the pts device that is used and also need a way to get a
valid fd when we need one. This patch solves this by ensuring that a valid file
descriptor to the pts device is sent via a unix socket and SCM_RIGHTS to the
socket and display handling part of screen. However, screen also sends around
the name of the associated pts device or, in the case where the pts device
exists in another namespace, the symlink /proc/self/fd/0. But after having sent
the fd this part of the codebase cannot simply operate on /proc/self/fd/0 since
it very likely refers to a different file. So we need to operate on
/proc/self/fd/<fd-sent-via-SCM_RIGHTS> but also need to ensure that we haven't
been tricked into operating on a tampered with file or device. So we cannot
simply sent /proc/self/fd/0 via the unix socket. Instead we read the contents
of the symbolic link /proc/self/fd/0 in the main function and sent it via the
unix socket. Then in the socket and display handling part of screen, we read
the contents of the /proc/self/fd/<fd-sent-via-SCM_RIGHTS> as well and compare
the pts device names. If they match we know that everything is well. However,
now we also need to update any tty handling code to directly operate on
/proc/self/fd/<fd-sent-via-SCM_RIGHTS>.