openpty fails in xenial buildd chroots

Hello Colin, or anyone else affected,

Accepted livecd-rootfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.408.52 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted livecd-rootfs (2.408.52) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-image/1.7+16.04ubuntu1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#livecd-rootfs

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

This seems to affect the build (ftbfs) of 'util-linux' for the SRU of LP: #1842437.

Hi Colin,

I've been analyzing another build failure related to openpty().

The same upload on Xenial passed on Sep 3rd and failed on Sep 17th,
due to openpty() failing in tests of util-linux/script.

The strace of the failing test-case shows no permission on /dev/ptmx:

  open("/dev/ptmx", O_RDWR) = -1 EACCES (Permission denied)

That's because /dev/ptmx is a symlink to /dev/pts/ptmx, which has
no permissions for non-root users on the livecd-rootfs chroot [1],
used with sbuild-launchpad-chroot locally.

  $ ls -lR /dev/pt*
  lrwxrwxrwx 1 root root 8 Sep 5 08:13 /dev/ptmx -> pts/ptmx
  ...
  /dev/pts:
  ...
  c--------- 1 root root 5, 2 Sep 17 21:23 ptmx

...

I don't understand enough of the live-build/livecd-rootfs/makedev
relation, but I could still reproduce that problem with -proposed
and https://git.launchpad.net/~cjwatson/livecd-rootfs/commit/?id=6049019a8b7bfe299ae8836a74e8db73042f3d96 (steps below).

I'm not exactly sure how makedev would affect it specifically
(maybe I didnt find/run the right hooks that gate on it, possibly?)

You mentioned you're 'fairly sure' about this, so would sincerely
appreciate a comment about it, if you have the time/chance. :- )

...

Other than that, for further debugging on that other problem..

Is there a way to find the specific chroot_url [1] used in a PPA for a particular build?
(buildlog only mentions 'in-target unpack-chroot' for a /home/buildd/file-cache-default/<hash>)

...

Otherwise, is there a way to find out when/on which release the
livecd-rootfs image was built?

Specifically looking for the deboostrap version used, as there's an
(old) change to deboostrap that changed /dev/ptmx from static node
to this symlink, but it's been a while ago, so not very sure of it.
(Debian bugs 817236 for discussion/issue, and 571136 for change)

Any comments would be really appreciated!

Thanks,
Mauricio

...

[1] https://api.launchpad.net/devel/ubuntu/xenial/amd64

Steps used to build livecd.ubuntu-base.rootfs.tar.gz
(according to python-lpbuildd's build_livefs.py,
then patched sbuild-launchpad-chroot to use that.)

  sudo apt-get install -y livecd-rootfs.

  mkdir auto
  ln -s /usr/share/livecd-rootfs/live-build/auto/{clean,config,build} auto/
  mkdir -p auto/functions
  echo 'set -x' > auto/functions/debug.sh

  lb clean --purge
       PROJECT=ubuntu-base SUBPROJECT=buildd ARCH=amd64 SUITE=xenial IMAGEFORMAT=none MIRROR=http://archive.ubuntu.com/ubuntu lb config
  sudo PROJECT=ubuntu-base SUBPROJECT=buildd ARCH=amd64 SUITE=xenial IMAGEFORMAT=none MIRROR=http://archive.ubuntu.com/ubuntu lb build

Hello Colin, or anyone else affected,

Accepted livecd-rootfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.408.53 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

On Tue, Sep 24, 2019 at 03:00:16PM -0000, Mauricio Faria de Oliveira wrote:
> The same upload on Xenial passed on Sep 3rd and failed on Sep 17th,
> due to openpty() failing in tests of util-linux/script.

We switched over to the new xenial chroots on 2019-09-05.

> The strace of the failing test-case shows no permission on /dev/ptmx:
>
> open("/dev/ptmx", O_RDWR) = -1 EACCES (Permission
> denied)
>
> That's because /dev/ptmx is a symlink to /dev/pts/ptmx, which has
> no permissions for non-root users on the livecd-rootfs chroot [1],
> used with sbuild-launchpad-chroot locally.

Grepping for dev/pt in the old (pre-2019-09-05) xenial chroots:

  drwxr-xr-x root/root 0 2011-11-14 17:30 chroot-autobuild/dev/pts/
  crw-rw-rw- root/tty 5,2 2011-05-23 11:44 chroot-autobuild/dev/ptmx

In the ones used from 2019-09-05:

  lrwxrwxrwx 0/0 0 2019-09-05 08:13 chroot-autobuild/dev/ptmx -> pts/ptmx
  drwxr-xr-x 0/0 0 2019-09-05 08:13 chroot-autobuild/dev/pts/

In a chroot built with my changes:

  crw-rw-rw- 0/0 5,2 2019-09-19 12:30 chroot-autobuild/dev/ptmx
  drwxr-xr-x 0/0 0 2019-09-19 12:30 chroot-autobuild/dev/pts/

(That said, I was building my test on bionic, so the different
debootstrap version could affect things.)

> I'm not exactly sure how makedev would affect it specifically
> (maybe I didnt find/run the right hooks that gate on it, possibly?)
>
> You mentioned you're 'fairly sure' about this, so would sincerely
> appreciate a comment about it, if you have the time/chance. :- )

It was an educated guess given that makedev obviously does stuff with
device nodes and is a glaring difference between old and new chroots,
but I'm certainly prepared to believe that it might be wrong. If things
still don't work after it's got all the way through the normal process
then I'll debug some more.

> Other than that, for further debugging on that other problem..
>
> Is there a way to find the specific chroot_url [1] used in a PPA for a particular build?
> (buildlog only mentions 'in-target unpack-chroot' for a /home/buildd/file-cache-default/<hash>)

I don't think so, sorry.

> Otherwise, is there a way to find out when/on which release the
> livecd-rootfs image was built?

You'd have to hunt it down from
https://launchpad.net/~cloud-images/+livefs/ubuntu/xenial/cpc-buildd
(possibly via the API, as the web UI there doesn't show older builds).
We don't have great tools for this yet.

At the moment my ad-hoc workaround is to mention chroot changes in
#launchpad-ops on irc.canonical.com, so they can be tracked down via
correlation with logs there. I know this needs improvement.

> Specifically looking for the deboostrap version used, as there's an
> (old) change to deboostrap that changed /dev/ptmx from static node
> to this symlink, but it's been a while ago, so not very sure of it.
> (Debian bugs 817236 for discussion/issue, and 571136 for change)

It'll be debootstrap 1.0.78+nmu1ubuntu1.9, since that's what's in
xenial-updates and has been for some months.

> Steps used to build livecd.ubuntu-base.rootfs.tar.gz
> (according to python-lpbuildd's build_livefs.py,
> then p...

All autopkgtests for the newly accepted livecd-rootfs (2.408.53) for xenial have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-image/1.7+16.04ubuntu1 (i386)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/xenial/update_excuses.html#livecd-rootfs

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Colin,

Thanks for the details and launchpad/livecd-rootfs comments.
That's very good info.

> In a chroot built with my changes:
>
> crw-rw-rw- 0/0 5,2 2019-09-19 12:30 chroot-autobuild/dev/ptmx
> drwxr-xr-x 0/0 0 2019-09-19 12:30 chroot-autobuild/dev/pts/

That does look better :-)

> (That said, I was building my test on bionic, so the different
> debootstrap version could affect things.)

Can you please clarify? I didn't follow how this livecd-rootfs
fix is for Xenial but the build happened on Bionic.

Doesn't the build uses the livecd-rootfs from the 'host'?

(Sorry, it's all new stuff to me; and I see there are wrapper tools,
scripts, so not sure I'm aware of the 'right/valid' build steps.)

> What release were you building the rootfs on? Was makedev present in
> the resulting rootfs? What did /dev/pt* look like in it?

I've been building on Xenial to use the livecd-rootfs package from Xenial.

With this fix from xenial-proposed, makedev is not present (ok), but /dev/ptmx is still a symlink:

    $ tar tvf livecd.ubuntu-base.rootfs.tar.gz | grep dev/pt
    lrwxrwxrwx 0/0 0 2019-09-24 17:45 chroot-autobuild/dev/ptmx -> pts/ptmx
    drwxr-xr-x 0/0 0 2019-09-24 17:45 chroot-autobuild/dev/pts/

    $ lsb_release -cs
    xenial

    $ dpkg -s livecd-rootfs | grep ^Version:
    Version: 2.408.53

    $ sudo <...> lb build
    ...
    [2019-09-24 17:47:43] lb_chroot_early_hooks
    ...
    Removing makedev (2.3.1-93ubuntu1) ...
    ...

    $ grep '^Package: makedev' chroot/var/lib/dpkg/status
    $

    $ grep makedev chroot/var/log/dpkg.log
    ...
    2019-09-24 17:47:44 remove makedev:all 2.3.1-93ubuntu1 <none>
    ...
    2019-09-24 17:47:44 status not-installed makedev:all <none>

Per your comment above I tried to build on Bionic + series Xenial,
but that failed/didn't complete.

    ...
    [2019-09-24 18:02:50] lb_chroot_install-packages install
    P: Begin installing packages (install pass)...
    ...
    E: Package 'gpg' has no installation candidate
    E: Unable to locate package gpg-agent
    P: Begin unmounting filesystems...
    ...

I'm using these build steps -- please let me know if that's missing something
or if standard/required practice is different (eg, '--backend=lxc' in build logs).

    sudo apt-get install -y livecd-rootfs.

    mkdir -p auto/functions
    ln -s /usr/share/livecd-rootfs/live-build/auto/{clean,config,build} auto/
    echo 'set -x' > auto/functions/debug.sh

    lb clean --purge
         PROJECT=ubuntu-base SUBPROJECT=buildd ARCH=amd64 SUITE=xenial IMAGEFORMAT=none MIRROR=http://archive.ubuntu.com/ubuntu lb config
    sudo PROJECT=ubuntu-base SUBPROJECT=buildd ARCH=amd64 SUITE=xenial IMAGEFORMAT=none MIRROR=http://archive.ubuntu.com/ubuntu lb build

I was running the build on bionic locally for convenience, and because it hadn't occurred to me that it might be a debootstrap problem. On LP the images are built in a xenial container.

I'll look again tomorrow.

Looks like you're right. I think cherry-picking that sort of change to debootstrap several years into a stable release is rather too risky, but I'm testing (in xenial this time!) a change to fix up /dev/ptmx in livecd-rootfs.

Hi Colin, that's very good news; thanks!

OK. This time my test procedure was more careful:

 * run a full livefs test build in a VM running launchpad-buildd, pushing in the new livecd-rootfs package at the relevant point
 * attempt to build util-linux in xenial from launchpad-buildd with the old image, verifying that this fails
 * build util-linux in xenial from launchpad-buildd with the new image, which now works

Colin,

> * run a full livefs test build in a VM running launchpad-buildd,
> pushing in the new livecd-rootfs package at the relevant point

For my own education (and possible future endeavours.. :-)
could you please describe the steps/commands for this?

It's been certainly an interesting journey around launchpad
tools, scripts, git repos, people on IRC.. until eventually
getting here -- but it would be great to know what would be
the "more-or-less official" steps to build livecd-rootfs on
non-Launchpad infrastructure.. from people who really do it.

And did you possibly find out how this change ended up into
xenial buildd chroot? I couldn't match the pass/fail dates
to livecd-rootfs/debootstrap changes; just out of curiosity.

Thanks again,
Mauricio

Hello Colin, or anyone else affected,

Accepted livecd-rootfs into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/livecd-rootfs/2.408.54 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Verification successful with livecd-rootfs 2.408.54 from xenial-proposed.

The /dev/ptmx node is a static device node (no longer a symlink to [/dev/]pts/ptmx),
and the util-linux test-case ("script -c 'Hallo World' /dev/null") is successful.
(for test-case details, see LP bug 1842437 comment 12).

Colin, thanks a bunch for fixing this.

I'll mark this is as verification-done as you're probably EOD by now,
but please feel free to change if for some reason it fails your test.

cheers,
Mauricio

Testing
===

Local build of livecd.ubuntu-base.rootfs.tar.gz:
(now the /dev/ptmx device is static device node)
---

$ dpkg -s livecd-rootfs | grep ^Version:
Version: 2.408.54

$ lsb_release -cs
xenial

$ mkdir -p auto/functions
$ ln -s /usr/share/livecd-rootfs/live-build/auto/{clean,config,build} auto/
$ echo 'set -x' > auto/functions/debug.sh

$ lb clean --purge

$ PROJECT=ubuntu-base \
  SUBPROJECT=buildd \
  ARCH=amd64 \
  SUITE=xenial \
  IMAGEFORMAT=none \
  MIRROR=http://archive.ubuntu.com/ubuntu \
  lb config

$ grep mknod config/hooks/00-ptmx-chardev.chroot_early
        mknod -m 666 /dev/ptmx.new c 5 2

$ sudo \
  PROJECT=ubuntu-base \
  SUBPROJECT=buildd \
  ARCH=amd64 \
  SUITE=xenial \
  IMAGEFORMAT=none \
  MIRROR=http://archive.ubuntu.com/ubuntu \
  lb build

$ tar tvf livecd.ubuntu-base.rootfs.tar.gz | grep /dev/pt
crw-rw-rw- 0/0 5,2 2019-09-26 18:26 chroot-autobuild/dev/ptmx
drwxr-xr-x 0/0 0 2019-09-26 18:23 chroot-autobuild/dev/pts/

Checking /dev/ptmx and openpty/script on launchpad image:
(symlink to /dev/pts/ptmx, openpty/script failure.)
---

$ sudo apt install sbuild-launchpad-chroot --no-install-recommends

$ sudo vim /usr/bin/sbuild-launchpad-chroot

  Remove (this is done/fixed in sbuild-launchpad-chroot_0.15)

    if not chroot_url.endswith(".tar.bz2"):
        print("[%s] Remote tarball isn't .tar.bz2." % chroot)
        return

$ sudo sbuild-launchpad-chroot create -n lp-xenial-amd64 -s xenial -a amd64

$ sudo schroot -c chroot:lp-xenial-updates-amd64 -u ubuntu -d / -- ls -lR /dev/pt* 2>&1 | grep ptmx
lrwxrwxrwx 1 root root 8 Sep 5 08:13 /dev/ptmx -> pts/ptmx
c--------- 1 root root 5, 2 Sep 26 18:56 ptmx

$ sudo schroot -c chroot:lp-xenial-updates-amd64 -u ubuntu -d / -- script -c 'echo Hello World' /dev/null
[...]
script: openpty failed: No such file or directory
E: Child terminated by signal ‘Terminated’

Checking /dev/ptmx and openpty/script on locally-built image:
(static device node, openpty/script succeeds!)
---

$ sudo vim /usr/bin/sbuild-launchpad-chroot

  Edit:
    use /tmp/livecd.ubuntu-base.rootfs.tar.gz instead of download from launchpad.

$ sudo sbuild-launchpad-chroot create -n lp1844504-xenial-amd64 -s xenial -a amd64

$ sudo schroot -c chroot:lp1844504-xenial-updates-amd64 -u ubuntu -d / -- ls -lR /dev/pt* 2>&1 | grep ptmx
crw-rw-rw- 1 root root 5, 2 Sep 26 18:26 /dev/ptmx
c--------- 1 root root 5, 2 Sep 26 18:56 ptmx

$ sudo schroot -c chroot:lp1844504-xenial-updates-amd64 -u ubuntu -d / -- script -c 'echo Hello World' /dev/null
[...]
Script started, file is /dev/null
Hello World
Script done, file is /dev/null

On Thu, Sep 26, 2019 at 10:39:56AM -0000, Mauricio Faria de Oliveira wrote:
> For my own education (and possible future endeavours.. :-)
> could you please describe the steps/commands for this?

I'm afraid we don't have sensible docs for this yet.
https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally#Set_up_a_builder
and
https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally#Drive_builder_through_rpc
(but not necessarily the other sections of that page) are sort of part
of it but probably not sufficient to run through the whole process. We
need to document this more clearly but it's not done yet.

> And did you possibly find out how this change ended up into
> xenial buildd chroot? I couldn't match the pass/fail dates
> to livecd-rootfs/debootstrap changes; just out of curiosity.

There's a manual process to update chroots to new livefs builds, which I
executed for xenial on 2019-09-05. That was the first time that this
version of the process had been run for xenial; before that, the chroots
were hand-crafted. As such, you're not really going to get anything
interesting out of trying to correlate with archive changes in this
case.

Thanks for the verification. util-linux builds cleanly here with an
updated chroot, so I agree.

Thanks for the links; that's an interesting read.

> [snip] As such, you're not really going to get anything
> interesting out of trying to correlate with archive
> changes in this case.

Got it -- actually glad to hear that, because I actually couldn't
make any sense of the issue w/ versions/release dates in archive. :)

> Thanks for the verification. util-linux builds cleanly here with an
> updated chroot, so I agree.

Cool, thanks!

This bug was fixed in the package livecd-rootfs - 2.408.54

---------------
livecd-rootfs (2.408.54) xenial; urgency=medium

  * Revert exclusion of makedev from buildd chroots, as it turned out not to
    be the problem. Instead, fix up /dev/ptmx to be a character device node
    rather than a symlink to /dev/pts/ptmx, in line with the discussion in
    https://bugs.debian.org/817236; I think this is safer than
    cherry-picking the fix to debootstrap at this point in a stable release
    cycle (LP: #1844504).

livecd-rootfs (2.408.53) xenial; urgency=medium

  * Fix exclusion of makedev from buildd chroots; debootstrap doesn't
    respect --exclude for "Priority: required" packages, so we have to purge
    makedev later (LP: #1844504).

livecd-rootfs (2.408.52) xenial; urgency=medium

  * Exclude makedev from buildd chroots, since it was historically excluded
    and apparently breaks some builds (e.g. mir; LP: #1844504).

livecd-rootfs (2.408.51) xenial; urgency=medium

  * ubuntu-cpc: Only produce explicitly specified artifacts (LP: #1837254)

 -- Colin Watson <email address hidden> Thu, 26 Sep 2019 10:53:23 +0100

The verification of the Stable Release Update for livecd-rootfs has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

I've rolled out the new xenial chroots with this fix.

Colin, thanks for the update.

For the record, I just confirmed that /dev/ptmx is OK
with a test package build on Launchpad PPAs right now.

  ...
  dh_clean
  # Check /dev/ptmx|pts/
  ls -l /dev/ptmx
  crw-rw-rw- 1 root root 5, 2 Oct 4 08:13 /dev/ptmx
  ls -l /dev/pts/*
  c--------- 1 root root 5, 2 Oct 4 11:25 /dev/pts/ptmx
  make[1]: Leaving directory '/<<BUILDDIR>>/test-pkg-1.0+devpts1'
  ...