apt-key add fails in overlayfs

Bug #1618572 reported by Andres Rodriguez
58
This bug affects 5 people
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Fix Released
High
Unassigned
Xenial
Fix Released
High
Andy Whitcroft

Bug Description

Sending a custom APT config to cloud-init fails to:
1. add keys
2. configure sources
3. configura additional repository.

The same config is being sent to curtin, and curtin doesn't seem to fail (curtin install log http://paste.ubuntu.com/23112826/ just in case).

config sent by maas = http://pastebin.ubuntu.com/23112834/
cloud-init.log = http://paste.ubuntu.com/23112820/
cloud-init-output.log = http://paste.ubuntu.com/23112822/
sources.list = http://paste.ubuntu.com/23112824/
ubuntu@node03:/var/log$ ls -l /etc/apt/sources.list.d/
total 0

ubuntu@node03:/var/log$ sudo apt-get update
Hit:2 http://us.archive.ubuntu.com/ubuntu yakkety-updates InRelease
Get:3 http://us.archive.ubuntu.com/ubuntu yakkety-backports InRelease [92.2 kB]
Err:2 http://us.archive.ubuntu.com/ubuntu yakkety-updates InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
Ign:3 http://us.archive.ubuntu.com/ubuntu yakkety-backports InRelease
Hit:4 http://us.archive.ubuntu.com/ubuntu yakkety-security InRelease
Get:1 http://us.archive.ubuntu.com/ubuntu yakkety InRelease [247 kB]
Err:4 http://us.archive.ubuntu.com/ubuntu yakkety-security InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
Err:1 http://us.archive.ubuntu.com/ubuntu yakkety InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
Fetched 339 kB in 0s (388 kB/s)
Reading package lists... Error!
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://us.archive.ubuntu.com/ubuntu yakkety-updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
W: GPG error: http://us.archive.ubuntu.com/ubuntu yakkety-backports InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
W: The repository 'http://us.archive.ubuntu.com/ubuntu yakkety-backports InRelease' is not signed.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://us.archive.ubuntu.com/ubuntu yakkety-security InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://us.archive.ubuntu.com/ubuntu yakkety InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/yakkety/InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/yakkety-updates/InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
W: Failed to fetch http://us.archive.ubuntu.com/ubuntu/dists/yakkety-security/InRelease The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 40976EAF437D05B5 NO_PUBKEY 3B4FE6ACC0B21F32
W: Some index files failed to download. They have been ignored, or old ones used instead.
E: Problem renaming the file /var/cache/apt/srcpkgcache.bin.3HKvbX to /var/cache/apt/srcpkgcache.bin - rename (116: Stale file handle)
E: Problem renaming the file /var/cache/apt/pkgcache.bin.d0JUHJ to /var/cache/apt/pkgcache.bin - rename (116: Stale file handle)
W: You may want to run apt-get update to correct these problems
E: The package cache file is corrupted

summary: - cloud-init failing to configure additional repository
+ cloud-init failing to configure sources and additional repository
tags: added: maas
Revision history for this message
Jon Grimm (jgrimm) wrote : Re: cloud-init failing to configure sources and additional repository

From cloud-init.log:

Aug 30 17:42:59 node03 [CLOUDINIT] util.py[DEBUG]: Running command ['apt-key', 'add', '-'] with allowed return codes [0] (shell=False, capture=True)
Aug 30 17:42:59 node03 [CLOUDINIT] cc_apt_configure.py[ERROR]: failed to add apt GPG Key to apt keyring#012Traceback (most recent call last):#012 File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 333, in add_apt_key_raw#012 util.subp(['apt-key', 'add', '-'], data=key.encode(), target=target)#012 File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1832, in subp#012 cmd=args)#012cloudinit.util.ProcessExecutionError: Unexpected error while running command.#012Command: ['apt-key', 'add', '-']#012Exit code: 1#012Reason: -#012Stdout: ''#012Stderr: 'Warning: apt-key output should not be parsed (stdout is not a terminal)\ngpg: renaming `/etc/apt/trusted.gpg.tmp\' to `/etc/apt/trusted.gpg\' failed: Stale file handle\ngpg: error writing keyring `/etc/apt/trusted.gpg\': file rename error\ngpg: can\'t open `/etc/apt/trusted.gpg\'\ngpg: keydb_search failed: file open error\ngpg: key 93EE8CC5: public key "[User ID not found]" imported\ngpg: error reading `[stdin]\': file rename error\ngpg: import from `[stdin]\' failed: file rename error\ngpg: Total number processed: 0\ngpg: imported: 1 (RSA: 1)\n'
Aug 30 17:42:59 node03 [CLOUDINIT] handlers.py[DEBUG]: finish: modules-config/config-apt-configure: FAIL: running config

Revision history for this message
Scott Moser (smoser) wrote :

Aug 30 17:42:59 node03 [CLOUDINIT] cc_apt_configure.py[ERROR]: failed to add apt GPG Key to apt keyring
Traceback (most recent call last):
 File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 333, in add_apt_key_raw
 util.subp(['apt-key', 'add', '-'], data=key.encode(), target=target)
 File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1832, in subp cmd=args)
cloudinit.util.ProcessExecutionError: Unexpected error while running command.
Command: ['apt-key', 'add', '-']
Exit code: 1
Reason: -
Stdout: ''
Stderr: 'Warning: apt-key output should not be parsed (stdout is not a terminal)gpg: renaming `/etc/apt/trusted.gpg.tmp' to `/etc/apt/trusted.gpg' failed: Stale file handle
gpg: error writing keyring `/etc/apt/trusted.gpg': file rename error
gpg: can't open `/etc/apt/trusted.gpg'
gpg: keydb_search failed: file open error
gpg: key 93EE8CC5: public key "[User ID not found]" imported
gpg: error reading `[stdin]': file rename error
gpg: import from `[stdin]' failed: file rename error
gpg: Total number processed: 0
gpg: imported: 1 (RSA: 1)
'
Aug 30 17:42:59 node03 [CLOUDINIT] handlers.py[DEBUG]: finish: modules-config/config-apt-configure: FAIL: running config

Some googlign for the apt-key rename error hits
  https://forums.plex.tv/discussion/218956/ubuntu-16-04-lts-plex-install
  https://bugs.gnupg.org/gnupg/issue1249
Another race condition in gpg:
  https://bugs.gnupg.org/gnupg/issue1675

Although there really shouldn't be any concurrent gpg usage going on.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Given that this might be racy - is that reproducible for you Andres?

Revision history for this message
Andres Rodriguez (andreserl) wrote :

100% reproduce rate.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Simplified the config to http://paste.ubuntu.com/23115334/

working, I found this in the image then:
 pub 1024R/93EE8CC5 2012-01-19
      Key fingerprint = E67C 0712 C07B 255B 6974 B05E 5CFF 1EA9 93EE 8CC5
 uid Launchpad PPA for MaaS Maintainers

Using the non-simplified version (I had to remove the instant poweroff thou): http://paste.ubuntu.com/23115353/

I ran a loop and it always worked, see script http://paste.ubuntu.com/23115502/

Revision history for this message
Andres Rodriguez (andreserl) wrote :

Just to provide some more information:

1. The APT config sent is exactly the same both for Curtin and for Cloud-init.
2. The APT config only fails inside the ephemeral environment when cloud-init is applying it to the ephemeral env.
3. The APT config done inside the target (by curtin) succeeds and both sources and extra repository work as expected.

Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1618572

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Scott Moser (smoser) wrote : Re: cloud-init failing to configure sources and additional repository

This seems to be a kernel regression that happened either in 4.4 or 4.2.

I'll get more information on recreate soon. to get this off my system, here are my very sloppy notes http://paste.ubuntu.com/23117619/

essentially, you boot a cloud image with overlayroot enabled then 'apt-key add - <some-file'.
Also can be recreated with a tempfile rename for some files. Ie, the way that atomic writes are often done, by writing to afile in the same dir and then renaming.

Revision history for this message
Andres Rodriguez (andreserl) wrote :

Don't think it matters, but:
http://pastebin.ubuntu.com/23119431/

Scott Moser (smoser)
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Changed in cloud-init:
status: New → Confirmed
Changed in linux (Ubuntu):
importance: Undecided → Medium
tags: added: kernel-da-key
Scott Moser (smoser)
summary: - cloud-init failing to configure sources and additional repository
+ cloud-init failing to configure sources and additional repository in
+ overlayfs
summary: - cloud-init failing to configure sources and additional repository in
- overlayfs
+ apt-key add fails in overlayfs
Changed in linux (Ubuntu):
importance: Medium → High
tags: added: kernel-key
removed: kernel-da-key
Revision history for this message
Scott Moser (smoser) wrote :
Download full text (3.2 KiB)

Here is some information on how to reproduce this.
I use uvt-kvm, just as an easy way to get a remote system to work with.
I assume that any boot of xenial or yakkety will reproduce the problem.

The failure shows itself in a couple different ways:
a.) original report: apt-key add my.pubkey
  This show error messages and exit non-zero.

b.) console error logs with cloud-init failing renames
  Note, these will not get to /var/log/cloud-init.log due to where they
  get raised.
    OSError: [Errno 116] Stale file handle: '/var/lib/cloud/data/tmpj6uweivq' -> '/var/lib/cloud/data/status.json'

c.) reproduce with atomic_write.py (see attached)
   sudo ./atomic_write.py /var/lib/cloud/data/status.json
   Very interestingly most file paths do not fail, but that one does.

4.2 (linux-virtual-lts-wily) passes, 4.4 fails (yakkety or xenial).

## Sync yakkety images to your uvt
uvt-simplestreams-libvirt -vv sync --source http://cloud-images.ubuntu.com/daily 'release~yakkety'

## put the key in my.pubkey locally
cat > my.pubkey <<"EOF"
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: SKS 1.1.5
Comment: Hostname: keyserver.ubuntu.com

mI0ETxf8lwEEAMAqtJVUnlCVzjTOsohaE/M4mGHFl4Py1cuE9ryOgmTWje+6BrNjtWLSfTQJ
Kp1V6hViUoxPBck1qkZoAz7VU5nDuBWDybCsolliUX4zzTYNiDnPS74fs4CDUWx9qpl5Sdb6
7aygIid/mFXubhJnTPR6Bq9ptGmc0Ks6ttNs3WJ/ABEBAAG0IkxhdW5jaHBhZCBQUEEgZm9y
IE1hYVMgTWFpbnRhaW5lcnOIuAQTAQIAIgUCTxf8lwIbAwYLCQgHAwIGFQgCCQoLBBYCAwEC
HgECF4AACgkQXP8eqZPujMU2zQP/W9OCzaU7HvFrqEt6nHGej2PEanIunxo7J8D5OR+Yl578
FpRkHRgvcdQnGuZUpdBnOFatDDFME7ClN9qUrD1wDN1r9ip2luaKiO2cZOW4Uu5Z0n/3Qc6J
eh9TNspyDMuHVVZ5GiAk+GXgF1m7ps5lCnOCZK/pXUEEUOS8AWnt3sM=
=1RpZ

-----END PGP PUBLIC KEY BLOCK-----
EOF

# launch a guest
name="sm-y1"
uvt-kvm create "$name"

# wait for it to come up
uvt-kvm wait --insecure "$name"

# add the key file for easy use there.
uvt-kvm ssh --insecure "$name" "cat > my.pubkey" <<my.pubkey

# ssh in
uvt-kvm ssh --insecure "$name"

## add trusty and get wily kernel where this is not an issue.
## This installs us a 4.2.0 kernel (currently 4.2.0-42-generic)
% echo "deb http://archive.ubuntu.com/ubuntu trusty-updates main" | sudo tee -a /etc/apt/sources.list.d/trusty.list
% sudo apt-get update -q
% sudo eatmydata apt-get install -qy -t trusty-updates linux-virtual-lts-wily
% ls /boot/vmlinuz*
/boot/vmlinuz-4.2.0-42-generic /boot/vmlinuz-4.4.0-36-generic

##
## make 'grub-reboot' work.
##
% sudo sed -i 's,^GRUB_DEFAULT=.*,GRUB_DEFAULT=saved,' /etc/default/grub /etc/default/grub.d/*

% grep -r GRUB_DEFAULT /etc/default/grub /etc/default/grub.d/*
/etc/default/grub:GRUB_DEFAULT=saved

% sudo update-grub
% sudo grub-set-default 0

## Set grub to reboot into the kernel you want.
#
# Assuming you want kernel 4.4.0-36-generic, to boot into that do:
sudo sh -c 'grub-reboot "$0"' "Advanced options for Ubuntu>Ubuntu, with Linux 4.4.0-36-generic"

# if you're already in overlayfs, you need to modify the root
# so you need overlayroot-chroot
sudo overlayroot-chroot sh -c 'grub-reboot "$0"' "Advanced options for Ubuntu>Ubuntu, with Linux 4.4.0-36-generic"

# then reboot

## Toggle overlayroot on or off
#
# to turn off:
sudo overlayroot-chroot rm /etc/overlayroot.local.conf

# to turn *on*
...

Read more...

Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Stefan Bader (smb) wrote :

Looking at a test VM the problematic files seem to be present in the read-only base but then have been at some point deleted. So when the rename/move is attempted there are whiteout softlinks present in the overlay.

E.g.:
/media/root-ro/var/lib/cloud/data/status.json
/media/root-rw/overlay/var/lib/cloud/data/status.json -> (overlayfs-whiteout)

#> cd /var/lib/cloud/data
#> touch tmpfile
#> mv tmpfile status.json
mv: cannot move 'tmpfile' to 'status.json': Stale file handle

Revision history for this message
Stefan Bader (smb) wrote :

Adding a script which can reproduce the problem without the need of having an active overlayfs-root. Not elegant but does the job.

Revision history for this message
Andy Whitcroft (apw) wrote :

Ok looks like I have found the underlying cause, a bad port of an upstream stable patch to our Ubuntu code base. I will send out the fix shortly.

Changed in linux (Ubuntu Xenial):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Andy Whitcroft (apw)
Revision history for this message
Andy Whitcroft (apw) wrote :
tags: added: kernel-da-key
removed: kernel-key
Tim Gardner (timg-tpi)
Changed in linux (Ubuntu Xenial):
status: In Progress → Fix Committed
Revision history for this message
Andres Rodriguez (andreserl) wrote :

Hi All,

I have come across what seems to be a related issue across the board. This is not exactly the same way to reproduce but a simple 'apt-get update' shows this:

In Yakkety: http://pastebin.ubuntu.com/23149838/ on 4.4.0-9136-generic
In Xenial: http://pastebin.ubuntu.com/23149864/
Trusty similar issues: https://bugs.launchpad.net/maas/+bug/1621344

Revision history for this message
Andres Rodriguez (andreserl) wrote :

Ok, I confirm the issue as per previous comment with Trusty+hwe-x

tags: added: oil
Revision history for this message
Brad Figg (brad-figg) wrote :

@andres

Please verify that the Xenial and Trusty+hwe-x in -proposed both fix this problem that you are now seeing.

Revision history for this message
Scott Moser (smoser) wrote :

I booted a xenial guest as shown in my comment 10, added -proposed and rebooted into overlayfs

ubuntu@sm-y1:~$ uname -r
4.4.0-38-generic
ubuntu@sm-y1:~$ dpkg -S /boot/vmlinuz-4.4.0-38-generic
linux-image-4.4.0-38-generic: /boot/vmlinuz-4.4.0-38-generic

$ grep overlayroot /proc/mounts
overlayroot / overlayfs rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root-rw//overlay,workdir=/media/root-rw//overlay-workdir 0 0

$ sudo apt-key add my.pubkey
OK

So xenial looks good to me.

Revision history for this message
Scott Moser (smoser) wrote :

trusty + linux-virtual-lts-xenial looks good also.

ubuntu@sm-y1:~$ uname -r
4.4.0-38-generic
ubuntu@sm-y1:~$ dpkg -S /boot/vmlinuz-$(uname -r)
linux-image-4.4.0-38-generic: /boot/vmlinuz-4.4.0-38-generic
ubuntu@sm-y1:~$ grep overlay /proc/mounts
overlayroot / overlayfs rw,relatime,lowerdir=/media/root-ro,upperdir=/media/root
-rw//overlay,workdir=/media/root-rw//overlay-workdir 0 0
ubuntu@sm-y1:~$ lsb_release -sc
trusty

$ dpkg-query --show | grep linux- | grep -v headers
linux-image-3.13.0-95-generic 3.13.0-95.142
linux-image-3.13.0-96-generic 3.13.0-96.143
linux-image-4.4.0-36-generic 4.4.0-36.55~14.04.1
linux-image-4.4.0-38-generic 4.4.0-38.57~14.04.1
linux-image-virtual 3.13.0.96.104
linux-image-virtual-lts-xenial 4.4.0.38.28
linux-virtual 3.13.0.96.104
linux-virtual-lts-xenial 4.4.0.38.28

$ sudo apt-key add my.pubkey
OK

Revision history for this message
Matteo Panella (mpanella) wrote :

I can confirm that trusty + linux-generic-lts-xenial from -proposed looks good. I can now run sbuild with a tmpfs overlay, with 4.4.0-36-generic debconf would die with ESTALE during chroot setup.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (22.8 KiB)

This bug was fixed in the package linux - 4.4.0-38.57

---------------
linux (4.4.0-38.57) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1620658

  * CIFS client: access problems after updating to kernel 4.4.0-29-generic
    (LP: #1612135)
    - Revert "UBUNTU: SAUCE: (namespace) Bypass sget() capability check for nfs"
    - fs: Call d_automount with the filesystems creds

  * apt-key add fails in overlayfs (LP: #1618572)
    - SAUCE: overlayfs: fix regression in whiteout detection

linux (4.4.0-37.56) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1618040

  * [Feature] Instruction decoder support for new SKX instructions- AVX512
    (LP: #1591655)
    - x86/insn: perf tools: Fix vcvtph2ps instruction decoding
    - x86/insn: Add AVX-512 support to the instruction decoder
    - perf tools: Add AVX-512 support to the instruction decoder used by Intel PT
    - perf tools: Add AVX-512 instructions to the new instructions test

  * [Ubuntu 16.04] FCoE Lun not visible in OS with inbox driver - Issue with
    ioremap() call on 32bit kernel (LP: #1608652)
    - lpfc: Correct issue with ioremap() call on 32bit kernel

  * [Feature] turbostat support for Skylake-SP server (LP: #1591802)
    - tools/power turbostat: decode more CPUID fields
    - tools/power turbostat: CPUID(0x16) leaf shows base, max, and bus frequency
    - tools/power turbostat: decode HWP registers
    - tools/power turbostat: Decode MSR_MISC_PWR_MGMT
    - tools/power turbostat: allow sub-sec intervals
    - tools/power turbostat: Intel Xeon x200: fix erroneous bclk value
    - tools/power turbostat: Intel Xeon x200: fix turbo-ratio decoding
    - tools/power turbostat: re-name "%Busy" field to "Busy%"
    - tools/power turbostat: add --out option for saving output in a file
    - tools/power turbostat: fix compiler warnings
    - tools/power turbostat: make fewer systems calls
    - tools/power turbostat: show IRQs per CPU
    - tools/power turbostat: show GFXMHz
    - tools/power turbostat: show GFX%rc6
    - tools/power turbostat: detect and work around syscall jitter
    - tools/power turbostat: indicate SMX and SGX support
    - tools/power turbostat: call __cpuid() instead of __get_cpuid()
    - tools/power turbostat: correct output for MSR_NHM_SNB_PKG_CST_CFG_CTL dump
    - tools/power turbostat: bugfix: TDP MSRs print bits fixing
    - tools/power turbostat: SGX state should print only if --debug
    - tools/power turbostat: print IRTL MSRs
    - tools/power turbostat: initial BXT support
    - tools/power turbostat: decode BXT TSC frequency via CPUID
    - tools/power turbostat: initial SKX support

  * [BYT] display hotplug doesn't work on console (LP: #1616894)
    - drm/i915/vlv: Make intel_crt_reset() per-encoder
    - drm/i915/vlv: Reset the ADPA in vlv_display_power_well_init()
    - drm/i915/vlv: Disable HPD in valleyview_crt_detect_hotplug()
    - drm/i915: Enable polling when we don't have hpd

  * [Feature]intel_idle enabling on Broxton-P (LP: #1520446)
    - intel_idle: add BXT support

  * [Feature] EDAC: Update driver for SKX-SP (LP: #1591815)
    - [Config] CONFIG_EDAC_SKX=m
    - EDAC, skx_edac: Ad...

Changed in linux (Ubuntu Xenial):
status: Fix Committed → Fix Released
Scott Moser (smoser)
no longer affects: cloud-init
Changed in linux (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.