Ubuntu
initramfs-tools package

[SRU] update-initramfs should sync only the filesystem containing the initramfs image

Bug #1667512 reported by tbenst
88
This bug affects 18 people
Affects Status Importance Assigned to Milestone
initramfs-tools (Debian)
Fix Released
Unknown
initramfs-tools (Ubuntu)
Fix Released
Medium
Unassigned
Xenial
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Unassigned

Bug Description

[Impact]
If there is lots of activity on a different filesystem, high io/cache pressure, or a stale mount updating initramfs might hang because the update-initramfs script calls sync. This leaves the system unresponsive and will require the user to forcefully kill dpkg.

[Test Case]
One easy way to simulate this is by copying a huge file over nfs. Note: do not use localhost, it is too fast for this purpose, a network bridge over qemu is good enough.

1. Setup a nfs server and export a directory with write permissions for the client

2. Copy a big file over NFS (assumes it is mounted at /mnt in the client)
$ dd if=/dev/zero of=/mnt/big_zeros bs=512 count=1M status=progress
Note: adjust the "count" accordingly to how fast/slow your system is.

3. Call sync and verify that it only finished after "dd" is done.

4. Repeat step #2 and now call "sync -f /boot/<some-initrd-image>". Verify that it finished immediatly.

[Regression Potential]
Although syncfs(2) man page [2] states that it works just the same as sync - except being limited to a filesystem -, any actual differences in the implementation can affect how the initrd image file is synced to the disk, leading to unexpected behavior.

[Other]
If syncfs is unsupported then the 'sync' binary will ensure that a normal system-wide sync is called [1]. The proposed patch won't help in scenarios where the initrd file is located in a busy filesystem (eg. no separated /boot partition).

The 'sync' and 'syncfs' specifications are pretty much similar and syncfs is expected to work the same as a normal sync regarding data integrity. Also altought the patch introduces the initrd file as an extra argument which would cause 'sync -f' to fail if the file does not exist, the update-initramfs script would fail earlier than that because that file is created by the 'mv' call a few lines above. Thus no regressions are expected.

[References]
[1] http://git.savannah.gnu.org/cgit/coreutils.git/commit/src/sync.c?id=8b2bf5295f353016d4f5e6a2317d55b6a8e7fd00
[2] http://manpages.ubuntu.com/manpages/xenial/man2/syncfs.2.html

[Original bug report]
At least four users, including myself, are having an issue with update-initramfs hanging while updating ubuntu 16.04. The bug has been documented while attempting an update to multiple kernel versions ( 4.4.0-24, 4.4.0-62, 4.4.0-63). The bug causes any apt-get update or install to fail, and may also lead to an unbootable system.

User #1 (me):
$ uname -a
Linux <computername> 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ sudo apt-get upgrade
Fetched 1,571 MB in 2min 9s (12.2 MB/s)
Extracting templates from packages: 100%
Preconfiguring packages ...
(Reading database ... 344634 files and directories currently installed.)
Preparing to unpack .../base-files_9.4ubuntu4.4_amd64.deb ...
Unpacking base-files (9.4ubuntu4.4) over (9.4ubuntu4.3) ...
Processing triggers for plymouth-theme-ubuntu-text (0.9.2-3ubuntu13.1) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for cracklib-runtime (2.9.2-1build2) ...
Processing triggers for install-info (6.1.0.dfsg.1-5) ...
Processing triggers for initramfs-tools (0.122ubuntu8.8) ...
update-initramfs: Generating /boot/initrd.img-4.4.0-63-generic

<hangs overnight> Uh oh! Let's try to troubleshoot.

$ sudo killall dpkg
$ sudo dpkg --configure -a
Setting up initramfs-tools (0.122ubuntu8.8) ...
update-initramfs: deferring update (trigger activated)
Setting up base-files (9.4ubuntu4.4) ...
Installing new version of config file /etc/issue ...
Installing new version of config file /etc/issue.net ...
Installing new version of config file /etc/lsb-release ...
Setting up linux-image-4.4.0-63-generic (4.4.0-63.84) ...
Running depmod.
update-initramfs: deferring update (hook will be called later)
initrd.img(/boot/initrd.img-4.4.0-63-generic
) points to /boot/initrd.img-4.4.0-63-generic
 (/boot/initrd.img-4.4.0-63-generic) -- doing nothing at /var/lib/dpkg/info/linux-image-4.4.0-63-generic.postinst line 491.
vmlinuz(/boot/vmlinuz-4.4.0-63-generic
) points to /boot/vmlinuz-4.4.0-63-generic
 (/boot/vmlinuz-4.4.0-63-generic) -- doing nothing at /var/lib/dpkg/info/linux-image-4.4.0-63-generic.postinst line 491.
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.0-63-generic /boot/vmlinuz-4.4.0-63-generic
run-parts: executing /etc/kernel/postinst.d/dkms 4.4.0-63-generic /boot/vmlinuz-4.4.0-63-generic
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.0-63-generic /boot/vmlinuz-4.4.0-63-generic
update-initramfs: Generating /boot/initrd.img-4.4.0-63-generic
^CFailed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-4.4.0-63-generic.postinst line 1052.
dpkg: error processing package linux-image-4.4.0-63-generic (--configure):
 subprocess installed post-installation script returned error exit status 2
dpkg: dependency problems prevent configuration of linux-image-generic:
 linux-image-generic depends on linux-image-4.4.0-63-generic; however:
  Package linux-image-4.4.0-63-generic is not configured yet.

dpkg: error processing package linux-image-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-signed-image-4.4.0-63-generic:
 linux-signed-image-4.4.0-63-generic depends on linux-image-4.4.0-63-generic (= 4.4.0-63.84); however:
  Package linux-image-4.4.0-63-generic is not configured yet.

dpkg: error processing package linux-signed-image-4.4.0-63-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-signed-image-generic:
 linux-signed-image-generic depends on linux-signed-image-4.4.0-63-generic; however:
  Package linux-signed-image-4.4.0-63-generic is not configured yet.

dpkg: error processing package linux-signed-image-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-image-extra-4.4.0-63-generic:
 linux-image-extra-4.4.0-63-generic depends on linux-image-4.4.0-63-generic; however:
  Package linux-image-4.4.0-63-generic is not configured yet.

dpkg: error processing package linux-image-extra-4.4.0-63-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-generic:
 linux-generic depends on linux-image-generic (= 4.4.0.63.67); however:
  Package linux-image-generic is not configured yet.

dpkg: error processing package linux-generic (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of linux-signed-generic:
 linux-signed-generic depends on linux-signed-image-generic (= 4.4.0.63.67); however:
  Package linux-signed-image-generic is not configured yet.

dpkg: error processing package linux-signed-generic (--configure):
 dependency problems - leaving unconfigured
Processing triggers for initramfs-tools (0.122ubuntu8.8) ...
update-initramfs: Generating /boot/initrd.img-4.4.0-63-generic
^Cdpkg: error processing package initramfs-tools (--configure):
 subprocess installed post-installation script was interrupted
Processing triggers for plymouth-theme-ubuntu-text (0.9.2-3ubuntu13.1) ...
update-initramfs: deferring update (trigger activated)
Errors were encountered while processing:
 linux-image-4.4.0-63-generic
 linux-image-generic
 linux-signed-image-4.4.0-63-generic
 linux-signed-image-generic
 linux-image-extra-4.4.0-63-generic
 linux-generic
 linux-signed-generic
 initramfs-tools

I verified that /boot has enough space:

/dev/nvme0n1p2 496M 314M 157M 67% /boot

Here's my /boot:

$ ls -l /boot
total 325378
-rw-r--r-- 1 root root 1244118 Jan 6 16:44 abi-4.4.0-59-generic
-rw-r--r-- 1 root root 1244118 Jan 18 07:59 abi-4.4.0-62-generic
-rw-r--r-- 1 root root 1245512 Feb 1 11:39 abi-4.4.0-63-generic
-rw-r--r-- 1 root root 1245512 Feb 20 05:40 abi-4.4.0-64-generic
-rw-r--r-- 1 root root 190047 Jan 6 16:44 config-4.4.0-59-generic
-rw-r--r-- 1 root root 190047 Jan 18 07:59 config-4.4.0-62-generic
-rw-r--r-- 1 root root 190247 Feb 1 11:39 config-4.4.0-63-generic
-rw-r--r-- 1 root root 190247 Feb 20 05:40 config-4.4.0-64-generic
drwx------ 3 root root 4096 Dec 31 1969 efi
drwxr-xr-x 5 root root 1024 Feb 23 13:23 grub
-rw-r--r-- 1 root root 10931153 Feb 23 13:56 initrd.img-4.4.0-57-generic
-rw-r--r-- 1 root root 46506399 Feb 9 05:37 initrd.img-4.4.0-59-generic
-rw-r--r-- 2 root root 39526162 Feb 23 13:27 initrd.img-4.4.0-62-generic
-rw-r--r-- 2 root root 39526162 Feb 23 13:27 initrd.img-4.4.0-62-generic.dpkg-bak
-rw-r--r-- 1 root root 0 Feb 23 13:56 initrd.img-4.4.0-62-generic.new
-rw-r--r-- 2 root root 26019417 Feb 23 13:11 initrd.img-4.4.0-63-generic
-rw-r--r-- 2 root root 26019417 Feb 23 13:11 initrd.img-4.4.0-63-generic.dpkg-bak
-rw-r--r-- 1 root root 0 Feb 23 13:56 initrd.img-4.4.0-63-generic.new
-rw-r--r-- 1 root root 39516686 Feb 23 14:50 initrd.img-4.4.0-64-generic
-rw-r--r-- 1 root root 39516505 Feb 23 13:33 initrd.img-4.4.0-64-generic.dpkg-bak
drwx------ 2 root root 12288 Jul 31 2016 lost+found
-rw-r--r-- 1 root root 182704 Jan 28 2016 memtest86+.bin
-rw-r--r-- 1 root root 184380 Jan 28 2016 memtest86+.elf
-rw-r--r-- 1 root root 184840 Jan 28 2016 memtest86+_multiboot.bin
-rw------- 1 root root 3875594 Jan 6 16:44 System.map-4.4.0-59-generic
-rw------- 1 root root 3875553 Jan 18 07:59 System.map-4.4.0-62-generic
-rw------- 1 root root 3883990 Feb 1 11:39 System.map-4.4.0-63-generic
-rw------- 1 root root 3883990 Feb 20 05:40 System.map-4.4.0-64-generic
-rw------- 1 root root 7069136 Jan 6 16:44 vmlinuz-4.4.0-59-generic
-rw------- 1 root root 7071064 Jan 11 07:08 vmlinuz-4.4.0-59-generic.efi.signed
-rw------- 1 root root 7070992 Jan 18 07:59 vmlinuz-4.4.0-62-generic
-rw------- 1 root root 7072920 Feb 3 08:04 vmlinuz-4.4.0-62-generic.efi.signed
-rw------- 1 root root 7087088 Feb 1 11:39 vmlinuz-4.4.0-63-generic
-rw------- 1 root root 7087152 Feb 20 05:40 vmlinuz-4.4.0-64-generic

After uninstall of linux-image-4.4.0-63-generic:

$ sudo dpkg --status linux-image-4.4.0-63-genericPackage: linux-image-4.4.0-63-generic
Status: deinstall ok config-files
Priority: optional
Section: kernel
Installed-Size: 64800
Maintainer: Ubuntu Kernel Team <email address hidden>
Architecture: amd64
Source: linux
Version: 4.4.0-63.84
Provides: fuse-module, ivtv-modules, kvm-api-4, linux-image, redhat-cluster-modules, spl-dkms, virtualbox-guest-modules, zfs-dkms
Depends: initramfs-tools | linux-initramfs-tool, kmod
Recommends: grub-pc | grub-efi-amd64 | grub-efi-ia32 | grub | lilo
Suggests: fdutils, linux-doc-4.4.0 | linux-source-4.4.0, linux-tools, linux-headers-4.4.0-63-generic

There does seem to be a problem with linux-image-extra-4.4.0-63-generic. Trying to uninstall this triggers update-initramfs.

$ sudo dpkg --status linux-image-extra-4.4.0-63-generic
Package: linux-image-extra-4.4.0-63-generic
Status: purge ok half-installed
Priority: optional
Section: kernel
Installed-Size: 148939
Maintainer: Ubuntu Kernel Team <email address hidden>
Architecture: amd64
Source: linux
Version: 4.4.0-63.84
Depends: linux-image-4.4.0-63-generic, crda | wireless-crda

I've tried http://askubuntu.com/questions/650732/apt-fails-to-remove-partially-installed-kernel-and-cant-install-any-other-packa/651596 with no luck. At this point I'm not sure if a partially installed kernel is the problem, and I've run out of troubleshooting insight.

User #2:
https://ubuntuforums.org/showthread.php?t=2351895

User #3:
https://ubuntuforums.org/showthread.php?t=2353606

User #4 (original question):

When upgrading (apt-get upgrade) on 16.04 (ubuntu-mate) I get a hang (left from 9:50am til 16:00 or thereabouts)

I tracked down as far as the call to /etc/kernel/postinst.d/initramfs-tools and added a -v option, the last relevant lines on the terminal when re-running `apt-get upgrade` are as follows:

...
Adding binary /bin/date
Adding binary /sbin/hwclock
Adding binary /sbin/dumpe2fs
Calling hook dmsetup
Calling hook reiserfsprogs
Building cpio /boot/initrd.img-4.4.0-24-generic.new initramfs
^CFailed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-4.4.0-24-generic.postinst line 1052.
dpkg: error processing package linux-image-4.4.0-24-generic (--configure):
 subprocess installed post-installation script returned error exit status 2
dpkg: dependency problems prevent configuration of linux-image-extra-4.4.0-24-generic:
 linux-image-extra-4.4.0-24-generic depends on linux-image-4.4.0-24-generic; however:
  Package linux-image-4.4.0-24-generic is not configured yet.

Checking with `ps fax` before the interrupt, I get this tree - indicating a hang on `sync` task

sudo dpkg --configure -a
 \_ dpkg --configure -a
     \_ /usr/bin/perl /var/lib/dpkg/info/linux-image-4.4.0-24-generic.postinst configure
         \_ run-parts --verbose --exit-on-error --arg=4.4.0-24-generic --arg=/boot/vmlinuz-4.4.0-24-generic /etc/kernel/postinst.d
             \_ /bin/sh -e /etc/kernel/postinst.d/initramfs-tools 4.4.0-24-generic /boot/vmlinuz-4.4.0-24-generic
                 \_ /bin/sh /usr/sbin/update-initramfs -v -c -t -k 4.4.0-24-generic -b /boot
                     \_ sync

I tried moving the resulting initramfs img file in /boot so a new one would be created - a new one is indeed created, hang still occurs on `sync`

See original description
Revision history for this message
tbenst (tbenst) wrote :

Just bumping and noting that this problem may date back to 2010: https://ubuntuforums.org/showthread.php?t=1611122

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in initramfs-tools (Ubuntu):
status: New → Confirmed
Revision history for this message
mesiu84 (nesiu84) wrote :
Download full text (3.3 KiB)

Have the same problem since yesterday. I'm having 4.4.0-47 4.4.0-59 4.4.0-62 4.4.0-64 installed. I was trying to upgrade my system yesterday and 4.4.0-66 was about to be installed, but it hangs on update-initramfs or to be more precise it's hanging on "Building cpio" when running update-initramfs -u -v

Problem is related probably to last 2 upgrades which I did.
Here is the list of packages installed yesterday (aptitude):

[REMOVE, NOT USED] ubuntu-core-launcher:amd64 2.22.6
[UPGRADE] snapd:amd64 2.22.3 -> 2.22.6
[UNCONFIGURED] linux-generic:amd64 4.4.0.66.70
[UNCONFIGURED] linux-headers-4.4.0-66:amd64 4.4.0-66.87
[UNCONFIGURED] linux-headers-4.4.0-66-generic:amd64 4.4.0-66.87
[UNCONFIGURED] linux-headers-generic:amd64 4.4.0.66.70
[UNCONFIGURED] linux-image-4.4.0-66-generic:amd64 4.4.0-66.87
[UNCONFIGURED] linux-image-extra-4.4.0-66-generic:amd64 4.4.0-66.87
[UNCONFIGURED] linux-image-generic:amd64 4.4.0.66.70
[UNCONFIGURED] linux-libc-dev:amd64 4.4.0-66.87
[UNCONFIGURED] lxd:amd64 2.0.9-0ubuntu1~16.04.2
[UNCONFIGURED] lxd-client:amd64 2.0.9-0ubuntu1~16.04.2
[UNCONFIGURED] mdadm:amd64 3.3-2ubuntu7.2
[UNCONFIGURED] snap-confine:amd64 2.22.6
[UNCONFIGURED] ubuntu-snappy-cli:amd64 2.22.6

here is the list of previous upgrade (apt):

Start-Date: 2017-02-21 23:48:52
Commandline: apt upgrade
Install: linux-headers-4.4.0-64:amd64 (4.4.0-64.85, automatic), linux-image-4.4.0-64-generic:amd64 (4.4.0-64.85, automatic), linux-image-extra-4.4.0-64-generic:amd64 (4.4.0-64.85, automatic), linux-headers-4.4.0-64-generic:amd64 (4.4.0-64.85, automatic)
Upgrade: lxc-common:amd64 (2.0.6-0ubuntu1~ubuntu16.04.2, 2.0.7-0ubuntu1~16.04.1), libssh2-1:amd64 (1.5.0-2, 1.5.0-2ubuntu0.1), libdns-export162:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.4, 1:9.10.3.dfsg.P4-8ubuntu1.5), libisccfg140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.4, 1:9.10.3.dfsg.P4-8ubuntu1.5), linux-headers-generic:amd64 (4.4.0.62.65, 4.4.0.64.68), linux-libc-dev:amd64 (4.4.0-62.83, 4.4.0-64.85), linux-image-generic:amd64 (4.4.0.62.65, 4.4.0.64.68), snapd:amd64 (2.21, 2.22.3), bind9-host:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.4, 1:9.10.3.dfsg.P4-8ubuntu1.5), snap-confine:amd64 (2.21, 2.22.3), libisc160:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.4, 1:9.10.3.dfsg.P4-8ubuntu1.5), lxd:amd64 (2.0.8-0ubuntu1~ubuntu16.04.2, 2.0.9-0ubuntu1~16.04.1), liblxc1:amd64 (2.0.6-0ubuntu1~ubuntu16.04.2, 2.0.7-0ubuntu1~16.04.1), libisc-export160:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.4, 1:9.10.3.dfsg.P4-8ubuntu1.5), liblwres141:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.4, 1:9.10.3.dfsg.P4-8ubuntu1.5), ubuntu-core-launcher:amd64 (2.21, 2.22.3), libdns162:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.4, 1:9.10.3.dfsg.P4-8ubuntu1.5), ubuntu-snappy-cli:amd64 (2.21, 2.22.3), libisccc140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.4, 1:9.10.3.dfsg.P4-8ubuntu1.5), libbind9-140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.4, 1:9.10.3.dfsg.P4-8ubuntu1.5), linux-generic:amd64 (4.4.0.62.65, 4.4.0.64.68), lxd-client:amd64 (2.0.8-0ubuntu1~ubuntu16.04.2, 2.0.9-0ubuntu1~16.04.1), lxcfs:amd64 (2.0.5-0ubuntu1~ubuntu16.04.1, 2.0.6-0ubuntu1~16.04.1)
End-Date: 2017-02-21 23:49:31

I'm able to update initrd.img file using following command, it doesnt hang on that "Building cpio":
mkinitramfs -v -c -k 4.4.0-66-generic -o...

Read more...

Revision history for this message
mesiu84 (mesiu84) wrote :

Any updates? As far as I can see the problem still exist.

Revision history for this message
davidg1982 (davidg1982) wrote :

Confirmed. Having same issue.

dpkg --configure -a
Setting up linux-image-4.4.0-57-generic (4.4.0-57.78) ...
Running depmod.
update-initramfs: deferring update (hook will be called later)
The link /initrd.img is a dangling linkto /boot/initrd.img-4.4.0-77-generic
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.0-57-generic /boot/vmlinuz-4.4.0-57-generic
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.0-57-generic /boot/vmlinuz-4.4.0-57-generic
update-initramfs: Generating /boot/initrd.img-4.4.0-57-generic
/usr/sbin/mkinitramfs: 66: .: Can't open /etc/initramfs-tools/initramfs.conf
update-initramfs: failed for /boot/initrd.img-4.4.0-57-generic with 2.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 2
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-4.4.0-57-generic.postinst line 1052.
dpkg: error processing package linux-image-4.4.0-57-generic (--configure):
 subprocess installed post-installation script returned error exit status 2
Setting up linux-image-4.4.0-77-generic (4.4.0-77.98) ...
Running depmod.
update-initramfs: deferring update (hook will be called later)
The link /initrd.img is a dangling linkto /boot/initrd.img-4.4.0-57-generic
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.0-77-generic /boot/vmlinuz-4.4.0-77-generic
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.0-77-generic /boot/vmlinuz-4.4.0-77-generic
update-initramfs: Generating /boot/initrd.img-4.4.0-77-generic
/usr/sbin/mkinitramfs: 66: .: Can't open /etc/initramfs-tools/initramfs.conf
update-initramfs: failed for /boot/initrd.img-4.4.0-77-generic with 2.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 2
Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-4.4.0-77-generic.postinst line 1052.
dpkg: error processing package linux-image-4.4.0-77-generic (--configure):
 subprocess installed post-installation script returned error exit status 2
Errors were encountered while processing:
 linux-image-4.4.0-57-generic
 linux-image-4.4.0-77-generic

Revision history for this message
Robert Wilson (sbcompleteit) wrote :

I'll +1 this thread. Strangest thing though. Earlier this week I spun up four different VMs from the image 'ubuntu-16.04.1-server-amd64.iso'. All configured and updated exactly the same. They continue to work great. Today, on the same VM server, I created another VM EXACTLY the same way I created and updated the others. The update hangs in the exact same place as everyone else is describing. Completely repeatable.

So what broke the update in the last two days?

Revision history for this message
atom (atom) wrote :

Me too.

Turns out the problem for me is updating *while* copying a hard-drive (in another terminal).

The "update-initramfs" script calls "sync" in the "generate_initramfs" function. For me, copying hard-drives at the same time caused "sync" to hang: Then the whole thing hangs.

I CTRL-Z'd (TSTP) my rsync process (doing the copying), and a few seconds later update-initramfs un-hung. Finished without a problem.

Proposed workarounds:
1- stop/pause large disk i/o operations while updating, or
2- edit "/usr/sbin/update-initramfs": comment out the "sync" command in the "generate_initramfs" function

This is why Ubuntu (and other Linux distros) needs a user-space "fsync" command; so instead of trying to sync busy drives that could be busy for days, it can just "fsync ${file}" and get on with the day.

Not sure if this will fix anyone else's problems, but it's all unicorns and rainbows over here, now.

Revision history for this message
amichair (amichai2) wrote :

Thanks for the analysis @atom, I think that explains my issue as well - I had a long-running rsync in parallel. Once killed, the update worked.

Revision history for this message
Jamie (solowinter) wrote :

Commenting out the sync command in the generate_initramfs function of /usr/sbin/update-initramfs worked for me.

Revision history for this message
B. Milde (bmilde) wrote :

Thanks atom!

Indeed, we also had a stale nfs share and an automatic apt-get security update that left our ubuntu 16.04. server in an unclean state since it would silently hang on update-initramfs. Because the mounted nfs share remained unresponsive, update-initramfs was hanging for several days. Killing it and rerunning sudo dpkg --configure -a would result in a deadlock and no other apt-get was possible since dpkg couldn't finish its last operation. Calling update-initramfs manually would also hang and could only be stopped with kill -9 (ctrl+c did not work).

Commenting out the sync command in the generate_initramfs function of /usr/sbin/update-initramfs worked to restore the system to a rebootable state! However a soft shut down didn't work either (shutdown -r now) presumably because it tried to sync the stale nfs mount again.

Revision history for this message
Nikhil Verma (nikhilweee) wrote :

Commenting out the sync command worked for me as well! Thanks!

Revision history for this message
Michael Neuffer (neuffer) wrote :

This problem also still exists with much newer kernels, for me with 4.15.0-12 on Ubuntu 18.04 (alpha)

------------------------------------
 diff -uN /var/lib/dpkg/info/initramfs-tools.postinst~ /var/lib/dpkg/info/initramfs-tools.postinst
--- /var/lib/dpkg/info/initramfs-tools.postinst~ 2018-02-20 22:34:46.000000000 +0100
+++ /var/lib/dpkg/info/initramfs-tools.postinst 2018-03-22 13:08:09.989213260 +0100
@@ -38,10 +38,10 @@
 # Regenerate initramfs whenever we go to dpkg state `installed'
 if [ "x$1" != xtriggered ]; then
  # this activates the trigger, if triggers are working
- update-initramfs -u
+ update-initramfs -u -v
 else
  # force it to actually happen
- DPKG_MAINTSCRIPT_PACKAGE='' update-initramfs -u
+ DPKG_MAINTSCRIPT_PACKAGE='' update-initramfs -u -v
 fi

 # Automatically added by dh_installdeb/11.1.4ubuntu1
-----------------------------------------
diff -uN /usr/sbin/update-initramfs~ /usr/sbin/update-initramfs
--- /usr/sbin/update-initramfs~ 2018-01-05 13:15:45.000000000 +0100
+++ /usr/sbin/update-initramfs 2018-03-22 13:11:34.440124524 +0100
@@ -158,7 +158,9 @@
   mv -f "${initramfs}.new" "${initramfs}"
   set_sha1
   # Guard against an unclean shutdown
- sync
+ # The sync never returns and seems to deadlock for some reason
+ # so comment it out for the time beeing
+ #sync
  else
   mkinitramfs_return="$?"
   remove_initramfs_bak
---------------------------------------------------------------
Also take sync out of commission in order to allow for a clean shutdown (important: re-enable after the reboot!). Also give the system enough time to sync all its buffers on its own before restarting.

cd /bin
mv sync sync.real
cp ls sync

This helped me before and now I have the deadlocked sync issue again.
root@charion:~# ps auxwww | grep sync
systemd+ 1949 0.0 0.0 146076 3364 ? Ssl 11:56 0:00 /lib/systemd/systemd-timesyncd
root 19420 0.0 0.0 7796 780 pts/7 D+ 12:03 0:00 sync
root 22919 0.0 0.0 14756 1120 pts/2 S+ 13:20 0:00 grep --color=auto sync

So I'll try to restart and try to run the
update-initramfs -u -v manually with these changes and then try to finish the apt-get upgrade

Revision history for this message
azizLIGHT (azizlight) wrote :

atom's suggestion to stop all disk operations was the key to fixing this. I had a ddrescue operation going on in the background that I stopped after reading this thread, and BAM it started moving up from 37% and eventually finished.

Basically, this issue affected me as well on Ubuntu 14.04 x64, when trying to update my nvidia drivers from graphics-drivers ppa. I did 'sudo apt install nvidia-384' and it was stuck at 37% saying 'update-initramfs: Generating /boot/initrd.img-3.13.0-144-generic' for 20+ minutes. At that point I went to /boot/ and saw that a 'initrd.img-3.13.0-144-generic.new' was sitting around with a 0 file size since 20 minutes ago. Something was stuck but I didn't know until reading this issue.

Revision history for this message
Colin Watson (cjwatson) wrote :

Re comment #7, coreutils 8.24 added the ability to do "sync -f FILENAME" to sync only the file system containing FILENAME. I think update-initramfs should probably be changed to use that in xenial and newer.

Revision history for this message
Colin Watson (cjwatson) wrote :

(A versioned Depends or versioned Breaks as appropriate on coreutils would probably be a good idea, at least in xenial to avoid upgrade problems from trusty.)

Steve Langasek (vorlon)
Changed in initramfs-tools (Ubuntu):
status: Confirmed → Triaged
importance: Undecided → Medium
Francis Ginther (fginther)
tags: added: id-5b2a8145636ebaee3f5ca982
Revision history for this message
Colin Watson (cjwatson) wrote :

I've posted a merge request upstream:

  https://salsa.debian.org/kernel-team/initramfs-tools/merge_requests/6

Bug Watch Updater (bug-watch-updater)
Changed in initramfs-tools (Debian):
status: Unknown → New
Bug Watch Updater (bug-watch-updater)
Changed in initramfs-tools (Debian):
status: New → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package initramfs-tools - 0.131ubuntu2

---------------
initramfs-tools (0.131ubuntu2) cosmic; urgency=medium

  * Fix permissions on /usr/share/initramfs-tools/init, which were mangled
    by merge-o-matic. LP: #1782854.

 -- Steve Langasek <email address hidden> Fri, 20 Jul 2018 22:54:00 -0400

Changed in initramfs-tools (Ubuntu):
status: Triaged → Fix Released
Tiago Stürmer Daitx (tdaitx)
tags: added: bionic xenial
Zakrapovic (zakrapovic)
no longer affects: initramfs-tools
Tiago Stürmer Daitx (tdaitx)
description: updated
Tiago Stürmer Daitx (tdaitx)
summary: - update-initramfs hangs on upgrade, dpkg unusable, unbootable system
+ [SRU] update-initramfs should sync only the initram hangs on upgrade,
+ dpkg unusable, unbootable system
summary: - [SRU] update-initramfs should sync only the initram hangs on upgrade,
- dpkg unusable, unbootable system
+ [SRU] update-initramfs should sync only the filesystem containing the
+ initramfs image
Revision history for this message
Tiago Stürmer Daitx (tdaitx) wrote :

SRU as a merge proposal for both this and bug 1791959 available at

Bionic: https://code.launchpad.net/~tdaitx/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+merge/355189
Xenial: https://code.launchpad.net/~tdaitx/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+merge/355190

Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello tbenst, or anyone else affected,

Accepted initramfs-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.130ubuntu3.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in initramfs-tools (Ubuntu Bionic):
status: New → Fix Committed
tags: added: verification-needed verification-needed-bionic
Changed in initramfs-tools (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed-xenial
Revision history for this message
Chris J Arges (arges) wrote :

Hello tbenst, or anyone else affected,

Accepted initramfs-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.122ubuntu8.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Tiago Stürmer Daitx (tdaitx) wrote :

Verification done in Xenial and Bionic as per the test case. It works fine and will prevent update-initramfs from being stuck in the described scenarios.

tags: added: verification-done-bionic verification-done-xenial
removed: verification-needed-bionic verification-needed-xenial
tags: removed: verification-needed
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hello tbenst, or anyone else affected,

Accepted initramfs-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.130ubuntu3.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed verification-needed-bionic
removed: verification-done-bionic
Revision history for this message
Tiago Stürmer Daitx (tdaitx) wrote :

Did the testcase verification again for the new 0.130ubuntu3.5 bionic-proposed upload and I confirm again that it works as expected.

tags: added: verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package initramfs-tools - 0.130ubuntu3.5

---------------
initramfs-tools (0.130ubuntu3.5) bionic; urgency=medium

  * debian/initramfs-tools.postinst: fix version comparison and
    so it will run for any version earlier than 0.130ubuntu3.5.

initramfs-tools (0.130ubuntu3.4) bionic; urgency=medium

  [ Colin Watson ]
  * [b4804dd] Only sync the filesystem containing the initramfs
    (Closes: #882380) (LP: #1667512)

  [ Tiago Stürmer Daitx ]
  * debian/initramfs-tools.postinst: remove orphaned old-dkms initrd files
    in /boot. (LP: #1791959)

 -- Tiago Stürmer Daitx <email address hidden> Thu, 27 Sep 2018 16:21:37 +0000

Changed in initramfs-tools (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for initramfs-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package initramfs-tools - 0.122ubuntu8.13

---------------
initramfs-tools (0.122ubuntu8.13) xenial; urgency=medium

  [ Colin Watson ]
  * [b4804dd] Only sync the filesystem containing the initramfs
    (Closes: #882380) (LP: #1667512)

  [ Tiago Stürmer Daitx ]
  * debian/initramfs-tools.postinst: remove orphaned old-dkms initrd files
    in /boot. (LP: #1791959)

 -- Tiago Stürmer Daitx <email address hidden> Tue, 18 Sep 2018 09:18:47 +0000

Changed in initramfs-tools (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
toby schneider (tes) wrote :

This bug fix causes a regression in Xenial when installing initramfs-tools in QEMU, since the version of QEMU in Xenial does not support syncfs():

/bin/sync: error syncing '/boot/initrd.img-4.14.44-v7ooicgsn': Function not implemented

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Related questions

Remote bug watches

Bug watches keep track of this bug in other bug trackers.