CentOS 7 1901 causes Curtin to hang when deploying with LVM

Bug #1815666 reported by Lee Trager
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
curtin
Fix Released
High
Unassigned

Bug Description

Upstream has released a new CentOS 7 image [1]. I built a test image[2] for MAAS using lp:maas-images and tried to deploy it with MAAS and Curtin master. When using a flat layout the image deploys fine however when I use LVM Curtin hangs until the installation times out.

This is blocking lp:maas-images from updating the CentOS image

[1] https://cloud.centos.org/centos/7/images/
[2] http://162.213.35.187/centos/centos70/amd64/20190101_01/root-tgz

Related branches

Revision history for this message
Lee Trager (ltrager) wrote :
Revision history for this message
Lee Trager (ltrager) wrote :
Revision history for this message
Lee Trager (ltrager) wrote :
Revision history for this message
Lee Trager (ltrager) wrote :

It looks like GRUB probing for LVM is what is hanging.

 1052 ? Ss 0:00 /usr/bin/python3 /usr/bin/cloud-init modules --mode=final
 1089 ? S 0:00 \_ /bin/sh -c tee -a /var/log/cloud-init-output.log
 1090 ? S 0:00 | \_ tee -a /var/log/cloud-init-output.log
 1095 ? S 0:00 \_ python3 -m curtin.commands.main --install-deps install --config=configs/config-000.cfg --config=configs/config-001.cfg --config=configs/config-002.cfg --config=configs/config-003.cfg --config=configs/co
 2543 ? S 0:00 \_ python3 -m curtin.commands.main curthooks
 2554 ? S 0:00 \_ python3 /tmp/tmptfgz0yyr/target/curtin/curtin-hooks.py
16076 ? S 0:00 \_ /bin/bash /curtin/helpers/install-grub --os-family=redhat /tmp/tmptfgz0yyr/target /dev/vda
16087 ? S 0:00 \_ sh -exc pkg=$1; shift; bootid=$1; shift; bootver=$1; shift; grubpost="" case $bootid in debian|ubuntu)
16236 ? S 0:00 \_ /bin/sh /usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg
16243 ? S 0:00 \_ /bin/sh /usr/sbin/grub2-mkconfig -o /boot/grub2/grub.cfg
16244 ? S 0:00 \_ /usr/sbin/grub2-probe --device /dev/mapper/vg0-lv0 --target=fs_uuid
16245 ? S 0:00 \_ vgs --options vg_uuid,pv_name --noheadings --separator :

Revision history for this message
Ryan Harper (raharper) wrote :

Well, that's grub2 inside centos7 ...

Can we get target rpm versions of centos7 grub2 and lvm packages?

I might guess that newer lvm is wanting to connect to lvmetad or some other daemon that's not running in the target...

Changed in curtin:
status: New → Incomplete
Revision history for this message
Lee Trager (ltrager) wrote :

Both grub2 and lvm2 have been updated since the last image build but nothing has stands out

LVM2:
* Mon Dec 17 2018 Marian Csontos <email address hidden> - 7:2.02.180-10.el7_6.3
- Fix component detection for MD RAID version 1.0 and 0.90.
- Use sync io if async io_setup fails, or when use_aio=0 is set in config.
- Fix dmstats report printing no output.

* Wed Oct 31 2018 Marian Csontos <email address hidden> - 7:2.02.180-10.el7_6.2
- Fix possible write beyond metadata area.

* Wed Oct 10 2018 Marian Csontos <email address hidden> - 7:2.02.180-10.el7_6.1
- Fix clustered mirror repair.

GRUB2:
* Tue Jan 29 2019 CentOS Sources <email address hidden> - 2.02-0.76.el7.centos.1
- Roll in CentOS Secureboot keys
- Move the edidir to be CentOS, so people can co-install fedora, rhel and centos

* Mon Nov 12 2018 Javier Martinez Canillas <email address hidden> - 2.02-0.76.e7_6.1
- Re-enable regexp module
  Resolves: rhbz#1647527

Changed in curtin:
status: Incomplete → New
Revision history for this message
Lee Trager (ltrager) wrote :

I attached strace to the running $(vgs --options vg_uuid,pv_name --noheadings --separator :) process. Its stuck in the following loop. /run isn't being bind mounted into the target environment.

nanosleep({0, 100000000}, NULL) = 0
readlink("/sys/dev/block/252:1", "../../devices/pci0000:00/0000:00"..., 1024) = 60
stat("/sys/devices/pci0000:00/0000:00:07.0/virtio3/block/vda/vda1/uevent", {st_mode=S_IFREG|0644, st_size=4096, ...}) = 0
readlink("/sys/devices/pci0000:00/0000:00:07.0/virtio3/block/vda/vda1/subsystem", "../../../../../../../class/block", 1024) = 32
open("/sys/devices/pci0000:00/0000:00:07.0/virtio3/block/vda/vda1/uevent", O_RDONLY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=4096, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f594227f000
read(4, "MAJOR=252\nMINOR=1\nDEVNAME=vda1\nD"..., 4096) = 57
read(4, "", 4096) = 0
close(4) = 0
munmap(0x7f594227f000, 4096) = 0
open("/run/udev/data/b252:1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

Revision history for this message
Lee Trager (ltrager) wrote :

Patching[1] Curtin to bind mount /run fixes CentOS 1901 deployments to both BIOS and UEFI systems.

[1] http://paste.ubuntu.com/p/schWRKZZzH/

Ryan Harper (raharper)
Changed in curtin:
importance: Undecided → High
status: New → In Progress
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

This bug is fixed with commit 0dc77168 to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=0dc77168

Changed in curtin:
status: In Progress → Fix Committed
Revision history for this message
Dan Watkins (oddbloke) wrote : Fixed in curtin version 19.1.

This bug is believed to be fixed in curtin in version 19.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Changed in curtin:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.