Unable to deploy Xenial on a s390x KVM

Bug #1825099 reported by Po-Hsu Lin
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
Undecided
Unassigned
ubuntu-kernel-tests
Fix Released
High
Unassigned
s390-tools (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Undecided
Unassigned

Bug Description

[Impact]

 * chreipl is a tool that can set/change the IPL settings of a guest, meaning what should it boot next upon a reboot call (ie. which device to boot from, or to like dumpkernel, etc). v1.34 has a processing bug, meaning that chreipl does not work with virtio disk drives, as seen in VMs. The fix is to cherrypick a patch which was most likely introduced in teh v1.35 release. Please note the v1.XX series upstream code base is not public, so the patch i'm cherrypicking is a hunk from the diff between xenial and yakkety releases.

[Test Case]

 * Use chreipl in a qemu-kvm VM, it should work.

[Regression Potential]

 * This changes the codepath, of detecting which drive to use. Other usage by user specified id are not affected, and default usage of chreipl in xenial so far has been on best effort basis (ie. guarded with || true, in d-i) because normally it is trivial to control IPL methods from the underlying hypervisor (HMC, z/VM, qemu/libvirt)
 * However, this patch is needed to enable MAAS KVM pods to be able to deploy Xenial guests, as that deployment mechanism depends on working chreipl.

[Other Info]

 * Original bug report:

When trying to deploy Xenial on a s390x KVM, the deployment will fail. (it works for B/C)

Looks like it will fail with:
chreipl: Could not find DASD CCW device "virtio0"
curtin: Installation failed with exception: Unexpected error while running command.
Command: chreipl node /dev/vda
Exit code: 1
Reason: -
Stdout: chreipl: Could not find DASD CCW device "virtio0"

Complete output from the MaaS UI:
curtin: Installation started. (18.1-59-g0f993084-0ubuntu1~18.04.1)
third party drivers not installed or necessary.
Hit:1 http://ports.ubuntu.com/ubuntu-ports xenial InRelease
Get:2 http://ports.ubuntu.com/ubuntu-ports xenial-updates InRelease [109 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports xenial-security InRelease [109 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports xenial-backports InRelease [107 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports xenial/universe s390x Packages [7190 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports xenial/multiverse s390x Packages [119 kB]
Get:7 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x Packages [672 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports xenial-updates/universe s390x Packages [613 kB]
Get:9 http://ports.ubuntu.com/ubuntu-ports xenial-updates/multiverse s390x Packages [11.0 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports xenial-security/main s390x Packages [432 kB]
Get:11 http://ports.ubuntu.com/ubuntu-ports xenial-security/universe s390x Packages [340 kB]
Get:12 http://ports.ubuntu.com/ubuntu-ports xenial-security/multiverse s390x Packages [1716 B]
Get:13 http://ports.ubuntu.com/ubuntu-ports xenial-backports/main s390x Packages [7280 B]
Get:14 http://ports.ubuntu.com/ubuntu-ports xenial-backports/universe s390x Packages [6536 B]
Fetched 9719 kB in 2s (4674 kB/s)
Reading package lists...
zfs-dkms set on hold.
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  crda iw libnl-3-200 libnl-genl-3-200 linux-firmware linux-headers-4.4.0-145
  linux-headers-4.4.0-145-generic linux-headers-generic
  linux-image-4.4.0-145-generic linux-image-generic
  linux-modules-4.4.0-145-generic linux-modules-extra-4.4.0-145-generic
  wireless-regdb
Suggested packages:
  fdutils linux-doc-4.4.0 | linux-source-4.4.0 linux-tools
The following NEW packages will be installed:
  crda iw libnl-3-200 libnl-genl-3-200 linux-firmware linux-generic
  linux-headers-4.4.0-145 linux-headers-4.4.0-145-generic
  linux-headers-generic linux-image-4.4.0-145-generic linux-image-generic
  linux-modules-4.4.0-145-generic linux-modules-extra-4.4.0-145-generic
  wireless-regdb
0 upgraded, 14 newly installed, 0 to remove and 8 not upgraded.
Need to get 74.5 MB of archives.
After this operation, 375 MB of additional disk space will be used.
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x libnl-3-200 s390x 3.2.27-1ubuntu0.16.04.1 [49.7 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x libnl-genl-3-200 s390x 3.2.27-1ubuntu0.16.04.1 [11.0 kB]
Get:3 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x wireless-regdb all 2018.05.09-0ubuntu1~16.04.1 [11.7 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports xenial/main s390x iw s390x 3.17-1 [59.8 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports xenial/main s390x crda s390x 3.13-1 [59.7 kB]
Get:6 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x linux-firmware all 1.157.21 [49.8 MB]
Get:7 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x linux-modules-4.4.0-145-generic s390x 4.4.0-145.171 [7359 kB]
Get:8 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x linux-image-4.4.0-145-generic s390x 4.4.0-145.171 [3882 kB]
Get:9 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x linux-modules-extra-4.4.0-145-generic s390x 4.4.0-145.171 [2689 kB]
Get:10 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x linux-image-generic s390x 4.4.0.145.153 [2708 B]
Get:11 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x linux-headers-4.4.0-145 all 4.4.0-145.171 [10.0 MB]
Get:12 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x linux-headers-4.4.0-145-generic s390x 4.4.0-145.171 [481 kB]
Get:13 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x linux-headers-generic s390x 4.4.0.145.153 [2566 B]
Get:14 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main s390x linux-generic s390x 4.4.0.145.153 [1786 B]
Fetched 74.5 MB in 0s (86.4 MB/s)
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)
Selecting previously unselected package libnl-3-200:s390x.
(Reading database ... 25868 files and directories currently installed.)
Preparing to unpack .../libnl-3-200_3.2.27-1ubuntu0.16.04.1_s390x.deb ...
Unpacking libnl-3-200:s390x (3.2.27-1ubuntu0.16.04.1) ...
Selecting previously unselected package libnl-genl-3-200:s390x.
Preparing to unpack .../libnl-genl-3-200_3.2.27-1ubuntu0.16.04.1_s390x.deb ...
Unpacking libnl-genl-3-200:s390x (3.2.27-1ubuntu0.16.04.1) ...
Selecting previously unselected package wireless-regdb.
Preparing to unpack .../wireless-regdb_2018.05.09-0ubuntu1~16.04.1_all.deb ...
Unpacking wireless-regdb (2018.05.09-0ubuntu1~16.04.1) ...
Selecting previously unselected package iw.
Preparing to unpack .../archives/iw_3.17-1_s390x.deb ...
Unpacking iw (3.17-1) ...
Selecting previously unselected package crda.
Preparing to unpack .../archives/crda_3.13-1_s390x.deb ...
Unpacking crda (3.13-1) ...
Selecting previously unselected package linux-firmware.
Preparing to unpack .../linux-firmware_1.157.21_all.deb ...
Unpacking linux-firmware (1.157.21) ...
Selecting previously unselected package linux-modules-4.4.0-145-generic.
Preparing to unpack .../linux-modules-4.4.0-145-generic_4.4.0-145.171_s390x.deb ...
Unpacking linux-modules-4.4.0-145-generic (4.4.0-145.171) ...
Selecting previously unselected package linux-image-4.4.0-145-generic.
Preparing to unpack .../linux-image-4.4.0-145-generic_4.4.0-145.171_s390x.deb ...
Unpacking linux-image-4.4.0-145-generic (4.4.0-145.171) ...
Selecting previously unselected package linux-modules-extra-4.4.0-145-generic.
Preparing to unpack .../linux-modules-extra-4.4.0-145-generic_4.4.0-145.171_s390x.deb ...
Unpacking linux-modules-extra-4.4.0-145-generic (4.4.0-145.171) ...
Selecting previously unselected package linux-image-generic.
Preparing to unpack .../linux-image-generic_4.4.0.145.153_s390x.deb ...
Unpacking linux-image-generic (4.4.0.145.153) ...
Selecting previously unselected package linux-headers-4.4.0-145.
Preparing to unpack .../linux-headers-4.4.0-145_4.4.0-145.171_all.deb ...
Unpacking linux-headers-4.4.0-145 (4.4.0-145.171) ...
Selecting previously unselected package linux-headers-4.4.0-145-generic.
Preparing to unpack .../linux-headers-4.4.0-145-generic_4.4.0-145.171_s390x.deb ...
Unpacking linux-headers-4.4.0-145-generic (4.4.0-145.171) ...
Selecting previously unselected package linux-headers-generic.
Preparing to unpack .../linux-headers-generic_4.4.0.145.153_s390x.deb ...
Unpacking linux-headers-generic (4.4.0.145.153) ...
Selecting previously unselected package linux-generic.
Preparing to unpack .../linux-generic_4.4.0.145.153_s390x.deb ...
Unpacking linux-generic (4.4.0.145.153) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up libnl-3-200:s390x (3.2.27-1ubuntu0.16.04.1) ...
Setting up libnl-genl-3-200:s390x (3.2.27-1ubuntu0.16.04.1) ...
Setting up wireless-regdb (2018.05.09-0ubuntu1~16.04.1) ...
Setting up iw (3.17-1) ...
Setting up crda (3.13-1) ...
Setting up linux-firmware (1.157.21) ...
Setting up linux-modules-4.4.0-145-generic (4.4.0-145.171) ...
Setting up linux-image-4.4.0-145-generic (4.4.0-145.171) ...
I: /boot/vmlinuz.old is now a symlink to vmlinuz-4.4.0-145-generic
I: /boot/initrd.img.old is now a symlink to initrd.img-4.4.0-145-generic
I: /boot/vmlinuz is now a symlink to vmlinuz-4.4.0-145-generic
I: /boot/initrd.img is now a symlink to initrd.img-4.4.0-145-generic
Setting up linux-modules-extra-4.4.0-145-generic (4.4.0-145.171) ...
Setting up linux-image-generic (4.4.0.145.153) ...
Setting up linux-headers-4.4.0-145 (4.4.0-145.171) ...
Setting up linux-headers-4.4.0-145-generic (4.4.0-145.171) ...
Setting up linux-headers-generic (4.4.0.145.153) ...
Setting up linux-generic (4.4.0.145.153) ...
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Processing triggers for linux-image-4.4.0-145-generic (4.4.0-145.171) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.4.0-145-generic
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Preparing boot device: vda (0000).
Done.
/etc/kernel/postinst.d/zz-zipl:
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Preparing boot device: vda (0000).
Done.
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Preparing boot device: vda (0000).
Done.
Setting up swapspace version 1, size = 2 GiB (2147479552 bytes)
no label, UUID=91c4722d-4d92-4235-ac13-e2e659044c2e
update-initramfs: Generating /boot/initrd.img-4.4.0-145-generic
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
Using config file '/etc/zipl.conf'
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Preparing boot device: vda (0000).
Done.
--2019-04-17 06:42:38-- http://10.246.72.3:5248/MAAS/metadata/latest/by-id/qnwagm/
Connecting to 10.246.72.3:5248... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2 [text/plain]
Saving to: '/dev/null'

     0K 100% 739K=0s

2019-04-17 06:42:38 (739 KB/s) - '/dev/null' saved [2/2]

chreipl: Could not find DASD CCW device "virtio0"
curtin: Installation failed with exception: Unexpected error while running command.
Command: chreipl node /dev/vda
Exit code: 1
Reason: -
Stdout: chreipl: Could not find DASD CCW device "virtio0"

Stderr: ''

MAAS version: 2.5.2 (7523-ge4ecbd54d-0ubuntu1~18.04.1)

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

This issue is blocking SRU testing on s390x KVM

description: updated
Changed in ubuntu-kernel-tests:
importance: Undecided → High
Frank Heimes (fheimes)
tags: added: s390x
Revision history for this message
Sean Feole (sfeole) wrote :

raising the prioirty to critical as this is blocking xenial s390x-kvm SRU testing

Changed in maas (Ubuntu):
importance: Undecided → Critical
Revision history for this message
Andres Rodriguez (andreserl) wrote :

We have never supported s390x on Xenial. s390x was enabled for bionic+.

Changed in maas (Ubuntu):
status: New → Won't Fix
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

s390x in general, is supported including qemu/kvm/openstack in xenial.

it would be nice if maas could deploy xenial kvm pod.

And this looks like a simple bug in chreipl, as in s390-tools?

Revision history for this message
Frank Heimes (fheimes) wrote :

+1 for Xenial KVM guest support (not for xenial as base OS for MAAS)

Revision history for this message
Andres Rodriguez (andreserl) wrote :

Then if this is a bug in chreipl, then there shouldn't really be any reason why MAAS couldn't deploy in Xenial. That said, what I'm surprised about is s390x VM off MAAS when [1] was never backported to Xenial.

[1]: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1790901

Revision history for this message
Andres Rodriguez (andreserl) wrote :

oh i see what's happening here, just to confirm:

1. You have a Bionic host
2. You are trying to deploy Xenial.
3. Xenial will use the rom available in bionic from comment #6 to PXE.
4. chreipl fails to configure.

So based on 4 and comment #4, then this would be like an issue on chreipl which is external to MAAS.

Also just to be clear, MAAS cannot support Xenial as a KVM host, but provided the bug in question gets fixed, there should be no reason why Xenial cannot be deployed on a Bionic+ KVM host.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Adjusting bug tasks

Changed in maas (Ubuntu Xenial):
status: New → Invalid
Changed in maas (Ubuntu):
status: Won't Fix → Invalid
Changed in s390-tools (Ubuntu Xenial):
status: New → Confirmed
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Hello Andres,

1. Yes, maas is running on a Bionic server
2. Yes, we are trying to deploy a Xenial guest
3. Not sure what is the rom here, isn't is an imported image on MAAS
4. From the log it looks like this

Thanks

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Hopefully this fixes s390-tools chreipl for xenial.

Cherrypick from s390-tools v1.35 release.

Changed in s390-tools (Ubuntu Xenial):
status: Confirmed → In Progress
Changed in s390-tools (Ubuntu):
status: New → Fix Released
description: updated
Revision history for this message
Frank Heimes (fheimes) wrote :

I wondering if it makes sense to cherrypick also for lsreipl, since both tools are almost used every time in combination, hence I think it would be good if they are on a similar level.
Not totally sure if it is used here, but could be, for example for verification purposes ...

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

cherrypick what for lsreipl? there are no code changes in lsreipl since forever...

note this is a bug in trying to resolve chreipl / -> where one needs to transverse the mountpoint of /, device, and finally get the CCW id of the device. Which was performed wrong for virtio devices.

Revision history for this message
Frank Heimes (fheimes) wrote :

I just recognized that you cherry-picked just the minimal needed changes in function ccw_busid_get_sysfs_new.
I thought you picked the entire latest chreipl command from a newer s390-tools version - and in this case it would have been good if lsreipl is aligned to it and on the same level (in case it had changes, too).
But with your minimal cherry pick this shouldn't be needed - I agree - and regression risk is also minimal.

Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello Po-Hsu, or anyone else affected,

Accepted s390-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/s390-tools/1.34.0-0ubuntu8.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in s390-tools (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-xenial
Mathew Hodson (mhodson)
no longer affects: maas (Ubuntu Xenial)
no longer affects: maas (Ubuntu)
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

$ dpkg-query -W s390-tools
s390-tools 1.34.0-0ubuntu8.9
$ sudo chreipl /dev/vda1
chreipl: Could not find DASD CCW device "virtio1"

Upgraded:

$ dpkg-query -W s390-tools
s390-tools 1.34.0-0ubuntu8.10

ubuntu@ubuntu:~$ sudo chreipl /dev/vda1
Re-IPL type: ccw
Device: 0.0.0000
Loadparm: ""
ubuntu@ubuntu:~$ echo $?
0

Everything is awesome!

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Sean Feole (sfeole)
Changed in ubuntu-kernel-tests:
status: New → In Progress
Revision history for this message
Sean Feole (sfeole) wrote :

Reviewed the changes made by Frank and Dimitri, thank you. As Dimitiri has verified this fixed I also verified the fix in -proposed, all appears to work, now waiting for it to land in main.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package s390-tools - 1.34.0-0ubuntu8.10

---------------
s390-tools (1.34.0-0ubuntu8.10) xenial; urgency=medium

  * chreipl: fix chreipl node for virtio devices LP: #1825099

 -- Dimitri John Ledkov <email address hidden> Fri, 26 Apr 2019 01:10:10 +0100

Changed in s390-tools (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for s390-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: New → Fix Released
Revision history for this message
Sean Feole (sfeole) wrote :

Verified fix released as kernel c-i is now passing for xenial s390x-kvm

Changed in ubuntu-kernel-tests:
status: In Progress → Fix Released
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

yeah \o/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.