Optimistic approach of using zkey for encrypted installations on s390x not working

same with latest 20.10 image (RC from Oct 20th)
works with DASD, but not with zFCP

UUID=0751262d-b006-45e5-84b1-d4e233ed8e9b as the root= arg is wrong. When using dm-crypt encryption, the root= arg should be a dm name.....

what is the contents of /etc/fstab of the installed system? what is curtin log?

It seems as if https://wiki.ubuntu.com/FSTAB is not followed.

Indeed, the cmdline defines root=UUID=288689a9-116a-4867-93a2-75a963062c61 while /target/etc/fstab contains:
/dev/disk/by-id/dm-uuid-LVM-wZilMCEJKlgdikVkEQdHhMiC8wYpJXVuDn9HuJvMYqYZu5DCbwb7nmGjMthndwJE

After reboot, inside the initramfs, it isn't able to setup or find any LVM LVs or VGs and nothing which would contain this UUID.

Some more information about the installed system, after installation, but before reboot (incl /etc/fstab):

root@ubuntu-server:/# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
loop0 7:0 0 765M 1 loop /target/cdrom
loop1 7:1 0 328.7M 1 loop /media/filesystem
loop2 7:2 0 231.4M 1 loop
loop3 7:3 0 15.6M 1 loop /usr/lib/modules
loop4 7:4 0 47.9M 1 loop /snap/core18/1945
loop5 7:5 0 50.8M 1 loop /snap/subiquity/2278
loop6 7:6 0 29M 1 loop /snap/snapd/10704
loop7 7:7 0 62.9M 1 loop /snap/lxd/19189
sda 8:0 0 64G 0 disk
|-sda1 8:1 0 1G 0 part
|-sda2 8:2 0 63G 0 part
`-mpathb 253:1 0 64G 0 mpath
  |-mpathb-part1 253:2 0 1G 0 part /target/boot
  `-mpathb-part2 253:3 0 63G 0 part
    `-dm_crypt-0 253:4 0 63G 0 crypt
      `-ubuntu--vg-ubuntu--lv 253:5 0 31.5G 0 lvm /target
sdb 8:16 0 64G 0 disk
`-mpatha 253:0 0 64G 0 mpath
sdc 8:32 0 64G 0 disk
|-sdc1 8:33 0 1G 0 part
|-sdc2 8:34 0 63G 0 part
`-mpathb 253:1 0 64G 0 mpath
  |-mpathb-part1 253:2 0 1G 0 part /target/boot
  `-mpathb-part2 253:3 0 63G 0 part
    `-dm_crypt-0 253:4 0 63G 0 crypt
      `-ubuntu--vg-ubuntu--lv 253:5 0 31.5G 0 lvm /target
sdd 8:48 0 64G 0 disk
`-mpatha 253:0 0 64G 0 mpath
sde 8:64 0 64G 0 disk
|-sde1 8:65 0 1G 0 part
|-sde2 8:66 0 63G 0 part
`-mpathb 253:1 0 64G 0 mpath
  |-mpathb-part1 253:2 0 1G 0 part /target/boot
  `-mpathb-part2 253:3 0 63G 0 part
    `-dm_crypt-0 253:4 0 63G 0 crypt
      `-ubuntu--vg-ubuntu--lv 253:5 0 31.5G 0 lvm /target
sdf 8:80 0 64G 0 disk
`-mpatha 253:0 0 64G 0 mpath
sdg 8:96 0 64G 0 disk
|-sdg1 8:97 0 1G 0 part
|-sdg2 8:98 0 63G 0 part
`-mpathb 253:1 0 64G 0 mpath
  |-mpathb-part1 253:2 0 1G 0 part /target/boot
  `-mpathb-part2 25...

I don't know a lot about the "zkey" tool, but I wonder if this is problematic:

root@ubuntu-server:/home/installer# zkey generate /tmp/test_zkey
WARNING: APQN 00.0010: No master key is set.
WARNING: Not all APQNs have the same master key or fulfill the requirements.

CARD.DOMAIN NEW MK CURRENT MK OLD MK TYPE
---------------------------------------------------------------------
00.0010 - - - CEX5C

zkey: Your master key setup is improper
root@ubuntu-server:/home/installer# echo $?
1

@slyon it appears you are running zkey generate on a host that has incomplete or incorrect APQN setup and needs master key setup to be completed. Last time frank did it. I did try to do it once and it required downloading proprietary java apps and run tools that were had .exe extensions and yet worked to finish that setup.

Can you please check with frank if the machine you are on, is setup correctly?

Imho zkey generate should not generate any warnings.

@slyon comment #5

So if fstab is sane, but cmdline is not.... what is inside /etc/zipl.conf then ? cause the command line there should be like in fstab.

Well, I am not sure if master keys are set on all the crypto domains of the 'clound' lpars.
I thought they are, but doesn't look like.
Since setting them is more effort than doing a reinstall and a system that has one set,
I thought I just do a quick install on one of the hwe systems (lp15),
where I could reproduce the issue:

ALERT! UUID=404a301e-d8cd-4abf-ae77-b0582122ba7f does not exist. Dropping to a
shell!

and I can check that the system is one with a master key properly set:
root@ubuntu-server:/# zkey generate /tmp/test_zkey
root@ubuntu-server:/# ls -l /tmp/test_zkey
-rw------- 1 root root 64 Jul 27 17:44 /tmp/test_zkey
root@ubuntu-server:/#

So this can be used as reference, too. (Ping me on MM.)

Just wanted to quickly mention (kind of for clarification) that the installation needs to work in both cases.

In case a master key is set, it is used for the LUKS install,
in case no master key is set, the installation should fall back to a 'standard' LUKS install.

So there is still value having the lpar lp9 and the issue happens in both cases.

(Sorry if this was already clear .)

This code in curtin needs to be updated, it seems: https://git.launchpad.net/curtin/tree/curtin/commands/curthooks.py#n173

Aha! Thanks to all of you for your input. Indeed the etc/zipl.conf seems to be off and apparently Michael already pointed out the relevant code where this can be fixed:

root@ubuntu-server:/# cat target/etc/zipl.conf

# This has been modified by the MAAS curtin installer
[defaultboot]
default=ubuntu

[ubuntu]
target = /boot
image = /boot/vmlinuz
ramdisk = /boot/initrd.img
parameters = root=UUID=288689a9-116a-4867-93a2-75a963062c61

So after testing MP#406305 on a live LPAR installation, I can see that target/etc/zipl.conf now has a proper "root=/dev/disk/by-id/dm-uuid-LVM-3Bb5WPhx3mleOG6kyqWO4aDu1FP63GeB4HVGUbMW8r8UIO7" parameter (that can also be read from /proc/cmdline after reboot), but the same issue still happens and the reboot drops into initramfs:

"Gave up waiting for root file system device. Common problems:"
"- Boot args (cat /proc/cmdline)"
"- Check rootdelay= (did the system wait long enough?)"
"- Missing modules (cat /proc/modules; ls /dev)"
"ALERT! /dev/disk/by-id/dm-uuid-LVM-3Bb5WPhx3mleOG6kyqWO4aDu1FP63GeB4HVGUbMW8r8U"
"IO7iZBmzvTE5dfbppRKi does not exist. Dropping to a shell!"

"BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu6.3) built-in shell (ash)"
"Enter 'help' for a list of built-in commands."

"(initramfs) [6n"

For some reason initramfs does not seem to find the LVM LV/VG after the LUKS2 partition was unlocked... I need to dig a bit deeper.

# ls -la /dev/disk/by-id
lrwxrwxrwx 1 10 wwn-0x6005076306ffd6b6000000000000260b -> ../../dm-0
lrwxrwxrwx 1 10 dm-uuid-mpath-36005076306ffd6b6000000000000260b -> ../
../dm-0
lrwxrwxrwx 1 10 dm-name-mpatha -> ../../dm-0
lrwxrwxrwx 1 10 scsi-36005076306ffd6b6000000000000260b -> ../../dm-0
lrwxrwxrwx 1 10 scsi-36005076306ffd6b6000000000000260a -> ../../dm-1
lrwxrwxrwx 1 10 wwn-0x6005076306ffd6b6000000000000260a -> ../../dm-1
lrwxrwxrwx 1 10 dm-name-mpathb -> ../../dm-1
lrwxrwxrwx 1 10 dm-uuid-mpath-36005076306ffd6b6000000000000260a -> ../
../dm-1
lrwxrwxrwx 1 10 dm-uuid-part2-mpath-36005076306ffd6b6000000000000260a
-> ../../dm-3
lrwxrwxrwx 1 10 scsi-36005076306ffd6b6000000000000260a-part2 -> ../../
dm-3
lrwxrwxrwx 1 10 dm-name-mpathb-part2 -> ../../dm-3
lrwxrwxrwx 1 10 wwn-0x6005076306ffd6b6000000000000260a-part2 -> ../../
dm-3
lrwxrwxrwx 1 10 scsi-36005076306ffd6b6000000000000260a-part1 -> ../../
dm-2
lrwxrwxrwx 1 10 dm-uuid-part1-mpath-36005076306ffd6b6000000000000260a
-> ../../dm-2
lrwxrwxrwx 1 10 wwn-0x6005076306ffd6b6000000000000260a-part1 -> ../../
dm-2
lrwxrwxrwx 1 10 dm-name-mpathb-part1 -> ../../dm-2
lrwxrwxrwx 1 10 dm-uuid-CRYPT-LUKS2-35211c8c1c494d85a860600de64ae1f7-d
m_crypt-0 -> ../../dm-4
lrwxrwxrwx 1 10 lvm-pv-uuid-frApMx-rUVa-E40s-DnKL-6lQd-6G8z-HnBshC ->

@slyon for your next debugging step i wonder if we have ordering issues between multipath; lvm2; cryptsetup => normally initrd should have a loop to setup multipath / lvm2 / cryptsetup in circles until it's not doing anything.

Maybe it's not iterating enough times, or is done out of loop by udev => hence it thinks that everything is done and nothing will change anymore, but in practice there is more things to do.

In the past we had bugs like these between lvm2 / cryptsetup / mdadm. But I don't recall any with multipath. Maybe we lack integrating them all correctly.

From the output it looks like we have the LVM physical volume for what will ultimately be the rootfs set up but not the logical volume (I don't think we can tell from this output if the VG is set up). What happens if you run vgs, vgchange -a, that sort of thing from the prompt?

I don't see anything in any initramfs script that explicitly loops around multipath / lvm2 / cryptsetup but shouldn't udev handle this these days? Maybe we can make udev super verbose somehow like adding udev.log_priority=DEBUG (or is that rd.udev.log_priority=DEBUG?) to the kernel commandline.

OK. I think I finally found the root-cause:

Inside the initramfs /etc/lvm/lvm.conf contains this line:
```
filter = [ "a|/dev/disk/by-id/dm-uuid-.*mpath-.*|", "r|.*|" ]
```

This makes LVM only search the multipath devices for LVM volumes, but in our case the multipath device contains a LUKS container, that contains the LVM volume.
So LVM needs to search the /dev/mapper/dm_crypt-0 block device, which is filtered out by the "r|.*|" regex.

Removing that line from /etc/lvm/lvm.conf (or adopting it accordingly) and running vgchange -a ay makes the ubuntu-vg show up:

```
$ blkid
/dev/mapper/dm_crypt-0: UUID="RP50EB-4DhN-LV9t-9Grx-QSNz-qfK8-lBEY56" TYPE="LVM2
_member"
/dev/mapper/mpatha-part1: UUID="b65d0cc1-0976-48b8-b6bb-e91049700380" BLOCK_SIZE
="4096" TYPE="ext4" PARTUUID="9bc9801d-70b4-42a8-8607-1ca971af476b"
/dev/mapper/mpatha-part2: UUID="b3ebe2f9-10e6-47c0-b30e-256e613eabdd" TYPE="cryp
to_LUKS" PARTUUID="453e1838-94ff-49da-b02e-268a0dabc1de"
/dev/mapper/ubuntu--vg-ubuntu--lv: UUID="e91c7591-b3b0-43db-aad4-092545325799" B
LOCK_SIZE="4096" TYPE="ext4"
/dev/sda: PTUUID="833ca1cf-e90e-4443-b7d3-b41ec73fc35b" PTTYPE="gpt"
/dev/sdc: PTUUID="833ca1cf-e90e-4443-b7d3-b41ec73fc35b" PTTYPE="gpt"
/dev/sde: PTUUID="833ca1cf-e90e-4443-b7d3-b41ec73fc35b" PTTYPE="gpt"
/dev/sdg: PTUUID="833ca1cf-e90e-4443-b7d3-b41ec73fc35b" PTTYPE="gpt"
/dev/mapper/mpatha: PTUUID="833ca1cf-e90e-4443-b7d3-b41ec73fc35b" PTTYPE="gpt"
```

Now let's find out where and why this file is generated.

The file is generated/modified in curtin's curthooks.py "detect_and_handle_multipath()" method.

This needs to be fixed to make it boot an encrypted LVM VG, but to make the "Optimistic" encryption work (using the paes_s390.ko kernel module), we probably also need the zkey tool inside the initramfs (which is currently not included).

So https://code.launchpad.net/~raharper/curtin/+git/curtin/+merge/381784 is where this code was added and has some comments about why -- basically without it, vgchange -a sees the PV multiple times (once per path) so it's necessary to tell lvm to only consider the multipath dm device, not the underlying disk devices.

This sounds to me like a udev configuration bug tbh -- surely the lvm2 package's rules shouldn't be running until after we know if a disk is multipathed or not? But maybe there's some reason this is hard that I don't understand.

OK, having done a bit more reading: I don't think anything in here has to do with zkey at all does it? This is just root being an lv on a vg on an encrypted multipath disk not working afaict... and indeed I can reproduce this in an amd64 vm locally.

The other thing that I don't understand is why the linked merge proposal deals with a failure of lvcreate at install time by writing a lvm.conf for the target. AFAICT, there are udev rules in place so that LVM will look at the multipath device and not the component drives already.

I don't understand what caused lvcreate to fail with the "Cannot update volume group ubuntu-vg with duplicate PV devices" message in the bug report linked from the merge proposal. probert used to be a bit enthusiastic with running lvm commands, and we used to occasionally run it while the install was running which did lead to all sorts of fun problems. I can see from the logs that probert was indeed run while curtin was running so I'm going to blame the lvcreate failure on that with no other evidence.

tl;dr it looks like we can delete curtin's customization of lvm.conf (certainly, this works locally).

Yes, I agree that this is pretty much unrelated to zkey. I discussed this with Frank and we found that the optimistic decryption is being done by the paes_s390.ko kernel module alone, so we do not need the "zkey" tool inside the initramfs.

I only realized after preparing another MP that you already proposed a change of removing the lvm.conf modification, so I'll leave it to @mwhudson and the subiquity devs to choose which MP best fits their needs. I tested both MPs and can confirm that both are fixing this bug (using a different approach).

I could only test the non-optimistic approach on the LPAR that I have access to, so I'd like to ask @fheimes to re-test on another LPAR with the HSM/CryptoCard enabled.

Steps to test:
Git clone one of the two MPs (or both, one after another):
https://git.launchpad.net/~slyon/curtin?h=slyon%2Flp1895192-2
https://git.launchpad.net/~mwhudson/curtin?h=lp-1895192

Boot the live-installer, SSH into it as usual.
Choose: Help -> Enter shell

Inside the live installer:
# adduser ubuntu # Add ubuntu:ubuntu user, to enable scp access
# mkdir /override
# chown -R ubuntu:ubuntu /override

On your local machine:
% scp curtin/block/lvm.py ubuntu@s1lp9:/override
% scp curtin/commands/curthooks.py ubuntu@s1lp9:/override

Inside the live installer:
# mount --bind /override/lvm.py /snap/subiquity/current/lib/python3.6/site-packages/curtin/block/lvm.py
# mount --bind /override/curthooks.py /snap/subiquity/current/lib/python3.6/site-packages/curtin/commands/curthooks.py
# exit # Drop back into the installer
-> Finish normal Multipass+LUKS+LVM installation (using ubuntu:ubuntu here)
-> reboot after installation finished
-> unlock encrypted volume on console

On your local machine and the freshly installed & rebooted system:
% ssh ubuntu@s1lp9
ubuntu@s390x-test:~$ sudo cryptsetup status $(awk '{ print $1 }' /etc/crypttab )
[sudo] password for ubuntu:
/dev/mapper/dm_crypt-0 is active and is in use.
  type: LUKS2
  cipher: aes-xts-plain64
  keysize: 512 bits
  key location: keyring
  device: /dev/mapper/mpatha-part2
  sector size: 512
  offset: 32768 sectors
  size: 132083712 sectors
  mode: read/write

Hi, yes sorry for the duplication of effort. I think I'm going to merge my version (it deletes more code!) but we can go back to yours if testing reveals problems with mine!

This bug is fixed with commit 3673687e to curtin on branch master.
To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=3673687e

Subiquity edge should have the fix for this now btw

On Tue, 3 Aug 2021, 18:01 Frank Heimes, <email address hidden> wrote:

> ** Changed in: ubuntu-z-systems
> Status: New => In Progress
>
> --
> You received this bug notification because you are a member of Canonical
> Foundations Team, which is a bug assignee.
> https://bugs.launchpad.net/bugs/1895192
>
> Title:
> Optimistic approach of using zkey for encrypted installations on s390x
> not working
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1895192/+subscriptions
>
>

I followed the steps on an LPAR that has an active crypto domain with a master key set, using FCP multipath disk storage.
I did two installations (using Impish), one for each of the MPs, and the results are the same.
The LUKS installations complete and I can boot into the OS afterwards.

However, in both cases a normal (fallback) LUKS installation is done and 'aes-xts-plain64' is used instead of 'paes-xts-plain64'.

$ sudo cryptsetup status $(awk '{ print $1 }' /etc/crypttab ) | grep cipher
  cipher: aes-xts-plain64

Wanting to be sure I checked the master key using the commands from the CCA proprietary package:
/opt/IBM/CCA/bin/panel.exe -mk-query --mktype=AES --mkregister=CURRENT
Preparing to QUERY master key verification pattern

Query of Key Verification Pattern for Master key [AES-MK ] [KEY-KM ] returned:

RND[0000000000000000]
VER[E113AC74BB40CCA1]
as well as using zkey:
# sudo zkey generate --xts --name key-for-verification
# sudo zkey list
Key : key-for-verification
-------------------------------------------------------------------------------------
        Description :
        Secure key size : 128 bytes
        Clear key size : 512 bits
        XTS type key : Yes
        Key type : CCA-AESDATA
        Volumes : (none)
        APQNs : (none)
        Key file name : /etc/zkey/repository/key-for-verification.skey
        Sector size : (system default)
        Volume type : LUKS2
        Verification pattern : 1d6f7ea0dfebdd8b65f0970e573522fd
                               85ebee9ee93ca80cc87dc020aa1eb309
        Created : 2021-08-03 06:38:28
        Changed : (never)
        Re-enciphered : (never)
and both looked fine.

To be sure I re-did the same installation using a single DASD with LVM instead and this worked and resulted in the use of 'paes-xts-plain64':

$ sudo cryptsetup status $(awk '{ print $1 }' /etc/crypttab ) | grep cipher
  cipher: paes-xts-plain64

This shows that an installation using zkey is generally possible, but that there seems to be another issue in the multipath code path that does not properly trigger the zkey usage (like with DASDs).

[I've attached a zip containing two files with the detailed steps that I followed.]

Just to be clear, the issue with the initramfs not being able to find and open the disk:
"ALERT! UUID=404a301e-d8cd-4abf-ae77-b0582122ba7f does not exist. Dropping to a shell!"
is solved.

from a DASD install

from an fcp install

Thank you for the fcp install log. Here is the relevant excerpt, that clearly shows the error:

[...]
return volume path /dev/mapper/mpatha-part2
Checking if zkey encryption is supported...
Skipping kernel module load, pkey already loaded
Running command ['zkey', 'generate', '/tmp/tmp5ortqgw5'] with allowed return codes [0] (capture=True)
zkey encryption supported.
Attempting to setup zkey for None:dm_crypt-0
Running command ['zkey', 'generate', '--xts', '--volume-type', 'luks2', '--sector-size', '4096', '--name', 'dm_crypt-0', '--description', 'curtin generated zkey for None:dm_crypt-0', '--volumes', 'None:dm_crypt-0'] with allowed return codes [0] (capture=True)
Unexpected error while running command.
Command: ['zkey', 'generate', '--xts', '--volume-type', 'luks2', '--sector-size', '4096', '--name', 'dm_crypt-0', '--description', 'curtin generated zkey for None:dm_crypt-0', '--volumes', 'None:dm_crypt-0']
Exit code: 1
Reason: -
Stdout: ''
Stderr: zkey: Volume 'None' is not a block device or is not available

Traceback (most recent call last):
  File "/snap/subiquity/2499/lib/python3.6/site-packages/curtin/commands/block_meta.py", line 1445, in dm_crypt_handler
    util.subp(gen_cmd, capture=True)
  File "/snap/subiquity/2499/lib/python3.6/site-packages/curtin/util.py", line 275, in subp
    return _subp(*args, **kwargs)
  File "/snap/subiquity/2499/lib/python3.6/site-packages/curtin/util.py", line 141, in _subp
    cmd=args)
curtin.util.ProcessExecutionError: Unexpected error while running command.
Command: ['zkey', 'generate', '--xts', '--volume-type', 'luks2', '--sector-size', '4096', '--name', 'dm_crypt-0', '--description', 'curtin generated zkey for None:dm_crypt-0', '--volumes', 'None:dm_crypt-0']
Exit code: 1
Reason: -
Stdout: ''
Stderr: zkey: Volume 'None' is not a block device or is not available

Setup of zkey on /dev/mapper/mpatha-part2 failed, fallback to cryptsetup.
[...]

I've created another MP (https://code.launchpad.net/~slyon/curtin/+git/curtin/+ref/slyon/lp1895192-zkey) that should fix this issue.
@mwhudson: let me know what you think of that.
@fheimes: Could you give this other MP another test on real hardware? (i.e. by also scp'ing and bind-mounting the curtin/block/__init__.py file)

@slyon: You nailed it!
I've used curtin/block/lvm.py and curtin/commands/curthooks.py from your first MP and curtin/block/__init__.py from your second MP.

That finally led me complete an FCP impish installation with LUKS using protected keys (paes_s390)!

$ sudo cryptsetup status $(awk '{ print $1 }' /etc/crypttab )
/dev/mapper/dm_crypt-0 is active and is in use.
  type: LUKS2
  cipher: paes-xts-plain64
  keysize: 1024 bits
  key location: keyring
  device: /dev/mapper/mpatha-part2
  sector size: 4096
  offset: 32768 sectors
  size: 132083712 sectors
  mode: read/write
$ sudo dmsetup ls --tree -o nodevice
mpatha-part1
 └─mpatha
ubuntu--vg-ubuntu--lv
 └─dm_crypt-0
    └─mpatha-part2
       └─mpatha

Great - many thx!!

Since 20.04.3 got released today I'm closing this ticket as Fix Released.