Comment 6 for bug 1967956

Revision history for this message
Corey Bryant (corey.bryant) wrote :

I was able to recreate this using the exact rules for hardening compute nodes as the bug reporter used.

$ openstack server create c1 --image cirros --flavor m1.tiny # this is successful

On the compute node where that server lands:

ubuntu@juju-da8cbf-zaza-41517351cce1-27:~$ sudo ls -al /var/lib/nova/instances/ed133a88-4f6e-4585-b208-2133c348ff35
total 20724
drwxr-xr-x 2 nova nova 4096 Aug 31 16:52 .
drwxr-xr-x 7 nova nova 4096 Aug 31 16:52 ..
-rw------- 1 root root 28411 Aug 31 16:52 console.log
-rw-r--r-- 1 libvirt-qemu kvm 21233664 Aug 31 16:53 disk
-rw-r--r-- 1 nova nova 79 Aug 31 16:52 disk.info

$ openstack server resize --flavor 2 c1

This results in server going into ERROR state and destination nova-compute.log contains:
libvirt.libvirtError: Cannot access storage file '/var/lib/nova/instances/ed133a88-4f6e-4585-b208-2133c348ff35/disk' (as uid:64055, gid:108): Permission denied

On the source compute node there's a slight change in ownership which is probably fine, but might as well document that here:

ubuntu@juju-da8cbf-zaza-41517351cce1-27:~$ ls -al /var/lib/nova/instances/ed133a88-4f6e-4585-b208-2133c348ff35_resize
total 20784
drwxr-xr-x 2 nova nova 4096 Aug 31 16:52 .
drwxr-xr-x 7 nova nova 4096 Aug 31 16:58 ..
-rw------- 1 root root 28571 Aug 31 16:58 console.log
-rw-r--r-- 1 nova nova 21299200 Aug 31 16:58 disk
-rw-r--r-- 1 nova nova 79 Aug 31 16:52 disk.info

On the compute node where the resized instance lands, we see the following:

ubuntu@juju-da8cbf-zaza-41517351cce1-28:~$ sudo ls -al /var/lib/nova/instances/ed133a88-4f6e-4585-b208-2133c348ff35
total 20816
drwxrwx--- 2 nova nova 4096 Aug 31 16:58 .
drwxr-xr-x 6 nova nova 4096 Aug 31 16:58 ..
-rw------- 1 root root 0 Aug 31 16:58 console.log
-rw-r----- 1 nova nova 21299520 Aug 31 16:58 disk
-rw-r----- 1 nova nova 79 Aug 31 16:58 disk.info

The file permissions look like they may have been created with a new default umask as they have 640 mode set instead of 644. This is likely cause of the access denial because libvirt-qemu no longer has read access.