Permissions 0644 for '/var/lib/nova/.ssh/id_rsa' are too open

Also seen on cs:nova-compute-327.

For the record, this was interfering with live migrations of instances.

This is very odd as the place where this is set up is using ssh-keygen (which does the right thing with permissions) with this bit of code:

def initialize_ssh_keys(user='root'):
    home_dir = pwd.getpwnam(user).pw_dir
    ssh_dir = os.path.join(home_dir, '.ssh')
    if not os.path.isdir(ssh_dir):
        os.mkdir(ssh_dir)

    priv_key = os.path.join(ssh_dir, 'id_rsa')
    if not os.path.isfile(priv_key):
        log('Generating new ssh key for user %s.' % user)
        cmd = ['ssh-keygen', '-q', '-N', '', '-t', 'rsa', '-b', '2048',
               '-f', priv_key]
        check_output(cmd)

I wonder how it is getting changed to 644?

We could put a change in to just always set the permissions to 600 in that function.

What's the provenance of the system in terms of what it was initially installed as? (i.e. has it been upgraded, charms upgraded, etc.) Thanks.

[Expired for OpenStack nova-compute charm because there has been no activity for 60 days.]

I just ran into this bug myself.

2021-08-04 13:04:35.136 5167 ERROR oslo_messaging.rpc.server Command: scp -r blade04-openstack:/var/lib/nova/instances/_base/629cc7c40ea2b906422db31fa818aa88b8886310 /var/lib/nova/instances/_base/629cc7c40ea2b906422db31fa818aa88b8886310
2021-08-04 13:04:35.136 5167 ERROR oslo_messaging.rpc.server Exit code: 1
2021-08-04 13:04:35.136 5167 ERROR oslo_messaging.rpc.server Stdout: ''
2021-08-04 13:04:35.136 5167 ERROR oslo_messaging.rpc.server Stderr: '@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@ WARNING: UNPROTECTED PRIVATE KEY FILE! @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nPermissions 0644 for \'/var/lib/nova/.ssh/id_rsa\' are too open.\r\nIt is required that your private key files are NOT accessible by others.\r\nThis private key will be ignored.\r\nLoad key "/var/lib/nova/.ssh/id_rsa": bad permissions\r\nnova@blade04-openstack: Permission denied (publickey).\r\n'
2021-08-04 13:04:35.136 5167 ERROR oslo_messaging.rpc.server

09:10




ok interesting that changed during the package upgrades all other servers are correct

Before permissions were broken we have the following packages.
libnova-0.16-0/bionic 0.16-2 amd64
libnova-dev/bionic 0.16-2 amd64
libtamuanova-0.2/bionic 0.2-4build1 amd64
libtamuanova-dev/bionic 0.2-4build1 amd64
nova-ajax-console-proxy/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-api/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-api-metadata/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all [upgradable from: 2:21.2.0-0ubuntu1~cloud0]
nova-api-os-compute/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-api-os-volume/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-cells/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-common/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all [upgradable from: 2:21.2.0-0ubuntu1~cloud0]
nova-compute/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all [upgradable from: 2:21.2.0-0ubuntu1~cloud0]
nova-compute-kvm/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all [upgradable from: 2:21.2.0-0ubuntu1~cloud0]
nova-compute-libvirt/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all [upgradable from: 2:21.2.0-0ubuntu1~cloud0]
nova-compute-lxc/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-compute-lxd/bionic-updates-stein 19.0.0-0ubuntu1~cloud0 all
nova-compute-qemu/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-compute-vmware/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-compute-xen/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-conductor/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-console/bionic-updates-train 2:20.6.0-0ubuntu1~cloud1 all
nova-consoleauth/bionic-updates-stein 2:19.3.2-0ubuntu1~cloud1 all
nova-doc/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-lxd-common/bionic-updates-stein 19.0.0-0ubuntu1~cloud0 all
nova-network/bionic-updates-train 2:20.6.0-0ubuntu1~cloud1 all
nova-novncproxy/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-placement-api/bionic-updates-stein 2:19.3.2-0ubuntu1~cloud1 all
nova-scheduler/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-serialproxy/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-spiceproxy/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-volume/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all
nova-xvpvncproxy/bionic-updates-train 2:20.6.0-0ubuntu1~cloud1 all
puppet-module-nova/bionic 9.4.0-1 all
python-django-nova/bionic 0.3~git20110711-0ubuntu3 amd64
python-nova/bionic-updates 2:17.0.13-0ubuntu2 all
python-nova-adminclient/bionic 0.1.8-0ubuntu2 amd64
python-nova-lxd/bionic-updates 17.0.1-0ubuntu1 all
python-nova.lxd/bionic-updates 17.0.1-0ubuntu1 all
python-novaclient/bionic-updates-stein 2:13.0.0-0ubuntu1~cloud0 all
python-novaclient-doc/bionic-updates-ussuri 2:17.0.0-0ubuntu1~cloud0 all
python3-nova/bionic-updates-ussuri 2:21.2.1-0ubuntu1~cloud0 all [upgradable from: 2:21.2.0-0ubuntu1~cloud0]
python3-nova-lxd/bionic-updates-stein 19.0.0-0ubuntu1~cloud0 all
python3-novaagent/bionic-updates 2.1.18-0ubuntu1~18.04.0 all
python3-novaclient/bionic-updates-ussuri,now 2:17.0.0-0ubuntu1~cloud0 all [installed]

After the upgrade there are the packages.
ibnova-0.16-0/bionic 0.16-2 amd64
libnova-dev/bionic 0.16-2 amd64
libtamuanova-0.2/bionic 0.2-4bui...

Using this charm as provided by juju status

nova-compute 21.2.1 active 9 nova-compute jujucharms 327 ubuntu

Charms were not upgraded while this broke. We simply upgrade the packages.

Status changed to 'Confirmed' because the bug affects multiple users.

> Charms were not upgraded while this broke. We simply upgrade the packages.

If that's the case, package maintainer script might be related? For example,

$ grep /var/lib/nova /var/lib/dpkg/info/nova-common.postinst
            --home /var/lib/nova \
        chown -R nova:nova /var/lib/nova/
    find /var/lib/nova -type f -exec chmod 0644 "{}" + -o -type d -exec chmod 0755 "{}" +
    find /var/lib/nova -name "console.log" -exec chmod 0600 "{}" +
    find /var/lib/nova -name "console.log" -exec chown root:root "{}" +

root@casual-condor:/var/lib/nova# ll .ssh/
total 28
drwxr-xr-x 2 nova root 4096 Aug 3 10:43 ./
drwxr-xr-x 10 nova nova 4096 Aug 3 10:25 ../
-rw-r--r-- 1 root root 1197 Aug 3 10:54 authorized_keys
-rw------- 1 nova root 1823 Aug 3 10:25 id_rsa
-rw-r--r-- 1 nova root 400 Aug 3 10:25 id_rsa.pub
-rw-r--r-- 1 root root 5526 Aug 3 10:54 known_hosts

^^^ 600 to id_rsa

root@casual-condor:/var/lib/nova# find /var/lib/nova -type f -exec chmod 0644 "{}" + -o -type d -exec chmod 0755 "{}" +

root@casual-condor:/var/lib/nova# ll .ssh/
total 28
drwxr-xr-x 2 nova root 4096 Aug 3 10:43 ./
drwxr-xr-x 10 nova nova 4096 Aug 3 10:25 ../
-rw-r--r-- 1 root root 1197 Aug 3 10:54 authorized_keys
-rw-r--r-- 1 nova root 1823 Aug 3 10:25 id_rsa
-rw-r--r-- 1 nova root 400 Aug 3 10:25 id_rsa.pub
-rw-r--r-- 1 root root 5526 Aug 3 10:54 known_hosts

^^^ 644 to id_rsa

Previously reported as https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1904745

Just had a recurrence. This happened when doing a series upgrade from bionic/ussuri to focal/ussuri. All compute nodes have /var/lib/nova/.ssh/id_rsa with 0644 permissions.

Status changed to 'Confirmed' because the bug affects multiple users.

Subscribing to field-high since we've seen several customers already affected by this, it will only get worse as more keep upgrading to Ussuri.

I just did a test myself and the bug is still present. Simply re-install your nova-common package and there goes the SSH key permissions. As others mentioned, it is indeed this line in the postinst script:

find /var/lib/nova -type f -exec chmod 0644 "{}" + -o -type d -exec chmod 0755 "{}" +

I have a patch in this branch https://git.launchpad.net/~freyes/ubuntu/+source/nova/commit/?id=88c97dc9332b97edf06618b6d4d2c770153821a6 , although I haven't been able to test it, I'm removing myself from the bug since I won't have cycles to dedicate to this task in the short term.

Any update here ? This is blocking critical node evacuation. Thanks !

Thank you Felipe and Rodrigo for this fix. It's been uploaded to focal, impish, jammy unapproved queues, and victoria/wallaby staging for the cloud archive.

This bug was fixed in the package nova - 3:25.0.0-0ubuntu2

---------------
nova (3:25.0.0-0ubuntu2) kinetic; urgency=medium

  * d/nova-common.postinst: Don't change file permissions under
    /var/lib/nova/.ssh (LP: #1904580).

 -- Felipe Reyes <email address hidden> Fri, 6 May 2022 17:03:39 -0300

As I understand it this patch will stop the permissions from being changed to 0644 going forward but it doesn't do anything to change the permission from 0644 to 0600. Shouldn't that also be fixed?

Hello Diko, or anyone else affected,

Accepted nova into impish-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/3:24.1.0-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-impish to verification-done-impish. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-impish. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Diko, or anyone else affected,

Accepted nova into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/3:25.0.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Diko, or anyone else affected,

Accepted nova into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:21.2.4-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Diko, or anyone else affected,

Accepted nova into yoga-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:yoga-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-yoga-needed to verification-yoga-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-yoga-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Diko, or anyone else affected,

Accepted nova into xena-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:xena-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-xena-needed to verification-xena-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-xena-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Diko, or anyone else affected,

Accepted nova into wallaby-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:wallaby-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-wallaby-needed to verification-wallaby-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-wallaby-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Diko, or anyone else affected,

Accepted nova into victoria-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:victoria-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-victoria-needed to verification-victoria-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-victoria-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Diko, or anyone else affected,

Accepted nova into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package nova - 3:25.0.0+git2022060214.d869163608-1~cloud0
---------------

 nova (3:25.0.0+git2022060214.d869163608-1~cloud0) jammy-zed; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 nova (3:25.0.0+git2022060214.d869163608-1) kinetic; urgency=medium
 .
   * New upstream snapshot for OpenStack Zed.
   * d/control: Align (Build-)Depends with upstream.
   * d/control: Update standards version to 4.6.1.
 .
 nova (3:25.0.0-0ubuntu2) kinetic; urgency=medium
 .
   * d/nova-common.postinst: Don't change file permissions under
     /var/lib/nova/.ssh (LP: #1904580).

@Diko, @Rodrigo, would you be able to help us verify this on the various releases?

attached file for ussuri validation. There is an unrelated error that apparently was not there before. I can move forward with the other verifications, but I'd prefer a green light on the issue. I did a quick search and found nothing related, but maybe it is worth another set of eyes and double checking.

@Rodrigo, Sorry I think I'm missing something. What's the unrelated error?

@Corey, see the message: invoke-rc.d: syntax error: unknown option "--skip-systemd-native"

as discussed internally with Corey, the error is unrelated and I'm proceeding with the verifications

victoria validation done

wallaby validation done

xena validation done

jammy validation done (yoga as well?)

does yoga need to be validated separately as UCA given jammy has already been validated?

same question for focal and impish, given that xena and ussuri UCAs have been validated

As per the instruction, renaming the tags instead of removing so it can be tracked in:
https://people.canonical.com/~ubuntu-archive/pending-sru.html

@Nobuto: Thank you so much, it has been such a long time since my last SRU I forgot about those details.

yoga validation done

only impish needed now

impish validation attached

All validations completed

This bug was fixed in the package nova - 3:25.0.0-0ubuntu1.1

---------------
nova (3:25.0.0-0ubuntu1.1) jammy; urgency=medium

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/yoga branch.

  [ Felipe Reyes ]
  * d/nova-common.postinst: Don't change file permissions under
    /var/lib/nova/.ssh (LP: #1904580).

 -- Corey Bryant <email address hidden> Mon, 16 May 2022 13:45:34 -0400

The verification of the Stable Release Update for nova has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package nova - 3:25.0.0-0ubuntu1.1~cloud0
---------------

 nova (3:25.0.0-0ubuntu1.1~cloud0) focal-yoga; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (3:25.0.0-0ubuntu1.1) jammy; urgency=medium
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/yoga branch.
 .
   [ Felipe Reyes ]
   * d/nova-common.postinst: Don't change file permissions under
     /var/lib/nova/.ssh (LP: #1904580).

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package nova - 3:24.1.0-0ubuntu2~cloud0
---------------

 nova (3:24.1.0-0ubuntu2~cloud0) focal-xena; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 nova (3:24.1.0-0ubuntu2) impish; urgency=medium
 .
   [ Felipe Reyes ]
   * d/nova-common.postinst: Don't change file permissions under
     /var/lib/nova/.ssh (LP: #1904580).
 .
 nova (3:24.1.0-0ubuntu1) impish; urgency=medium
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/xena branch.
 .
   [ Felipe Reyes ]
   * New stable point release for OpenStack Xena (LP: #1972665).

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package nova - 3:23.2.0-0ubuntu1~cloud1
---------------

 nova (3:23.2.0-0ubuntu1~cloud1) focal-wallaby; urgency=medium
 .
   [ Felipe Reyes ]
   * d/nova-common.postinst: Don't change file permissions under
     /var/lib/nova/.ssh (LP: #1904580).

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.

This bug was fixed in the package nova - 2:21.2.4-0ubuntu2

---------------
nova (2:21.2.4-0ubuntu2) focal; urgency=medium

  [ Felipe Reyes ]
  * d/nova-common.postinst: Don't change file permissions under
    /var/lib/nova/.ssh (LP: #1904580).

 -- Corey Bryant <email address hidden> Mon, 16 May 2022 13:52:02 -0400

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package nova - 2:21.2.4-0ubuntu2~cloud0
---------------

 nova (2:21.2.4-0ubuntu2~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (2:21.2.4-0ubuntu2) focal; urgency=medium
 .
   [ Felipe Reyes ]
   * d/nova-common.postinst: Don't change file permissions under
     /var/lib/nova/.ssh (LP: #1904580).