Permission denied: '/var/log/nova/nova-manage.log'

Bug #1671681 reported by Diana Clarke
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
puppet-nova
Fix Released
Undecided
Oliver Walsh

Bug Description

The following traceback has been reported multiple times downstream of puppet-nova (here are a few examples).

    https://bugzilla.redhat.com/show_bug.cgi?id=1044155
    https://bugzilla.redhat.com/show_bug.cgi?id=1289761
    https://bugzilla.redhat.com/show_bug.cgi?id=1368588
    https://bugzilla.redhat.com/show_bug.cgi?id=1428572

Traceback (most recent call last):
  File "/bin/nova-manage", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/site-packages/nova/cmd/manage.py", line 1517, in main
    logging.setup(CONF, "nova")
  File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 243, in setup
    _setup_logging_from_conf(conf, product_name, version)
  File "/usr/lib/python2.7/site-packages/oslo_log/log.py", line 319, in _setup_logging_from_conf
    filelog = file_handler(logpath)
  File "/usr/lib64/python2.7/logging/handlers.py", line 392, in __init__
    logging.FileHandler.__init__(self, filename, mode, encoding, delay)
  File "/usr/lib64/python2.7/logging/__init__.py", line 902, in __init__
    StreamHandler.__init__(self, self._open())
  File "/usr/lib64/python2.7/logging/__init__.py", line 925, in _open
    stream = open(self.baseFilename, self.mode)
IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log'

When you execute nova-manage commands, oslo logs to the following location (the log file name is dynamically created based on the command name).

    /var/log/nova/nova-manage.log

Because puppet-nova is executing these commands as root, nova-manage.log is owned by root:root, preventing the 'nova-manage db archive_deleted_rows' entry in nova's crontab from later executing successfully.

    $ sudo crontab -u nova -l
    PATH=/bin:/usr/bin:/usr/sbin SHELL=/bin/sh
    1 0 * * * nova-manage db archive_deleted_rows --max_rows 100 >>/dev/null 2>&1

This log file is also an outlier, as all other log files in /var/log/nova/ are owned by nova:nova. For example:

    $ sudo ls -la /var/log/nova/
    total 102700
    drwxr-x---. 2 nova root 252 Mar 3 03:25 .
    drwxr-xr-x. 40 root root 4096 Mar 2 02:58 ..
    -rw-r--r--. 1 nova nova 49046149 Mar 3 20:21 nova-api.log
    -rw-r--r--. 1 nova nova 3419964 Mar 3 03:25 nova-api.log-20170303.gz
    -rw-r--r--. 1 nova nova 178353 Mar 2 03:07 nova-cert.log
    -rw-r--r--. 1 nova nova 25453688 Mar 3 20:21 nova-compute.log
    -rw-r--r--. 1 nova nova 2528760 Mar 3 03:25 nova-compute.log-20170303.gz
    -rw-r--r--. 1 nova nova 14367109 Mar 3 20:21 nova-conductor.log
    -rw-r--r--. 1 nova nova 1806036 Mar 3 03:25 nova-conductor.log-20170303.gz
    -rw-r--r--. 1 root root 215298 Mar 2 03:07 nova-manage.log
    -rw-r--r--. 1 nova nova 2666163 Mar 3 20:21 nova-scheduler.log

** Heads Up! **

Note that a fix for this bug should consider the upgrades case where that log file is already owned by root:root.

PS. Here's an example of a past workaround:

    https://github.com/redhat-openstack/openstack-utils/pull/8/files

Changed in puppet-nova:
assignee: nobody → Diana Clarke (diana-clarke)
Revision history for this message
Diana Clarke (diana-clarke) wrote :

Here's a work in progress attempt at this bug:

- Run nova-manage commands as the nova user
- https://review.openstack.org/#/c/442713/

It does not yet address the upgrades case.

Revision history for this message
Christopher Brown (snecklifter) wrote :

Hi Diana,

Any interest in reviving the abandoned patch here and trying to get this merged?

Revision history for this message
Diana Clarke (diana-clarke) wrote :

Hi Christopher,

I'm not working on OpenStack anymore. I'll un-assign myself from the bug in case someone else wants to give it a shot.

Have a great week!

Changed in puppet-nova:
assignee: Diana Clarke (diana-clarke) → nobody
Revision history for this message
Oliver Walsh (owalsh) wrote :

I've proposed a fix in the nova RPM spec https://review.rdoproject.org/r/7474

Changed in puppet-nova:
assignee: nobody → Oliver Walsh (owalsh)
status: New → In Progress
Revision history for this message
Maciej Kucia (maciejkucia) wrote :

This is happening on Ocata as well.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/498723

Changed in puppet-nova:
assignee: Oliver Walsh (owalsh) → Maciej Kucia (maciejkucia)
Changed in puppet-nova:
assignee: Maciej Kucia (maciejkucia) → Oliver Walsh (owalsh)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-nova (master)

Reviewed: https://review.openstack.org/498723
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=e128ba6538eb806589ebc01c16ee5e298bb0ee86
Submitter: Zuul
Branch: master

commit e128ba6538eb806589ebc01c16ee5e298bb0ee86
Author: Diana Clarke <email address hidden>
Date: Tue Mar 7 14:07:16 2017 -0500

    Correct permissions on the nova logfiles

    When you execute nova-manage commands, oslo logs to the following
    location (file name is dynamically created based on command name).

        /var/log/nova/nova-manage.log

    Because puppet-nova is executing these commands as root,
    nova-manage.log is owned by root, preventing the 'nova-manage
    db archive_deleted_rows' entry in nova's crontab from executing.

        Permission denied: '/var/log/nova/nova-manage.log'

    This log file is also an outlier, as all other log files in
    /var/log/nova/ are owned by nova:nova.

    Similar issues are possible for other nova logs, if for example
    a nova services is initially started manually as root, so the
    ownership of all nova logs is corrected before configuring nova.

    Co-Authored-By: Oliver Walsh <email address hidden>
    Co-Authored-By: Diana Clarke <email address hidden>
    Co-Authored-By: Maciej Kucia <email address hidden>
    Closes-Bug: #1671681
    Change-Id: I0ca0110cbf9139c79074cf603dcab9135f96e765

Changed in puppet-nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-nova 12.2.0

This issue was fixed in the openstack/puppet-nova 12.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/561695

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/561742

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-nova (stable/pike)

Reviewed: https://review.openstack.org/561695
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=0d6f9e10749959b0fe88c876319ca03ddac1b34e
Submitter: Zuul
Branch: stable/pike

commit 0d6f9e10749959b0fe88c876319ca03ddac1b34e
Author: Diana Clarke <email address hidden>
Date: Tue Mar 7 14:07:16 2017 -0500

    Correct permissions on the nova logfiles

    When you execute nova-manage commands, oslo logs to the following
    location (file name is dynamically created based on command name).

        /var/log/nova/nova-manage.log

    Because puppet-nova is executing these commands as root,
    nova-manage.log is owned by root, preventing the 'nova-manage
    db archive_deleted_rows' entry in nova's crontab from executing.

        Permission denied: '/var/log/nova/nova-manage.log'

    This log file is also an outlier, as all other log files in
    /var/log/nova/ are owned by nova:nova.

    Similar issues are possible for other nova logs, if for example
    a nova services is initially started manually as root, so the
    ownership of all nova logs is corrected before configuring nova.

    Co-Authored-By: Oliver Walsh <email address hidden>
    Co-Authored-By: Diana Clarke <email address hidden>
    Co-Authored-By: Maciej Kucia <email address hidden>
    Closes-Bug: #1671681
    Change-Id: I0ca0110cbf9139c79074cf603dcab9135f96e765
    (cherry picked from commit e128ba6538eb806589ebc01c16ee5e298bb0ee86)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-nova (stable/ocata)

Reviewed: https://review.openstack.org/561742
Committed: https://git.openstack.org/cgit/openstack/puppet-nova/commit/?id=482bf9695fb49c9bc59785510978cd2f973b3bba
Submitter: Zuul
Branch: stable/ocata

commit 482bf9695fb49c9bc59785510978cd2f973b3bba
Author: Diana Clarke <email address hidden>
Date: Tue Mar 7 14:07:16 2017 -0500

    Correct permissions on the nova logfiles

    When you execute nova-manage commands, oslo logs to the following
    location (file name is dynamically created based on command name).

        /var/log/nova/nova-manage.log

    Because puppet-nova is executing these commands as root,
    nova-manage.log is owned by root, preventing the 'nova-manage
    db archive_deleted_rows' entry in nova's crontab from executing.

        Permission denied: '/var/log/nova/nova-manage.log'

    This log file is also an outlier, as all other log files in
    /var/log/nova/ are owned by nova:nova.

    Similar issues are possible for other nova logs, if for example
    a nova services is initially started manually as root, so the
    ownership of all nova logs is corrected before configuring nova.

    Co-Authored-By: Oliver Walsh <email address hidden>
    Co-Authored-By: Diana Clarke <email address hidden>
    Co-Authored-By: Maciej Kucia <email address hidden>
    Closes-Bug: #1671681
    Change-Id: I0ca0110cbf9139c79074cf603dcab9135f96e765
    (cherry picked from commit e128ba6538eb806589ebc01c16ee5e298bb0ee86)
    (cherry picked from commit 0d6f9e10749959b0fe88c876319ca03ddac1b34e)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-nova 11.6.0

This issue was fixed in the openstack/puppet-nova 11.6.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.