OpenStack Compute (nova)

Error during ComputeManager.update_available_resource: AttributeError: '_TransactionContextManager' object has no attribute 'async_

Bug #1788833 reported by Dilip Renkila
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Stephen Finucane
Rocky
Fix Committed
Medium
Matthew Thode
Ubuntu Cloud Archive
Fix Released
High
Unassigned
Rocky
Fix Released
High
Unassigned
nova (Ubuntu)
Fix Released
High
Unassigned
Cosmic
Fix Released
High
Unassigned

Bug Description

Hi all,

I have a rocky openstack cluster. I am using Mariadb galera cluster (3 galera nodes Active/Active) behind haproxy.

When i resize or migrate an instance, I am hitting with the following errors.

2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task [req-67381a5e-24e2-4dd2-bfc6-693bd1fabb8d 290bb90f6cbc46548951cbcaee0c0a34 9804c6f8ffe148bc9fa7ed409d41cb16 - default default] Error during ComputeManager._heal_instance_info_cache: AttributeError: '_TransactionContextManager' object has no attribute 'async_'
Traceback (most recent call last):

  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 126, in _object_dispatch
    return getattr(target, method)(*args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
    result = fn(cls, context, *args, **kwargs)

  File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1351, in get_by_host
    use_slave=use_slave)

  File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 218, in wrapper
    reader_mode = get_context_manager(context).async_

AttributeError: '_TransactionContextManager' object has no attribute 'async_'
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task Traceback (most recent call last):
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_service/periodic_task.py", line 220, in run_periodic_tasks
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task task(self, context)
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6632, in _heal_instance_info_cache
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task context, self.host, expected_attrs=[], use_slave=True)
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task args, kwargs)
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 241, in object_class_action_versions
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task args=args, kwargs=kwargs)
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 179, in call
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task retry=self.retry)
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 133, in _send
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task retry=retry)
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task call_monitor_timeout, retry=retry)
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 575, in _send
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task raise result
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task AttributeError: '_TransactionContextManager' object has no attribute 'async_'
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task Traceback (most recent call last):
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 126, in _object_dispatch
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task return getattr(target, method)(*args, **kwargs)
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task result = fn(cls, context, *args, **kwargs)
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1351, in get_by_host
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task use_slave=use_slave)
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 218, in wrapper
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task reader_mode = get_context_manager(context).async_
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task AttributeError: '_TransactionContextManager' object has no attribute 'async_'
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task
2018-08-24 12:06:37.668 19857 ERROR oslo_service.periodic_task
2018-08-24 12:07:29.064 19857 INFO nova.compute.resource_tracker [req-67381a5e-24e2-4dd2-bfc6-693bd1fabb8d 290bb90f6cbc46548951cbcaee0c0a34 9804c6f8ffe148bc9fa7ed409d41cb16 - default default] Final resource view: name=kvm4 phys_ram=257618MB used_ram=512MB phys_disk=100816GB used_disk=0GB total_vcpus=40 used_vcpus=0 pci_stats=[]

Full log file is at

Revision history for this message
Matt Riedemann (mriedem) wrote :

(10:56:34 AM) diliprenkila: <mriedem> diliprenkila oslo.db==4.40.0 ,oslo.messaging==8.1.0 on the compute nodes and oslo.db==4.38.0, oslo.messaging==6.4.1 on the nova controller node

Stephen already pointed out that this is the breaking nova change:

https://github.com/openstack/nova/commit/2d532963fa2e013e16cc403f2674a4488c4170ab#diff-8fec546e4c39f78d233f8e21dadaa3ff

which depends on this oslo.db change:

https://github.com/openstack/oslo.db/commit/df6bf3401266f42271627c1e408f87c71a06cef7

Which was released in oslo.db 4.40 but we don't require that min version in rocky for nova, so that's the failure. You need oslo.db >= 4.40.

Revision history for this message
Dilip Renkila (dilip-renkila278) wrote :

I will upgrade oslo db to min 4.40 and come back

OpenStack Infra (hudson-openstack)
Changed in nova:
assignee: nobody → Stephen Finucane (stephenfinucane)
status: New → In Progress
Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/597173
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bd7d991309ea2bea5d175cb1f2710519936fd0c2
Submitter: Zuul
Branch: master

commit bd7d991309ea2bea5d175cb1f2710519936fd0c2
Author: Stephen Finucane <email address hidden>
Date: Tue Aug 28 17:01:07 2018 +0100

    Don't use '_TransactionContextManager._async'

    In commit 2d532963, all instances of 'async' were replaced with 'async_'
    in preparation for Python 3.7. However, one of these should not have
    been changed as it refers to an oslo.db object attribute. That attribute
    has actually been renamed itself but that rename is only present from
    oslo.db 4.40.0 [1]. Thankfully, an alias to the older name is provided
    so we use that.

    [1] https://github.com/openstack/oslo.db/commit/df6bf34

    Change-Id: I1afd0ba34a9ebcb63edb91e880ef60580befb32e
    Closes-Bug: #1788833

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/597174
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=964832d37dd244f4f4ebc0dba46e4316241a2120
Submitter: Zuul
Branch: master

commit 964832d37dd244f4f4ebc0dba46e4316241a2120
Author: Stephen Finucane <email address hidden>
Date: Tue Aug 28 17:15:24 2018 +0100

    Revert "Don't use '_TransactionContextManager._async'"

    This reverts commit bd7d991309ea2bea5d175cb1f2710519936fd0c2 and bumps
    the minimum version of oslo.db to 4.40.0, as that is the first version
    of the library to include the renamed attribute.

    Change-Id: Ic9e7864be3af7ef362cad5648dfc7bdecd104465
    Related-Bug: #1788833

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/597421

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/rocky)

Reviewed: https://review.openstack.org/597421
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=40e18681bc26c1e8f51aa280f600d72ad0846588
Submitter: Zuul
Branch: stable/rocky

commit 40e18681bc26c1e8f51aa280f600d72ad0846588
Author: Stephen Finucane <email address hidden>
Date: Tue Aug 28 17:01:07 2018 +0100

    Don't use '_TransactionContextManager._async'

    In commit 2d532963, all instances of 'async' were replaced with 'async_'
    in preparation for Python 3.7. However, one of these should not have
    been changed as it refers to an oslo.db object attribute. That attribute
    has actually been renamed itself but that rename is only present from
    oslo.db 4.40.0 [1]. Thankfully, an alias to the older name is provided
    so we use that.

    [1] https://github.com/openstack/oslo.db/commit/df6bf34

    Change-Id: I1afd0ba34a9ebcb63edb91e880ef60580befb32e
    Closes-Bug: #1788833
    (cherry picked from commit bd7d991309ea2bea5d175cb1f2710519936fd0c2)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 18.0.1

This issue was fixed in the openstack/nova 18.0.1 release.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Ubuntu Rocky py3.7 is tripping over the async keyword issue. Since we have python-oslo.db 4.40.0 in Ubuntu Rocky I'm planning to cherry-pick the following patch to the nova rocky package:

commit 964832d37dd244f4f4ebc0dba46e4316241a2120
Author: Stephen Finucane <email address hidden>
Date: Tue Aug 28 17:15:24 2018 +0100

    Revert "Don't use '_TransactionContextManager._async'"
    ...

Changed in nova (Ubuntu Cosmic):
status: New → Incomplete
status: Incomplete → Triaged
importance: Undecided → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 2:18.0.1-0ubuntu1

---------------
nova (2:18.0.1-0ubuntu1) cosmic; urgency=medium

  * New stable point release for OpenStack Rocky.
  * d/p/skip-openssl-1.1.1-tests.patch: Skip tests for >= cosmic that fail
    with openssl 1.1.1 (LP: #1771506).
  * d/p/py3.7-get-context-manager-async.patch: Cherry-picked from upstream
    master. This is needed for py3.7 where async is a keyword (LP: #1788833).
  * d/rules: Skip py3.7 tests until they run successfully upstream.

 -- Corey Bryant <email address hidden> Wed, 03 Oct 2018 07:59:30 -0400

Changed in nova (Ubuntu Cosmic):
status: Triaged → Fix Released
James Page (james-page)
Changed in cloud-archive:
status: Triaged → Fix Committed
Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package nova - 2:18.0.1-0ubuntu1~cloud0
---------------

 nova (2:18.0.1-0ubuntu1~cloud0) bionic-rocky; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 nova (2:18.0.1-0ubuntu1) cosmic; urgency=medium
 .
   * New stable point release for OpenStack Rocky.
   * d/p/skip-openssl-1.1.1-tests.patch: Skip tests for >= cosmic that fail
     with openssl 1.1.1 (LP: #1771506).
   * d/p/py3.7-get-context-manager-async.patch: Cherry-picked from upstream
     master. This is needed for py3.7 where async is a keyword (LP: #1788833).
   * d/rules: Skip py3.7 tests until they run successfully upstream.

Changed in cloud-archive:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 19.0.0.0rc1

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.