Cinder

DetachedInstanceError: Parent instance <VolumeAttachment at > is not bound to a Session

Bug #1834845 reported by Patrick Oberdorf on 2019-07-01

This bug affects 5 people

	Status	Importance	Assigned to
Cinder	Fix Released	Undecided	Gorka Eguileor
Ubuntu Cloud Archive	Fix Released	High	James Page
Stein	Fix Released	High	James Page
Train	Fix Released	High	James Page
cinder (Ubuntu)	Fix Released	High	James Page
Disco	Won't Fix	High	James Page
Eoan	Fix Released	High	James Page

Bug Description

Hey there.

We upgraded from rocky to stein and facing a strange error. Our cinder-ceph volume services gets marked as down after a few seconds. So we looked into this and found a error in the logs:

Error starting thread.: sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <VolumeAttachment at 0x7f04ba5ed3c8> is not bound to a Session; lazy load operation of attribute 'volume' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service Traceback (most recent call last):
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_service/service.py", line 796, in run_service
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service service.start()
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/service.py", line 222, in start
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service service_id=Service.service_id)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 445, in init_host
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service self._init_host(added_to_cluster, **kwargs)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 523, in _init_host
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service volumes = self._get_my_volumes(ctxt)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 3007, in _get_my_volumes
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service limit, offset)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 3003, in _get_my_resources
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service offset=offset)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume.py", line 617, in get_all
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service volumes, expected_attrs=expected_attrs)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 1133, in obj_make_list
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service **extra_args)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume.py", line 290, in _from_db_object
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service db_volume.get('volume_attachment'))
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 1133, in obj_make_list
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service **extra_args)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/cinder/objects/volume_attachment.py", line 102, in _from_db_object
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service if 'volume' in expected_attrs and hasattr(db_attachment, 'volume'):
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/attributes.py", line 242, in __get__
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service return self.impl.get(instance_state(instance), dict_)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/attributes.py", line 601, in get
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service value = self.callable_(state, passive)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service File "/usr/lib/python3/dist-packages/sqlalchemy/orm/strategies.py", line 596, in _load_for_state
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service (orm_util.state_str(state), self.key)
2019-07-01 10:29:22.947 16745 ERROR oslo_service.service sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <VolumeAttachment at 0x7f04ba5ed3c8> is not bound to a Session; lazy load operation of attribute 'volume' cannot proceed (Background on this error at: http://sqlalche.me/e/bhk3)

It looks like, because of this error the service gets marked down. This issue is definitly linked to this: https://bugs.launchpad.net/cinder/+bug/1812913 but the fix that is released is not fixing it.
If we put a try-catch block around this block, the services stays up, but i am not sure if something else will break.
This is in our perspective a high priority bug, because we are not able to use one of our ceph backends.

ii cinder-api 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - API server
ii cinder-common 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - common files
ii cinder-scheduler 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - Scheduler server
ii cinder-volume 2:14.0.0-0ubuntu1~cloud0 all Cinder storage service - Volume server
ii python3-cinder 2:14.0.0-0ubuntu1~cloud0 all Cinder Python 3 libraries

Tags:

Revision history for this message

Sean McGinnis (sean-mcginnis) wrote on 2019-07-01:

Gorka, could you take a look at this?

Changed in cinder:
assignee:	nobody → Gorka Eguileor (gorka)

Revision history for this message

Patrick Oberdorf (obi12341) wrote on 2019-07-01:

really_dirty_workaround.patch Edit (1.7 KiB, text/plain)

_Dirty_ workaround for this

Revision history for this message

Ubuntu Foundations Team Bug Bot (crichton) wrote on 2019-07-01:

The attachment "really_dirty_workaround.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags:

added: patch

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-02: Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/668646

Changed in cinder:
status:	New → In Progress

Revision history for this message

Patrick Oberdorf (obi12341) wrote on 2019-07-02:

I can confirm the commit is working :)

OpenStack Infra (hudson-openstack) on 2019-07-02

Changed in cinder:
assignee:	Gorka Eguileor (gorka) → Eric Harney (eharney)

Eric Harney (eharney) on 2019-07-02

Changed in cinder:
assignee:	Eric Harney (eharney) → Gorka Eguileor (gorka)

OpenStack Infra (hudson-openstack) on 2019-07-03

Changed in cinder:
assignee:	Gorka Eguileor (gorka) → Eric Harney (eharney)

Eric Harney (eharney) on 2019-07-03

Changed in cinder:
assignee:	Eric Harney (eharney) → Gorka Eguileor (gorka)

OpenStack Infra (hudson-openstack) on 2019-07-08

Changed in cinder:
assignee:	Gorka Eguileor (gorka) → Eric Harney (eharney)

Eric Harney (eharney) on 2019-07-09

Changed in cinder:
assignee:	Eric Harney (eharney) → Gorka Eguileor (gorka)

Revision history for this message

Matt Riedemann (mriedem) wrote on 2019-07-11:

Note that bug 1626499 and bug 1640920 seem like older versions of a similar thing but on different parts of the data model.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-01: Fix merged to cinder (master)

Reviewed: https://review.opendev.org/668646
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=2e73bede80cc2acdb3527f06bc5c5f9c1a8463a7
Submitter: Zuul
Branch: master

commit 2e73bede80cc2acdb3527f06bc5c5f9c1a8463a7
Author: Gorka Eguileor <email address hidden>
Date: Tue Jul 2 11:06:17 2019 +0200

Fix DetachedInstanceError for VolumeAttachment

    Patch I253123d5451b32f0e3143916e41aaa1af75561c2 fixed the
    DetachedInstanceError for VolumeAttachment OVOs but only partially, as
    apparently it was dependent on the SQLAlchemy version due to the use os
    "hasattr".

This patch replaces "hasattr" with a check on the object's dictionary,
which will never trigger a Lazy Load.

Closes-Bug: #1834845
Change-Id: Iac785eef9be4b9cdb5c739ee0a87949805282867

Changed in cinder:
status:	In Progress → Fix Released

Revision history for this message

Launchpad Janitor (janitor) wrote on 2019-08-07:

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in cinder (Ubuntu):
status:	New → Confirmed

Peter Sabaini (peter-sabaini) on 2019-08-07

tags:

added: canonical-bootstack

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-07: Fix proposed to cinder (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/675081

James Page (james-page) on 2019-08-07

Changed in cinder (Ubuntu Eoan):
status:	Confirmed → Triaged
Changed in cinder (Ubuntu Disco):
status:	New → Triaged
importance:	Undecided → High
Changed in cinder (Ubuntu Eoan):
importance:	Undecided → High

James Page (james-page) on 2019-08-07

Changed in cinder (Ubuntu Eoan):
status:	Triaged → In Progress
assignee:	nobody → James Page (james-page)

Revision history for this message

James Page (james-page) wrote on 2019-08-07:

#10

Ubuntu SRU information

[Impact]
After upgrading from Ubuntu with OpenStack Rocky to Ubuntu with OpenStack Stein, the cinder-volume service fails to restart resulting in loss of ability to create and manage volumes

[Test Case]
Deploy OpenStack Rocky
Create volume, attach to instance
Upgrade to OpenStack Stein
openstack volume create --size 5 test-volume fails.

[Regression Potential]
The patch for this fix is fairly defensive in approach and fixes an issue with an associated failure fixed in an earlier commit; low regression potential.

Revision history for this message

James Page (james-page) wrote on 2019-08-07:

#11

I've uploaded fixes for eoan and disco; disco-proposed current contains 14.0.1-0ubuntu1 which will need to clear first.

Changed in cinder (Ubuntu Disco):
status:	Triaged → In Progress
assignee:	nobody → James Page (james-page)

Revision history for this message

Launchpad Janitor (janitor) wrote on 2019-08-07:

#12

This bug was fixed in the package cinder - 2:15.0.0~b1~git2019080709.0a0d55d8a-0ubuntu1

---------------
cinder (2:15.0.0~b1~git2019080709.0a0d55d8a-0ubuntu1) eoan; urgency=medium

* New upstream snapshot for OpenStack Train (LP: #1834845).

-- James Page <email address hidden> Wed, 07 Aug 2019 13:13:56 +0100

Changed in cinder (Ubuntu Eoan):
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-08-08: Fix merged to cinder (stable/stein)

#13

Reviewed: https://review.opendev.org/675081
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=44a15be6a71ebdaf0e5d1663fc31011ff43ef37b
Submitter: Zuul
Branch: stable/stein

commit 44a15be6a71ebdaf0e5d1663fc31011ff43ef37b
Author: Gorka Eguileor <email address hidden>
Date: Tue Jul 2 11:06:17 2019 +0200

Fix DetachedInstanceError for VolumeAttachment

This patch replaces "hasattr" with a check on the object's dictionary,
which will never trigger a Lazy Load.

    Closes-Bug: #1834845
    Change-Id: Iac785eef9be4b9cdb5c739ee0a87949805282867
    (cherry picked from commit 2e73bede80cc2acdb3527f06bc5c5f9c1a8463a7)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-04: Fix proposed to cinder (stable/rocky)

#14

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/680000

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2019-09-25:

#15

I've uploaded 2:14.0.1-0ubuntu3 to the disco unapproved queue which includes 2:14.0.1-0ubuntu2 and provides a fix for disco autopkgtest failure.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-27: Fix included in openstack/cinder 15.0.0.0rc1

#16

This issue was fixed in the openstack/cinder 15.0.0.0rc1 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-01: Fix proposed to cinder (stable/queens)

#17

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/685978

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-03: Fix merged to cinder (stable/rocky)

#18

Reviewed: https://review.opendev.org/680000
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=eabf648b7acce33cb93d667e8f6e70505093b4c9
Submitter: Zuul
Branch: stable/rocky

commit eabf648b7acce33cb93d667e8f6e70505093b4c9
Author: Gorka Eguileor <email address hidden>
Date: Tue Jul 2 11:06:17 2019 +0200

Fix DetachedInstanceError for VolumeAttachment

This patch replaces "hasattr" with a check on the object's dictionary,
which will never trigger a Lazy Load.

    Closes-Bug: #1834845
    Change-Id: Iac785eef9be4b9cdb5c739ee0a87949805282867
    (cherry picked from commit 2e73bede80cc2acdb3527f06bc5c5f9c1a8463a7)
    (cherry picked from commit 44a15be6a71ebdaf0e5d1663fc31011ff43ef37b)
    Conflicts:
        cinder/tests/unit/volume/drivers/test_lvm_driver.py

tags:

added: in-stable-rocky

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-04: Fix merged to cinder (stable/queens)

#19

Reviewed: https://review.opendev.org/685978
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=2abd8a68bb1bde9adc7c870a1b827731ad4e42e9
Submitter: Zuul
Branch: stable/queens

commit 2abd8a68bb1bde9adc7c870a1b827731ad4e42e9
Author: Gorka Eguileor <email address hidden>
Date: Tue Jul 2 11:06:17 2019 +0200

Fix DetachedInstanceError for VolumeAttachment

This patch replaces "hasattr" with a check on the object's dictionary,
which will never trigger a Lazy Load.

tags:

added: in-stable-queens

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-04: Fix included in openstack/cinder 14.0.2

#20

This issue was fixed in the openstack/cinder 14.0.2 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-10-24: Fix included in openstack/cinder 12.0.10

#21

This issue was fixed in the openstack/cinder 12.0.10 release.

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2019-10-28: Please test proposed package

#22

Hello Patrick, or anyone else affected,

Accepted cinder into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

sudo add-apt-repository cloud-archive:stein-proposed
sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags:

added: verification-stein-needed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2020-01-06: Fix included in openstack/cinder 13.0.8

#23

This issue was fixed in the openstack/cinder 13.0.8 release.

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2020-01-09:

#24

This was fixed in Ubuntu Train version 2:15.0.0~b1~git2019080709.0a0d55d8a-0ubuntu1.

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2020-01-09:

#25

This is fixed in Ubuntu Disco version 2:14.0.1-0ubuntu2, so this is currently in disco-proposed.

Changed in cinder (Ubuntu Disco):
status:	In Progress → Fix Committed

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2020-01-09:

#26

Since disco is EOL this month, we can work this through the stein cloud archive.

Changed in cinder (Ubuntu Disco):
status:	Fix Committed → Won't Fix
Changed in cloud-archive:
status:	Fix Committed → Fix Released

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2020-01-09:

#27

Verified successfully:

$ juju ssh cinder/0 apt policy cinder-common
cinder-common:
  Installed: 2:13.0.8-0ubuntu1~cloud0
  Candidate: 2:13.0.8-0ubuntu1~cloud0
  Version table:
*** 2:13.0.8-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/rocky/main amd64

$ juju config cinder openstack-origin=cloud:bionic-stein/proposed

$ juju ssh cinder/0 apt policy cinder-common
cinder-common:
  Installed: 2:14.0.2-0ubuntu1~cloud0
  Candidate: 2:14.0.2-0ubuntu1~cloud0
  Version table:
*** 2:14.0.2-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/stein/main amd64

$ openstack volume create --size 5 v2

$ openstack volume list
+--------------------------------------+------+-----------+------+-----------------------------+
| ID | Name | Status | Size | Attached to |
+--------------------------------------+------+-----------+------+-----------------------------+
| 41b29ac2-c580-444c-ba65-17d016fb136f | v2 | available | 5 | |
| 760b2fe5-bfa3-4ec6-964d-6fd05ccba3b0 | v1 | in-use | 1 | Attached to x1 on /dev/vdb |
+--------------------------------------+------+-----------+------+-----------------------------+

tags:

added: verification-stein-done
removed: verification-stein-needed

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2020-01-13: Update Released

#28

The verification of the Stable Release Update for cinder has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2020-01-13:

#29

This bug was fixed in the package cinder - 2:14.0.2-0ubuntu1~cloud0
---------------

cinder (2:14.0.2-0ubuntu1~cloud0) bionic-stein; urgency=medium
.
   * New upstream release for the Ubuntu Cloud Archive.
.
cinder (2:14.0.2-0ubuntu1) disco; urgency=medium
.
   * New upstream point release for OpenStack Stein (LP: #1849192):
     - Includes fix for SIGABRT in cinder-backup/ceph (LP: #1838691).
     - d/p/Fix-DetachedInstanceError-for-VolumeAttachment.patch: Drop,
       included in release.
.
cinder (2:14.0.1-0ubuntu3) disco; urgency=medium
.
   * d/tests/cinder-daemons: As of disco, sqlalchemy defaults to mysqldb so
     we must override the driver to use pymysql (LP: #1845321).
.
cinder (2:14.0.1-0ubuntu2) disco; urgency=medium
.
   * d/p/Fix-DetachedInstanceError-for-VolumeAttachment.patch: Cherry
     pick fix for issue with DetachedInstanceError being generated due
     to compatibility issues with SQLAlchemy (LP: #1834845)