nova-cells fails when using JSON file to store cell information

Bug #1314677 reported by Liam Young
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Liam Young
Icehouse
Fix Released
Undecided
Unassigned
nova (Ubuntu)
Fix Released
Undecided
Liam Young
Trusty
Fix Released
Undecided
Liam Young

Bug Description

As recommended in http://docs.openstack.org/havana/config-reference/content/section_compute-cells.html#cell-config-optional-json I'm creating the nova-cells config with the cell information stored in a json file. However, when I do this nova-cells fails to start with this error in the logs:

2014-04-29 11:52:05.240 16759 CRITICAL nova [-] __init__() takes exactly 3 arguments (1 given)
2014-04-29 11:52:05.240 16759 TRACE nova Traceback (most recent call last):
2014-04-29 11:52:05.240 16759 TRACE nova File "/usr/bin/nova-cells", line 10, in <module>
2014-04-29 11:52:05.240 16759 TRACE nova sys.exit(main())
2014-04-29 11:52:05.240 16759 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/cmd/cells.py", line 40, in main
2014-04-29 11:52:05.240 16759 TRACE nova manager=CONF.cells.manager)
2014-04-29 11:52:05.240 16759 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 257, in create
2014-04-29 11:52:05.240 16759 TRACE nova db_allowed=db_allowed)
2014-04-29 11:52:05.240 16759 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/service.py", line 139, in __init__
2014-04-29 11:52:05.240 16759 TRACE nova self.manager = manager_class(host=self.host, *args, **kwargs)
2014-04-29 11:52:05.240 16759 TRACE nova File "/usr/lib/python2.7/dist-packages/nova/cells/manager.py", line 87, in __init__
2014-04-29 11:52:05.240 16759 TRACE nova self.state_manager = cell_state_manager()
2014-04-29 11:52:05.240 16759 TRACE nova TypeError: __init__() takes exactly 3 arguments (1 given)

I have had a dig into the code and it appears that CellsManager creates an instance of CellStateManager with no arguments. CellStateManager __new__ runs and creates an instance of CellStateManagerFile which runs __new__ and __init__ with cell_state_cls and cells_config_path set. At this point __new__ returns CellStateManagerFile and the new instance's __init__() method is invoked (CellStateManagerFile.__init__) with the original arguments (there weren't any) which then results in the stack trace.

It seems reasonable for CellStateManagerFile to derive the cells_config_path info for itself so I've patched it locally with

=== modified file 'state.py'
--- state.py 2014-04-30 15:10:16 +0000
+++ state.py 2014-04-30 15:10:26 +0000
@@ -155,7 +155,7 @@
             config_path = CONF.find_file(cells_config)
             if not config_path:
                 raise cfg.ConfigFilesNotFoundError(config_files=[cells_config])
- return CellStateManagerFile(cell_state_cls, config_path)
+ return CellStateManagerFile(cell_state_cls)

         return CellStateManagerDB(cell_state_cls)

@@ -450,7 +450,9 @@

 class CellStateManagerFile(CellStateManager):
- def __init__(self, cell_state_cls, cells_config_path):
+ def __init__(self, cell_state_cls=None):
+ cells_config = CONF.cells.cells_config
+ cells_config_path = CONF.find_file(cells_config)
         self.cells_config_path = cells_config_path
         super(CellStateManagerFile, self).__init__(cell_state_cls)

Ubuntu: 14.04
nova-cells: 1:2014.1-0ubuntu1

nova.conf:

[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
iscsi_helper=tgtadm
libvirt_use_virtio_for_bridges=True
connection_type=libvirt
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
verbose=True
ec2_private_dns_show_ip=True
api_paste_config=/etc/nova/api-paste.ini
volumes_path=/var/lib/nova/volumes
enabled_apis=ec2,osapi_compute,metadata
auth_strategy=keystone
compute_driver=libvirt.LibvirtDriver
quota_driver=nova.quota.NoopQuotaDriver

[cells]
enable=True
name=cell
cell_type=compute
cells_config=/etc/nova/cells.json

cells.json:
{
    "parent": {
        "name": "parent",
        "api_url": "http://api.example.com:8774",
        "transport_url": "rabbit://rabbit.example.com",
        "weight_offset": 0.0,
        "weight_scale": 1.0,
        "is_parent": true
    }
}

Related branches

Tracy Jones (tjones-i)
tags: added: cells
Changed in nova:
assignee: nobody → Pranav Salunke (dguitarbite)
Revision history for this message
Liam Young (gnuoy) wrote :

As there hasn't been much movement on this I'm going to take the bug and submit the patch I described when the bug was submitted

Changed in nova:
assignee: Pranav Salunke (dguitarbite) → Liam Young (gnuoy)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/108733

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/108733
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=695191fa89387d96e60120ff32965493c844e7f5
Submitter: Jenkins
Branch: master

commit 695191fa89387d96e60120ff32965493c844e7f5
Author: Liam Young <email address hidden>
Date: Tue Jul 22 16:25:00 2014 +0100

    Fix CellStateManagerFile init to failure

    Currently, specifying a cells_config file in nova.conf causes
    CellStateManager to fail and in turn stops the nova-cells service from
    starting. The reason is that CellsManager creates an instance of
    CellStateManager with no arguments. CellStateManager __new__ runs and
    creates an instance of CellStateManagerFile which runs __new__ and
    __init__ with cell_state_cls and cells_config_path set. At this point
    __new__ returns CellStateManagerFile and the new instance's __init__
    method is invoked (CellStateManagerFile.__init__) with the original
    arguments (there weren't any) which then results in:
    2014-04-29 11:52:05.240 16759 TRACE nova self.state_manager =
    cell_state_manager()
    2014-04-29 11:52:05.240 16759 TRACE nova TypeError: __init__() takes
    exactly 3 arguments (1 given)

    It seems reasonable for CellStateManagerFile to derive the
    cells_config_path info for itself so I have updated the code with that
    change and added unit tests to catch this bug and to check that the
    correct managers are still returned

    Change-Id: I9021640515142a3ca95c2d9e7b03e19b529bc175
    Closes-Bug: #1314677

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
RedBaron (dheeraj-gupta4) wrote :
Download full text (3.4 KiB)

I know this bug has been patched, but I ran into a weird situation with the unpatched master branch.
If I specified invalid JSON in the file, the CellStateManagerFile was correctly instantiated (going beyond the point where the bug pops up) and nova-cells went as far as trying to read cell state from the file and failing with invalid JSON.
However, on correcting the JSON, the CellStateManager failed to instantiate with the error message in the original bug.
I couldn't figure out why this happened. Maybe, if you have time, you can investigate it

Appendix: Here are the logs for starting an unpatched nova with invalid JSON

CRITICAL nova [-] ValueError: Expecting property name enclosed in double quotes: line 9 column 9 (char 273)

2014-08-29 14:28:09.030 TRACE nova Traceback (most recent call last):
2014-08-29 14:28:09.030 TRACE nova File "/usr/bin/nova-cells", line 10, in <module>
2014-08-29 14:28:09.030 TRACE nova sys.exit(main())
2014-08-29 14:28:09.030 TRACE nova File "/opt/stack/nova/nova/cmd/cells.py", line 45, in main
2014-08-29 14:28:09.030 TRACE nova manager=CONF.cells.manager)
2014-08-29 14:28:09.030 TRACE nova File "/opt/stack/nova/nova/service.py", line 275, in create
2014-08-29 14:28:09.030 TRACE nova db_allowed=db_allowed)
2014-08-29 14:28:09.030 TRACE nova File "/opt/stack/nova/nova/service.py", line 148, in __init__
2014-08-29 14:28:09.030 TRACE nova self.manager = manager_class(host=self.host, *args, **kwargs)
2014-08-29 14:28:09.030 TRACE nova File "/opt/stack/nova/nova/cells/manager.py", line 91, in __init__
2014-08-29 14:28:09.030 TRACE nova self.state_manager = cell_state_manager()
2014-08-29 14:28:09.030 TRACE nova File "/opt/stack/nova/nova/cells/state.py", line 159, in __new__
2014-08-29 14:28:09.030 TRACE nova return CellStateManagerFile(cell_state_cls, config_path)
2014-08-29 14:28:09.030 TRACE nova File "/opt/stack/nova/nova/cells/state.py", line 466, in __init__
2014-08-29 14:28:09.030 TRACE nova super(CellStateManagerFile, self).__init__(cell_state_cls)
2014-08-29 14:28:09.030 TRACE nova File "/opt/stack/nova/nova/cells/state.py", line 176, in __init__
2014-08-29 14:28:09.030 TRACE nova self._cell_data_sync(force=True)
2014-08-29 14:28:09.030 TRACE nova File "/opt/stack/nova/nova/cells/state.py", line 480, in _cell_data_sync
2014-08-29 14:28:09.030 TRACE nova self.cells_config_data = jsonutils.loads(data)
2014-08-29 14:28:09.030 TRACE nova File "/opt/stack/nova/nova/openstack/common/jsonutils.py", line 176, in loads
2014-08-29 14:28:09.030 TRACE nova return json.loads(strutils.safe_decode(s, encoding), **kwargs)
2014-08-29 14:28:09.030 TRACE nova File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
2014-08-29 14:28:09.030 TRACE nova return _default_decoder.decode(s)
2014-08-29 14:28:09.030 TRACE nova File "/usr/lib64/python2.7/json/decoder.py", line 365, in decode
2014-08-29 14:28:09.030 TRACE nova obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2014-08-29 14:28:09.030 TRACE nova File "/usr/lib64/python2.7/json/decoder.py", line 381, in raw_decode
2014-08-29 14:28:09.030 TRACE nova obj, end = self.scan_once(s, idx)
2014-08-...

Read more...

Thierry Carrez (ttx)
Changed in nova:
milestone: none → juno-3
status: Fix Committed → Fix Released
Robie Basak (racb)
Changed in nova (Ubuntu):
assignee: nobody → Liam Young (gnuoy)
Changed in nova (Ubuntu Trusty):
assignee: nobody → Liam Young (gnuoy)
Revision history for this message
Liam Young (gnuoy) wrote :

debdiff nova_2014.1.2-0ubuntu1.1.dsc nova_2014.1.2-0ubuntu1.2.dsc

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "trusty-ubuntu1.1-ubuntu1.2.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/124811

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/icehouse)

Change abandoned by Liam Young (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/124811
Reason: One of the chaged files went awol

Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Liam, or anyone else affected,

Accepted nova into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nova/1:2014.1.3-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in nova (Ubuntu):
status: New → Fix Released
Changed in nova (Ubuntu Trusty):
status: New → Fix Committed
tags: added: verification-needed
Revision history for this message
Liam Young (gnuoy) wrote :

I've tested with nova-cells 1:2014.1.3-0ubuntu1 and this fixes the problem

tags: added: verification-done
removed: verification-needed
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-3 → 2014.2
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (3.9 KiB)

This bug was fixed in the package nova - 1:2014.1.3-0ubuntu1

---------------
nova (1:2014.1.3-0ubuntu1) trusty; urgency=medium

  [ Liam Young ]
  * d/p/cells-json-store.patch: Fix issue with nova-cells failing when using
    JSON file to store cell information (LP: #1314677).

  [ Corey Bryant ]
  * Resynchronize with stable/icehouse (a058646) (LP: #1377136):
    - [1a95c95] Adds tests for Hyper-V VM Utils
    - [bb47d55] Removes unnecessary instructions in test_hypervapi
    - [4f41d37] Fixes a Hyper-V list_instances localization issue
    - [9015410] Adds list_instance_uuids to the Hyper-V driver
    - [3371ad8] Add _wrap_db_error() support to Session.commit()
    - [dfb0e0f] Neutron: Atomic update of instance info cache
    - [bce481c] Ensure info cache updates don't overwhelm cells
    - [f58d95c] Sync process and str utils from oslo
    - [4e6371b] remove test_multiprocess_api
    - [7e09173] Fixes Hyper-V agent force_hyperv_utils_v1 flag issue
    - [7523ab4] Fix attaching config drive issue on Hyper-V when migrate instances
    - [74e0ba7] Fix live-migration failure in FC multipath case
    - [b61aa4d] libvirt: Save device_path in connection_info when booting from volume
    - [f93b8ee] Made unassigned networks visible in flat networking
    - [82cc3be] Do not fail cell's instance deletion, if it's missing info_cache
    - [d72c0a4] Fixes Hyper-V boot from volume root device issue
    - [0d3dad7] Fixes Hyper-V resize down exception
    - [5d5970a] db: Add @_retry_on_deadlock to service_update()
    - [9596f52] Add Hyper-V driver in the "compute_driver" option description
    - [4a8d6ca] Block sqlalchemy migrate 0.9.2 as it breaks all of nova
    - [311ab57] Move the error check for "brctl addif"
    - [df09c2a] Fix rootwrap for non openstack.org iqn's
    - [1613cd99] Fix instance boot when Ceph is used for ephemeral storage
    - [4bc680f] Make floatingip-ip-delete atomic with neutron
    - [0d69163] Fix race condition with vif plugging in finish migrate
    - [520aa4c] libvirt: Use VIR_DOMAIN_AFFECT_LIVE for paused instances
    - [3c34e37] add repr for event objects
    - [1b7ab22] make lifecycle event logs more clear
    - [e1d6e18] Catch missing Glance image attrs with None
    - [b591389] Update block_device_info to contain swap and ephemeral disks
    - [2155188] Adds get_instance_disk_info to compute drivers
    - [87f842d] Fixes Hyper-V vm state issue
    - [1106ef2] Fix expected error details from jsonschema
    - [e5e6bc7] Include next link when default limit is reached
    - [526853e] Fix FloatingIP.save() passing FixedIP object to sqlalchemy
    - [4e1e217] Read deleted instances during lifecycle events
    - [d8b9ba5] Add a retry_on_deadlock to reservations_expire
    - [b53adea] Add expire reservations in backport position.
    - [e874ee2] Fixes Hyper-V SCSI slot selection
    - [471e644] VMware: do not cache image when root_gb is 0
    - [825cfe4] Fix _parse_datetime in simple tenant usage extension
    - [073ee06] Avoid traceback logs from simple tenant usage extension
    - [9447203] replace NovaException with VirtualInterfaceCreate when neutron fails
    - [6b7cb1a] libvirt: convert cpu features attribute from list...

Read more...

Changed in nova (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/icehouse)

Reviewed: https://review.openstack.org/124811
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bfeae680bc0632e0129b647d45e4b80328a33535
Submitter: Jenkins
Branch: stable/icehouse

commit bfeae680bc0632e0129b647d45e4b80328a33535
Author: Liam Young <email address hidden>
Date: Tue Jul 22 16:25:00 2014 +0100

    Fix CellStateManagerFile init to failure

    Currently, specifying a cells_config file in nova.conf causes
    CellStateManager to fail and in turn stops the nova-cells service from
    starting. The reason is that CellsManager creates an instance of
    CellStateManager with no arguments. CellStateManager __new__ runs and
    creates an instance of CellStateManagerFile which runs __new__ and
    __init__ with cell_state_cls and cells_config_path set. At this point
    __new__ returns CellStateManagerFile and the new instance's __init__
    method is invoked (CellStateManagerFile.__init__) with the original
    arguments (there weren't any) which then results in:
    2014-04-29 11:52:05.240 16759 TRACE nova self.state_manager =
    cell_state_manager()
    2014-04-29 11:52:05.240 16759 TRACE nova TypeError: __init__() takes
    exactly 3 arguments (1 given)

    It seems reasonable for CellStateManagerFile to derive the
    cells_config_path info for itself so I have updated the code with that
    change and added unit tests to catch this bug and to check that the
    correct managers are still returned

    Closes-Bug: #1314677
    (cherry picked from commit 695191fa89387d96e60120ff32965493c844e7f5)

    Conflicts:
     nova/tests/cells/test_cells_state_manager.py

    Change-Id: I9021640515142a3ca95c2d9e7b03e19b529bc175

tags: added: in-stable-icehouse
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.