OpenStack Nova Cloud Controller Charm

No sql_connection parameter is established

Bug #1752402 reported by Ashley Lai
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Nova Cloud Controller Charm
Fix Released
High
Corey Bryant
OpenStack Nova Cloud Controller Charm 18.05

Bug Description

nova.conf file is missing the database connection line.
[database]
max_pool_size = 5

nova.conf file: https://pastebin.canonical.com/p/NmTGqZCqCP/

2018-02-28 16:21:02 ERROR juju-log cloud-compute:105: list_cells failed
An error has occurred:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 1797, in main
    ret = fn(*fn_args, **fn_kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/cmd/manage.py", line 1640, in list_cells
    context.get_admin_context())
  File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
    result = fn(cls, context, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/nova/objects/cell_mapping.py", line 137, in get_all
    db_mappings = cls._get_all_from_db(context)
  File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 978, in wrapper
    with self._transaction_scope(context):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1028, in _transaction_scope
    context=context) as resource:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 633, in _session
    bind=self.connection, mode=self.mode)
  File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 398, in _create_session
    self._start()
  File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 484, in _start
    engine_args, maker_args)
  File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 506, in _setup_for_connection
    "No sql_connection parameter is established")
CantStartEngineError: No sql_connection parameter is established

Revision history for this message
Ashley Lai (alai) wrote :
Revision history for this message
Ashley Lai (alai) wrote :
Revision history for this message
Chris Gregan (cgregan) wrote :

escalated to field high after additional instances of this issue hit in CI

Revision history for this message
John George (jog) wrote :
Revision history for this message
John George (jog) wrote :
Ryan Beisner (1chb1n)
Changed in charm-nova-cloud-controller:
assignee: nobody → Corey Bryant (corey.bryant)
importance: Undecided → High
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hi Ashley and John. Thanks for reporting this. I'm starting to dig into this.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

nova-cloud-controller/0
-----------------------
(3/lxd/5) db migration + list cells looks successful. unit is leader.

nova.conf shows:

[database]
connection = mysql://nova:PWR7Jm9CTrM27SqKRXYrCH8gShSMTXNJ@192.168.33.149/nova
max_pool_size = 5

[api_database]
connection = mysql://nova:PWR7Jm9CTrM27SqKRXYrCH8gShSMTXNJ@192.168.33.149/nova_api
max_pool_size = 5

nova-cloud-controller/1
-----------------------
(5/lxd/5) failure occurs here. all signs point to this unit not being leader.

nova.conf shows:

[database]
max_pool_size = 5

[api_database]
max_pool_size = 5

nova-cloud-controller/2
-----------------------
(6/lxd/5) all signs point to this unit not being leader, no failures.

nova.conf shows:

[database]
connection = mysql://nova:PWR7Jm9CTrM27SqKRXYrCH8gShSMTXNJ@192.168.33.149/nova
max_pool_size = 5

[api_database]
connection = mysql://nova:PWR7Jm9CTrM27SqKRXYrCH8gShSMTXNJ@192.168.33.149/nova_api
max_pool_size = 5

Revision history for this message
Corey Bryant (corey.bryant) wrote :

I think we're missing an is_leader() check in the following code:

@hooks.hook('cloud-compute-relation-changed')
def compute_changed(rid=None, unit=None):
    ...

    if is_cellv2_init_ready() and is_db_initialised():
        add_hosts_to_cell()

More thorough look at code: https://paste.ubuntu.com/p/9K7Xt3YwWt/

Revision history for this message
Corey Bryant (corey.bryant) wrote :

The log juju log for nova-cloud-controller/1 seems to confirm the above missing is_leader() check. Traceback occurs while in cloud-compute-relation-changed hook. Following are a few logs of interest right before the traceback occurs. Note the "Database is initialised" log comes from is_db_initialised().

2018-02-28 16:20:18 DEBUG juju.worker.leadership tracker.go:276 nova-cloud-controller/1 is not nova-cloud-controller leader
2018-02-28 16:20:18 DEBUG juju-log cloud-compute:105: Database is initialised
2018-02-28 16:20:18 INFO juju-log cloud-compute:105: Cell1 discover_hosts
2018-02-28 16:20:19 INFO juju-log cloud-compute:105: Listing cell, 'cell1'
2018-02-28 16:20:22 ERROR juju-log cloud-compute:105: list_cells failed
An error has occurred:
Traceback (most recent call last):
 ...
CantStartEngineError: No sql_connection parameter is established

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Attaching full nova-cloud-controller/1 juju log.

Changed in charm-nova-cloud-controller:
status: New → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-cloud-controller (master)

Fix proposed to branch: master
Review: https://review.openstack.org/554931

Changed in charm-nova-cloud-controller:
status: Triaged → In Progress
Corey Bryant (corey.bryant)
tags: added: stable-backport
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-cloud-controller (stable/18.02)

Fix proposed to branch: stable/18.02
Review: https://review.openstack.org/554999

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-cloud-controller (master)

Reviewed: https://review.openstack.org/554931
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=8893bace59853c6379659a94f81c1a4a5cc095d1
Submitter: Zuul
Branch: master

commit 8893bace59853c6379659a94f81c1a4a5cc095d1
Author: Corey Bryant <email address hidden>
Date: Wed Mar 21 12:40:22 2018 +0000

    Ensure only the leader can add hosts to cell

    Ensure that only the elected leader can add compute hosts to the
    cell in cloud-compute-relation-changed as database commands should
    only be run by the elected leader.

    Change-Id: I4806580f58e2a2feba82c74d9d5ff29dfb220c9e
    Closes-Bug: #1752402

Changed in charm-nova-cloud-controller:
status: In Progress → Fix Committed
Corey Bryant (corey.bryant)
Changed in charm-nova-cloud-controller:
milestone: none → 18.05
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-cloud-controller (stable/18.02)

Reviewed: https://review.openstack.org/554999
Committed: https://git.openstack.org/cgit/openstack/charm-nova-cloud-controller/commit/?id=9d3fb0f6cf1fef03feffd5e602081269b5dc83b0
Submitter: Zuul
Branch: stable/18.02

commit 9d3fb0f6cf1fef03feffd5e602081269b5dc83b0
Author: Corey Bryant <email address hidden>
Date: Wed Mar 21 12:40:22 2018 +0000

    Ensure only the leader can add hosts to cell

    Ensure that only the elected leader can add compute hosts to the
    cell in cloud-compute-relation-changed as database commands should
    only be run by the elected leader.

    Change-Id: I4806580f58e2a2feba82c74d9d5ff29dfb220c9e
    Closes-Bug: #1752402

Ryan Beisner (1chb1n)
Changed in charm-nova-cloud-controller:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.