Nova should retry connections if started before MySQL

Bug #876663 reported by IT Archmerc
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Wishlist
Unassigned
Diablo
Fix Released
Undecided
Unassigned
OpenStack Dashboard (Horizon)
Invalid
Undecided
Unassigned

Bug Description

I had a working installation of a cloud and compute node on a single server running Ubuntu 11.04. Everything was fine until I rebooted the server and found that upon each reboot, the nova services were starting prior to MySQL. Once the server came up, MySQL was running, but nova-compute, nova-network, nova-volume, and nova-scheduler had all failed with messages about not able to connect to MySQL.

Nova should probably be able to retry until MySQL is available.

Revision history for this message
Kiall Mac Innes (kiall) wrote :

I think the reason mysql is not included in the `start on` field is because installs with more than 1 server will likely not have mysql installed on every server...

Revision history for this message
Rafi Khardalian (rkhardalian) wrote :

What I'd actually prefer to see is to have nova-* keep trying to connect to the DB rather than exiting if it's unavailable at start time. Perhaps the frequency and and maximum number of attempts could be configurable (defaulting to trying forever).

Revision history for this message
Devin Carlen (devcamcar) wrote :

Marking as invalid in context of the dashboard since this is an order of operations issue.

Changed in nova:
status: New → Invalid
status: Invalid → New
Changed in horizon:
status: New → Invalid
Revision history for this message
Thierry Carrez (ttx) wrote :

This is actually an Ubuntu packaging issue.

affects: nova → ubuntu
affects: ubuntu → nova (Ubuntu)
Revision history for this message
Thierry Carrez (ttx) wrote :

Actually nova should support starting before MySQL is -- and retry if not. Will change description to match

Could you tell us what version you're running ?

affects: nova (Ubuntu) → nova
Changed in nova:
status: New → Incomplete
summary: - Nova starting before MySQL
+ Nova should be able to start before MySQL
description: updated
Revision history for this message
IT Archmerc (itarchmerc) wrote : Re: Nova should be able to start before MySQL

After reading all the comments and looking at the different scenarios, a retry does seem to be the best approach. Thanks for the feedback.

@ttx I'm not sure what version or software you're looking for, but here are a few. Please let me know if you need more.

ii libdbd-mysql-perl 4.016-1 Perl5 database interface to the MySQL database
ii libmysqlclient16 5.1.54-1ubuntu4 MySQL database client library
ii mysql-client-5.1 5.1.54-1ubuntu4 MySQL database client binaries
ii mysql-client-core-5.1 5.1.54-1ubuntu4 MySQL database core client binaries
ii mysql-common 5.1.54-1ubuntu4 MySQL database common files, e.g. /etc/mysql/my.cnf
ii mysql-server 5.1.54-1ubuntu4 MySQL database server (metapackage depending on the latest version)
ii mysql-server-5.1 5.1.54-1ubuntu4 MySQL database server binaries and system database setup
ii mysql-server-core-5.1 5.1.54-1ubuntu4 MySQL database server binaries
ii python-mysqldb 1.2.2-10build2 A Python interface to MySQL

ii nova-ajax-console-proxy 2011.3-0ubuntu2~ppa1~natty1 OpenStack Compute - AJAX console proxy
ii nova-api 2011.3-0ubuntu2~ppa1~natty1 OpenStack Compute - API frontend
ii nova-common 2011.3-0ubuntu2~ppa1~natty1 OpenStack Compute - common files
ii nova-compute 2011.3-0ubuntu2~ppa1~natty1 OpenStack Compute - compute node
ii nova-compute-kvm 2011.3-0ubuntu2~ppa1~natty1 OpenStack Compute - compute node (KVM)
ii nova-doc 2011.3-0ubuntu2~ppa1~natty1 OpenStack Compute - documetation
ii nova-network 2011.3-0ubuntu2~ppa1~natty1 OpenStack Compute - Network manager
ii nova-objectstore 2011.3-0ubuntu2~ppa1~natty1 OpenStack Compute - object store
ii nova-scheduler 2011.3-0ubuntu2~ppa1~natty1 OpenStack Compute - virtual machine scheduler
ii nova-vncproxy 2011.3-0ubuntu2~ppa1~natty1 OpenStack Compute - VNC proxy
ii nova-volume 2011.3-0ubuntu2~ppa1~natty1 OpenStack Compute - storage
ii python-nova 2011.3-0ubuntu2~ppa1~natty1 OpenStack Compute Python libraries
ii python-novaclient 2.6.4~bzr113-0ubuntu0ppa1~natty1 client library for OpenStack Compute API

Thierry Carrez (ttx)
summary: - Nova should be able to start before MySQL
+ Nova should retry connections if started before MySQL
Changed in nova:
importance: Undecided → Wishlist
status: Incomplete → Confirmed
Revision history for this message
Soren Hansen (soren) wrote :

I started digging into this a bit.

We seem to handle this reasonably gracefully once we're up and running. The attempted operation fails, but the service keeps running. However, if the connection fails during initialisation, we're in much more trouble, because the unhandled exception happens in the main thread which then dies.
If we see this error later on, it's (almost?) exclusively while handling incoming requests, which is done in separate threads.

There are a couple of different ways to attack this. I'll see what I can come up with.

Changed in nova:
assignee: nobody → Soren Hansen (soren)
assignee: Soren Hansen (soren) → nobody
Revision history for this message
Soren Hansen (soren) wrote :

First stab at this:

   https://github.com/sorenh/nova/tree/sql-retry

I need to work out a way to test this, though.

Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/1157
Committed: http://github.com/openstack/nova/commit/6ebc0ac3248b6ec7f1a221b4dae5f326406ab614
Submitter: Jenkins
Branch: master

 status fixcommitted
 done

commit 6ebc0ac3248b6ec7f1a221b4dae5f326406ab614
Author: Soren Hansen <email address hidden>
Date: Thu Oct 27 11:46:54 2011 +0200

    Retry failed SQL connections (LP #876663)

    Change-Id: I972d8debb2e3230bccfcca68aee5b08021ba16c7

Revision history for this message
Openstack Gerrit (openstack-gerrit) wrote : Fix merged to nova (stable/diablo)

Reviewed: https://review.openstack.org/1661
Committed: http://github.com/openstack/nova/commit/b0821f257e7fb512474a0da2b9d5c24d4cbf49ae
Submitter: Jenkins
Branch: stable/diablo

 tag in-stable-diablo
 done

commit b0821f257e7fb512474a0da2b9d5c24d4cbf49ae
Author: Soren Hansen <email address hidden>
Date: Thu Oct 27 11:46:54 2011 +0200

    Retry failed SQL connections (LP #876663)

    (cherry picked from commit 6ebc0ac3248b6ec7f1a221b4dae5f326406ab614)

    Change-Id: I972d8debb2e3230bccfcca68aee5b08021ba16c7

Revision history for this message
Martin Pitt (pitti) wrote : Please test proposed package

Hello IT, or anyone else affected,

Accepted nova into oneiric-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

tags: added: verification-needed
Thierry Carrez (ttx)
Changed in nova:
status: Confirmed → Fix Committed
status: Fix Committed → Fix Released
Revision history for this message
Kevin Jackson (kevin-linuxservices) wrote :

This affected me too - running packages from proposed I can see the services waiting for the database to become available and when it does it connects fine. After rebooting no further intervention was required.

tags: added: verification-done
removed: verification-needed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.