Glance does not recover database connection automatically after DB restart

Setting to low priority since the subsequent requests work fine...

Fix proposed to branch: master
Review: https://review.openstack.org/5552

Reviewed: https://review.openstack.org/5552
Committed: http://github.com/openstack/glance/commit/3f462fbf9a2b9cfe6ac39d87fe5b327ba82e9dac
Submitter: Jenkins
Branch: master

commit 3f462fbf9a2b9cfe6ac39d87fe5b327ba82e9dac
Author: Unmesh Gurjar <email address hidden>
Date: Tue Mar 20 12:07:39 2012 +0530

    Fixed db conn recovery issue. Fixes bug 954971.

    This implementation wraps the db method calls and retries the method if an attempt failed due to db connectivity issues. Defines 2 new configuration parameters viz sql_max_retries and sql_retry_interval (values default to 10 and 1 respectively).
    This implementation does not have any overhead in the normal case, as opposed to MySQLPingListener implementation, which does a 'select 1' query every time a connection is checkout from the pool.

    Change-Id: I24240f22bca445b9ab76a1594631b5eaca393b4d

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/5889

Reviewed: https://review.openstack.org/5889
Committed: http://github.com/openstack/glance/commit/98f53473008a04c113e35f9c61f0361fa9328420
Submitter: Jenkins
Branch: milestone-proposed

commit 98f53473008a04c113e35f9c61f0361fa9328420
Author: Unmesh Gurjar <email address hidden>
Date: Tue Mar 20 12:07:39 2012 +0530

    Fixed db conn recovery issue. Fixes bug 954971.

    This implementation wraps the db method calls and retries the method if an attempt failed due to db connectivity issues. Defines 2 new configuration parameters viz sql_max_retries and sql_retry_interval (values default to 10 and 1 respectively).
    This implementation does not have any overhead in the normal case, as opposed to MySQLPingListener implementation, which does a 'select 1' query every time a connection is checkout from the pool.

    Change-Id: I24240f22bca445b9ab76a1594631b5eaca393b4d

I'm not sure this is actually fixed the issue anymore and perhaps has regressed if the MySQLPingListener did in fact reconnect after servers have 'gone away' As a test, I'm starting glance, running 'glance index' and restarting the mysql server. The next run of 'glance index' produces a traceback on the client and server side, dumping the "Mysql server has gone away" error. I believe the functions in get_session + get_engine are not enough to catch the error when executed against the database connection. If someone else can confirm, this bug should be reopenned.

Turns out there are some differences in exception behavior between python-sqlalchemy 0.6.8-1 (oneiric) and 0.7.4-1 (precise).

AFAICS, the exceptions being caught in the wrap_db() handling are not being similarly raised in the newer version. On oneiric, we were catching the sqlalchemy.exc.OperationalError exception out of /usr/lib/python2.7/dist-packages/sqlalchemy/orm/query.py:1726 in _execute_and_instances(). This code has changed in some recent version, and no there is no exception to be caught. The one that ends up raising up and causing traceback is coming from the mysqldb back-end dialect, and it would be a bad idea to start worrying about catching that kind of thing from glance/nova. I'm not sure if this is a bug in sqlalchemy, or if we should not rely on wrapping + catching this kind of thing, and instead use MySQLPingListener that was used before, as it seems to at least be portable.

Correction: the issue is more that _execute_and_instances() previously was calling session.execute(), which we're wrapping in glance. Newer versions called execute directly on the connection. If we wished to wrap this as well, we'd need to be looking out for exceptions that are specific to the many backend connections supported by SQLA.