Merge lp:~cbehrens/nova/lp766404 into lp:~hudson-openstack/nova/trunk

[Chris Behrens]

XenAPI was not implemented to allow for multiple simultaneous XenAPI requests. A single XenAPIConnection (and thus XenAPISession) is used for all queries. XenAPISession's wait_for_task method would set a self.loop = for looping calls to _poll_task until task completion. Subsequent (parallel) calls to wait_for_task for another query would overwrite this. XenAPISession._poll_task was pulled into the XenAPISession.wait_for_task method to avoid having to store self.loop.

[Brian Lamar (blamar)]

Are there tests that can show this fix in action?

[Chris Behrens (cbehrens)]

Good question. There are XenAPI tests in general which exercise this code (nova/tests/xenapi/*). I don't think I'm going to easily be able to get 2 queries going at once in the tests due to how quickly they execute. Hmm.

[Vish Ishaya (vishvananda)]

You can probably get this to work with a greenthread and a mock method that uses time.sleep. If you want to get a little bit more elaborate you can make a method that waits on an eventlet Event() so you can be sure that it pauses until you tell it to finish. See CacheConcurrencyTestCase in test_virt for an example.

On May 11, 2011, at 10:31 AM, Chris Behrens wrote:

> Good question. There are XenAPI tests in general which exercise this code (nova/tests/xenapi/*). I don't think I'm going to easily be able to get 2 queries going at once in the tests due to how quickly they execute. Hmm.
> --
> https://code.launchpad.net/~cbehrens/nova/lp766404/+merge/60543
> You are subscribed to branch lp:nova.

[Chris Behrens (cbehrens)]

Ya, was thinking along those lines. I'll take a look at that example. Thnx!

[Chris Behrens (cbehrens)]

Ok, got a test in that fails before fixing this bug and succeeds after fixing. Also added a small improvement to LoopingCall to not sleep unnecessarily.

[Josh Kearney (jk0)]

Looks great and tests perform as expected! Very nice fix.

review: Approve

[Vish Ishaya (vishvananda)]

lgtm

review: Approve

[OpenStack Infra (hudson-openstack)]

The attempt to merge lp:~cbehrens/nova/lp766404 into lp:nova failed. Below is the output from the failed tests.

AccountsTest
    test_account_create OK
    test_account_delete OK
    test_account_update OK
    test_get_account OK
AdminAPITest
    test_admin_disabled OK
    test_admin_enabled OK
APITest
    test_exceptions_are_converted_to_faults OK
Test
    test_authorize_token OK
    test_authorize_user OK
    test_bad_token OK
    test_bad_user_bad_key OK
    test_bad_user_good_key OK
    test_no_user OK
    test_token_expiry OK
TestFunctional
    test_token_doesnotexist OK
    test_token_expiry OK
TestLimiter
    test_authorize_token OK
LimiterTest
    test_limiter_custom_max_limit OK
    test_limiter_limit_and_offset OK
    test_limiter_limit_medium OK
    test_limiter_limit_over_max OK
    test_limiter_limit_zero OK
    test_limiter_negative_limit OK
    test_limiter_negative_offset OK
    test_limiter_nothing OK
    test_limiter_offset_bad OK
    test_limiter_offset_blank OK
    test_limiter_offset_medium OK
    test_limiter_offset_over_max OK
    test_limiter_offset_zero OK
ActionExtensionTest
    test_extended_action OK
    test_invalid_action OK
    test_invalid_action_body OK
ExtensionControllerTest
    test_get_by_alias OK
    test_index OK
ExtensionManagerTest
    test_get_resources OK
ResourceExtensionTest
    test_get_resources OK
    test_get_resources_with_controller OK
    test_no_extension_present OK
ResponseExtensionTest
    test_get_resources_with_mgr OK
    test_get_resources_with_stub_mgr OK
TestFaults
    test_400_fault_json OK
    test_400_fault_xml OK
    test...

[Chris Behrens (cbehrens)]

Fixed the pep8 issues with the tests.

=== modified file 'nova/tests/test_xenapi.py'
--- nova/tests/test_xenapi.py	2011-05-09 19:57:56 +0000
+++ nova/tests/test_xenapi.py	2011-05-13 16:47:26 +0000
@@ -16,6 +16,7 @@
 
 """Test suite for XenAPI."""
 
+import eventlet
 import functools
 import json
 import os
@@ -198,6 +199,28 @@
         self.context = context.RequestContext('fake', 'fake', False)
         self.conn = xenapi_conn.get_connection(False)
 
+    def test_parallel_builds(self):
+        stubs.stubout_loopingcall_delay(self.stubs)
+
+        def _do_build(id, proj, user, *args):
+            values = {
+                'id': id,
+                'project_id': proj,
+                'user_id': user,
+                'image_id': 1,
+                'kernel_id': 2,
+                'ramdisk_id': 3,
+                'instance_type_id': '3',  # m1.large
+                'mac_address': 'aa:bb:cc:dd:ee:ff',
+                'os_type': 'linux'}
+            instance = db.instance_create(self.context, values)
+            self.conn.spawn(instance)
+
+        gt1 = eventlet.spawn(_do_build, 1, self.project.id, self.user.id)
+        gt2 = eventlet.spawn(_do_build, 2, self.project.id, self.user.id)
+        gt1.wait()
+        gt2.wait()
+
     def test_list_instances_0(self):
         instances = self.conn.list_instances()
         self.assertEquals(instances, [])

=== modified file 'nova/tests/xenapi/stubs.py'
--- nova/tests/xenapi/stubs.py	2011-03-25 23:43:21 +0000
+++ nova/tests/xenapi/stubs.py	2011-05-13 16:47:26 +0000
@@ -16,6 +16,7 @@
 
 """Stubouts, mocks and fixtures for the test suite"""
 
+import eventlet
 from nova.virt import xenapi_conn
 from nova.virt.xenapi import fake
 from nova.virt.xenapi import volume_utils
@@ -28,29 +29,6 @@
     @classmethod
     def fake_fetch_image(cls, session, instance_id, image, user, project,
                          type):
-        # Stubout wait_for_task
-        def fake_wait_for_task(self, task, id):
-            class FakeEvent:
-
-                def send(self, value):
-                    self.rv = value
-
-                def wait(self):
-                    return self.rv
-
-            done = FakeEvent()
-            self._poll_task(id, task, done)
-            rv = done.wait()
-            return rv
-
-        def fake_loop(self):
-            pass
-
-        stubs.Set(xenapi_conn.XenAPISession, 'wait_for_task',
-                  fake_wait_for_task)
-
-        stubs.Set(xenapi_conn.XenAPISession, '_stop_loop', fake_loop)
-
         from nova.virt.xenapi.fake import create_vdi
         name_label = "instance-%s" % instance_id
         #TODO: create fake SR record
@@ -62,11 +40,6 @@
         return vdi_uuid
 
     stubs.Set(vm_utils.VMHelper, 'fetch_image', fake_fetch_image)
-
-    def fake_parse_xmlrpc_value(val):
-        return val
-
-    stubs.Set(xenapi_conn, '_parse_xmlrpc_value', fake_parse_xmlrpc_value)
 
     def fake_wait_for_vhd_coalesce(session, instance_id, sr_ref, vdi_ref,
                               original_parent_uuid):
@@ -144,6 +117,16 @@
     stubs.Set(utils.LoopingCall, 'start', fake_start)
 
 
+def stubout_loopingcall_delay(stubs):
+    def fake_start(self, interval, now=True):
+        self._running = True
+        eventlet.sleep(1)
+        self.f(*self.args, **self.kw)
+        # This would fail before parallel xenapi calls were fixed
+        assert self._running == False
+    stubs.Set(utils.LoopingCall, 'start', fake_start)
+
+
 class FakeSessionForVMTests(fake.SessionBase):
     """ Stubs out a XenAPISession for VM tests """
     def __init__(self, uri):

=== modified file 'nova/utils.py'
--- nova/utils.py	2011-05-11 22:56:57 +0000
+++ nova/utils.py	2011-05-13 16:47:26 +0000
@@ -462,6 +462,8 @@
             try:
                 while self._running:
                     self.f(*self.args, **self.kw)
+                    if not self._running:
+                        break
                     greenthread.sleep(interval)
             except LoopingCallDone, e:
                 self.stop()

=== modified file 'nova/virt/xenapi_conn.py'
--- nova/virt/xenapi_conn.py	2011-05-09 19:57:56 +0000
+++ nova/virt/xenapi_conn.py	2011-05-13 16:47:26 +0000
@@ -347,7 +347,6 @@
                             "(is the Dom0 disk full?)"))
         with timeout.Timeout(FLAGS.xenapi_login_timeout, exception):
             self._session.login_with_password(user, pw)
-        self.loop = None
 
     def get_imported_xenapi(self):
         """Stubout point. This can be replaced with a mock xenapi module."""
@@ -384,57 +383,52 @@
 
     def wait_for_task(self, task, id=None):
         """Return the result of the given task. The task is polled
-        until it completes. Not re-entrant."""
+        until it completes."""
         done = event.Event()
-        self.loop = utils.LoopingCall(self._poll_task, id, task, done)
-        self.loop.start(FLAGS.xenapi_task_poll_interval, now=True)
-        rv = done.wait()
-        self.loop.stop()
-        return rv
-
-    def _stop_loop(self):
-        """Stop polling for task to finish."""
-        #NOTE(sandy-walsh) Had to break this call out to support unit tests.
-        if self.loop:
-            self.loop.stop()
+        loop = utils.LoopingCall(f=None)
+
+        def _poll_task():
+            """Poll the given XenAPI task, and return the result if the
+            action was completed successfully or not.
+            """
+            try:
+                name = self._session.xenapi.task.get_name_label(task)
+                status = self._session.xenapi.task.get_status(task)
+                if id:
+                    action = dict(
+                        instance_id=int(id),
+                        action=name[0:255],  # Ensure action is never > 255
+                        error=None)
+                if status == "pending":
+                    return
+                elif status == "success":
+                    result = self._session.xenapi.task.get_result(task)
+                    LOG.info(_("Task [%(name)s] %(task)s status:"
+                            " success    %(result)s") % locals())
+                    done.send(_parse_xmlrpc_value(result))
+                else:
+                    error_info = self._session.xenapi.task.get_error_info(task)
+                    action["error"] = str(error_info)
+                    LOG.warn(_("Task [%(name)s] %(task)s status:"
+                            " %(status)s    %(error_info)s") % locals())
+                    done.send_exception(self.XenAPI.Failure(error_info))
+
+                if id:
+                    db.instance_action_create(context.get_admin_context(),
+                            action)
+            except self.XenAPI.Failure, exc:
+                LOG.warn(exc)
+                done.send_exception(*sys.exc_info())
+            loop.stop()
+
+        loop.f = _poll_task
+        loop.start(FLAGS.xenapi_task_poll_interval, now=True)
+        return done.wait()
 
     def _create_session(self, url):
         """Stubout point. This can be replaced with a mock session."""
         return self.XenAPI.Session(url)
 
-    def _poll_task(self, id, task, done):
-        """Poll the given XenAPI task, and fire the given action if we
-        get a result.
-        """
-        try:
-            name = self._session.xenapi.task.get_name_label(task)
-            status = self._session.xenapi.task.get_status(task)
-            if id:
-                action = dict(
-                    instance_id=int(id),
-                    action=name[0:255],  # Ensure action is never > 255
-                    error=None)
-            if status == "pending":
-                return
-            elif status == "success":
-                result = self._session.xenapi.task.get_result(task)
-                LOG.info(_("Task [%(name)s] %(task)s status:"
-                        " success    %(result)s") % locals())
-                done.send(_parse_xmlrpc_value(result))
-            else:
-                error_info = self._session.xenapi.task.get_error_info(task)
-                action["error"] = str(error_info)
-                LOG.warn(_("Task [%(name)s] %(task)s status:"
-                        " %(status)s    %(error_info)s") % locals())
-                done.send_exception(self.XenAPI.Failure(error_info))
-
-            if id:
-                db.instance_action_create(context.get_admin_context(), action)
-        except self.XenAPI.Failure, exc:
-            LOG.warn(exc)
-            done.send_exception(*sys.exc_info())
-        self._stop_loop()
-
     def _unwrap_plugin_exceptions(self, func, *args, **kwargs):
         """Parse exception details"""
         try: