Merge lp:~vila/uci-engine/nova-failure into lp:uci-engine

[Vincent Ladeuil]

Turn nova exceptions into TestBedException to identify infra failures

Most of the testbed related issues were already raising TestBedException except for the nova issues.

With this patch *all* testbed issues (nova or not) can be identified at higher levels with that single exception.

[Para Siva (psivaa)]

One question inline, about removing the 'skipping' of the integration tests altogether. Apart from that, it looks good.

review: Approve

[Francis Ginther (fginther)]

Looks good. It looks odd to re-enable the Swift feature with this MP, but I have no good reason to create a separate MP just for that. Ultimately if there is trouble landing this due to the Swift tests having problems in tarmac, then we can rework in that event.

review: Approve

[Vincent Ladeuil (vila)]

Oops, got confused when submitting this, thanks for the heads-up !

> Ultimately if there is trouble landing this due to the Swift tests having problems in tarmac, then we can rework in that event.

/me nods

Yup, good summary, I'll try to land and will revisit a more focused MP if I encounter issues.

[Para Siva (psivaa)]

Just clarified my comment. +1

[Ubuntu CI Bot (uci-bot)]

The attempt to merge lp:~vila/uci-engine/nova-failure into lp:uci-engine failed. Below is the output from the failed tests.

Checking juju status
Private PPAs: disabled
Preparing local branch upload...
Uploading local branch, fingerprint 87df68656e30adc88219166e790e1cc78ebbc61b
Installing keys from bzr+ssh://bazaar.launchpad.net/~ci-engineering-private/+junk/ci-airline-dev-keys/
Running juju-deployer -v -c /tmp/tmpI70Mjg/deployer/relations.yaml -c /tmp/tmpI70Mjg/deployer/services.yaml ci-airline
Tests running...
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testConnectFailed ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testProcessQueue ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testRunForever ... OK (0.102 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testSent ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestProgressTrigger.testProgress ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testCancel ... OK (0.108 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoQueue ... OK (0.005 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoTicket ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageCalledProcessError ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageFail ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageKilled ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageSimple ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageUnexpected ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testSaveLastRun ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCBRuns ... OK (0.021 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCanCancel ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_invalid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_valid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_abspath ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_basename ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_relpath ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_empty_single_config ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_multiple_config ... OK (0.007 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_single_config ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_features.TestJujuControlBucket.test_no_control_bucket ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_features.TestJujuGetEnvsFromYaml.test_broken_environments_file ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_features.TestJujuGetEnvsFromYaml.test_inconsistent_environments_file ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_features.TestJujuNova.test_get_nova_id_malformed ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_features.TestJujuNov...

[Vincent Ladeuil (vila)]

FAIL: tests.test_test_runner.TestTestRunner.test_process_ticket

TestBedException: None of [m1.smaller,m1.small,standard.small] can be found

FAIL: tests.test_test_runner.TestTestRunner.test_run_tests

TestBedException: None of [m1.smaller,m1.small,standard.small] can be found

Huh ?

======================================================================
FAIL: tests.test_webui.WebUITest.test_edit_ticket
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tarmac/branch.ODIjlD/bin/../tests/test_webui.py", line 291, in test_edit_ticket
    wait_for(get_element_by_css, 'a.yui3-pagview-link-page-active')
  File "/dev/shm/venv-a55Hp9/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 145, in inner
    return func(*args, **kwargs)
  File "/dev/shm/venv-a55Hp9/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 852, in wait_for
    return _wait_for(condition, False, _TIMEOUT, _POLL, *args, **kwargs)
  File "/dev/shm/venv-a55Hp9/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 816, in _wait_for
    _raise(error)
  File "/dev/shm/venv-a55Hp9/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 118, in _raise
    raise AssertionError(msg)
AssertionError: Timed out waiting for: get_element_by_css
Error during wait: Could not identify element: 0 elements found
======================================================================
FAIL: tests.test_webui.WebUITest.test_edit_ticket_sources_and_mps
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tarmac/branch.ODIjlD/bin/../tests/test_webui.py", line 354, in test_edit_ticket_sources_and_mps
    wait_for(get_element_by_css, 'a.yui3-pagview-link-page-active')
  File "/dev/shm/venv-a55Hp9/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 145, in inner
    return func(*args, **kwargs)
  File "/dev/shm/venv-a55Hp9/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 852, in wait_for
    return _wait_for(condition, False, _TIMEOUT, _POLL, *args, **kwargs)
  File "/dev/shm/venv-a55Hp9/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 816, in _wait_for
    _raise(error)
  File "/dev/shm/venv-a55Hp9/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 118, in _raise
    raise AssertionError(msg)
AssertionError: Timed out waiting for: get_element_by_css
Error during wait: Could not identify element: 0 elements found
======================================================================
FAIL: tests.test_webui.WebUITest.test_edit_ticket_with_invalid_mps
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tarmac/branch.ODIjlD/bin/../tests/test_webui.py", line 490, in test_edit_ticket_with_invalid_mps
    wait_for(get_element_by_css, 'a.yui3-pagview-link-page-active')
  File "/dev/shm/venv-a55Hp9/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 145, in inner
    return func(*args, **kwargs)
  File "/dev...

Unmerged revisions

914. By Vincent Ladeuil on 2015-01-07

Raise TestBedException for nova related failures too so all testbed related issues can be detected at higher levels.

913. By Vincent Ladeuil on 2015-01-07

Fix missed fallout from an old refactoring moving the config part out of britney/post_request.py.

912. By Vincent Ladeuil on 2015-01-07

Activate Swift and Nova features again so integration tests are not skipped anymore.

=== modified file 'ci-utils/ci_utils/testing/features.py'
--- ci-utils/ci_utils/testing/features.py	2014-11-26 08:09:56 +0000
+++ ci-utils/ci_utils/testing/features.py	2015-01-07 16:35:28 +0000
@@ -1,5 +1,5 @@
 # Ubuntu CI Engine
-# Copyright 2014 Canonical Ltd.
+# Copyright 2014, 2015 Canonical Ltd.
 
 # This program is free software: you can redistribute it and/or modify it
 # under the terms of the GNU Affero General Public License version 3, as
@@ -36,10 +36,6 @@
 class Swift(features.Feature):
 
     def _probe(self):
-        # FIXME: swift needs to handle more transient failures, tests are
-        # currently failing when swift becomes too slow. Disable all tests that
-        # require a "perfect" swift -- vila 2014-11-26
-        return False
         conn = self.get_connection()
         if conn is None:
             return False
@@ -71,10 +67,6 @@
 class NovaCompute(features.Feature):
 
     def _probe(self):
-        # FIXME: nova needs to handle more transient failures, tests are
-        # currently failing for various known reasons. Disable all tests that
-        # require a "perfect" nova. -- vila 2014-11-26
-        return False
         client = self.get_client()
         if client is None:
             return False

=== modified file 'test_runner/tstrun/testbed.py'
--- test_runner/tstrun/testbed.py	2014-10-17 12:28:22 +0000
+++ test_runner/tstrun/testbed.py	2015-01-07 16:35:28 +0000
@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 # Ubuntu CI Engine
-# Copyright 2014 Canonical Ltd.
+# Copyright 2014, 2015 Canonical Ltd.
 
 # This program is free software: you can redistribute it and/or modify it
 # under the terms of the GNU Affero General Public License version 3, as
@@ -178,8 +178,19 @@
                 msg = ('Rate limit reached for {},'
                        ' will sleep for {} seconds')
                 self.logger.exception(msg.format(func.__name__, nap_time))
-        time.sleep(nap_time)
-        return func(*args, **kwargs)  # Retry once
+        except:
+            # All other exceptions are raised
+            self.logger.exception('{} failed'.format(func.__name__))
+            raise TestBedException('{} failed'.format(func.__name__))
+        try:
+            time.sleep(nap_time)
+            return func(*args, **kwargs)  # Retry once
+        except:
+            # FIXME: This duplicates the two lines above but this method will
+            # be refactored to retry more than once in the future. The
+            # duplication can be fixed then -- vila 2015-01-07
+            self.logger.exception('{} failed'.format(func.__name__))
+            raise TestBedException('{} failed'.format(func.__name__))
 
     def flavors_list(self):
         return self.retry(self.nova.flavors.list)

=== modified file 'test_runner/tstrun/tests/test_testbed.py'
--- test_runner/tstrun/tests/test_testbed.py	2014-11-04 12:07:12 +0000
+++ test_runner/tstrun/tests/test_testbed.py	2015-01-07 16:35:28 +0000
@@ -1,5 +1,5 @@
 # Ubuntu CI Engine
-# Copyright 2014 Canonical Ltd.
+# Copyright 2014, 2015 Canonical Ltd.
 
 # This program is free software: you can redistribute it and/or modify it
 # under the terms of the GNU Affero General Public License version 3, as
@@ -128,9 +128,9 @@
         # This mimics what will happen when we encounter unknown transient
         # failures we want to catch: an exception will bubble up and we'll have
         # to add it to NovaClient.retry().
-        with self.assertRaises(AssertionError) as cm:
+        with self.assertRaises(testbed.TestBedException) as cm:
             tb.find_flavor()
-        self.assertEqual('Boom!', unicode(cm.exception))
+        self.assertEqual('fail failed', '{}'.format(cm.exception))
 
 
 @features.requires(features.nova_creds)

=== modified file 'tests/test_britney.py'
--- tests/test_britney.py	2014-11-03 12:59:48 +0000
+++ tests/test_britney.py	2015-01-07 16:35:28 +0000
@@ -1,6 +1,6 @@
 #!/usr/bin/env python
 # Ubuntu Continuous Integration Engine
-# Copyright 2014 Canonical Ltd.
+# Copyright 2014, 2015 Canonical Ltd.
 
 # This program is free software: you can redistribute it and/or modify it
 # under the terms of the GNU Affero General Public License version 3, as
@@ -37,6 +37,7 @@
 
 
 from britney import (
+    config,
     post_request,
     process_results,
 )
@@ -186,15 +187,15 @@
         password = 's3cr3t'
         rabbit_ip = get_rabbit_ip(status)
         deployers.expose_rabbit(self, rabbit_ip, user, password)
-        os.makedirs(post_request.user_config_dir())
-        self.config = post_request.BritneyStack()
+        os.makedirs(config.user_config_dir())
+        self.config = config.BritneyStack()
         self.config.store._load_from_string('''\
 uci.rabbit.host = {rabbit_ip}
 uci.rabbit.user = {user}
 uci.rabbit.password = {password}
         '''.format(**locals()))
         fixtures.patch(self, amqp_utils, 'get_config',
-                       post_request.amqp_config(self.config))
+                       config.amqp_config(self.config))
         self.out = StringIO()
         self.err = StringIO()