Merge lp:~vila/uci-engine/1295324-testbed-unreachable into lp:uci-engine

[Vincent Ladeuil]

Handle testbed disappearing when getting files out of it.

Fix the rare case where the testbed can't be reached even when it's up.

To test, I simulated the case by killing the nova instance after it has been
setup and the tests run there.

As a drive-by, I also fixed the case where an HTTP 413 Rate limiting
exception was breaking the testbed monitoring.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:419
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/466/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/466/rebuild

review: Approve (continuous-integration)

[Andy Doan (doanac)]

40 + time.sleep(30)
41 + log.exception('Rate limit while acquiring nova console'
42 + ' for {}'.format(self.instance.id))

nothing huge, but putting the log message before the time.sleep might make tailing the logfile more friendly.

[Vincent Ladeuil (vila)]

/me facepalms

Thanks for the catch, fixed ;)

[lp:~vila/uci-engine/1295324-testbed-unreachable]

420. By Vincent Ladeuil on 2014-04-01

Sleep after logging the exception makes tailing the log more friendly (Andy).

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:420
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/468/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/468/rebuild

review: Approve (continuous-integration)

[Andy Doan (doanac)]

review: Approve

=== modified file 'test_runner/tstrun/testbed.py'
--- test_runner/tstrun/testbed.py	2014-03-17 20:49:20 +0000
+++ test_runner/tstrun/testbed.py	2014-04-01 17:30:56 +0000
@@ -31,7 +31,7 @@
 log = logging.getLogger(__name__)
 
 
-class NovaClientException(Exception):
+class TestBedException(Exception):
     pass
 
 
@@ -179,7 +179,7 @@
                 pass
         # MISSINGTEST: some cloud doesn't provide one of our expected flavors
         # -- vila 2014-01-06
-        raise NovaClientException(
+        raise TestBedException(
             'None of {} can be found'.format(self.flavors))
 
     def setup(self):
@@ -235,9 +235,9 @@
             time.sleep(5)
         # MISSINGTEST: What if the instance doesn't come up after max_tries ?
         if status == 'BUILD':
-            raise NovaClientException('Instance never came up')
+            raise TestBedException('Instance never came up')
         else:
-            raise NovaClientException('Instance never provided an IP')
+            raise TestBedException('Instance never provided an IP')
 
     def get_cloud_init_console(self, length=None):
         return self.instance.get_console_output(length)
@@ -252,6 +252,15 @@
             # the console for the specific message we specified in user-data).
             try:
                 console = self.get_cloud_init_console(10)
+            except exceptions.OverLimit:
+                # This happens rarely but breaks badly if not caught. elmo
+                # recommended a 30 seconds nap in that case.
+                nap_time = 30
+                msg = ('Rate limit while acquiring nova console for {},'
+                       ' will sleep for {} seconds')
+                log.exception(msg.format(self.instance.id, nap_time))
+                time.sleep(nap_time)
+                continue
             except requests.ConnectionError:
                 # The reported status is the one known before the last attempt.
                 log.warn('Received connection error for {},'
@@ -264,7 +273,7 @@
                     'cloud-init completed for {}'.format(self.instance.id))
                 return
             time.sleep(5)
-        raise NovaClientException('Instance never completed cloud-init')
+        raise TestBedException('Instance never completed cloud-init')
 
     def teardown(self):
         self.nova.servers.delete(self.instance.id)
@@ -294,7 +303,12 @@
         return proc, out, err
 
     def get_remote_content(self, path):
-        _, content, _ = self.ssh('cat', path, out=subprocess.PIPE)
+        proc, content, err = self.ssh('cat', path, out=subprocess.PIPE,
+                                      err=subprocess.PIPE)
+        if proc.returncode:
+            # We didn't get a proper content, report it instead
+            content = ("{} couldn't be copied from testbed {}:\n"
+                       "error: {}\n".format(path, self.ip, err))
         return content
 
     def run_test(self, package):