Merge lp:~cjohnston/uci-engine/wait into lp:uci-engine

[Chris Johnston]

Fix issue where get_ip_and_port returns KeyError

For a while now on some of the amulet tests where we need to get the IP and port we have been seeing KeyError because open-ports was not found. There are two things going on here that contributed to the problem.

1) The way that juju works is that just because it says it's done deploying, that doesn't mean that it actually is done deploying. This was causing the tests to start running before things were actually ready.

2) The call to check the status was made during setUp, so, no matter how many times we may check, the status didn't change as it wasn't called again.

I moved the status check into the try so that it gets refreshed every time we need to check it. I also added a loop to try every 5 seconds for 40 seconds to get the IP and open-ports. This will allow juju time to actually finish what it's supposed to do.

[Andy Doan (doanac)]

30 + except KeyError:
31 + tries += 1
32 + import time
33 + time.sleep(5)

can we move the import to the top of the module. Also - might be worth a log.info('open-ports not found waiting...')

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:474
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/612/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/612/rebuild

review: Approve (continuous-integration)

[lp:~cjohnston/uci-engine/wait]

475. By Chris Johnston on 2014-05-12

Changes per Andy's review

[Andy Doan (doanac)]

LGTM

review: Approve

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:475
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/616/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/616/rebuild

review: Approve (continuous-integration)

=== modified file 'tests/deployers.py'
--- tests/deployers.py	2014-05-07 10:01:50 +0000
+++ tests/deployers.py	2014-05-12 14:18:01 +0000
@@ -17,10 +17,14 @@
 import os
 import subprocess
 import sys
-
+import time
+import logging
 import yaml
 import amulet
 
+logging.basicConfig(level=logging.INFO, format='%(msg)s')
+log = logging.getLogger('integration-test-runner-deployers')
+
 HERE = os.path.abspath(os.path.dirname(__file__))
 # We need access to ci_utils
 sys.path.append(os.path.join(HERE, '..', 'ci-utils'))
@@ -75,15 +79,25 @@
             os.path.join(os.environ['JUJU_DEPLOYER_DIR'], self.deployer_cfg))
         self.addCleanup(self.deployer.tearDown)
         self.deployer.setUp(self.timeout)
-        self.status = amulet.waiter.status(self.deployer.deployment.juju_env)
 
     def get_ip_and_port(self, service, unit=0):
-        units = self.status['services'][service]['units']
-        ip = units['%s/%d' % (service, unit)]['public-address']
-        # FIXME: Can fail with KeyError: 'open-ports'. Need to refresh status ?
-        # -- vila 2014-04-10
-        port, _ = units['%s/%d' % (service, unit)]['open-ports'][0].split('/')
-        return ip, int(port)
+        tries = 0
+        # this sometimes takes up to 30 seconds for the port to become
+        # available Lets try for 40 seconds just incase.
+        while tries < 8:
+            try:
+                status = amulet.waiter.status(
+                    self.deployer.deployment.juju_env)
+                units = status['services'][service]['units']
+                ip = units['%s/%d' % (service, unit)]['public-address']
+                port, _ = units[
+                    '%s/%d' % (service, unit)]['open-ports'][0].split('/')
+                return ip, int(port)
+                break
+            except KeyError:
+                tries += 1
+                log.warn("open-ports not found waiting...")
+                time.sleep(5)
 
     def assert_job_running(self, upstart_job, service, unit=0):
         '''ensure the given upstart job is running on a unit'''