Merge lp:~sinzui/juju-ci-tools/wait-for-upgrade into lp:juju-ci-tools

[Curtis Hovey]

Wait for list-models to work to know when the controller has upgraded.

Older jujus went dark for many minutes during an upgrade. We polled status for a time and when it worked, we started checking for the upgraded agent version. New juju does not go completely dark. status will works, and it will report the agents are upgraded, but the controller is *still* upgrading.

This branch treats list-models like status. After we see the agent are upgraded, we try list-models for a period of time. When the command works the script proceeds. Otherwise a timeout is raised.

Neither status nor show controller indicates that mongos collections are being upgraded. Our only clue is that we cannot work with any model.

[Nicholas Skaggs (nskaggs)]

This looks good, but I'd prefer we put a small comment in as to why we are doing the additional wait. A summary of what you've said above for future reference in the test. Namely, upgrades occur on agent and potentially db as well. Juju doesn't tell us when it's truly done. We are using list-models as the proxy for 'juju done'.

review: Approve

[lp:~sinzui/juju-ci-tools/wait-for-upgrade]

1784. By Curtis Hovey on 2016-12-01

Added comments.

=== modified file 'deploy_stack.py'
--- deploy_stack.py	2016-11-28 22:17:20 +0000
+++ deploy_stack.py	2016-12-01 16:40:43 +0000
@@ -401,8 +401,25 @@
         timeout = 600
 
     for client in all_clients:
+        logging.info('Upgrading {}'.format(client.env.environment))
         upgrade_juju(client)
         client.wait_for_version(client.get_matching_agent_version(), timeout)
+        logging.info('Agents upgraded in {}'.format(client.env.environment))
+        client.show_status()
+        logging.info('Waiting for model {}'.format(client.env.environment))
+        # While the agents are upgraded, the controller/model may still be
+        # upgrading. We are only certain that the upgrade as is complete
+        # when we can list models.
+        for ignore in until_timeout(600):
+            try:
+                client.list_models()
+                break
+            except subprocess.CalledProcessError:
+                pass
+        # The upgrade will trigger the charm hooks. We want the charms to
+        # return to active state to know they accepted the upgrade.
+        client.wait_for_workloads()
+        logging.info('Upgraded model {}'.format(client.env.environment))
 
 
 def _get_clients_to_upgrade(old_client, juju_path):

=== modified file 'tests/test_deploy_stack.py'
--- tests/test_deploy_stack.py	2016-11-28 22:04:46 +0000
+++ tests/test_deploy_stack.py	2016-12-01 16:40:43 +0000
@@ -1207,17 +1207,22 @@
             'juju', '--show-log', 'upgrade-juju', '-m', 'foo:controller',
             '--agent-version', '2.0-rc2'), 0)
         assert_juju_call(self, cc_mock, new_client, (
+            'juju', '--show-log', 'show-status', '-m', 'foo:controller',
+            '--format', 'yaml'), 1)
+        assert_juju_call(self, cc_mock, new_client, (
+            'juju', '--show-log', 'list-models', '-c', 'foo'), 2)
+        assert_juju_call(self, cc_mock, new_client, (
             'juju', '--show-log', 'upgrade-juju', '-m', 'foo:foo',
-            '--agent-version', '2.0-rc2'), 1)
-        self.assertEqual(cc_mock.call_count, 2)
+            '--agent-version', '2.0-rc2'), 3)
+        self.assertEqual(cc_mock.call_count, 6)
         assert_juju_call(self, co_mock, new_client, self.LIST_MODELS, 0)
         assert_juju_call(self, co_mock, new_client, self.GET_CONTROLLER_ENV, 1)
         assert_juju_call(self, co_mock, new_client, self.GET_CONTROLLER_ENV, 2)
         assert_juju_call(self, co_mock, new_client, self.CONTROLLER_STATUS, 3)
-        assert_juju_call(self, co_mock, new_client, self.GET_ENV, 4)
         assert_juju_call(self, co_mock, new_client, self.GET_ENV, 5)
-        assert_juju_call(self, co_mock, new_client, self.STATUS, 6)
-        self.assertEqual(co_mock.call_count, 7)
+        assert_juju_call(self, co_mock, new_client, self.GET_ENV, 6)
+        assert_juju_call(self, co_mock, new_client, self.STATUS, 7)
+        self.assertEqual(co_mock.call_count, 9)
 
     def test__get_clients_to_upgrade_returns_new_version_class(self):
         env = SimpleEnvironment('foo', {'type': 'foo'})