Merge lp:~vila/uci-vms/nova into lp:uci-vms

[Vincent Ladeuil]

Add nova support.

Add support for OpenStack nova (vm.class = nova).

Most of the tests and code were written for the uci-engine.

A few tweaks were needed to re-sync the API that have diverged a bit but mostly this is copying known working test and code (hence the huge diff :-/).

There are still a few rough edges but the basics (setup, shell, stop, teardown) work.

[lp:~vila/uci-vms/nova]

151. By Vincent Ladeuil on 2015-09-01

Merge trunk resolving conflicts

=== modified file 'NEWS.rst'
--- NEWS.rst	2015-08-21 15:42:06 +0000
+++ NEWS.rst	2015-09-01 11:19:08 +0000
@@ -7,6 +7,8 @@
 dev
 ===
 
+* Add support for OpenStack nova (vm.class = nova).
+
 * Fix the script name in the help output.
 
 * Restore python2 support.

=== modified file 'TODO.rst'
--- TODO.rst	2015-07-22 15:22:18 +0000
+++ TODO.rst	2015-09-01 11:19:08 +0000
@@ -1,3 +1,16 @@
+* Instead of juggling in tests to acquire config options from the user, we'd
+  better create a specific config stack for tests. Whether it's an
+  independent one or one wrapping (patching) the existing one with an
+  additional file.
+
+* Add a vm.os_credentials_file option, a file to be sourced to set the OS_
+  env vars and populate the vm.os.* options without polluting the environ.
+
+* According to https://git.openstack.org/cgit/openstack/python-novaclient ,
+  openstack is moving to python3. According to zul/jamespage ubuntu will see
+  progress starting with wily. The short term for uci-vms seems to be to
+  support python2 again...
+
 * The error when vm.release is not set is obscure.
 
 * Ensure that vms can be created/used without any vm.ssh_keys at all. With

=== modified file 'ucivms/commands.py'
--- ucivms/commands.py	2015-08-21 15:42:06 +0000
+++ ucivms/commands.py	2015-09-01 11:19:08 +0000
@@ -38,6 +38,10 @@
                                   'Linux container virtual machine')
 config.vm_class_registry.register('ephemeral-lxc', vms.EphemeralLxc,
                                   'Linux container ephemeral virtual machine')
+if sys.version_info < (3,):
+    from ucivms.vms import nova
+    config.vm_class_registry.register(
+        'nova', nova.NovaServer, 'Openstack Nova instance')
 
 
 class ArgParser(argparse.ArgumentParser):
@@ -361,10 +365,10 @@
         # FIXME: states need to be defined uniquely across the various vms
         # implementations -- vila 2014-01-17
         if state in('shut off', 'STOPPED'):
-            self.vm.undefine()
+            self.vm.teardown()
         elif state in ('running', 'RUNNING'):
             raise errors.VmRunning(self.vm_name)
-        self.vm.install()
+        self.vm.setup()
         return 0
 
 
@@ -453,7 +457,7 @@
         # implementations -- vila 2014-01-17
         if state not in ('running', 'RUNNING'):
             raise errors.VmNotRunning(self.vm_name)
-        self.vm.poweroff()
+        self.vm.stop()
         return 0
 
 
@@ -473,7 +477,7 @@
         # implementations -- vila 2014-01-17
         if state in ('running', 'RUNNING'):
             raise errors.VmRunning(self.vm_name)
-        self.vm.undefine()
+        self.vm.teardown()
         return 0
 
 

=== modified file 'ucivms/config.py'
--- ucivms/config.py	2015-07-21 17:10:34 +0000
+++ ucivms/config.py	2015-09-01 11:19:08 +0000
@@ -112,6 +112,22 @@
         super(VmStack, self).__init__(
             section_getters, user_store, mutable_section_id=name)
 
+    def get_nova_creds(self):
+        """Get nova credentials from a config.
+
+        This defines the set of options needed to authenticate against nova in
+        a single place.
+
+        :raises: uciconfig.errors.OptionMandatoryValueError if one of the
+            options is not set.
+        """
+        creds = {}
+        for k in ('username', 'password', 'tenant_name',
+                  'auth_url', 'region_name'):
+            opt_name = 'vm.os.{}'.format(k)
+            creds[opt_name] = self.get(opt_name)
+        return creds
+
 
 def path_from_unicode(path_string):
     return os.path.expanduser(path_string)
@@ -427,3 +443,83 @@
 - up_to: seconds after which to give up
 - retries: how many attempts after the first try
 '''))
+
+# nova options
+register(options.Option('vm.os.username', default_from_env=['OS_USERNAME'],
+                        default=options.MANDATORY,
+                        help_string='''The Open Stack user name.
+
+This is generally set via OS_USERNAME, sourced from a novarc file
+(~/.novarc, ~/.canonistack/novarc).
+'''))
+register(options.Option('vm.os.password', default_from_env=['OS_PASSWORD'],
+                        default=options.MANDATORY,
+                        help_string='''The Open Stack password.
+
+This is generally set via OS_PASSWORD, sourced from a novarc file
+(~/.novarc, ~/.canonistack/novarc).
+'''))
+register(options.Option('vm.os.region_name',
+                        default_from_env=['OS_REGION_NAME'],
+                        default=options.MANDATORY,
+                        help_string='''The Open Stack region name.
+
+This is generally set via OS_REGION_NAME, sourced from a novarc file
+(~/.novarc, ~/.canonistack/novarc).
+'''))
+register(options.Option('vm.os.tenant_name',
+                        default_from_env=['OS_TENANT_NAME'],
+                        default=options.MANDATORY,
+                        help_string='''The Open Stack tenant name.
+
+This is generally set via OS_TENANT_NAME, sourced from a novarc file
+(~/.novarc, ~/.canonistack/novarc).
+'''))
+register(options.Option('vm.os.auth_url', default_from_env=['OS_AUTH_URL'],
+                        default=options.MANDATORY,
+                        help_string='''The Open Stack keystone url.
+
+This is generally set via OS_AUTH_URL, sourced from a novarc file
+(~/.novarc, ~/.canonistack/novarc).
+'''))
+register(options.ListOption('vm.os.flavors', default=None,
+                            help_string='''\
+A list of flavors for all supported clouds.
+
+The first known one is used.
+'''))
+register(options.Option('vm.image', default=options.MANDATORY,
+                        help_string='''The glance image to boot from.'''))
+register(options.Option('vm.net_id', default=None,
+                        help_string='''The network id for the vm.'''))
+register(PathOption('vm.ssh_key_path', default='~/.ssh/id_rsa',
+                    help_string='''The ssh key for the vm.'''))
+register(options.ListOption('vm.apt_get.update.timeouts',
+                            default='15.0, 90.0, 240.0',
+                            help_string='''apt-get update timeouts in seconds.
+
+When apt-get update fails on hash sum mismatches, retry after the specified
+timeouts. More values mean more retries.
+'''))
+# FIXME: According to the help string this can (and should) be fixed
+# -- vila 2015-07-20
+register(options.ListOption('vm.ppas',
+                            default=None,
+                            help_string='''PPAs to be added to the testbed.
+
+This works around cloud-init not activating the deb-src line and not providing
+a way to do so. This is intended to be fixed in uci-vms so vm.apt_sources can
+be used again.
+'''))
+register(options.Option('vm.nova.boot_timeout', default='300',
+                        from_unicode=options.float_from_store,
+                        help_string='''\
+Max time to boot a nova instance (in seconds).'''))
+register(options.Option('vm.nova.set_ip_timeout', default='300',
+                        from_unicode=options.float_from_store,
+                        help_string='''\
+Max time for a nova instance to get an IP (in seconds).'''))
+register(options.Option('vm.nova.cloud_init_timeout', default='1200',
+                        from_unicode=options.float_from_store,
+                        help_string='''\
+Max time for cloud-init to fisnish (in seconds).'''))

=== modified file 'ucivms/tests/__init__.py'
--- ucivms/tests/__init__.py	2014-01-15 13:28:13 +0000
+++ ucivms/tests/__init__.py	2015-09-01 11:19:08 +0000
@@ -1,6 +1,6 @@
 # This file is part of Ubuntu Continuous Integration virtual machine tools.
 #
-# Copyright 2014 Canonical Ltd.
+# Copyright 2014, 2015 Canonical Ltd.
 #
 # This program is free software: you can redistribute it and/or modify it under
 # the terms of the GNU General Public License version 3, as published by the
@@ -13,3 +13,153 @@
 #
 # You should have received a copy of the GNU General Public License along with
 # this program.  If not, see <http://www.gnu.org/licenses/>.
+from __future__ import unicode_literals
+import functools
+import io
+import logging
+import unittest
+
+
+from ucitests import (
+    assertions,
+    results,
+)
+
+
+class TestLogger(logging.Logger):
+    """A logger dedicated to a given test.
+
+    Log messages are captured in string buffer.
+    """
+
+    def __init__(self, test, level=logging.DEBUG,
+                 fmt='%(asctime)-15s %(message)s'):
+        super(TestLogger, self).__init__(test.id(), level)
+        self.stream = io.StringIO()
+        handler = logging.StreamHandler(self.stream)
+        handler.setFormatter(logging.Formatter(fmt))
+        self.addHandler(handler)
+
+    def getvalue(self):
+        return self.stream.getvalue()
+
+
+class log_on_failure(object):
+    """Decorates a test to display log on failure.
+
+    This adds a 'logger' attribute to the parameters of the decorated
+    test. Using this logger the test can display its content when it fails or
+    errors but stay silent otherwise.
+    """
+
+    def __init__(self, level=logging.INFO, fmt='%(message)s'):
+        self.level = level
+        self.fmt = fmt
+
+    def __call__(self, func):
+
+        @functools.wraps(func)
+        def decorator(*args):
+            test = args[0]
+            logger = TestLogger(test, level=self.level, fmt=self.fmt)
+            display_log = True
+
+            # We need to delay the log acquisition until we attempt to display
+            # it (or we get no content).
+            def delayed_display_log():
+                msg = 'Failed test log: >>>\n{}\n<<<'.format(logger.getvalue())
+                if display_log:
+                    raise Exception(msg)
+
+            test.addCleanup(delayed_display_log)
+
+            # Run the test without the decoration
+            func(*args + (logger,))
+            # If it terminates properly, disable log display
+            display_log = False
+
+        return decorator
+
+
+class TestLogOnFailure(unittest.TestCase):
+
+    def setUp(self):
+        self.result = results.TextResult(io.StringIO(), verbosity=2)
+
+        # We don't care about timing here so we always return 0 which
+        # simplifies matching the expected result
+        def zero(atime):
+            return 0.0
+
+        self.result._delta_to_float = zero
+        # Inner tests will set this from the logger they receive so outter
+        # tests can assert the content.
+        self.logger = None
+
+    def test_log_not_displayed(self):
+
+        class Test(unittest.TestCase):
+
+            @log_on_failure()
+            def test_pass(inner, logger):
+                self.logger = logger
+                logger.info('pass')
+
+        t = Test('test_pass')
+        t.run(self.result)
+        self.assertEqual('pass\n', self.logger.getvalue())
+        self.assertEqual('ucivms.tests.Test.test_pass ... OK (0.000 secs)\n',
+                         self.result.stream.getvalue())
+        self.assertEqual([], self.result.errors)
+
+    def test_log_displayed(self):
+
+        class Test(unittest.TestCase):
+
+            @log_on_failure()
+            def test_fail(inner, logger):
+                self.logger = logger
+                logger.info("I'm broken")
+                inner.fail()
+
+        t = Test('test_fail')
+        t.run(self.result)
+        self.assertEqual("I'm broken\n", self.logger.getvalue())
+        # FAILERROR: The test FAIL, the cleanup ERRORs out.
+        self.assertEqual(
+            'ucivms.tests.Test.test_fail ... FAILERROR (0.000 secs)\n',
+            self.result.stream.getvalue())
+        assertions.assertLength(self, 1, self.result.errors)
+        failing_test, traceback = self.result.errors[0]
+        self.assertIs(t, failing_test)
+        expected = traceback.endswith("Failed test log:"
+                                      " >>>\nI'm broken\n\n<<<\n")
+        self.assertTrue(expected, 'Actual traceback: {}'.format(traceback))
+
+    def test_log_debug_not_displayed(self):
+
+        class Test(unittest.TestCase):
+
+            @log_on_failure()
+            def test_debug_silent(inner, logger):
+                self.logger = logger
+                logger.debug('more info')
+                self.fail()
+
+        t = Test('test_debug_silent')
+        t.run(self.result)
+        self.assertEqual('', self.logger.getvalue())
+
+    def test_log_debug_displayed(self):
+
+        class Test(unittest.TestCase):
+
+            @log_on_failure(level=logging.DEBUG)
+            def test_debug_verbose(inner, logger):
+                self.logger = logger
+                logger.debug('more info')
+                self.fail()
+
+        t = Test('test_debug_verbose')
+        t.run(self.result)
+        self.assertEqual('more info\n', self.logger.getvalue())

=== modified file 'ucivms/tests/features.py'
--- ucivms/tests/features.py	2015-07-21 08:07:22 +0000
+++ ucivms/tests/features.py	2015-09-01 11:19:08 +0000
@@ -17,9 +17,16 @@
 import errno
 import os
 import subprocess
-
+import sys
+
+
+if sys.version_info < (3,):
+    # novaclient doesn't support python3 (yet)
+    from ucivms.vms import nova
+
+
+from uciconfig import errors
 from ucitests import features
-
 from ucivms import config
 
 
@@ -103,3 +110,52 @@
     user_conf = config.VmStack(vm_name)
     if user_conf.get('vm.name') != vm_name:
         test.skipTest('{} does not exist'.format(vm_name))
+
+
+class NovaCompute(features.Feature):
+
+    def _probe(self):
+        if sys.version_info >= (3,):
+            # novaclient doesn't support python3 (yet)
+            return False
+        client = self.get_client()
+        if client is None:
+            return False
+        try:
+            # can transiently fail with requests.exceptions.ConnectionError
+            # (converted from MaxRetryError).
+            client.authenticate()
+        except nova.exceptions.ClientException:
+            return False
+        return True
+
+    def get_client(self):
+        test_vm_conf = config.VmStack('uci-vms-tests-nova')
+        return nova.get_os_nova_client(test_vm_conf)
+
+
+# The single instance shared by all tests
+nova_compute = NovaCompute()
+
+
+class NovaCredentials(features.Feature):
+
+    def __init__(self):
+        super(NovaCredentials, self).__init__()
+
+    def _probe(self):
+        if sys.version_info >= (3,):
+            # novaclient doesn't support python3 (yet)
+            return False
+        try:
+            config.VmStack('uci-vms-tests-nova').get_nova_creds()
+        except errors.OptionMandatoryValueError:
+            return False
+        return True
+
+    def feature_name(self):
+        return 'Valid nova credentials'
+
+
+# The single instance shared by all tests
+nova_creds = NovaCredentials()

=== modified file 'ucivms/tests/fixtures.py'
--- ucivms/tests/fixtures.py	2014-01-13 21:58:22 +0000
+++ ucivms/tests/fixtures.py	2015-09-01 11:19:08 +0000
@@ -1,6 +1,6 @@
 # This file is part of Ubuntu Continuous Integration virtual machine tools.
 #
-# Copyright 2014 Canonical Ltd.
+# Copyright 2014, 2015 Canonical Ltd.
 #
 # This program is free software: you can redistribute it and/or modify it under
 # the terms of the GNU General Public License version 3, as published by the
@@ -36,3 +36,23 @@
     test.etc_dir = os.path.join(test.uniq_dir, 'etc')
     os.mkdir(test.etc_dir)
     fixtures.patch(test, config, 'system_config_dir', lambda: test.etc_dir)
+
+
+def share_nova_test_creds(test):
+    """Set default nova test creds in user config.
+
+    :note: This should be called early during setUp so the user configuration
+        is still available (i.e. the test has not yet been isolated from disk).
+
+    """
+    test_vm_conf = config.VmStack('uci-vms-tests-nova')
+    credentials = test_vm_conf.get_nova_creds()
+    isolate_from_disk(test)
+    user_conf = config.VmStack(None)
+    for k, v in credentials.items():
+        user_conf.set(k, v)
+    user_conf.set('vm.os.flavors',
+                  test_vm_conf.get('vm.os.flavors', convert=False))
+    # Avoid triggering the 'atexit' hook as the config files are long gone
+    # when atexit run.
+    test.addCleanup(user_conf.store.save_changes)

=== modified file 'ucivms/tests/test_commands.py'
--- ucivms/tests/test_commands.py	2015-08-21 15:42:06 +0000
+++ ucivms/tests/test_commands.py	2015-09-01 11:19:08 +0000
@@ -25,6 +25,7 @@
 from ucitests import (
     assertions,
     fixtures,
+    scenarii,
 )
 from ucivms import (
     commands,
@@ -36,6 +37,8 @@
     fixtures as vms_fixtures,
 )
 
+load_tests = scenarii.load_tests_with_scenarios
+
 
 class TestHelpOptions(unittest.TestCase):
 
@@ -112,19 +115,19 @@
     def __init__(self, conf, vm_states=None):
         super(FakeVM, self).__init__(conf)
         self.states = vm_states
-        self.install_called = False
+        self.setup_called = False
         self.start_called = False
         self.shell_called = False
         self.shell_command = None
         self.shell_cmd_args = None
-        self.undefine_called = False
-        self.poweroff_called = False
+        self.teardown_called = False
+        self.stop_called = False
 
     def state(self):
         return self.states.get(self.conf.get('vm.name'), None)
 
-    def install(self):
-        self.install_called = True
+    def setup(self):
+        self.setup_called = True
 
     def start(self):
         self.start_called = True
@@ -134,11 +137,11 @@
         self.shell_command = command
         self.shell_cmd_args = cmd_args
 
-    def poweroff(self):
-        self.poweroff_called = True
+    def stop(self):
+        self.stop_called = True
 
-    def undefine(self):
-        self.undefine_called = True
+    def teardown(self):
+        self.teardown_called = True
 
 
 def setup_fake_vm(test):
@@ -436,16 +439,16 @@
         self.conf.set('vm.name', 'foo')
         self.states = {'foo': 'shut off'}
         self.run_setup(['foo'])
-        self.assertTrue(self.vm.install_called)
-        self.assertTrue(self.vm.undefine_called)
+        self.assertTrue(self.vm.setup_called)
+        self.assertTrue(self.vm.teardown_called)
 
     def test_while_running(self):
         self.conf.set('vm.name', 'foo')
         self.states = {'foo': 'running'}
         with self.assertRaises(errors.VmRunning):
             self.run_setup(['foo'])
-        self.assertFalse(self.vm.install_called)
-        self.assertFalse(self.vm.undefine_called)
+        self.assertFalse(self.vm.setup_called)
+        self.assertFalse(self.vm.teardown_called)
 
 
 class TestStatus(unittest.TestCase):
@@ -478,6 +481,28 @@
         self.assertEqual('RUNNING\n', self.out.getvalue())
 
 
+class TestStatusPerVm(unittest.TestCase):
+
+    scenarios = [(k, dict(kls=k)) for k in config.vm_class_registry.keys()]
+
+    def setUp(self):
+        super(TestStatusPerVm, self).setUp()
+        # FIXME: If tests config was properly shared across all tests we could
+        # get rid of the specific reference to nova below -- vila 2015-08-26
+        vms_fixtures.share_nova_test_creds(self)
+        self.conf = config.VmStack('foo')
+        self.conf.set('vm.name', 'foo')
+        self.conf.set('vm.class', self.kls)
+        self.conf.set('vm.release', 'trusty')
+        self.conf.set('vm.cpu_model', 'amd64')
+        self.addCleanup(self.conf.store.save_changes)
+
+    def test_unknown_vm(self):
+        vm_class = self.conf.get('vm.class')
+        vm = vm_class(self.conf)
+        self.assertEqual('UNKNOWN', vm.state())
+
+
 class TestStart(unittest.TestCase):
 
     def setUp(self):
@@ -598,20 +623,20 @@
         self.states = {'foo': 'shut off'}
         with self.assertRaises(errors.VmNotRunning):
             self.run_stop()
-        self.assertFalse(self.vm.poweroff_called)
+        self.assertFalse(self.vm.stop_called)
 
     def test_while_running(self):
         self.conf.set('vm.name', 'foo')
         self.states = {'foo': 'running'}
         self.run_stop()
-        self.assertTrue(self.vm.poweroff_called)
+        self.assertTrue(self.vm.stop_called)
 
     def test_unknown(self):
         self.conf.set('vm.name', 'I-dont-exist')
         self.states = {}
         with self.assertRaises(errors.VmUnknown):
             self.run_stop()
-        self.assertFalse(self.vm.poweroff_called)
+        self.assertFalse(self.vm.stop_called)
 
 
 class TestTeardown(unittest.TestCase):
@@ -630,18 +655,18 @@
         self.conf.set('vm.name', 'foo')
         self.states = {'foo': 'shut off'}
         self.run_teardown(['foo'])
-        self.assertTrue(self.vm.undefine_called)
+        self.assertTrue(self.vm.teardown_called)
 
     def test_while_running(self):
         self.conf.set('vm.name', 'foo')
         self.states = {'foo': 'running'}
         with self.assertRaises(errors.VmRunning):
             self.run_teardown(['foo'])
-        self.assertFalse(self.vm.undefine_called)
+        self.assertFalse(self.vm.teardown_called)
 
     def test_unknown(self):
         self.conf.set('vm.name', 'I-dont-exist')
         self.states = {}
         with self.assertRaises(errors.VmUnknown):
             self.run_teardown(['I-dont-exist'])
-        self.assertFalse(self.vm.undefine_called)
+        self.assertFalse(self.vm.teardown_called)

=== modified file 'ucivms/tests/test_ssh.py'
--- ucivms/tests/test_ssh.py	2015-07-22 13:02:27 +0000
+++ ucivms/tests/test_ssh.py	2015-09-01 11:19:08 +0000
@@ -99,7 +99,7 @@
 
     def setUp(self):
         super(TestSsh, self).setUp()
-        vms_features.requires_existing_vm(self, 'uci-vms-tests-trusty')
+        vms_features.requires_existing_vm(self, 'uci-vms-tests-lxc')
         vms_fixtures.isolate_from_disk(self)
         # To isolate tests from each other, created vms needs a unique name. To
         # keep those names legal and still user-readable we use the class name
@@ -112,15 +112,15 @@
         conf.store._load_from_string('''
 vm.ssh_opts=-oUserKnownHostsFile=/dev/null,-oStrictHostKeyChecking=no
 vm.config_dir={config_dir}
-[uci-vms-tests-trusty]
+[uci-vms-tests-lxc]
 # /!\ Should match the one defined by the user
-vm.name = uci-vms-tests-trusty
+vm.name = uci-vms-tests-lxc
 vm.class = lxc
 vm.release = trusty
 [{vm_name}]
 vm.name = {vm_name}
 vm.class = ephemeral-lxc
-vm.backing = uci-vms-tests-trusty
+vm.backing = uci-vms-tests-lxc
 vm.release = trusty
 '''.format(config_dir=config_dir, vm_name=self.vm_name))
         conf.store.save()

=== modified file 'ucivms/tests/test_vms.py'
--- ucivms/tests/test_vms.py	2015-07-22 12:03:03 +0000
+++ ucivms/tests/test_vms.py	2015-09-01 11:19:08 +0000
@@ -16,15 +16,21 @@
 from __future__ import unicode_literals
 import io
 import os
+import subprocess
 import sys
 import unittest
 import yaml
 
 
-from uciconfig import errors as conf_errors
+from uciconfig import (
+    errors as conf_errors,
+    options,
+)
 from ucitests import (
+    assertions,
     features,
     fixtures,
+    scenarii,
 )
 from ucivms import (
     config,
@@ -32,11 +38,17 @@
     vms,
     ssh,
     subprocesses,
+    tests,
 )
 from ucivms.tests import (
     features as vms_features,
     fixtures as vms_fixtures,
 )
+if sys.version_info < (3,):
+    from ucivms.vms import nova
+
+
+load_tests = scenarii.load_tests_with_scenarios
 
 
 def requires_known_reference_image(test):
@@ -701,13 +713,13 @@
 
 
 @features.requires(vms_features.use_sudo_for_tests_feature)
-class TestInstallWithSeed(unittest.TestCase):
+class TestSetupWithSeed(unittest.TestCase):
 
     def setUp(self):
         (download_cache,
          reference_cloud_image_name,
          images_dir) = requires_known_reference_image(self)
-        super(TestInstallWithSeed, self).setUp()
+        super(TestSetupWithSeed, self).setUp()
         vms_fixtures.isolate_from_disk(self)
         # We need to allow other users to read this dir
         os.chmod(self.uniq_dir, 0o755)
@@ -733,9 +745,9 @@
            download_cache=download_cache,
            cloud_image_name=reference_cloud_image_name))
 
-    def test_install_with_seed(self):
-        self.addCleanup(self.vm.undefine)
-        self.vm.install()
+    def test_setup_with_seed(self):
+        self.addCleanup(self.vm.teardown)
+        self.vm.setup()
         self.assertEqual('shut off', self.vm.state())
         # As a side-effect, the console and the interface file are created
         # MISSINGTEST: This applies to Kvm only, lxc needs the same,
@@ -745,13 +757,13 @@
 
 
 @features.requires(vms_features.use_sudo_for_tests_feature)
-class TestInstallWithBacking(unittest.TestCase):
+class TestSetupWithBacking(unittest.TestCase):
 
     def setUp(self):
         (download_cache_dir,
          reference_cloud_image_name,
          images_dir) = requires_known_reference_image(self)
-        super(TestInstallWithBacking, self).setUp()
+        super(TestSetupWithBacking, self).setUp()
         vms_fixtures.isolate_from_disk(self)
         # We need to allow other users to read this dir
         os.chmod(self.uniq_dir, 0o755)
@@ -788,10 +800,10 @@
         self.addCleanup(subprocesses.run,
                         ['sudo', 'rm', '-f', temp_vm._disk_image_path])
 
-    def test_install_with_backing(self):
+    def test_setup_with_backing(self):
         vm = vms.Kvm(config.VmStack('selftest-backing'))
-        self.addCleanup(vm.undefine)
-        vm.install()
+        self.addCleanup(vm.teardown)
+        vm.setup()
         self.assertEqual('shut off', vm.state())
         # As a side-effect, the console and the interface files are created
         # MISSINGTEST: This applies to Kvm only, lxc needs the same,
@@ -959,7 +971,7 @@
 
     def setUp(self):
         super(TestEphemeralLXC, self).setUp()
-        vms_features.requires_existing_vm(self, 'uci-vms-tests-trusty')
+        vms_features.requires_existing_vm(self, 'uci-vms-tests-lxc')
         vms_fixtures.isolate_from_disk(self)
         # To isolate tests from each other, created vms needs a unique name. To
         # keep those names legal and still user-readable we use the class name
@@ -973,14 +985,14 @@
 vm.ssh_opts=-oUserKnownHostsFile=/dev/null,-oStrictHostKeyChecking=no
 vm.config_dir={config_dir}
 # /!\ Should match the one defined by the user
-[uci-vms-tests-trusty]
-vm.name = uci-vms-tests-trusty
+[uci-vms-tests-lxc]
+vm.name = uci-vms-tests-lxc
 vm.class = lxc
 vm.release = trusty
 [{vm_name}]
 vm.name = {vm_name}
 vm.class = ephemeral-lxc
-vm.backing = uci-vms-tests-trusty
+vm.backing = uci-vms-tests-lxc
 vm.release = trusty
 '''.format(config_dir=config_dir, vm_name=self.vm_name))
         conf.store.save()
@@ -1011,3 +1023,344 @@
 #
 #    def test_fail_once_then_succeed(self):
 #        pass
+
+
[e-mail](vms_features.nova_creds)
+class TestUciImageName(unittest.TestCase):
+
+    def test_valid_britney_image(self):
+        self.assertEqual(
+            'uci/britney/precise-amd64.img',
+            nova.uci_image_name('britney', 'precise', 'amd64'))
+
+    def test_valid_cloud_image(self):
+        self.assertEqual(
+            'uci/cloudimg/precise-amd64.img',
+            nova.uci_image_name('cloudimg', 'precise', 'amd64'))
+
+    def test_invalid_image(self):
+        with self.assertRaises(ValueError) as cm:
+            nova.uci_image_name('I-dont-exist', 'precise', 'amd64')
+        self.assertEqual('Invalid image domain', '{}'.format(cm.exception))
+
+
[e-mail](vms_features.nova_creds)
[e-mail](vms_features.nova_compute)
+class TestNovaClient(unittest.TestCase):
+    """Check the nova client behavior when it encounters exceptions.
+
+    This is achieved by overriding specific methods from NovaClient and
+    exercising it through the TestBed methods.
+    """
+
+    def setUp(self):
+        super(TestNovaClient, self).setUp()
+        vms_fixtures.share_nova_test_creds(self)
+        conf = config.VmStack('testing-nova-client')
+        # Default to precise
+        conf.set('vm.release', 'precise')
+        # Avoid triggering the 'atexit' hook as the config files are long gone
+        # when atexit run.
+        self.addCleanup(conf.store.save_changes)
+        self.conf = conf
+        os.makedirs(self.conf.get('vm.vms_dir'))
+
+    def get_image_id(self, series, arch):
+        return nova.uci_image_name('cloudimg', series, arch)
+
+    @tests.log_on_failure()
+    def test_retry_is_called(self, logger):
+        self.retry_calls = []
+
+        class RetryingNovaClient(nova.NovaClient):
+
+            def __init__(inner, conf, **kwargs):
+                # We don't want to wait, it's enough to retry
+                super(RetryingNovaClient, inner).__init__(
+                    conf, first_wait=0, wait_up_to=0, **kwargs)
+
+            def retry(inner, func, *args, **kwargs):
+                self.retry_calls.append((func, args, kwargs))
+                return super(RetryingNovaClient, inner).retry(
+                    func, *args, **kwargs)
+
+        image_id = self.get_image_id('trusty', 'amd64')
+        self.conf.set('vm.image', image_id)
+        fixtures.patch(self, nova.NovaServer,
+                       'nova_client_class', RetryingNovaClient)
+        tb = nova.NovaServer(self.conf, logger)
+        self.assertEqual(image_id, tb.find_nova_image().name)
+        assertions.assertLength(self, 1, self.retry_calls)
+
+    @tests.log_on_failure()
+    def test_known_failure_is_retried(self, logger):
+        self.nb_calls = 0
+
+        class FailingOnceNovaClient(nova.NovaClient):
+
+            def __init__(inner, conf, **kwargs):
+                # We don't want to wait, it's enough to retry
+                super(FailingOnceNovaClient, inner).__init__(
+                    conf, first_wait=0, wait_up_to=0, retries=1,
+                    **kwargs)
+
+            def fail_once(inner):
+                self.nb_calls += 1
+                if self.nb_calls == 1:
+                    raise nova.client.requests.ConnectionError()
+                else:
+                    return inner.nova.flavors.list()
+
+            def flavors_list(inner):
+                return inner.retry(inner.fail_once)
+
+        fixtures.patch(self, nova.NovaServer,
+                       'nova_client_class', FailingOnceNovaClient)
+        tb = nova.NovaServer(self.conf, logger)
+        tb.find_flavor()
+        self.assertEqual(2, self.nb_calls)
+
+    @tests.log_on_failure()
+    def test_unknown_failure_is_raised(self, logger):
+
+        class FailingNovaClient(nova.NovaClient):
+
+            def __init__(inner, conf, **kwargs):
+                # We don't want to wait, it's enough to retry
+                super(FailingNovaClient, inner).__init__(
+                    conf, first_wait=0, wait_up_to=0,
+                    **kwargs)
+
+            def fail(inner):
+                raise AssertionError('Boom!')
+
+            def flavors_list(inner):
+                return inner.retry(inner.fail)
+
+        fixtures.patch(self, nova.NovaServer,
+                       'nova_client_class', FailingNovaClient)
+        tb = nova.NovaServer(self.conf, logger)
+        # This mimics what will happen when we encounter unknown transient
+        # failures we want to catch: an exception will bubble up and we'll have
+        # to add it to NovaClient.retry().
+        with self.assertRaises(nova.NovaServerException) as cm:
+            tb.find_flavor()
+        self.assertEqual('fail failed', '{}'.format(cm.exception))
+
+
[e-mail](vms_features.nova_creds)
[e-mail](vms_features.nova_compute)
+class TestTestbed(unittest.TestCase):
+
+    def setUp(self):
+        super(TestTestbed, self).setUp()
+        vms_fixtures.share_nova_test_creds(self)
+        conf = config.VmStack('testing-testbed')
+        conf.set('vm.name', 'testing-testbed')
+        # Default to precise
+        conf.set('vm.release', 'precise')
+        # Avoid triggering the 'atexit' hook as the config files are long gone
+        # at that point.
+        self.addCleanup(conf.store.save_changes)
+        self.conf = conf
+        os.makedirs(self.conf.get('vm.vms_dir'))
+
+    def get_image_id(self, series='precise', arch='amd64'):
+        return nova.uci_image_name('cloudimg', series, arch)
+
+    @tests.log_on_failure()
+    def test_create_no_image(self, logger):
+        tb = nova.NovaServer(self.conf, logger)
+        with self.assertRaises(options.errors.OptionMandatoryValueError) as cm:
+            tb.setup()
+        self.assertEqual('vm.image must be set.', '{}'.format(cm.exception))
+
+    @tests.log_on_failure()
+    def test_create_unknown_image(self, logger):
+        image_name = "I don't exist and eat kittens"
+        self.conf.set('vm.image', image_name)
+        tb = nova.NovaServer(self.conf, logger)
+        with self.assertRaises(nova.NovaServerException) as cm:
+            tb.setup()
+        self.assertEqual('Image "{}" cannot be found'.format(image_name),
+                         '{}'.format(cm.exception))
+
+    @tests.log_on_failure()
+    def test_create_unknown_flavor(self, logger):
+        flavors = "I don't exist and eat kittens"
+        self.conf.set('vm.os.flavors', flavors)
+        tb = nova.NovaServer(self.conf, logger)
+        with self.assertRaises(nova.NovaServerException) as cm:
+            tb.setup()
+        self.assertEqual('None of [{}] can be found'.format(flavors),
+                         '{}'.format(cm.exception))
+
+    @tests.log_on_failure()
+    def test_existing_home_ssh(self, logger):
+        # The first request for the worker requires creating ~/.ssh if it
+        # doesn't exist, but it may happen that this directory already exists
+        # (see http://pad.lv/1334146).
+        ssh_home = os.path.expanduser('~/sshkeys')
+        os.mkdir(ssh_home)
+        self.conf.set('vm.ssh_key_path', os.path.join(ssh_home, 'id_rsa'))
+        tb = nova.NovaServer(self.conf, logger)
+        tb.ensure_ssh_key_is_available()
+        self.assertTrue(os.path.exists(ssh_home))
+        self.assertTrue(os.path.exists(os.path.join(ssh_home, 'id_rsa')))
+        self.assertTrue(os.path.exists(os.path.join(ssh_home, 'id_rsa.pub')))
+
+    @tests.log_on_failure()
+    def test_create_new_ssh_key(self, logger):
+        self.conf.set('vm.image', self.get_image_id())
+        # We use a '~' path to cover proper uci-vms user expansion
+        self.conf.set('vm.ssh_key_path', '~/sshkeys/id_rsa')
+        tb = nova.NovaServer(self.conf, logger)
+        tb.ensure_ssh_key_is_available()
+        self.assertTrue(os.path.exists(os.path.expanduser('~/sshkeys/id_rsa')))
+        self.assertTrue(
+            os.path.exists(os.path.expanduser('~/sshkeys/id_rsa.pub')))
+
+    @tests.log_on_failure()
+    def test_ssh_failure(self, logger):
+        self.conf.set('vm.release', 'trusty')
+        self.conf.set('vm.image', self.get_image_id())
+        tb = nova.NovaServer(self.conf, logger)
+        self.addCleanup(tb.teardown)
+        tb.setup()
+        # Sabotage ssh access
+        os.remove(self.conf.get('vm.ssh_key_path'))
+        # Oh, we can't ssh anymore !
+        with self.assertRaises(nova.NovaServerException) as cm:
+            tb.ensure_ssh_works()
+        msg = 'No ssh access to {}, IP: {}'
+        msg = msg.format(tb.instance.id, tb.ip)
+        self.assertEqual(msg, '{}'.format(cm.exception))
+
+    @tests.log_on_failure()
+    def test_apt_get_update_retries(self, logger):
+        self.conf.set('vm.image', self.get_image_id())
+        self.conf.set('vm.apt_get.update.timeouts', '0.1, 0.1')
+        tb = nova.NovaServer(self.conf, logger)
+        self.nb_calls = 0
+
+        class Proc(object):
+            returncode = 0
+
+        def failing_update():
+            self.nb_calls += 1
+            if self.nb_calls > 1:
+                return Proc(), 'stdout success', 'stderr success'
+            else:
+                # Fake a failed apt-get update
+                proc = Proc()
+                proc.returncode = 1
+                return proc, 'stdout error', 'stderr error'
+
+        tb.apt_get_update = failing_update
+        tb.safe_apt_get_update()
+        self.assertEqual(2, self.nb_calls)
+
+    @tests.log_on_failure()
+    def test_apt_get_update_fails(self, logger):
+        self.conf.set('vm.image', self.get_image_id())
+        self.conf.set('vm.apt_get.update.timeouts', '0.1, 0.1, 0.1')
+        tb = nova.NovaServer(self.conf, logger)
+
+        def failing_update():
+            class Proc(object):
+                pass
+
+            proc = Proc()
+            proc.returncode = 1
+            return proc, 'stdout', 'stderr'
+
+        tb.apt_get_update = failing_update
+        with self.assertRaises(nova.NovaServerException) as cm:
+            tb.safe_apt_get_update()
+        self.assertEqual('apt-get update never succeeded',
+                         '{}'.format(cm.exception))
+
+    @tests.log_on_failure()
+    def test_wait_for_instance_fails(self, logger):
+        self.conf.set('vm.image', self.get_image_id())
+        # Force a 0 timeout so the instance can't finish booting
+        self.conf.set('vm.nova.boot_timeout', '0')
+        tb = nova.NovaServer(self.conf, logger)
+        self.addCleanup(tb.teardown)
+        with self.assertRaises(nova.NovaServerException) as cm:
+            tb.setup()
+        msg = 'Instance {} never came up (last status: BUILD)'
+        msg = msg.format(tb.instance.id)
+        self.assertEqual(msg, '{}'.format(cm.exception))
+
+    @tests.log_on_failure()
+    def test_wait_for_instance_errors(self, logger):
+        self.conf.set('vm.image', self.get_image_id())
+        tb = nova.NovaServer(self.conf, logger)
+        self.addCleanup(tb.teardown)
+
+        def update_instance_to_error():
+            # Fake an instance starting in error state
+            tb.instance.status = 'ERROR'
+            return True
+        tb.update_instance = update_instance_to_error
+        with self.assertRaises(nova.NovaServerException) as cm:
+            tb.setup()
+        msg = 'Instance {} never came up (last status: ERROR)'
+        msg = msg.format(tb.instance.id)
+        self.assertEqual(msg, '{}'.format(cm.exception))
+
+    @tests.log_on_failure()
+    def test_wait_for_ip_fails(self, logger):
+        self.conf.set('vm.image', self.get_image_id())
+        # Force a 0 timeout so the instance never get an IP
+        self.conf.set('vm.nova.set_ip_timeout', '0')
+        tb = nova.NovaServer(self.conf, logger)
+        self.addCleanup(tb.teardown)
+        with self.assertRaises(nova.NovaServerException) as cm:
+            tb.setup()
+        msg = 'Instance {} never provided an IP'.format(tb.instance.id)
+        self.assertEqual(msg, '{}'.format(cm.exception))
+
+
[e-mail](vms_features.nova_creds)
[e-mail](vms_features.nova_compute)
+class TestUsableTestbed(unittest.TestCase):
+
+    scenarios = scenarii.multiply_scenarios(
+        # series
+        ([('precise', dict(series='precise', result='skip')),
+          ('trusty', dict(series='trusty', result='pass')),
+          ('utopic', dict(series='utopic', result='pass')),
+          ('vivid', dict(series='vivid', result='pass')),
+          ('wily', dict(series='wily', result='pass'))]),
+        # architectures
+        ([('amd64', dict(arch='amd64')), ('i386', dict(arch='i386'))]))
+
+    def setUp(self):
+        super(TestUsableTestbed, self).setUp()
+        vms_fixtures.share_nova_test_creds(self)
+        tb_name = 'testing-testbed-{}-{}'.format(self.series, self.arch)
+        conf = config.VmStack(tb_name)
+        conf.set('vm.name', tb_name)
+        conf.set('vm.release', self.series)
+        # Avoid triggering the 'atexit' hook as the config files are long gone
+        # at that point.
+        self.addCleanup(conf.store.save_changes)
+        self.conf = conf
+        os.makedirs(self.conf.get('vm.vms_dir'))
+
+    def get_image_id(self):
+        return nova.uci_image_name('cloudimg', self.series, self.arch)
+
+    @tests.log_on_failure()
+    def test_create_usable_testbed(self, logger):
+        self.conf.set('vm.image', self.get_image_id())
+        tb = nova.NovaServer(self.conf, logger)
+        self.addCleanup(tb.teardown)
+        tb.setup()
+        # We should be able to ssh with the right user
+        proc, out, err = tb.ssh('whoami',
+                                out=subprocess.PIPE, err=subprocess.PIPE)
+        self.assertEqual(0, proc.returncode)
+        self.assertEqual('ubuntu\n', out)

=== added directory 'ucivms/vms'
=== renamed file 'ucivms/vms.py' => 'ucivms/vms/__init__.py'
--- ucivms/vms.py	2015-07-22 07:48:25 +0000
+++ ucivms/vms/__init__.py	2015-09-01 11:19:08 +0000
@@ -1,4 +1,3 @@
-#!/usr/bin/env python
 # This file is part of Ubuntu Continuous Integration virtual machine tools.
 #
 # Copyright 2014, 2015 Canonical Ltd.
@@ -384,6 +383,32 @@
         self._user_data_path = None
         self.ci_user_data = None
 
+    def download(self, force=False):
+        raise NotImplementedError(self.download)
+
+    def setup(self):
+        raise NotImplementedError(self.setup)
+
+    def start(self):
+        raise NotImplementedError(self.start)
+
+    def stop(self):
+        raise NotImplementedError(self.stop)
+
+    def teardown(self):
+        raise NotImplementedError(self.teardown)
+
+    # MISSINGTEST
+    def shell(self, command, *args):
+        ssh_cmd = self.get_ssh_command(command, *args)
+        retcode = subprocesses.raw_run(ssh_cmd)
+        return retcode
+
+    def shell_captured(self, command, *args):
+        ssh_cmd = self.get_ssh_command(command, *args)
+        retcode, out, err = subprocesses.run(ssh_cmd)
+        return retcode, out, err
+
     def _download_in_cache(self, source_url, name, force=False):
         """Download ``name`` from ``source_url`` in ``vm.download_cache``.
 
@@ -475,9 +500,6 @@
             f.write(ci_user_data.dump())
         self.ci_user_data = ci_user_data
 
-    def download(self, force=False):
-        raise NotImplementedError(self.download)
-
     def iface_path(self, iface):
         return os.path.join(self.config_dir_path(),
                             'interface.{}'.format(iface))
@@ -519,8 +541,8 @@
             else:
                 raise
 
-    def scan_console_during_install(self, console_size, cmd):
-        """Scan the console output until the end of the install.
+    def scan_console_during_setup(self, console_size, cmd):
+        """Scan the console output until the end of the setup.
 
         We add a specific part for cloud-init to ensure we properly detect
         the end of the run.
@@ -528,7 +550,7 @@
         :param console_size: The size of the console file before 'install' is
             run.
 
-        :param cmd: The install command (used for error display).
+        :param cmd: The setup command (used for error display).
         """
         console = monitors.TailMonitor(self.console_path(), console_size)
         iface = 'eth0'
@@ -604,17 +626,6 @@
                 cmd += args
         return cmd
 
-    # MISSINGTEST
-    def shell(self, command, *args):
-        ssh_cmd = self.get_ssh_command(command, *args)
-        retcode = subprocesses.raw_run(ssh_cmd)
-        return retcode
-
-    def shell_captured(self, command, *args):
-        ssh_cmd = self.get_ssh_command(command, *args)
-        retcode, out, err = subprocesses.run(ssh_cmd)
-        return retcode, out, err
-
 
 def kvm_states(source=None):
     """A dict of states for kvms indexed by name.
@@ -646,7 +657,7 @@
         try:
             state = states[self.conf.get('vm.name')]
         except KeyError:
-            state = None
+            state = 'UNKNOWN'
         return state
 
     def download_iso(self, force=False):
@@ -732,17 +743,17 @@
              disk_image_path, self.conf.get('vm.disk_size')])
         self._disk_image_path = disk_image_path
 
-    def scan_console_during_install(self, console_size, cmd):
-        """See Vm.scan_console_during_install."""
+    def scan_console_during_setup(self, console_size, cmd):
+        """See Vm.scan_console_during_setup."""
         # The console is re-created by virt-install (even if we created it
         # before) which requires sudo but creates the file 0600 for
         # libvirt-qemu. We give read access to all otherwise 'tail -f' requires
         # sudo and can't be killed anymore.
         subprocesses.run(['sudo', 'chmod', '0644', self.console_path()])
         # While `virt-install` is running, let's connect to the console
-        super(Kvm, self).scan_console_during_install(console_size, cmd)
+        super(Kvm, self).scan_console_during_setup(console_size, cmd)
 
-    def install(self):
+    def setup(self):
         # Create a kvm, relying on cloud-init to customize the base image.
         #
         # There are two processes involvded here:
@@ -788,13 +799,13 @@
         ]
         console_size = monitors.actual_file_size(self.console_path())
         subprocesses.run(virt_install)
-        self.scan_console_during_install(console_size, virt_install)
+        self.scan_console_during_setup(console_size, virt_install)
         # We've seen the console signaling halt, but the vm will need a bit
         # more time to get there so we help it a bit.
         if self.conf.get('vm.release') in ('precise', 'quantal'):
             # cloud-init doesn't implement power_state until raring and need a
             # gentle nudge.
-            self.poweroff()
+            self.stop()
         while True:
             state = self.state()
             # We expect the vm's state to be 'in shutdown' but in some rare
@@ -808,8 +819,8 @@
                 break
             # FIXME: No idea on how to test the following. Manually tested by
             # altering the expected state above and running 'selftest.py -v'
-            # which errors out for test_install_with_seed and
-            # test_install_backing. Also reproduced when 'running' wasn't
+            # which errors out for test_setup_with_seed and
+            # test_setup_backing. Also reproduced when 'running' wasn't
             # expected before 'in shutdown' -- vila 2013-02-19
             # Unexpected state reached, bad.
             raise errors.UciVmsError(
@@ -834,17 +845,11 @@
         # available -- vila 2015-06-18
         return proc
 
-    # FIXME: Something is wrong in the API, all call sites should use stop()
-    # not poweroff() and stop() should be defined in the base class -- vila
-    # 2015-06-18
     def stop(self):
-        self.poweroff()
-
-    def poweroff(self):
         return subprocesses.run(
             ['sudo', 'virsh', 'destroy', self.conf.get('vm.name')])
 
-    def undefine(self):
+    def teardown(self):
         return subprocesses.run(
             ['sudo', 'virsh', 'undefine', self.conf.get('vm.name'),
              '--remove-all-storage'])
@@ -928,7 +933,7 @@
                     self._lxc_seed_path]
         _, out, err = subprocesses.run(cp_seeds)
 
-    def install(self):
+    def setup(self):
         '''Create an lxc, relying on cloud-init to customize the base image.
 
         There are two processes involvded here:
@@ -980,7 +985,7 @@
         # print 'cmd: %s' % (' '.join(lxc_start),)
         console_size = monitors.actual_file_size(self.console_path())
         proc = subprocesses.run(lxc_start)
-        self.scan_console_during_install(console_size, lxc_start)
+        self.scan_console_during_setup(console_size, lxc_start)
         return proc
 
     # MISSINGTEST
@@ -1054,14 +1059,11 @@
         self.wait_for_ip()
         self.wait_for_ssh()
 
-    def poweroff(self):
+    def stop(self):
         return subprocesses.run(
             ['sudo', 'lxc-stop', '-n', self.conf.get('vm.name')])
 
-    def stop(self):
-        self.poweroff()
-
-    def undefine(self):
+    def teardown(self):
         try:
             return subprocesses.run(
                 ['sudo', 'lxc-destroy', '-n', self.conf.get('vm.name')])
@@ -1082,11 +1084,11 @@
     def download(self, force=False):
         raise NotImplementedError(self.download)
 
-    def install(self):
-        raise NotImplementedError(self.download)
+    def setup(self):
+        raise NotImplementedError(self.setup)
 
-    def undefine(self):
-        raise NotImplementedError(self.download)
+    def teardown(self):
+        raise NotImplementedError(self.teardown)
 
     def start(self):
         start_cmd = ['sudo', 'lxc-start-ephemeral',
@@ -1098,6 +1100,3 @@
         subprocesses.run(start_cmd)
         self.wait_for_ip()
         self.wait_for_ssh()
-
-    def stop(self):
-        self.poweroff()

=== added file 'ucivms/vms/nova.py'
--- ucivms/vms/nova.py	1970-01-01 00:00:00 +0000
+++ ucivms/vms/nova.py	2015-09-01 11:19:08 +0000
@@ -0,0 +1,475 @@
+# This file is part of Ubuntu Continuous Integration virtual machine tools.
+#
+# Copyright 2015 Canonical Ltd.
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License version 3, as published by the
+# Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranties of MERCHANTABILITY,
+# SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program.  If not, see <http://www.gnu.org/licenses/>.
+from __future__ import unicode_literals
+
+import errno
+import logging
+import os
+import subprocess
+import time
+
+
+from novaclient import exceptions
+from novaclient import client
+
+
+from ucivms import (
+    timeouts,
+    vms,
+)
+
+
+def uci_image_name(domain, series, architecture):
+    """Returns an image name.
+
+    The images are uploaded to glance for specific needs.
+
+    :param domain: 'cloudimg' or 'britney'.
+
+    :param series: The ubuntu series (precise, utopic, etc).
+
+    :param architecture: The processor architecture ('amd64', i386, etc).
+    """
+    if domain not in ('cloudimg', 'britney'):
+        raise ValueError('Invalid image domain')
+    return 'uci/{}/{}-{}.img'.format(domain, series, architecture)
+
+
+def get_os_nova_client(conf, debug=False):
+    os_nova_client = client.Client(
+        '1.1',
+        conf.get('vm.os.username'), conf.get('vm.os.password'),
+        conf.get('vm.os.tenant_name'),
+        conf.get('vm.os.auth_url'),
+        region_name=conf.get('vm.os.region_name'),
+        service_type='compute')
+    return os_nova_client
+
+
+# FIXME: This should inherit from ucivms.errors.UciVmsError (respecting the
+# API) so it can be caught by upper layers -- vila 2015-08-26
+class NovaServerException(Exception):
+    pass
+
+
+class NovaClient(object):
+    """A nova client re-trying requests on known transient failures."""
+
+    def __init__(self, conf, **kwargs):
+        self.logger = kwargs.pop('logger')
+        self.first_wait = kwargs.pop('first_wait', 30)
+        self.wait_up_to = kwargs.pop('wait_up_to', 600)
+        self.retries = kwargs.pop('retries', 8)
+        debug = kwargs.pop('debug', False)
+        # Activating debug will output the http requests issued by nova and the
+        # corresponding responses (/!\ including credentials).
+        if debug:
+            self.logger.setLevel(logging.DEBUG)
+        self.nova = get_os_nova_client(conf, debug)
+
+    def retry(self, func, *args, **kwargs):
+        no_404_retry = kwargs.pop('no_404_retry', False)
+        sleeps = timeouts.ExponentialBackoff(
+            self.first_wait, self.wait_up_to, self.retries)
+        for attempt, sleep in enumerate(sleeps, start=1):
+            try:
+                if attempt > 1:
+                    self.logger.info('Re-trying {} {}/{}'.format(
+                        func.__name__, attempt, self.retries))
+                return func(*args, **kwargs)
+            except client.requests.ConnectionError:
+                # Most common transient failure: the API server is unreachable
+                msg = 'Connection error for {}, will sleep for {} seconds'
+                self.logger.warn(msg.format(func.__name__, sleep))
+            except (exceptions.OverLimit, exceptions.RateLimit):
+                msg = ('Rate limit reached for {},'
+                       ' will sleep for {} seconds')
+                # This happens rarely but breaks badly if not caught. elmo
+                # recommended a 30 seconds nap in that case.
+                sleep += 30
+                self.logger.exception(msg.format(func.__name__, sleep))
+            except exceptions.ClientException as e:
+                if no_404_retry and e.http_status == 404:
+                    raise
+                msg = '{} failed will sleep for {} seconds'
+                self.logger.exception(msg.format(func.__name__, sleep))
+            except:
+                # All other exceptions are raised
+                self.logger.exception('{} failed'.format(func.__name__))
+                raise NovaServerException('{} failed'.format(func.__name__))
+            # Take a nap before retrying
+            self.logger.info('Sleeping {} seconds for {} {}/{}'.format(
+                sleep, func.__name__, attempt, self.retries))
+            time.sleep(sleep)
+        # Raise if we didn't succeed at all
+        raise NovaServerException("Failed to '{}' after {} retries".format(
+            func.__name__, attempt))
+
+    def flavors_list(self):
+        return self.retry(self.nova.flavors.list)
+
+    def images_list(self):
+        return self.retry(self.nova.images.list)
+
+    def create_server(self, name, flavor, image, user_data, nics,
+                      availability_zone):
+        return self.retry(self.nova.servers.create, name=name,
+                          flavor=flavor, image=image, userdata=user_data,
+                          nics=nics, availability_zone=availability_zone)
+
+    def delete_server(self, server_id):
+        # FIXME: 404 shouldn't be retried, if it's not there anymore, there is
+        # nothing to delete. -- vila 2015-07-16
+        return self.retry(self.nova.servers.delete, server_id)
+
+    def start_server(self, instance):
+        return self.retry(instance.start)
+
+    def stop_server(self, instance):
+        return self.retry(instance.stop)
+
+    def create_floating_ip(self):
+        return self.retry(self.nova.floating_ips.create)
+
+    def delete_floating_ip(self, floating_ip):
+        return self.retry(self.nova.floating_ips.delete, floating_ip)
+
+    def add_floating_ip(self, instance, floating_ip):
+        return self.retry(instance.add_floating_ip, floating_ip)
+
+    def get_server_details(self, server_id):
+        return self.retry(self.nova.servers.get, server_id,
+                          no_404_retry=True)
+
+    def get_server_console(self, server, length=None):
+        return self.retry(server.get_console_output, length)
+
+
+class NovaServer(vms.VM):
+
+    nova_client_class = NovaClient
+
+    def __init__(self, conf, logger=None):
+        super(NovaServer, self).__init__(conf)
+        if logger is None:
+            # FIXME: We probably want to generalize logging -- vila 2015-08-25
+            self.ensure_dir(self.config_dir_path())
+            logging.basicConfig(
+                level=logging.INFO,
+                format="%(asctime)s %(levelname)s %(message)s",
+                filename=os.path.join(self.config_dir_path(), 'uci-vms.log'))
+            logger = logging.getLogger()
+        self.logger = logger
+        self.instance = None
+        self.floating_ip = None
+        self.nova = self.build_nova_client()
+        self.test_bed_key_path = None
+        self.conf.set('vm.ssh_authorized_keys',
+                      self.conf.get('vm.ssh_key_path') + '.pub')
+        # No need to reboot a nova instance
+        self.conf.set('vm.poweroff', 'False')
+        self.conf.set('vm.final_message', 'testbed setup completed.')
+
+    # MISSINGTEST
+    def state(self):
+        try:
+            with open(self.nova_id_path()) as f:
+                nova_id = f.read().strip()
+        except IOError as e:
+            # python2 does not provide FileNotFoundError
+            if e.errno == errno.ENOENT:
+                # Unknown interface
+                return 'UNKNOWN'
+        try:
+            self.instance = self.nova.get_server_details(nova_id)
+        except exceptions.NotFound:
+            return 'UNKNOWN'
+        # The instance may remain in the DELETED state for some time.
+        nova_states = dict(BUILD='STARTING',
+                           ACTIVE='RUNNING',
+                           SHUTOFF='STOPPED',
+                           DELETED='UNKNOWN')
+        return nova_states[self.instance.status]
+
+    def build_nova_client(self):
+        nova_client = self.nova_client_class(self.conf, logger=self.logger)
+        return nova_client
+
+    def ensure_ssh_key_is_available(self):
+        self.test_bed_key_path = self.conf.get('vm.ssh_key_path')
+        # FIXME: Needs to be unified for all vm classes -- vila 2015-08-26
+        # From the test runner, we need an ssh key that can be used to connect
+        # to the testbed. For testing purposes, we rely on self.auth_conf to
+        # provide this key.
+        if not os.path.exists(self.test_bed_key_path):
+            base_dir = os.path.dirname(self.test_bed_key_path)
+            try:
+                # Try to create needed dirs
+                os.makedirs(base_dir)
+            except OSError as e:
+                # They are already there, no worries
+                if e.errno != errno.EEXIST:
+                    raise
+            # First time the test runner instance needs to create a test bed
+            # instance, we need to create the ssh key pair.
+            subprocess.call(
+                ['ssh-keygen', '-t', 'rsa', '-q',
+                 '-f', self.test_bed_key_path, '-N', ''])
+
+    def find_flavor(self):
+        flavors = self.conf.get('vm.os.flavors')
+        existing_flavors = self.nova.flavors_list()
+        for flavor in flavors:
+            for existing in existing_flavors:
+                if flavor == existing.name:
+                    return existing
+        raise NovaServerException(
+            'None of [{}] can be found'.format(','.join(flavors)))
+
+    def find_nova_image(self):
+        image_name = self.conf.get('vm.image')
+        existing_images = self.nova.images_list()
+        for existing in existing_images:
+            if image_name == existing.name:
+                return existing
+        raise NovaServerException(
+            'Image "{}" cannot be found'.format(image_name))
+
+    def find_nics(self):
+        net_id = self.conf.get('vm.net_id')
+        if net_id:
+            return [{'net-id': self.conf.get('vm.net_id')}]
+        return None
+
+    # FIXME: This should save the console whether or not the setup fails
+    # -- vila 2015-08-26
+    def setup(self):
+        flavor = self.find_flavor()
+        image = self.find_nova_image()
+        nics = self.find_nics()
+        self.ensure_ssh_key_is_available()
+        self.create_user_data()
+        with open(self._user_data_path) as f:
+            user_data = f.read()
+        self.instance = self.nova.create_server(
+            name=self.conf.get('vm.name'), flavor=flavor, image=image,
+            user_data=user_data, nics=nics,
+            # FIXME: We probably want at least a vm.az_name option. And get
+            # that option from higher levels too -- vila 2014-10-13
+            availability_zone=None)
+        self.create_nova_id_file(self.instance.id)
+        self.wait_for_active_instance()
+# FIXME: We want a vm.create_floating_ip option ? -- vila 2015-08-24
+#        if unit_config.is_hpcloud(self.conf.get('os.auth_url')):
+#            self.floating_ip = self.nova.create_floating_ip()
+#            self.nova.add_floating_ip(self.instance, self.floating_ip)
+        self.wait_for_ip()
+        self.wait_for_cloud_init()
+        self.ensure_ssh_works()
+        ppas = self.conf.get('vm.ppas')
+        if ppas:
+            cmd = ['sudo', 'add-apt-repository']
+            if self.conf.get('vm.release') > 'precise':
+                cmd.append('--enable-source')
+            for ppa in ppas:
+                self.ssh(*(cmd + [ppa]))
+        # Now we can apt-get update (doing it earlier would lead to the wrong
+        # source package to be installed).
+        self.safe_apt_get_update()
+
+    def apt_get_update(self):
+        return self.ssh('sudo', 'apt-get', 'update')
+
+    def safe_apt_get_update(self):
+        for timeout in self.conf.get('vm.apt_get.update.timeouts'):
+            proc, out, err = self.apt_get_update()
+            if proc.returncode == 0:
+                # We're done
+                return
+            else:
+                msg = ('apt-get update failed, wait {}s\n'
+                       'stdout:\n{}\n'
+                       'stderr:\n{}\n')
+                self.logger.info(msg.format(timeout, out, err))
+                time.sleep(float(timeout))
+        raise NovaServerException('apt-get update never succeeded')
+
+    def update_instance(self, nova_id=None):
+        if nova_id is None:
+            nova_id = self.instance.id
+        try:
+            # Always query nova to get updated data about the instance
+            self.instance = self.nova.get_server_details(nova_id)
+            return True
+        except:
+            # But catch exceptions if something goes wrong. Higher levels will
+            # deal with the instance not replying.
+            return False
+
+    def wait_for_active_instance(self):
+        timeout_limit = time.time() + self.conf.get('vm.nova.boot_timeout')
+        while (time.time() < timeout_limit
+               and self.instance.status not in ('ACTIVE', 'ERROR')):
+            time.sleep(5)
+            self.update_instance()
+        if self.instance.status != 'ACTIVE':
+            msg = 'Instance {} never came up (last status: {})'.format(
+                self.instance.id, self.instance.status)
+            raise NovaServerException(msg)
+
+    def nova_id_path(self):
+        return os.path.join(self.config_dir_path(), 'nova_id')
+
+    def create_nova_id_file(self, nova_id):
+        nova_id_path = self.nova_id_path()
+        self.ensure_dir(self.config_dir_path())
+        with open(nova_id_path, 'w') as f:
+            f.write(nova_id + '\n')
+
+    def wait_for_ip(self):
+        timeout_limit = time.time() + self.conf.get('vm.nova.set_ip_timeout')
+        while time.time() < timeout_limit:
+            if not self.update_instance():
+                time.sleep(5)
+                continue
+            networks = self.instance.networks.values()
+            if networks:
+                # The network name is arbitrary, can vary for different clouds
+                # but there should be only one network so we get the first one
+                # and avoid the need for a config option for the network name.
+                # We take the last IP address so it's either the only one or
+                # the floating one. In both cases that gives us a reachable IP.
+                self.ip = networks[0][-1]
+                self.logger.info('Got IP {} for {}'.format(
+                    self.ip, self.instance.id))
+                # FIXME: Why not get it from the console ? -- vila 2015-08-26
+                # MISSINGTEST
+                self.create_iface_file('eth0', self.ip, 'unknown', 'unknown')
+                return
+            else:
+                self.logger.info(
+                    'IP not yet available for {}'.format(self.instance.id))
+            time.sleep(5)
+        msg = 'Instance {} never provided an IP'.format(self.instance.id)
+        raise NovaServerException(msg)
+
+    def get_cloud_init_console(self, length=None):
+        return self.nova.get_server_console(self.instance, length)
+
+    def wait_for_cloud_init(self):
+        timeout_limit = (time.time()
+                         + self.conf.get('vm.nova.cloud_init_timeout'))
+        final_message = self.conf.get('vm.final_message')
+        while time.time() < timeout_limit:
+            # A relatively cheap way to catch cloud-init completion is to watch
+            # the console for the specific message we specified in user-data).
+            # FIXME: or at least check that we don't miss when we sleep a
+            # significant time between two calls (like on canonistack where we
+            # can sleep for minute(s) -- vila 2015-07-17
+            console = self.get_cloud_init_console(10)
+            if final_message in console:
+                # We're good to go
+                self.logger.info(
+                    'cloud-init completed for {}'.format(self.instance.id))
+                return
+            time.sleep(5)
+        raise NovaServerException('Instance never completed cloud-init')
+
+    def ensure_ssh_works(self):
+        proc, out, err = self.ssh('whoami')
+        if proc.returncode:
+            msg = ('testbed {} IP {} cannot be reached with ssh, retcode: {}\n'
+                   'stdout:\n{}\n'
+                   'stderr:\n{}\n')
+            self.logger.info(msg.format(self.instance.id, self.ip,
+                                        proc.returncode, out, err))
+            raise NovaServerException('No ssh access to {}, IP: {}'.format(
+                self.instance.id, self.ip))
+
+    # MISSINGTEST
+    def start(self):
+        self.nova.start_server(self.instance)
+
+    # MISSINGTEST
+    def stop(self):
+        self.nova.stop_server(self.instance)
+
+    def teardown(self):
+        if self.instance is not None:
+            self.logger.info('Deleting instance {}'.format(self.instance.id))
+            self.nova.delete_server(self.instance.id)
+            self.instance = None
+            os.remove(self.nova_id_path())
+        if self.floating_ip is not None:
+            self.nova.delete_floating_ip(self.floating_ip)
+            self.floating_ip = None
+        # FIXME: Now we can remove the testbed key from known_hosts (see
+        # ssh()). -- vila 2014-01-30
+
+    # FIXME: Should be unified with 'shell' and 'shell_captured'
+    # -- vila 2015-08-26
+    def ssh(self, command, *args, **kwargs):
+        """Run a command in the testbed via ssh.
+
+        :param args: The command and its positional arguments.
+
+        :param kwargs: The named arguments for the command.
+
+        The stdout and stderr outputs are captured and returned to the caller.
+        """
+        user = 'ubuntu'
+        host = self.ip
+        cmd = ['ssh',
+               # FIXME: It would be better to ssh-keygen -f
+               # "~/.ssh/known_hosts" -R self.ip once we're done with the test
+               # bed to avoid polluting ssh commands stdout, but that will do
+               # for now (that's what juju is doing after all ;)
+               # -- vila 2014-01-29
+               '-oStrictHostKeyChecking=no',
+               '-i', self.test_bed_key_path,
+               '{}@{}'.format(user, host)]
+        if command is not None:
+            cmd += [command]
+        if args:
+            cmd += args
+        proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
+                                stderr=subprocess.PIPE)
+        out, err = proc.communicate()
+        return proc, out, err
+
+    # FIXME: Should be provided by the base class -- vila 2015-08-26
+    def scp(self, local_path, remote_path):
+        cmd = ['scp',
+               # FIXME: It would be better to ssh-keygen -f
+               # "~/.ssh/known_hosts" -R self.ip once we're done with the test
+               # bed to avoid polluting ssh commands stdout, but that will do
+               # for now (that's what juju is doing after all ;)
+               # -- vila 2014-01-29
+               '-oStrictHostKeyChecking=no',
+               '-i', self.test_bed_key_path,
+               local_path, remote_path]
+        proc = subprocess.Popen(cmd, stdout=subprocess.PIPE,
+                                stderr=subprocess.PIPE)
+        out, err = proc.communicate()
+        return proc, out, err
+
+    def get_remote_content(self, path):
+        proc, content, err = self.ssh('cat', path)
+        if proc.returncode:
+            # We didn't get a proper content, report it instead
+            content = ("{} couldn't be copied from testbed {}:\n"
+                       "error: {}\n".format(path, self.ip, err))
+        return content