Merge lp:~sbaldassin/snappy-ecosystem-tests/merge_driver_and_container into lp:snappy-ecosystem-tests

[Santiago Baldassin]

Splitting responsibilities between driver and manager

This mp makes a better use of the Manager and Builder and split the responsibilities between them

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:33
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/250/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2059/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/250/rebuild

review: Approve (continuous-integration)

[Heber Parrucci (heber013)]

Comment inline

review: Needs Fixing

[Santiago Baldassin (sbaldassin)]

Reply inline

[Omer Akram (om26er)]

The approach to start multiple containers and not waiting for them is problematic and its enforcing use to use random sleeps. Added inline comment on how that is a problem.

If you really want to start multiple containers, ask pylxd to do that for you in a loop.

review: Needs Fixing

[lp:~sbaldassin/snappy-ecosystem-tests/merge_driver_and_container]

34. By Santiago Baldassin on 2017-03-07

Addressing comments from the reviews

[Santiago Baldassin (sbaldassin)]

Thanks for the feedback Omer. All comments were addressed, however, I did leave the option to create containers and not wait and I also leave a wait period for starting the containers faster. Please do take a look

[Omer Akram (om26er)]

Added inline comment, yesterday I started working on actually fixing the snapd staging environment but when you proposed this branch, it had a potential of conflict. So I am waiting for this branch to either land or the comment that I just made inline addressed.

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:34
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/251/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2062/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/251/rebuild

review: Needs Fixing (continuous-integration)

[lp:~sbaldassin/snappy-ecosystem-tests/merge_driver_and_container]

35. By Santiago Baldassin on 2017-03-07

Fixing pylint

[Omer Akram (om26er)]

Added comments inline.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:35
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/252/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2063/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/252/rebuild

review: Approve (continuous-integration)

[Santiago Baldassin (sbaldassin)]

Thanks Omer, I riplied inline. We'll need a phone call to get everyone on the same page

[Santiago Baldassin (sbaldassin)]

Replied inline

[Omer Akram (om26er)]

Added comments inline. Lets aim to land this sooner.

[lp:~sbaldassin/snappy-ecosystem-tests/merge_driver_and_container]

36. By Santiago Baldassin on 2017-03-08

Addressing comments from the reviews

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:36
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/256/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2070/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/256/rebuild

review: Needs Fixing (continuous-integration)

[lp:~sbaldassin/snappy-ecosystem-tests/merge_driver_and_container]

37. By Santiago Baldassin on 2017-03-08

Minor refactor

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:37
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/257/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2072/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/257/rebuild

review: Needs Fixing (continuous-integration)

[Omer Akram (om26er)]

please merge with trunk, there are conflicts.

review: Needs Fixing

[lp:~sbaldassin/snappy-ecosystem-tests/merge_driver_and_container]

38. By Santiago Baldassin on 2017-03-08

Merge from trunk

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:38
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/258/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2073/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/258/rebuild

review: Needs Fixing (continuous-integration)

[Omer Akram (om26er)]

Seems jenkins is again having issues. Anyways can you please add this bug report into your `launch` https://github.com/lxc/pylxd/issues/221

[Heber Parrucci (heber013)]

> Seems jenkins is again having issues. Anyways can you please add this bug
> report into your `launch` https://github.com/lxc/pylxd/issues/221

Actually jenkins is not having issues. There is a pylint issue in the revision:
W: 26, 0: Unused sleep imported from time (unused-import)

[Omer Akram (om26er)]

hmm, I mis-interpretted the output https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/258/console

had the impression that line: error: <urlopen error Tunnel connection failed: 403 Forbidden> was the cause. But you are right, its pylint complaint that needs fixing.

[lp:~sbaldassin/snappy-ecosystem-tests/merge_driver_and_container]

39. By Santiago Baldassin on 2017-03-08

Fixing pylint

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:39
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/259/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2075/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/259/rebuild

review: Approve (continuous-integration)

[Omer Akram (om26er)]

LGTM.

review: Approve

[Heber Parrucci (heber013)]

Code LGTM. Let's land it.

review: Approve

=== added directory 'snappy_ecosystem_tests/environment/containers'
=== added file 'snappy_ecosystem_tests/environment/containers/__init__.py'
=== added file 'snappy_ecosystem_tests/environment/containers/lxd.py'
--- snappy_ecosystem_tests/environment/containers/lxd.py	1970-01-01 00:00:00 +0000
+++ snappy_ecosystem_tests/environment/containers/lxd.py	2017-03-08 15:34:35 +0000
@@ -0,0 +1,118 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+
+#
+# Snappy Ecosystem Tests
+# Copyright (C) 2017 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+"""Module top interact with lxd containers"""
+
+import os
+import shlex
+
+import logging
+
+from pylxd import Client
+from pylxd.exceptions import LXDAPIException, NotFound
+from retrying import retry
+
+from snappy_ecosystem_tests.environment.data.snapd import (
+    DEFAULT_CONTAINER_CONFIG)
+
+COMMAND_ALLOW_ENV_VARS = 'sed -i \'/^AcceptEnv/ s/$/ {}/\' ' \
+                         '/etc/ssh/sshd_config'
+
+LOGGER = logging.getLogger(__name__)
+DOMAIN_PING = 'ubuntu.com'
+
+class LxdDriver:
+    """Manage lxd containers"""
+
+    def __init__(self):
+        self.client = Client()
+
+    def launch(self, containers):
+        """Launch a container"""
+        created_containers = []
+        for cont in containers:
+
+            if self.client.containers.exists(cont):
+                container = self.client.containers.get(cont)
+                try:
+                    if container.status == "Running":
+                        container.stop(wait=True)
+                    container.delete(wait=True)
+                except NotFound:
+                    # If using ephemeral stop will delete the container and try
+                    # to sync which will fail because the container does not
+                    # exist. container.delete will be only execute if stop did
+                    # not raise an exception:
+                    # https://github.com/lxc/pylxd/issues/221
+                    pass
+
+            container_config = DEFAULT_CONTAINER_CONFIG.copy()
+            container_config['source']['alias'] = containers[cont]['release']
+            container_config['name'] = cont
+            created_containers.append(self.client.containers.create(
+                container_config, wait=True))
+        self.start(created_containers)
+        return created_containers
+
+    @staticmethod
+    def start(containers):
+        """start all containers received as parameter"""
+        for cont in containers:
+            if not cont.state().status == 'Running':
+                cont.start(wait=True)
+
+    @staticmethod
+    def install(container, packages, channel=None):
+        """Install packages"""
+        container.execute(shlex.split('apt-get update'))
+        install_command = 'apt-get install {} -y'.format(
+            ' '.join(packages))
+        if channel:
+            install_command += ' -t {}'.format(channel)
+        container.execute(shlex.split(install_command))
+
+    @staticmethod
+    @retry(stop_max_attempt_number=5, wait_fixed=3000,
+           retry_on_exception=lambda exception: isinstance(
+               exception, LXDAPIException))
+    def enable_ssh(container):
+        """Enable the ssh connection on the container"""
+        pub_key = open(
+            os.path.expanduser('~') + '/.ssh/id_rsa.pub').read().strip('\n')
+        container.files.put('/home/ubuntu/.ssh/authorized_keys', pub_key)
+
+    @staticmethod
+    def get_ip(container):
+        """Gets the ip address for a given container"""
+        networks = container.state().network['eth0']['addresses']
+
+        for network in networks:
+            if network['address']:
+                return network['address']
+        raise RuntimeError(
+            "The container {} does not have an IPV4 connection".format(
+                container.name))
+
+    @staticmethod
+    def export_environment_variables(container,
+                                     environment_variables):
+        """Allow export of environment variables over ssh."""
+        container.execute(shlex.split(COMMAND_ALLOW_ENV_VARS.format(
+            ' '.join(environment_variables))))
+        container.execute(shlex.split('service ssh restart'))

=== modified file 'snappy_ecosystem_tests/environment/data/snapd.py'
--- snappy_ecosystem_tests/environment/data/snapd.py	2017-03-02 10:51:37 +0000
+++ snappy_ecosystem_tests/environment/data/snapd.py	2017-03-08 15:34:35 +0000
@@ -93,5 +93,6 @@
         "server": "https://cloud-images.ubuntu.com/releases",
         "protocol": "simplestreams",
         "fingerprint": "16.04"
-    }
+    },
+    "ephemeral": True
 }

=== modified file 'snappy_ecosystem_tests/environment/managers.py'
--- snappy_ecosystem_tests/environment/managers.py	2017-03-07 17:41:31 +0000
+++ snappy_ecosystem_tests/environment/managers.py	2017-03-08 15:34:35 +0000
@@ -1,7 +1,7 @@
 # -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
 
 #
-# Ubuntu System Tests
+# Snappy Ecosystem Tests
 # Copyright (C) 2017 Canonical
 #
 # This program is free software: you can redistribute it and/or modify
@@ -17,134 +17,39 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
-
 """Managers to setup the environment"""
 
-import os
-import shlex
-from pylxd import Client
-from pylxd.exceptions import NotFound, LXDAPIException
-from retrying import retry
-
-from snappy_ecosystem_tests.environment import constants
 from snappy_ecosystem_tests.environment.constants import (
     PROFILES, DEPENDENCIES)
-from snappy_ecosystem_tests.environment.data.snapcraft import (
-    SNAPCRAFT_CONTAINER_NAME)
-from snappy_ecosystem_tests.environment.data.snapd import (
-    DEFAULT_CONTAINER_CONFIG, SNAPD_CONTAINER_NAME)
-
-COMMAND_ALLOW_ENV_VARS = 'sed -i \'/^AcceptEnv/ s/$/ {}/\' ' \
-                         '/etc/ssh/sshd_config'
-
-
-def get_manager(mode):
-    """Get the manager for the given mode"""
-    supported_modules = {
-        "lxd": LxdManager
+from snappy_ecosystem_tests.environment.containers.lxd import LxdDriver
+
+CONTAINERS = {
+    "snapd":{
+        "release": "16.04"
+    },
+    "snapcraft":{
+        "release": "16.04"
     }
-    try:
-        manager = supported_modules[mode]
-        return manager()
-    except KeyError:
-        raise RuntimeError("Mode: {} is not supported".format(mode))
-
-
-class LxdManager:
-    """Manage lxd containers"""
+}
+
+class _LxdManager:
+    """Manager for lxd containers"""
 
     def __init__(self):
-        self.client = Client()
-
-    def launch(self, container_name, release='16.04'):
-        """Launch a container"""
-        try:
-            container = self.client.containers.get(container_name)
-            if container.status == "Running":
-                container.stop(wait=True)
-            container.delete(wait=True)
-        except NotFound:
-            pass
-        container_config = DEFAULT_CONTAINER_CONFIG.copy()
-        container_config['source']['alias'] = release
-        container_config['name'] = container_name
-        container = self.client.containers.create(container_config, wait=True)
-        container.start(wait=True)
-
-    def execute_command(self, container_name, command):
-        """execute command"""
-        container = self.client.containers.get(container_name)
-        container.execute(command)
-
-    def install(self, container_name, packages, channel=None):
-        """Install a packages"""
-        container = self.client.containers.get(container_name)
-        container.execute(shlex.split('apt-get update'))
-        install_command = 'apt-get install {} -y'.format(
-            ' '.join(packages))
-        if channel:
-            install_command += ' -t {}'.format(channel)
-        container.execute(shlex.split(install_command))
-
-    def install_dependencies(self, container_name, container_type):
-        """Install a dependencies"""
-        self.install(
-            container_name,
-            DEPENDENCIES[container_type] + DEPENDENCIES['shared'])
-
-    def configure(self, container_name, container_type, profile):
-        """Configure a given container"""
-        container = self.client.containers.get(container_name)
-        for key, value in \
-                PROFILES[profile][container_type]['environment_variables'].\
-                        items():
-            container.config.update({'environment.{}'.format(
-                key): value})
-        container.save(wait=True)
-
-    @retry(stop_max_attempt_number=5, wait_fixed=3000,
-           retry_on_exception=lambda exception: isinstance(
-               exception, LXDAPIException))
-    def enable_ssh(self, container_name, container_type, profile):
-        """Enable the ssh connection on the container"""
-        container = self.client.containers.get(container_name)
-        pub_key = open(
-            os.path.expanduser('~') + '/.ssh/id_rsa.pub').read().strip('\n')
-        container.files.put('/home/ubuntu/.ssh/authorized_keys', pub_key)
-        # Allow export of environment variables over ssh.
-        container.execute(shlex.split(COMMAND_ALLOW_ENV_VARS.format(
-            ' '.join(PROFILES[profile][container_type]
-                     ['environment_variables'].keys()))))
-        container.execute(shlex.split('service ssh restart'))
+        self.driver = LxdDriver()
 
     def setup(self, profile):
         """setup the container based on the profile"""
-        self._setup(SNAPD_CONTAINER_NAME, constants.SNAPD, profile)
-        self._setup(
-            SNAPCRAFT_CONTAINER_NAME, constants.SNAPCRAFT, profile)
-
-    def _setup(self, container_name, container_type, profile):
-        """
-        Launch a container, enable ssh, install dependencies
-        and setup the needed environment variables
-        """
-        self.launch(container_name)
-        self.enable_ssh(container_name, container_type, profile)
-        self.install(
-            container_name,
-            packages=[PROFILES[profile][container_type]["package_name"]],
-            channel=PROFILES[profile][container_type]["channel"])
-        self.install_dependencies(container_name, container_type)
-        self.configure(container_name, container_type, profile)
-
-    def get_ip(self, container_name):
-        """Gets the ip address for a given container"""
-        networks = self.client.containers.get(
-            container_name).state().network['eth0']['addresses']
-
-        for network in networks:
-            if network['address']:
-                return network['address']
-        raise RuntimeError(
-            "The container {} does not have an IPV4 connection".format(
-                container_name))
+        containers = self.driver.launch(CONTAINERS)
+
+        for cont in containers:
+            deps = DEPENDENCIES[cont.name] + DEPENDENCIES["shared"]
+            self.driver.install(cont, packages=deps)
+            self.driver.install(
+                cont,
+                packages=[PROFILES[profile][cont.name]["package_name"]],
+                channel=PROFILES[profile][cont.name]["channel"])
+            self.driver.enable_ssh(cont)
+            self.driver.export_environment_variables(
+                cont,
+                PROFILES[profile][cont.name]["environment_variables"].keys())

=== modified file 'snappy_ecosystem_tests/environment/setup.py'
--- snappy_ecosystem_tests/environment/setup.py	2017-03-03 18:58:19 +0000
+++ snappy_ecosystem_tests/environment/setup.py	2017-03-08 15:34:35 +0000
@@ -1,7 +1,7 @@
 # -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
 
 #
-# Ubuntu System Tests
+# Snappy Ecosystem Tests
 # Copyright (C) 2017 Canonical
 #
 # This program is free software: you can redistribute it and/or modify
@@ -24,11 +24,26 @@
 
 import logging
 
-from snappy_ecosystem_tests.environment.managers import get_manager
+from snappy_ecosystem_tests.environment.managers import _LxdManager
 
 logging.basicConfig(level=logging.DEBUG)
 
 
+SUPPORTED_MODULES = {
+    "lxd": _LxdManager
+}
+
+
+def get_manager(mode):
+    """Get the manager for the given mode"""
+
+    try:
+        manager = SUPPORTED_MODULES[mode]
+        return manager()
+    except KeyError:
+        raise RuntimeError("Mode: {} is not supported".format(mode))
+
+
 def main(mode, profile):
     """Main script"""
     manager = get_manager(mode)

=== modified file 'snappy_ecosystem_tests/utils/user.py'
--- snappy_ecosystem_tests/utils/user.py	2017-03-01 21:51:29 +0000
+++ snappy_ecosystem_tests/utils/user.py	2017-03-08 15:34:35 +0000
@@ -23,7 +23,7 @@
 import os
 
 from snappy_ecosystem_tests.commons.config import USER_CONFIG_STACK
-from snappy_ecosystem_tests.environment.managers import get_manager
+from snappy_ecosystem_tests.environment.setup import get_manager
 
 MANAGER = get_manager(os.environ['target'])