Merge lp:~canonical-platform-qa/snappy-ecosystem-tests/build-snap into lp:snappy-ecosystem-tests

[Omer Akram]

Build a snap and pull it

We have many test cases that upload a new snap to the store, or upload a new revision of an already existing snap, we could ship snaps in our test suite but I am not sure how will that play out if we have to register and upload a new snap for a random new name. So in such cases we just quickly build a new snap in an isolated environment.

[lp:~canonical-platform-qa/snappy-ecosystem-tests/build-snap]

31. By Omer Akram on 2017-02-28

tweak existing

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:31
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/212/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2004/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/212/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/snappy-ecosystem-tests/build-snap]

32. By Omer Akram on 2017-03-01

Fix absolute path

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:32
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/216/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2009/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/216/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/snappy-ecosystem-tests/build-snap]

33. By Omer Akram on 2017-03-01

fix

34. By Omer Akram on 2017-03-01

some cleanup; also require less parameters to build

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:34
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/218/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2011/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/218/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/snappy-ecosystem-tests/build-snap]

35. By Omer Akram on 2017-03-01

extend paramiko client so that we don't have to provide credentials for every call

[platform-qa-bot (platform-qa-bot)]

FAILED: Continuous integration, rev:35
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/219/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2012/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/219/rebuild

review: Needs Fixing (continuous-integration)

[lp:~canonical-platform-qa/snappy-ecosystem-tests/build-snap]

36. By Omer Akram on 2017-03-01

push new revision for kicks

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:36
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/220/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2013/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/220/rebuild

review: Approve (continuous-integration)

[Santiago Baldassin (sbaldassin)]

Looks good. Minor comments/question inline

review: Needs Information

[lp:~canonical-platform-qa/snappy-ecosystem-tests/build-snap]

37. By Omer Akram on 2017-03-01

Make changes per questions

[Omer Akram (om26er)]

replied.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:37
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/224/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2018/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/224/rebuild

review: Approve (continuous-integration)

[Santiago Baldassin (sbaldassin)]

Code looks good to me

review: Approve

[Heber Parrucci (heber013)]

Just minor comments inline.

review: Needs Fixing

[lp:~canonical-platform-qa/snappy-ecosystem-tests/build-snap]

38. By Omer Akram on 2017-03-02

Add detailed docstrings

[Omer Akram (om26er)]

Replied inline and also updated docstrings.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:38
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/232/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2035/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/snappy-ecosystem-tests-ci/232/rebuild

review: Approve (continuous-integration)

[Heber Parrucci (heber013)]

Code LGTM

review: Approve

=== added file 'snappy_ecosystem_tests/helpers/snapcraft/build_snap.py'
--- snappy_ecosystem_tests/helpers/snapcraft/build_snap.py	1970-01-01 00:00:00 +0000
+++ snappy_ecosystem_tests/helpers/snapcraft/build_snap.py	2017-03-02 16:25:26 +0000
@@ -0,0 +1,140 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+
+#
+# Snappy Ecosystem Tests
+# Copyright (C) 2017 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+"""Module to build a snap in a remote machine."""
+
+import os
+from pathlib import PurePath
+
+from snappy_ecosystem_tests.utils.ssh import SSHManager
+
+COMMANDS_SETUP = [
+    'apt update',
+    'apt dist-upgrade -y',
+    'apt install snapcraft -y'
+]
+SNAPCRAFT_YAML = """\
+name: {snap_name}
+version: {version}
+summary: {summary}
+description: |
+  {description}
+
+grade: {grade}
+confinement: {confinement}
+
+parts:
+  {snap_name}:
+    plugin: nil
+
+apps:
+  {snap_name}:
+    command: echo ok
+"""
+
+
+class SnapBuilder:
+    """Class to build a snap on a remote machine and download its content."""
+    def __init__(self, hostname, username, port=22):
+        self.client = SSHManager.get_instance(hostname, username, port)
+
+    def is_host_setup(self):
+        """Return bool representing whether the remote host is setup by
+        checking if snapcraft package is installed."""
+        try:
+            self.client.exec_command('which snapcraft')
+            return True
+        except ValueError:
+            return False
+
+    def setup_host(self):
+        """Setup the host to build a snap, currently only installs
+        snapcraft."""
+        for command in COMMANDS_SETUP:
+            self.client.exec_command(command)
+
+    def build(self, name, version, summary, description, grade='stable',
+              confinement='strict'):
+        """Build a new snap in the remote host and return its remote path.
+
+        :param name: name of the snap to build.
+        :param version: version number for the snap to build.
+        :param summary: short summary of the snap to build.
+        :param description: long description of the snap.
+        :param grade: stability grade of the snap e.g. stable or devel.
+        :param confinement: confinement policy for the snap e.g.
+            classic, devmode or strict.
+        :raises ValueError: If an error is occurred during snap build.
+        :return: Remote path of the newly build snap package.
+        """
+        if not self.is_host_setup():
+            self.setup_host()
+        tempdir = self.client.exec_command('mktemp -d')
+        self.client.exec_command('mkdir snap', cwd=tempdir)
+        self.client.exec_command('echo "{}" > snap/snapcraft.yaml'.format(
+            SNAPCRAFT_YAML.format(
+                snap_name=name,
+                version=version,
+                summary=summary,
+                description=description,
+                grade=grade,
+                confinement=confinement
+            )
+        ), cwd=tempdir)
+        self.client.exec_command('snapcraft', cwd=tempdir)
+        return os.path.join(
+            tempdir,
+            '{name}_{version}_{arch}.snap'.format(
+                name=name,
+                version=version,
+                arch=self.client.exec_command('dpkg --print-architecture')
+            )
+        )
+
+    def build_and_pull(self, name, version, summary, description,
+                       grade='stable', confinement='strict',
+                       output_location='.'):
+        """Build a snap based on the provided arguments and download it.
+
+        :param name: name of the snap to build.
+        :param version: version number for the snap to build.
+        :param summary: short summary of the snap to build.
+        :param description: long description of the snap.
+        :param grade: stability grade of the snap e.g. stable or devel.
+        :param confinement: confinement policy for the snap e.g.
+            classic, devmode or strict.
+        :param output_location: Local location to download the newly built
+            snap to. Defaults to pwd.
+        :raises ValueError: If an error is occurred during snap build.
+        :return: Local path of the newly built and downloaded snap package.
+        """
+        remote_snap_path = self.build(
+            name=name,
+            version=version,
+            summary=summary,
+            description=description,
+            grade=grade,
+            confinement=confinement
+        )
+        output_path_absolute = os.path.abspath(
+            os.path.join(output_location, PurePath(remote_snap_path).name)
+        )
+        self.client.pull(remote_snap_path, output_path_absolute)
+        return output_path_absolute

=== modified file 'snappy_ecosystem_tests/helpers/snapd/snapd.py'
--- snappy_ecosystem_tests/helpers/snapd/snapd.py	2017-02-21 15:17:12 +0000
+++ snappy_ecosystem_tests/helpers/snapd/snapd.py	2017-03-02 16:25:26 +0000
@@ -56,10 +56,7 @@
         should run.
     :returns: stdout of the command
     """
-    if cwd:
-        return ssh.run_command(
-            'cd {}; {} {}'.format(cwd, PATH_SNAP, parameters))
-    return ssh.run_command('{} {}'.format(PATH_SNAP, parameters))
+    return ssh.run_command('{} {}'.format(PATH_SNAP, parameters), cwd=cwd)
 
 
 def login(email, password):

=== modified file 'snappy_ecosystem_tests/utils/ssh.py'
--- snappy_ecosystem_tests/utils/ssh.py	2017-02-22 18:56:04 +0000
+++ snappy_ecosystem_tests/utils/ssh.py	2017-03-02 16:25:26 +0000
@@ -20,10 +20,39 @@
 
 """Module to connect to ssh client for running tests."""
 
+import logging
 import paramiko
 
 from snappy_ecosystem_tests.utils.user import get_remote_host_credentials
 
+LOGGER = logging.getLogger(__name__)
+
+
+class SSHClient(paramiko.SSHClient):
+    """Extended SSHClient."""
+
+    def exec_command(self, command, bufsize=-1, timeout=None, get_pty=False,
+                     cwd=''):
+        """Execute the given command over ssh."""
+        if cwd:
+            command = 'cd {}; {}'.format(cwd, command)
+        _, stdout, stderr = super().exec_command(command, bufsize, timeout,
+                                                 get_pty)
+        if stdout.channel.recv_exit_status() != 0:
+            raise ValueError(stderr.read().decode().strip())
+        response = stdout.read().decode().strip()
+        LOGGER.info(response)
+        return response
+
+    def pull(self, remote_path, local_path):
+        """Pull file/directory from a remote location.
+
+        :param remote_path: Path on the remote location.
+        :param local_path: Path to download to.
+        """
+        sftp = self.open_sftp()
+        sftp.get(remote_path, local_path)
+
 
 class SSHManager:
     """Manager class to keep a pool of ssh connections."""
@@ -46,7 +75,7 @@
                 client.connect(
                     hostname, username=username, port=port, **kwargs)
         else:
-            client = paramiko.SSHClient()
+            client = SSHClient()
             client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
             client.connect(hostname, username=username, port=port, **kwargs)
             SSHManager.__connection_pool.append(client)
@@ -73,10 +102,11 @@
         return connections[0] if connections else None
 
 
-def run_command(command, hostname=None, username=None, port=None):
+def run_command(command, cwd=None, hostname=None, username=None, port=None):
     """Run the given command on remote machine over ssh.
 
     :param command: a string of the command to run.
+    :param cwd: Current working directory for the command.
     :param hostname: The host to run command on.
     :param username: Name of the user on the remote host to login to.
     :param port: SSH port number.
@@ -86,7 +116,4 @@
     _hostname, _username, _port = get_remote_host_credentials()
     ssh = SSHManager.get_instance(
         hostname or _hostname, username or _username, port or _port)
-    _, stdout, stderr = ssh.exec_command(command)
-    if stdout.channel.recv_exit_status() != 0:
-        raise ValueError(stderr.read().decode())
-    return stdout.read().decode()
+    return ssh.exec_command(command, cwd=cwd)