Merge lp:~canonical-platform-qa/ubuntu-system-tests/snap-channel-support into lp:ubuntu-system-tests

[Richard Huddie]

Add support for installing snaps from beta and edge channels.

Changes include:
- Check the beta channel first and use this version, or use edge channel if beta is not found.
- Update target_setup.py to do this during snap install
- Update Target class to do this during test using --upgrade option
- Get the core series and architecture from target rather than using parameters
- Update command helpers to check for snap channel revisions when creating --setup-commands for upgrade
- Update the Terminal app classes to work with both deb and snap versions as snap version would be required when testing on snap based image.
- Update self tests and add new ones for checking snap channel

This should be landed in parallel with: https://code.launchpad.net/~canonical-platform-qa/qa-jenkins-jobs/snap-channel-support/+merge/320420

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:523
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/1189/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2167/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/1189/rebuild

review: Approve (continuous-integration)

[lp:~canonical-platform-qa/ubuntu-system-tests/snap-channel-support]

524. By Richard Huddie on 2017-03-30

Merge from trunk.

[platform-qa-bot (platform-qa-bot)]

PASSED: Continuous integration, rev:524
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/1232/
Executed test runs:
    None: https://platform-qa-jenkins.ubuntu.com/job/generic-update-mp/2251/console

Click here to trigger a rebuild:
https://platform-qa-jenkins.ubuntu.com/job/ubuntu-system-tests-ci/1232/rebuild

review: Approve (continuous-integration)

[Santiago Baldassin (sbaldassin)]

Looks good in general. I just have a couple of questions:

1. Why is beta the default channel? Once a "snap" silo is ready for QA, will the snap in question be automatically available in the beta channel?

2. I'm not sure yet about the workflow with Bileto but if the app is not available in the beta channel, I would just make the setup to fail. Installing the snap from --edge could lead us to test the wrong version of the app. right?

For example, let's say that there's a new silo for the calc snap v1.2, stable has 1.0 and edge has 1.1, if 1.2 is not uploaded to --beta, then we would end up executing the test cases against 1.1 and approving the 1.2 silo based on the 1.1 results

Thoughts?

review: Needs Information

[Richard Huddie (rhuddie)]

> 1. Why is beta the default channel? Once a "snap" silo is ready for QA, will
> the snap in question be automatically available in the beta channel?

This change was requested due to the poor state of snap testing, so we could use more stable versions, rather than always taking the latest development version from edge which might not be fit for testing.

This doesn't take into account silos, I'm not sure how that process works for snaps. It would be possible to add a command option to force it to take a specific channel/revision for specific cases.

>
> 2. I'm not sure yet about the workflow with Bileto but if the app is not
> available in the beta channel, I would just make the setup to fail. Installing
> the snap from --edge could lead us to test the wrong version of the app.
> right?
>
> For example, let's say that there's a new silo for the calc snap v1.2, stable
> has 1.0 and edge has 1.1, if 1.2 is not uploaded to --beta, then we would end
> up executing the test cases against 1.1 and approving the 1.2 silo based on
> the 1.1 results
>
> Thoughts?

I'm also unsure how that process would work. But we could easily add some command options to force a specific channel and revision and fail if it can't find it.

These changes would also be related to the other app changes we're discussing about only having 1 version installed, so I think it would be better to look at adding these options again when we have that sorted.

[Santiago Baldassin (sbaldassin)]

Looks good

review: Approve

=== modified file 'ubuntu_system_tests/helpers/terminal/app.py'
--- ubuntu_system_tests/helpers/terminal/app.py	2017-03-13 14:28:26 +0000
+++ ubuntu_system_tests/helpers/terminal/app.py	2017-03-30 18:35:16 +0000
@@ -18,18 +18,50 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-from ubuntu_system_tests.helpers.application import Deb
+from ubuntu_system_tests.helpers import autopilot
+from ubuntu_system_tests.helpers.application import (
+    Deb,
+    Snap,
+)
+from ubuntu_system_tests.helpers.processes import get_process_id
 
+APP = 'terminal'
 APP_NAME = 'Terminal'
 APP_ID = 'com.ubuntu.terminal'
 APP_PACKAGE_ID = 'ubuntu-terminal-app'
 
 
-class Terminal(Deb):
+class TerminalDeb(Deb):
 
     def __init__(self):
         super().__init__(APP_NAME, APP_ID, APP_PACKAGE_ID)
 
     def get_main_view(self):
-        from ubuntu_system_tests.helpers.terminal import cpo  # NOQA
-        return self.get_proxy_object().main_view
+        return get_main_view()
+
+
+class TerminalSnap(Snap):
+
+    def __init__(self):
+        super().__init__(APP_NAME, APP_ID)
+
+    def get_main_view(self):
+        return get_main_view()
+
+
+def get_main_view():
+    from ubuntu_system_tests.helpers.terminal import cpo  # NOQA
+    pid = get_process_id(APP)
+    return autopilot.get_proxy_object(pid=pid).main_view
+
+
+def get_terminal_app(mode):
+    """Return terminal application class of required type.
+    :param mode: Either 'snap' or 'deb'.
+    :return: Either TerminalDeb or TerminalSnap class reference.
+    """
+    app_modes = {
+        'deb': TerminalDeb,
+        'snap': TerminalSnap
+    }
+    return app_modes[mode]

=== modified file 'ubuntu_system_tests/host/commands.py'
--- ubuntu_system_tests/host/commands.py	2017-03-09 17:21:07 +0000
+++ ubuntu_system_tests/host/commands.py	2017-03-30 18:35:16 +0000
@@ -19,6 +19,7 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
+import requests
 import subprocess
 
 from ubuntu_system_tests.common import (
@@ -44,6 +45,23 @@
 legacy_cmds = None
 
 XENIAL = '16.04'
+SNAP_STORE_QUERY_URL = (
+    'https://search.apps.ubuntu.com/api/v1/snaps/details/{n}?channel={c}'
+)
+# List of apps to be installed as snaps on a deb based session.
+# Anything else will be installed as a deb.
+# This is used for upgrade command where a list of packages is
+# specified which must be upgraded before running a test.
+SNAP_APPS = [
+    'address-book-app',
+    'camera-app',
+    'gallery-app',
+    'ubuntu-calculator-app',
+    'ubuntu-calendar-app',
+    'ubuntu-clock-app',
+    'ubuntu-filemanager-app',
+    'webbrowser-app',
+]
 
 
 def use_legacy_commands():
@@ -158,20 +176,48 @@
 def _get_upgrade_setup_command(target, upgrade):
     """Return setup command to upgrade csv list of components."""
     if upgrade:
+        upgrade_list = upgrade.split(',')
+        cmd = ''
+        series = target.core_series()
+        arch = target.architecture()
         if target.snap_session():
-            cmd = ''
-            for name in upgrade.split(','):
-                cmd += 'snap refresh --devmode --edge {}; '.format(name)
-            cmd = cmd.strip()
+            # For a snap session all apps must be snaps
+            for name in upgrade_list:
+                cmd += _get_snap_channel_upgrade_command(series, arch, name)
         else:
-            upgrade = upgrade.replace(',', ' ')
-            cmd = (
-                'apt-get -y --no-install-recommends install '
-                '{}'.format(upgrade))
-        return ['--setup-commands', _get_fs_rw_command(target) + cmd]
+            # For a deb session, the apps could be either deb or snap
+            for name in upgrade_list:
+                if name in SNAP_APPS:
+                    cmd += _get_snap_channel_upgrade_command(
+                        series, arch, name)
+                else:
+                    cmd += _get_deb_upgrade_command(name)
+        return ['--setup-commands', _get_fs_rw_command(target) + cmd.strip()]
     return []
 
 
+def _get_snap_channel_upgrade_command(series, arch, snap_name):
+    for channel in ['beta', 'edge']:
+        if _is_snap_published_for_channel(series, arch, snap_name, channel):
+            return _get_snap_upgrade_command(snap_name, channel)
+    raise RuntimeError('No channel found for {}.'.format(snap_name))
+
+
+def _get_snap_upgrade_command(app, channel):
+    return 'snap refresh --devmode --{c} {a}; '.format(c=channel, a=app)
+
+
+def _get_deb_upgrade_command(app):
+    return 'apt-get -y --no-install-recommends install {}; '.format(app)
+
+
+def _is_snap_published_for_channel(series, arch, snap, channel):
+    headers = {'X-Ubuntu-Series': series, 'X-Ubuntu-Architecture': arch}
+    url = SNAP_STORE_QUERY_URL.format(n=snap, c=channel)
+    response = requests.get(url, headers=headers)
+    return response.ok
+
+
 def _get_fs_ro_setup_command(target):
     """Return setup command to set file system read-only."""
     if target.config.get('mount_fs_ro', False):

=== modified file 'ubuntu_system_tests/host/target_setup.py'
--- ubuntu_system_tests/host/target_setup.py	2017-03-13 14:28:26 +0000
+++ ubuntu_system_tests/host/target_setup.py	2017-03-30 18:35:16 +0000
@@ -24,7 +24,9 @@
 import dbus
 import json
 import os
+import platform
 import pwd
+import requests
 import shutil
 import subprocess
 import sys
@@ -48,6 +50,10 @@
 XENIAL = 16.04
 YAKKETY = 16.10
 
+SNAP_STORE_QUERY_URL = (
+    'https://search.apps.ubuntu.com/api/v1/snaps/details/{n}?channel={c}'
+)
+
 
 class SetupRunner:
 
@@ -57,6 +63,8 @@
         self.config = config
         self.series = None
         self.release = None
+        self.core_series = None
+        self.arch = None
         self._set_cache_dir()
 
     def _set_cache_dir(self):
@@ -95,6 +103,23 @@
                 ['lsb_release', '-cs']).decode().strip()
         return self.series
 
+    def get_core_series(self):
+        """Return the Ubuntu Core series."""
+        if self.core_series is None:
+            self.core_series = subprocess.check_output(
+                'snap version | grep series | tr -s " " | cut -d" " -f2',
+                shell=True).decode().strip()
+        return self.core_series
+
+    def get_architecture(self):
+        """Return the current architecture."""
+        if self.arch is None:
+            if platform.architecture()[0] == '64bit':
+                self.arch = 'amd64'
+            else:
+                self.arch = 'i386'
+        return self.arch
+
     def update_apt(self):
         """Return command to apt-get update."""
         if self.config['update_apt']:
@@ -182,9 +207,26 @@
         if snaps:
             self.mount_fs_rw()
             for snap in snaps:
+                installed = False
                 cmd = 'refresh' if self.is_snap_installed(snap) else 'install'
-                subprocess.check_call(
-                    ['snap', cmd, '--devmode', '--edge', snap])
+                for channel in ['beta', 'edge']:
+                    if self.is_snap_published_for_channel(snap, channel):
+                        channel_arg = '--{}'.format(channel)
+                        subprocess.check_call(
+                            ['snap', cmd, '--devmode', channel_arg, snap])
+                        installed = True
+                        break
+                if not installed:
+                    raise RuntimeError(
+                        'Could not find channel for snap {}'.format(snap))
+
+    def is_snap_published_for_channel(self, snap, channel):
+        arch = self.get_architecture()
+        series = self.get_core_series()
+        headers = {'X-Ubuntu-Series': series, 'X-Ubuntu-Architecture': arch}
+        url = SNAP_STORE_QUERY_URL.format(n=snap, c=channel)
+        response = requests.get(url, headers=headers)
+        return response.ok
 
     def is_snap_installed(self, snap):
         return os.path.isdir(os.path.join('/snap', snap))

=== modified file 'ubuntu_system_tests/host/targets.py'
--- ubuntu_system_tests/host/targets.py	2017-03-07 16:30:47 +0000
+++ ubuntu_system_tests/host/targets.py	2017-03-30 18:35:16 +0000
@@ -131,6 +131,17 @@
     def release(self):
         return self.run('lsb_release -rs', log_stdout=False).output.strip()
 
+    def core_series(self):
+        return self.run(
+            'snap version | grep series | tr -s " " | cut -d" " -f2 ',
+            log_stdout=False).output.strip()
+
+    def architecture(self):
+        arch = self.run('uname --machine', log_stdout=False).output.strip()
+        if arch == 'x86_64':
+            return 'amd64'
+        return 'i386'
+
     def user_id(self, user):
         """Return user id for specified user name."""
         return self.run(

=== modified file 'ubuntu_system_tests/selftests/test_commands.py'
--- ubuntu_system_tests/selftests/test_commands.py	2017-02-24 14:22:54 +0000
+++ ubuntu_system_tests/selftests/test_commands.py	2017-03-30 18:35:16 +0000
@@ -228,12 +228,20 @@
 class MockTarget:
 
     def __init__(self, **kwargs):
+        self._core_series = '16'
+        self._architecture = 'amd64'
         for arg in kwargs.keys():
             setattr(self, arg, kwargs[arg])
 
     def snap_session(self):
         return self.snap
 
+    def core_series(self):
+        return self._core_series
+
+    def architecture(self):
+        return self._architecture
+
 
 class TestUpgradeSetupCommands(ConfigBaseTestCase):
 
@@ -241,14 +249,20 @@
         super().setUp()
         self.args = DummyArgs(upgrade='package1,package2')
 
-    def test_get_upgrade_setup_command_snap(self):
+    @mock.patch('ubuntu_system_tests.host.commands.'
+                '_is_snap_published_for_channel',
+                return_value=True)
+    def test_get_upgrade_setup_command_snap(self, mock_is_snap_published):
         target = MockTarget(snap=True, config={'mount_fs_rw': False})
         cmd = commands._get_upgrade_setup_command(target, self.args.upgrade)
+        mock_is_snap_published.assert_has_calls([
+            mock.call('16', 'amd64', 'package1', 'beta'),
+            mock.call('16', 'amd64', 'package2', 'beta')])
         self.assertEqual(
             cmd,
             ['--setup-commands',
-             'snap refresh --devmode --edge package1; '
-             'snap refresh --devmode --edge package2;'])
+             'snap refresh --devmode --beta package1; '
+             'snap refresh --devmode --beta package2;'])
 
     def test_get_upgrade_setup_command_deb(self):
         target = MockTarget(snap=False, config={'mount_fs_rw': False})
@@ -256,9 +270,51 @@
         self.assertEqual(
             cmd,
             ['--setup-commands',
-             'apt-get -y --no-install-recommends install package1 package2'])
+             'apt-get -y --no-install-recommends install package1; '
+             'apt-get -y --no-install-recommends install package2;'])
 
     def test_get_upgrade_setup_includes_fs_rw(self):
         target = MockTarget(snap=False, config={'mount_fs_rw': True})
         cmd = commands._get_upgrade_setup_command(target, self.args.upgrade)
         self.assertIn('mount -o remount,rw /', cmd[1])
+
+
+class TestSnapUpgradeChannelSelection(ConfigBaseTestCase):
+
+    @mock.patch('ubuntu_system_tests.host.commands.'
+                '_is_snap_published_for_channel',
+                return_value=True)
+    def test_beta_channel_first_priority(self, mock_is_snap_published):
+        target = MockTarget(snap=True, config={'mount_fs_rw': False})
+        cmd = commands._get_upgrade_setup_command(target, 'pkg1')
+        self.assertEqual(
+            cmd,
+            ['--setup-commands',
+             'snap refresh --devmode --beta pkg1;'])
+        mock_is_snap_published.assert_has_calls([
+            mock.call('16', 'amd64', 'pkg1', 'beta')])
+
+    @mock.patch('ubuntu_system_tests.host.commands.'
+                '_is_snap_published_for_channel',
+                side_effect=[False, True])
+    def test_edge_channel_second_priority(self, mock_is_snap_published):
+        target = MockTarget(snap=True, config={'mount_fs_rw': False})
+        cmd = commands._get_upgrade_setup_command(target, 'pkg1')
+        self.assertEqual(
+            cmd,
+            ['--setup-commands',
+             'snap refresh --devmode --edge pkg1;'])
+        mock_is_snap_published.assert_has_calls([
+            mock.call('16', 'amd64', 'pkg1', 'beta'),
+            mock.call('16', 'amd64', 'pkg1', 'edge')])
+
+    @mock.patch('ubuntu_system_tests.host.commands.'
+                '_is_snap_published_for_channel',
+                side_effect=[False, False])
+    def test_raises_on_no_channel_found(self, mock_is_snap_published):
+        target = MockTarget(snap=True, config={'mount_fs_rw': False})
+        with self.assertRaises(RuntimeError):
+            commands._get_upgrade_setup_command(target, 'pkg1')
+        mock_is_snap_published.assert_has_calls([
+            mock.call('16', 'amd64', 'pkg1', 'beta'),
+            mock.call('16', 'amd64', 'pkg1', 'edge')])

=== modified file 'ubuntu_system_tests/tests/test_launch_apps.py'
--- ubuntu_system_tests/tests/test_launch_apps.py	2017-03-14 17:05:13 +0000
+++ ubuntu_system_tests/tests/test_launch_apps.py	2017-03-30 18:35:16 +0000
@@ -31,7 +31,9 @@
 from ubuntu_system_tests.helpers.clock.app import Clock
 from ubuntu_system_tests.helpers.filemanager.app import FileManager
 from ubuntu_system_tests.helpers.gallery.app import Gallery
-from ubuntu_system_tests.helpers.terminal.app import Terminal
+from ubuntu_system_tests.helpers.terminal.app import (
+    get_terminal_app,
+)
 from ubuntu_system_tests.helpers.system_settings.app import (
     get_system_settings_app,
 )
@@ -96,7 +98,8 @@
         self.check_app_launch(FileManager, 1)
 
     def test_launch_terminal_app(self):
-        self.check_app_launch(Terminal, 1)
+        app = get_terminal_app(self.config_stack.get('unity8_mode'))
+        self.check_app_launch(app, 1)
 
     def test_launch_webbrowser_app(self):
         self.check_app_launch(WebBrowser, 1)
@@ -131,7 +134,8 @@
         self.check_app_launch(FileManager, 2)
 
     def test_launch_terminal_app_twice(self):
-        self.check_app_launch(Terminal, 2)
+        app = get_terminal_app(self.config_stack.get('unity8_mode'))
+        self.check_app_launch(app, 2)
 
     def test_launch_webbrowser_app_twice(self):
         self.check_app_launch(WebBrowser, 2)