Merge lp:~canonical-platform-qa/ubuntu-ota-tests/upgrade into lp:ubuntu-ota-tests

[Leo Arias]

Added an initial upgrade test.

Flash with:
ubuntu-device-flash --revision=-1 touch --developer-mode --password 0000 --channel="ubuntu-touch/devel-proposed" --wipe

Run with:
adt-run -d -B --unbuilt-tree=. --output-dir /tmp/output --- ssh -s ./adb-reboot-to-recovery

The adb-reboot-to-recovery script is copied from /usr/share/autopkgtest/ssh-setup/adb, which a change to use a reboot target. Once we send this upstream and a new version is release, we can remove the file.

[Leo Arias (elopio)]

I've been playing with this branch for a long time, so I think now it's good to give it a review. I think it achieves the purpose of investigating adb-reboot, and it actually has a test to proof it.
I couldn't get the dbus upgrade to work, and to finish that I need some of the other branches. So for now I removed the dbus code and left a cli upgrade. I will move the dbus code I had to the other reboot or upgrade card.
This works with the currently released version of system-image-cli. It doesn't work with the one from barry's PPA. But we have to use the new version with the dbus upgrade, so it will be done on one of those cards too.

[Christopher Lee (veebers)]

Just a couple of comments re: docs and logging. The code looks really good.
A note that we really need to get adb-reboot-to-recovery upstreamed so we're not hosting a fork.

I attempted to run the code but experienced what looked like a timeout error (Potentially due to the internet connection issues I currently have, will try again later tonight.)

If other people can successfully run this code then I would say this would be an approve (if the mentioned documentation is added) as we can iterate on the failure issue.

review: Needs Fixing

[Christopher Lee (veebers)]

Running it again with decent internet passes. I say we tackle this in a different branch (if we tackle it at all).

[Federico Gimenez (fgimenez)]

Worked just fine, +1

review: Approve

[Leo Arias (elopio)]

Thanks for the reviews. I added the comments and logging.

[Christopher Lee (veebers)]

LGTM

review: Approve

[PS Jenkins bot (ps-jenkins)]

FAILED: Autolanding.
No commit message was specified in the merge proposal. Hit 'Add commit message' on the merge proposal web page or follow the link below. You can approve the merge proposal yourself to rerun.
https://code.launchpad.net/~canonical-platform-qa/ubuntu-ota-tests/upgrade/+merge/252372/+edit-commit-message

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

review: Approve (continuous-integration)

=== added file 'adb-reboot-to-recovery'
--- adb-reboot-to-recovery	1970-01-01 00:00:00 +0000
+++ adb-reboot-to-recovery	2015-03-12 16:11:21 +0000
@@ -0,0 +1,337 @@
+#!/bin/sh
+
+# adb.ssh is part of autopkgtest
+# autopkgtest is a tool for testing Debian binary packages
+#
+# This script sets up an ssh connection to an adb host. If that adb host is an
+# Ubuntu Touch system, it also does some extra configuration like disabling the
+# screen timeout and allowing Autopilot to introspect running apps.
+#
+# Options:
+# -r/--reset  Do a factory reset of the device before running the test
+#             (Available on Ubuntu Phone only; disabled for now as it does not
+#             re-enable developer mode)
+# -b/--reboot Reboot the device before running the test
+# -s serial | --serial=serial
+#             Serial  ID  of  the device as returned by adb devices -l when
+#             several devices are connected to the same host.
+# -p PASSWORD | --password PASSWORD
+#             sudo password; if not given, tries "phablet" and "0000"; if
+#             neither works, tests cannot run as root
+#
+# autopkgtest is Copyright (C) 2006-2014 Canonical Ltd.
+#
+# Author: Jean-Baptiste Lallement <[e-mail]>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+# See the file CREDITS for a full list of credits information (often
+# installed as /usr/share/doc/autopkgtest/CREDITS).
+set -e
+
+SSH_USER=phablet
+SUDO_PASSWORD=
+CAPABILITIES='isolation-machine reboot'
+
+# allow using lp:phablet-tools from a checkout
+if [ -n "$PHABLET_TOOLS_PATH" ]; then
+    export PATH="$PHABLET_TOOLS_PATH:$PATH"
+fi
+
+# argument parsing
+ADBOPTS=""
+RESET=""
+REBOOT=""
+IDENTITY=
+
+open() {
+    # Setup a connection to an adb device
+    # - Configure ssh connection
+    # - optionally mount device RW
+    wait_booted
+
+    if [ -n "$REBOOT" ]; then
+        adb $ADBOPTS reboot
+        wait_booted
+    fi
+
+    # special setup on Ubuntu images
+    if [ -n "$(adb $ADBOPTS shell 'type unity8 2>/dev/null')" ]; then
+        if [ -n "$RESET" ]; then
+            revert
+        else
+            ubuntu_prepare_config
+            ubuntu_prepare_for_testing
+        fi
+        #CAPABILITIES="$CAPABILITIES revert"
+    fi
+
+
+    # Configure SSH
+    adb $ADBOPTS shell 'gdbus call -y -d com.canonical.PropertyService -o /com/canonical/PropertyService -m com.canonical.PropertyService.SetProperty ssh true >/dev/null'
+    for port in `seq 2222 2299`; do
+        adb $ADBOPTS forward tcp:$port tcp:22 && break
+    done
+
+    # Purge the device host key so that SSH doesn't print a scary warning about it
+    # (it changes every time the device is reflashed and this is expected)
+    ssh-keygen -f ~/.ssh/known_hosts -R [localhost]:$PORT 2>/dev/null || true
+
+    # Copy your ssh id down to the device so you never need a password.
+    IDENTITY=$HOME/.ssh/id_rsa
+    if [ ! -e $IDENTITY ]; then
+        IDENTITY=$HOME/.ssh/id_autopkgtest
+        if [ ! -e $IDENTITY ]; then
+            echo "No default ssh key, generating $IDENTITY" >&2
+            ssh-keygen -q -t ed25519 -f $IDENTITY -N ''
+        fi
+    fi
+    script=$(mktemp /tmp/$(basename $0).XXXXXX)
+    adb $ADBOPTS push ${IDENTITY}.pub /home/$SSH_USER/.ssh/authorized_keys
+    cat>$script <<EOF
+# Set right permissions
+chown $SSH_USER:$SSH_USER -R /home/$SSH_USER/.ssh/
+chmod 700 /home/$SSH_USER/.ssh
+chmod 600 /home/$SSH_USER/.ssh/authorized_keys
+EOF
+
+    adb $ADBOPTS push $script /tmp
+    adb $ADBOPTS shell sh $script
+    adb $ADBOPTS shell rm $script
+    rm $script
+
+    # verify that we can connect through ssh
+    local ssh_cmd="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i $IDENTITY -p $port -l $SSH_USER localhost"
+    for retry in `seq 10`; do
+        if $ssh_cmd true; then
+            can_ssh=1
+            break
+        fi
+        sleep 2
+    done
+    if [ -z "$can_ssh" ]; then
+        echo "ssh setup failed, cannot connect" >&2
+        exit 1
+    fi
+
+    # try to auto-detect sudo password from common ones used by CI
+    for p in phablet 0000; do
+        if $ssh_cmd "echo $p | sudo -S true" >/dev/null 2>&1; then
+            SUDO_PASSWORD="$p"
+            break
+        fi
+    done
+
+    # print info for adt-virt-ssh
+    cat<<EOF
+login=$SSH_USER
+hostname=localhost
+port=$port
+capabilities=$CAPABILITIES
+identity=$IDENTITY
+extraopts=--no-reset --fwd-port=$port
+EOF
+    if [ -n "$SUDO_PASSWORD" ]; then
+        echo "password=$SUDO_PASSWORD"
+    fi
+}
+
+wait_booted() {
+    echo "Waiting for device ADB to appear..." >&2
+    if ! timeout 300 adb $ADBOPTS wait-for-device >/dev/null; then
+        echo "ERROR: Timed out waiting for adb device" >&2
+    fi
+}
+
+# configure Ubuntu device for testing
+ubuntu_prepare_config() {
+    if [ -z "$(adb $ADBOPTS shell 'type unity8 2>/dev/null')" ]; then
+        # not an Ubuntu phone
+        return
+    fi
+
+    if ! type phablet-config >/dev/null 2>&1; then
+        echo "ERROR: phablet-config not found! Install phablet-tools package or" >&2
+        echo "bzr branch lp:phablet-tools and run with PHABLET_TOOLS_PATH=<checkout dir>" >&2
+        exit 1
+    fi
+
+    echo "Configuring Ubuntu phone for testing..." >&2
+
+    # disable first-time wizards; these fail due to adb instability from time
+    # to time, so retry
+    for retry in `seq 5`; do
+        if phablet-config $ADBOPTS welcome-wizard --disable >/dev/null; then
+            break
+        else
+            echo "Failed to disable welcome wizard, retrying (attempt $retry)..." >&2
+            sleep 1
+        fi
+    done
+    # this needs accounts-daemon running
+    adb $ADBOPTS shell 'while ! pidof accounts-daemon >/dev/null; do sleep 1; done'
+    for retry in `seq 5`; do
+        if phablet-config $ADBOPTS edges-intro --disable >/dev/null; then
+            break
+        else
+            echo "Failed to disable edges intro, retrying (attempt $retry)..." >&2
+            sleep 1
+        fi
+    done
+
+    # kill an already running welcome wizard
+    adb $ADBOPTS shell 'kill `pidof system-settings-wizard` 2>/dev/null'
+}
+
+# test run time setup for Ubuntu device
+ubuntu_prepare_for_testing() {
+    if [ -z "$(adb $ADBOPTS shell 'type unity8 2>/dev/null')" ]; then
+        # not an Ubuntu phone
+        return
+    fi
+
+    echo "Preparing Ubuntu phone for running tests..." >&2
+
+    echo "Waiting for desktop to boot" >&2
+    local timeout=$(($(date +%s) + 300))
+    while [ "$(date +%s)" -le $timeout ]; do
+        out=$(adb $ADBOPTS shell 'gdbus call --timeout 5 --session --dest com.canonical.UnityGreeter --object-path / --method org.freedesktop.DBus.Properties.Get com.canonical.UnityGreeter IsActive 2>/dev/null')
+        if [ -n "$out" ]; then
+            timeout=0
+            break
+        fi
+        sleep 5
+    done
+    if [ "$timeout" -gt 0 ]; then
+        echo "ERROR: timed out waiting for Unity greeter" >&2
+        exit 1
+    fi
+
+    # disable screen dimming; ugly, but pretty much everything else hangs forever
+    adb $ADBOPTS shell "setsid powerd-cli display </dev/null >/dev/null 2>&1 & disown;
+                        while ! pidof powerd-cli >/dev/null; do sleep 0.1; done"
+
+    # unlock the greeter
+    adb $ADBOPTS shell "gdbus call --session --dest com.canonical.UnityGreeter --object-path / --method com.canonical.UnityGreeter.HideGreeter && echo Greeter unlocked" >/dev/null
+}
+
+revert() {
+    # revert is only offered on Ubuntu images
+    echo "Performing factory reset, this will take a minute..." >&2
+
+    # save current network connections
+    NETCONF=$(phablet-config $ADBOPTS network --read) || true:
+
+    # do factory reset
+    adb $ADBOPTS shell 'gdbus call --system -d com.canonical.SystemImage -o /Service -m com.canonical.SystemImage.FactoryReset'
+    wait_booted
+    adb $ADBOPTS shell 'while ! pidof NetworkManager >/dev/null; do sleep 1; done'
+
+    # restore network connections
+    if [ -n "$NETCONF" ]; then
+        phablet-config $ADBOPTS network --write "$NETCONF"
+    fi
+
+    ubuntu_prepare_config
+    ubuntu_prepare_for_testing
+}
+
+reboot() {
+    if [ -z "$FWD_PORT" ]; then
+        echo "ERROR: Must pass --fwd-port" >&2
+        exit 1
+    fi
+    BOOT_TARGET=$(adb pull /run/adt_reboot_target /dev/stdout) || BOOT_TARGET=""
+    adb $ADBOPTS reboot $BOOT_TARGET
+    wait_booted
+    adb $ADBOPTS forward tcp:$FWD_PORT tcp:22
+    adb $ADBOPTS shell "gdbus call -y -d com.canonical.PropertyService -o /com/canonical/PropertyService -m com.canonical.PropertyService.SetProperty ssh true >/dev/null"
+    ubuntu_prepare_for_testing
+}
+
+cleanup() {
+    adb $ADBOPTS shell pkill powerd-cli
+}
+
+
+#
+# main
+#
+
+# argument parsing
+SHORTOPTS="l:,p:,s:,w,r,b"
+LONGOPTS="login:,password:,serial:,rw,apt-update,reset,reboot,no-reset,fwd-port:"
+
+TEMP=$(getopt -o $SHORTOPTS --long $LONGOPTS -- "$@")
+eval set -- "$TEMP"
+
+while true; do
+    case "$1" in
+        -l|--login)
+            SSH_USER=$2
+            shift 2;;
+        -p|--password)
+            SUDO_PASSWORD="$2"
+            shift 2;;
+        -s|--serial)
+            ADBOPTS="$ADBOPTS -s $2"
+            shift 2;;
+        -b|--reboot)
+            REBOOT="1"
+            shift;;
+        -r|--reset)
+            echo "--reset is currently broken as it does not restore PIN/developer mode. Use -b/--reboot instead." >&2
+            exit 1
+            RESET="1"
+            shift;;
+        # passed in "extraopts" so that --reset is only applied once, not
+        # in between tests that call "revert"
+        --no-reset)
+            RESET=""
+            shift;;
+        --fwd-port)
+            FWD_PORT="$2"
+            shift 2;;
+        --)
+            shift;
+            break;;
+        *)
+            echo "E: $(basename $0): Unsupported option $1" >&2
+            exit 1;;
+    esac
+done
+
+if [ -z "$1" ]; then
+    echo "Needs to be called with command as first argument" >&2
+    exit 1
+fi
+
+cmd=$(echo "$1"|tr [[:upper:]] [[:lower:]])
+shift
+
+case $cmd in
+    open)
+        open;;
+    revert)
+        revert;;
+    reboot)
+        reboot;;
+    cleanup)
+        cleanup;;
+    *)
+        echo "invalid command $cmd" >&2
+        exit 1
+        ;;
+esac

=== modified file 'debian/tests/control'
--- debian/tests/control	2015-03-12 14:57:16 +0000
+++ debian/tests/control	2015-03-12 16:11:21 +0000
@@ -3,4 +3,10 @@
 Depends: python3-autopilot,
          python3-coverage,
          python3-psutil,
+         system-image-dbus
 
+Test-Command: cd debian/tests; python3 -m unittest ubuntu_ota_tests.tests.test_basic_upgrade
+Restrictions: needs-root allow-stderr
+Depends: python3-autopilot,
+         python3-psutil,
+         system-image-cli
\ No newline at end of file

=== modified file 'debian/tests/ubuntu_ota_tests/__init__.py'
--- debian/tests/ubuntu_ota_tests/__init__.py	2015-03-06 04:26:55 +0000
+++ debian/tests/ubuntu_ota_tests/__init__.py	2015-03-12 16:11:21 +0000
@@ -17,11 +17,15 @@
 #
 
 from ubuntu_ota_tests import (
+    hooks,
     services,
     system,
+    upgrade,
 )
 
 __all__ = [
+    'hooks',
     'services',
     'system',
+    'upgrade',
 ]

=== added file 'debian/tests/ubuntu_ota_tests/hooks.py'
--- debian/tests/ubuntu_ota_tests/hooks.py	1970-01-01 00:00:00 +0000
+++ debian/tests/ubuntu_ota_tests/hooks.py	2015-03-12 16:11:21 +0000
@@ -0,0 +1,60 @@
+#
+# Ubuntu OTA Tests
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import logging
+import subprocess
+
+from systemimage import reboot
+
+
+log = logging.getLogger('systemimage')
+
+
+class ADTRebootToRecovery(reboot.BaseReboot):
+
+    """Hook to be used during system image upgrade to apply the upgrade."""
+
+    def reboot(self):
+        """Reboot the testbed to apply the upgrade.
+
+        This method does not execute the reboot, it tells adt-run that the
+        reboot is needed and adt executes it.
+
+        """
+        self._set_adt_reboot_to_recovery_flag()
+        try:
+            # When running tests with adt-run, we can't just execute a reboot
+            # because it will break the communication with the host. This
+            # script tell adt-run that a reboot is required. The reboot is
+            # executed from the host, saving the state so that the test can
+            # continue when the testbed comes back.
+            autopkgtest_reboot_script = '/tmp/autopkgtest-reboot'
+            # The reboot mark will be set to the $ADT_REBOOT_MARK environment
+            # variable after the machine is rebooted. It can be used in tests
+            # to tell when the reboot happened.
+            reboot_mark = 'rebooted'
+            subprocess.check_output([autopkgtest_reboot_script, reboot_mark])
+        except subprocess.CalledProcessError as error:
+            log.exception('reboot exit status: {}'.format(error.returncode))
+            log.exception('reboot command output: {}'.format(error.output))
+            raise
+
+    def _set_adt_reboot_to_recovery_flag(self):
+        """Set a flag to tell adt that the reboot needs to go into recovery."""
+        with open('/run/adt_reboot_target', 'w') as reboot_target:
+            reboot_target.write('recovery')

=== added directory 'debian/tests/ubuntu_ota_tests/tests'
=== added file 'debian/tests/ubuntu_ota_tests/tests/__init__.py'
=== added file 'debian/tests/ubuntu_ota_tests/tests/test_basic_upgrade.py'
--- debian/tests/ubuntu_ota_tests/tests/test_basic_upgrade.py	1970-01-01 00:00:00 +0000
+++ debian/tests/ubuntu_ota_tests/tests/test_basic_upgrade.py	2015-03-12 16:11:21 +0000
@@ -0,0 +1,34 @@
+#
+# Ubuntu OTA Tests
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import unittest
+
+from ubuntu_ota_tests import upgrade
+
+
+class OTABasicUpgradeTestCase(unittest.TestCase):
+
+    def test_basic_over_the_air_upgrade(self):
+        rebooted_mark = os.environ.get('ADT_REBOOT_MARK', None)
+        if not rebooted_mark:
+            # The upgrade with dbus needs more work, it's getting stuck.
+            # upgrade.upgrade_with_system_image_dbus()
+            upgrade.upgrade_with_system_image_cli()
+        else:
+            print('Rebooted')

=== added file 'debian/tests/ubuntu_ota_tests/tests/test_basic_upgrade.py.THIS'
--- debian/tests/ubuntu_ota_tests/tests/test_basic_upgrade.py.THIS	1970-01-01 00:00:00 +0000
+++ debian/tests/ubuntu_ota_tests/tests/test_basic_upgrade.py.THIS	2015-03-12 16:11:21 +0000
@@ -0,0 +1,34 @@
+#
+# Ubuntu OTA Tests
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import unittest
+
+from ubuntu_ota_tests import upgrade
+
+
+class OTABasicUpgradeTestCase(unittest.TestCase):
+
+    def test_basic_over_the_air_upgrade(self):
+        rebooted_mark = os.environ.get('ADT_REBOOT_MARK', None)
+        if not rebooted_mark:
+            # The upgrade with dbus needs more work, it's getting stuck.
+            # upgrade.upgrade_with_system_image_dbus()
+            upgrade.upgrade_with_system_image_cli()
+        else:
+            print('Rebooted')

=== added file 'debian/tests/ubuntu_ota_tests/upgrade.py'
--- debian/tests/ubuntu_ota_tests/upgrade.py	1970-01-01 00:00:00 +0000
+++ debian/tests/ubuntu_ota_tests/upgrade.py	2015-03-12 16:11:21 +0000
@@ -0,0 +1,28 @@
+#
+# Ubuntu OTA Tests
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import subprocess
+
+from ubuntu_ota_tests import hooks
+
+
+def upgrade_with_system_image_cli():
+    command = 'system-image-cli -v -g'
+    print('Upgrading with ' + command)
+    subprocess.check_call(command.split())
+    hooks.ADTRebootToRecovery().reboot()