Merge lp:~rhuddie/ubuntu-push/push-autopilot-tests into lp:ubuntu-push

[Richard Huddie]

A new autopilot test framework for testing push notifications.

The tests will configure the client to run against a specified push server and then send through a push notification message which is displayed on device's screen.

Currently the autopilot code for validating the notifications is not completed.

[Leo Arias (elopio)]

Hey Richard. You need to resubmit your MP against lp:ubuntu-push/automatic instead of trunk. I'll forward you the mail pedronis sent about this.

review: Needs Fixing

[Leo Arias (elopio)]

53 +5) autopilot3 list push-notifications
54 +6) autopilot3 run push-notifications

You have a typo there ^. It should be push_notifications.

166 +class PushNotificationMessage:
187 +class NotificationData:

I would put these classes in a push_notifications/data.py module.

210 + if dbus_info != None:
215 + elif copy_obj != None:

You should make checks like these with 'is not' instead of !=.
That's from pep8: "Comparisons to singletons like None should always be done with is or is not, never the equality operators."

203 + def __init__(self, dbus_info=None, copy_obj=None):

I find this to be a confusing constructor. I think it would be clearer to make two @classmethods, like
def from_dbus_info(dbus_info=None)
def copy(original_object=None)

Actually, I'm not sure you would need that copy. You could use the copy module.

review: Needs Fixing

[lp:~rhuddie/ubuntu-push/push-autopilot-tests]

110. By Richard Huddie on 2014-04-22

update for review comments

111. By Richard Huddie on 2014-04-22

fix pep8

112. By Richard Huddie on 2014-05-14

remove dbus dependency so it can run without root

113. By Richard Huddie on 2014-05-14

unity8 dialog validation

114. By Richard Huddie on 2014-05-15

added some new tests including locked greeter

115. By Richard Huddie on 2014-05-16

re-structure tests so that client is not re-started before each test

116. By Richard Huddie on 2014-05-16

restructure to use helper classes

117. By Richard Huddie on 2014-05-16

tidy and add test for device screen off

118. By Richard Huddie on 2014-05-16

fix flake8

119. By Richard Huddie on 2014-05-16

clear notifications before running test and move stuff into base class

120. By Richard Huddie on 2014-05-23

added cert_pem_file configuration to client

121. By Richard Huddie on 2014-05-27

certificate file path update

122. By Richard Huddie on 2014-05-27

rename tests for broadcast

123. By Richard Huddie on 2014-05-27

minor change

124. By Richard Huddie on 2014-05-27

tidy up

125. By Richard Huddie on 2014-05-27

update dialog assertion

126. By Richard Huddie on 2014-05-29

tidy up and add documentation to methods

127. By Richard Huddie on 2014-05-29

remove comment

128. By Richard Huddie on 2014-05-29

Readme updates

129. By Richard Huddie on 2014-06-02

disable mocking and minor updates

130. By Richard Huddie on 2014-06-02

move display message from base class

131. By Richard Huddie on 2014-06-04

fix review comments

132. By Richard Huddie on 2014-06-04

fix review comment

133. By Richard Huddie on 2014-06-05

Copyright header updates

134. By Richard Huddie on 2014-06-11

split greeter changes

135. By Richard Huddie on 2014-06-13

tidied up and added extra test

136. By Richard Huddie on 2014-06-13

merge with trunk

137. By Richard Huddie on 2014-06-13

update comment

138. By Richard Huddie on 2014-06-16

revert split greeter changes

139. By Richard Huddie on 2014-06-19

use /sbin/initctl

140. By Richard Huddie on 2014-06-24

update readme including emulator info

141. By Richard Huddie on 2014-07-04

wait for notification to dismiss automatically

Unmerged revisions

=== added directory 'tests'
=== added directory 'tests/autopilot'
=== added directory 'tests/autopilot/push_notifications'
=== added file 'tests/autopilot/push_notifications/README'
--- tests/autopilot/push_notifications/README	1970-01-01 00:00:00 +0000
+++ tests/autopilot/push_notifications/README	2014-04-22 14:02:46 +0000
@@ -0,0 +1,63 @@
+==================
+README
+==================
+
+To run ubuntu-push autopilot tests you need to have a push server available. This can be running locally using loopback (127.0.0.1) or remotely on the same network.
+
+----------------------------------
+To configure and build the server:
+----------------------------------
+
+1) export GOPATH=${PWD}/push
+2) mkdir -p push/src/launchpad.net
+3) cd push/src/launchpad.net
+4) bzr branch lp:ubuntu-push
+5) Edit ubuntu-push/sampleconfigs/dev.json:
+	"addr": "192.168.1.2:9090",
+   The ip should be changed to match your current environment
+   - use 127.0.0.1 if the client is also going to be running on the same machine as the server
+   - use real ip address if the client is going to be running on a different machine on the same network
+5) cd ubuntu-push
+6) make bootstrap
+7) make run-server-dev
+   Following output should be observed:
+     INFO listening for http on 192.168.1.2:8080
+     INFO listening for devices on 192.168.1.2:9090
+
+------------------------
+To configure the client:
+------------------------
+
+Note: Tests must be run as root user
+
+1) sudo apt-get install ubuntu-push-client
+2) bzr branch lp:ubuntu-push
+3) Edit ubuntu-push/tests/autopilot/push_notifications/config/push.conf:
+   [default]
+   environment = remote
+   [remote]
+   addr = 192.168.1.2
+   listener_port = 8080
+   device_port = 9090
+
+   The environment can be toggled using the 'environment' setting under [default] section. The addr and port settings for the chosen environment shold match required.
+
+4) cd ubuntu-push/tests/autopilot
+5) autopilot3 list push_notifications
+6) autopilot3 run push_notifications
+
+----------------
+Troubleshooting:
+----------------
+
+1) Ping from client to server to ensure connectivity is correct
+2) Delete .local/share/ubuntu-push-client/levels.db if no notifications are being displayed
+3) To send a notification manually:
+     echo '{"channel":"system", "data": {"ubuntu-touch/trusty-proposed/mako": [297, ""]}, "expire_on": "2015-12-19T16:39:57-08:00"}' | POST -c application/json http://192.168.1.2:8080/broadcast
+   Response should be:
+     {"ok":true}
+   Note that:
+     - The channel and device names must match the client. 
+     - The build number must be greater than current installed version in order to trigger an update message.
+     - The expiration time must be in the future.
+

=== added file 'tests/autopilot/push_notifications/__init__.py'
--- tests/autopilot/push_notifications/__init__.py	1970-01-01 00:00:00 +0000
+++ tests/autopilot/push_notifications/__init__.py	2014-04-22 14:02:46 +0000
@@ -0,0 +1,8 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+# Copyright 2014 Canonical
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+"""push-notifications autopilot tests."""

=== added directory 'tests/autopilot/push_notifications/config'
=== added file 'tests/autopilot/push_notifications/config/__init__.py'
--- tests/autopilot/push_notifications/config/__init__.py	1970-01-01 00:00:00 +0000
+++ tests/autopilot/push_notifications/config/__init__.py	2014-04-22 14:02:46 +0000
@@ -0,0 +1,19 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+# Copyright 2014 Canonical
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+
+import os
+
+CONFIG_FILE = 'push.conf'
+
+
+def get_config_file():
+    """
+    Return the path for the config file
+    """
+    config_dir = os.path.dirname(__file__)
+    return os.path.join(config_dir, CONFIG_FILE)

=== added file 'tests/autopilot/push_notifications/config/push.conf'
--- tests/autopilot/push_notifications/config/push.conf	1970-01-01 00:00:00 +0000
+++ tests/autopilot/push_notifications/config/push.conf	2014-04-22 14:02:46 +0000
@@ -0,0 +1,23 @@
+[default]
+environment = remote
+
+[local]
+addr = 127.0.0.1
+listener_port = 8080
+device_port = 9090
+
+[remote]
+addr = 192.168.1.2
+listener_port = 8080
+device_port = 9090
+
+[staging]
+addr = staging-url.com
+listener_port = 8080
+device_port = 9090
+
+[production]
+addr = production-url.com
+listener_port = 8080
+device_port = 9090
+

=== added file 'tests/autopilot/push_notifications/data.py'
--- tests/autopilot/push_notifications/data.py	1970-01-01 00:00:00 +0000
+++ tests/autopilot/push_notifications/data.py	2014-04-22 14:02:46 +0000
@@ -0,0 +1,81 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+# Copyright 2014 Canonical
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+"""Push-Notifications autopilot data structure classes"""
+
+
+class PushNotificationMessage:
+    """
+    Class to hold all the details required for a
+    push notification message
+    """
+    channel = None
+    expire_after = None
+    data = None
+
+    def __init__(self, channel='system', data='', expire_after=''):
+        self.channel = channel
+        self.data = data
+        self.expire_after = expire_after
+
+    def json(self):
+        """
+        Return json representation of message
+        """
+        json_str = '{{"channel":"{0}", "data":{{{1}}}, "expire_on":"{2}"}}'
+        return json_str.format(self.channel, self.data, self.expire_after)
+
+
+class NotificationData:
+    """
+    Class to represent notification data including
+    Device software channel
+    Device build number
+    Device model
+    Device last update
+    Data for the notification
+    """
+    channel = None
+    build_number = None
+    device = None
+    last_update = None
+    version = None
+    data = None
+
+    @classmethod
+    def from_dbus_info(cls, dbus_info=None):
+        """
+        Create a new object based on dbus_info if provided
+        """
+        nd = NotificationData()
+        if dbus_info is not None:
+            nd.device = dbus_info[1]
+            nd.channel = dbus_info[2]
+            nd.last_update = dbus_info[3]
+            nd.build_number = dbus_info[4]['version']
+        return nd
+
+    def inc_build_number(self):
+        """
+        Increment build number
+        """
+        self.build_number = str(int(self.build_number) + 1)
+
+    def dec_build_number(self):
+        """
+        Decrement build number
+        """
+        self.build_number = str(int(self.build_number) - 1)
+
+    def json(self):
+        """
+        Return json representation of info based:
+        "IMAGE-CHANNEL/DEVICE-MODEL": [BUILD-NUMBER, CHANNEL-ALIAS]"
+        """
+        json_str = '"{0}/{1}": [{2}, "{3}"]'
+        return json_str.format(self.channel, self.device, self.build_number,
+                               self.data)

=== added directory 'tests/autopilot/push_notifications/tests'
=== added file 'tests/autopilot/push_notifications/tests/__init__.py'
--- tests/autopilot/push_notifications/tests/__init__.py	1970-01-01 00:00:00 +0000
+++ tests/autopilot/push_notifications/tests/__init__.py	2014-04-22 14:02:46 +0000
@@ -0,0 +1,239 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+# Copyright 2014 Canonical
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+"""push-notifications autopilot tests."""
+
+
+import configparser
+import http.client as http
+import json
+import os
+import datetime
+import subprocess
+import dbus
+import copy
+
+from autopilot.testcase import AutopilotTestCase
+from push_notifications.data import PushNotificationMessage
+from push_notifications.data import NotificationData
+from push_notifications import config
+
+
+class PushNotificationTestBase(AutopilotTestCase):
+    """
+    Base class for push notification test cases
+    """
+
+    PUSH_CLIENT_DEFAULT_CONFIG_FILE = '/etc/xdg/ubuntu-push-client/config.json'
+    PUSH_CLIENT_CONFIG_FILE = '~/.config/ubuntu-push-client/config.json'
+    PUSH_SERVER_BROADCAST_URL = '/broadcast'
+    DEFAULT_DISPLAY_MESSAGE = 'There\'s an updated system image.'
+    PUSH_MIME_TYPE = 'application/json'
+    SECTION_DEFAULT = 'default'
+    KEY_ENVIRONMENT = 'environment'
+    KEY_ADDR = 'addr'
+    KEY_LISTENER_PORT = 'listener_port'
+    KEY_DEVICE_PORT = 'device_port'
+
+    def setUp(self):
+        """
+        Start the client running with the correct server config
+        """
+        # setup
+        super(PushNotificationTestBase, self).setUp()
+        # Read the config data
+        self.read_config_file()
+        # write server device address to the client config
+        self.write_client_test_config()
+        # restart the push client
+        self.restart_push_client()
+        # validate that the initialisation push message is displayed
+        self.validate_push_message(self.DEFAULT_DISPLAY_MESSAGE)
+        # dbus
+        self.get_device_info()
+
+    def create_notification_data_copy(self):
+        """
+        Return a copy of the device's notification data
+        """
+        return copy.deepcopy(self.notification_data)
+
+    def get_device_info(self):
+        """
+        Discover the device's model and build info
+        """
+        system_bus = dbus.SystemBus()
+        info_service = system_bus.get_object(
+            'com.canonical.SystemImage', '/Service')
+        info = info_service.Info()
+        # Create a NotificationData object based on the dbus info
+        self.notification_data = NotificationData.from_dbus_info(
+            dbus_info=info)
+
+    def read_config_file(self):
+        """
+        Read data from config file
+        """
+        config_file = config.get_config_file()
+        self.config = configparser.ConfigParser()
+        self.config.read(config_file)
+        # read the name of the environment to use (local/remote)
+        self.env = self.config[self.SECTION_DEFAULT][self.KEY_ENVIRONMENT]
+        # format the server device and listener address
+        addr_fmt = '{0}:{1}'
+        self.server_listener_addr = addr_fmt.format(
+            self.get_server_addr(), self.get_listener_port())
+        self.server_device_addr = addr_fmt.format(
+            self.get_server_addr(), self.get_device_port())
+
+    def get_server_addr(self):
+        """
+        Return the server address from config file
+        """
+        return self.config[self.env][self.KEY_ADDR]
+
+    def get_listener_port(self):
+        """
+        Return the server listener port from config file
+        """
+        return self.config[self.env][self.KEY_LISTENER_PORT]
+
+    def get_device_port(self):
+        """
+        Return the server listener port from config file
+        """
+        return self.config[self.env][self.KEY_DEVICE_PORT]
+
+    def _control_client(self, command):
+        """
+        start/stop/restart the ubuntu-push-client using initctl
+        """
+        subprocess.call(['initctl', command, 'ubuntu-push-client'])
+
+    def stop_push_client(self):
+        """
+        Stop the push client
+        """
+        self._control_client('stop')
+
+    def start_push_client(self):
+        """
+        Start the push client
+        """
+        self._control_client('start')
+
+    def restart_push_client(self):
+        """
+        Restart the push client
+        """
+        self.stop_push_client()
+        self.start_push_client()
+
+    def write_client_test_config(self):
+        """
+        Write the test server address to client config file
+        """
+        # read the original config file
+        with open(self.PUSH_CLIENT_DEFAULT_CONFIG_FILE) as config_file:
+            config = json.load(config_file)
+        # change server address
+        config['addr'] = self.server_device_addr
+        # write the config json out to the ~.local address
+        abs_config_file = os.path.expanduser(self.PUSH_CLIENT_CONFIG_FILE)
+        config_dir = os.path.dirname(abs_config_file)
+        if not os.path.exists(config_dir):
+            os.makedirs(config_dir)
+        with open(abs_config_file, 'w+') as outfile:
+            json.dump(config, outfile, indent=4)
+            outfile.close()
+
+    def send_push_broadcast_notification(self, msg_json):
+        """
+        Send the specified push message to the server broadcast url
+        using an HTTP POST command
+        """
+        headers = {'Content-type': self.PUSH_MIME_TYPE}
+        conn = http.HTTPConnection(self.server_listener_addr)
+        conn.request(
+            'POST',
+            self.PUSH_SERVER_BROADCAST_URL,
+            headers=headers,
+            body=msg_json)
+        return conn.getresponse()
+
+    def create_push_message(self, channel='system', data='', expire_after=''):
+        """
+        Return a new push msg
+        If no expiry time is given, a future date will be assigned
+        """
+        if expire_after == '':
+            expire_after = self.get_future_iso_time()
+        return PushNotificationMessage(
+            channel=channel,
+            data=data,
+            expire_after=expire_after)
+
+    def validate_push_message(self, display_message, timeout=10):
+        """
+        Validate that a notification message is displayed on screen
+        """
+
+    def get_past_iso_time(self):
+        """
+        Return time 1 year in past in ISO format
+        """
+        return self.get_iso_time(year_offset=-1)
+
+    def get_near_past_iso_time(self):
+        """
+        Return time 1 minute in past in ISO format
+        """
+        return self.get_iso_time(min_offset=-1)
+
+    def get_near_future_iso_time(self):
+        """
+        Return time 1 minute in future in ISO format
+        """
+        return self.get_iso_time(min_offset=1)
+
+    def get_future_iso_time(self):
+        """
+        Return time 1 year in future in ISO format
+        """
+        return self.get_iso_time(year_offset=1)
+
+    def get_current_iso_time(self):
+        """
+        Return current time in ISO format
+        """
+        return self.get_iso_time()
+
+    def get_iso_time(self, year_offset=0, month_offset=0, day_offset=0,
+                     hour_offset=0, min_offset=0, sec_offset=0,
+                     tz_hour_offset=0, tz_min_offset=0):
+        """
+        Return an ISO8601 format date-time string, including time-zone
+        offset: YYYY-MM-DDTHH:MM:SS-HH:MM
+        """
+        # calulate target time based on current time and format it
+        now = datetime.datetime.now()
+        target_time = datetime.datetime(
+            year=now.year + year_offset,
+            month=now.month + month_offset,
+            day=now.day + day_offset,
+            hour=now.hour + hour_offset,
+            minute=now.minute + min_offset,
+            second=now.second + sec_offset)
+        target_time_fmt = target_time.strftime('%Y-%m-%dT%H:%M:%S')
+        # format time zone offset
+        tz = datetime.time(
+            hour=tz_hour_offset,
+            minute=tz_min_offset)
+        tz_fmt = tz.strftime('%H:%M')
+        # combine target time and time zone offset
+        iso_time = '{0}-{1}'.format(target_time_fmt, tz_fmt)
+        return iso_time

=== added file 'tests/autopilot/push_notifications/tests/test_push_client.py'
--- tests/autopilot/push_notifications/tests/test_push_client.py	1970-01-01 00:00:00 +0000
+++ tests/autopilot/push_notifications/tests/test_push_client.py	2014-04-22 14:02:46 +0000
@@ -0,0 +1,73 @@
+# -*- Mode: Python; coding: utf-8; indent-tabs-mode: nil; tab-width: 4 -*-
+# Copyright 2014 Canonical
+#
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+"""Tests for Push Notifications client"""
+
+from __future__ import absolute_import
+
+from testtools.matchers import Equals
+from push_notifications.tests import PushNotificationTestBase
+
+
+class TestPushClient(PushNotificationTestBase):
+    """ Tests a Push notification can be sent and received """
+
+    def _validate_response(self, response, expected_status_code=200):
+        """
+        Validate the received response status code against expected code
+        """
+        self.assertThat(response.status, Equals(expected_status_code))
+
+    def test_broadcast_push_notification(self):
+        """
+        Positive test case to send a valid broadcast push notification
+        to the client and validate that a notification message is displayed
+        """
+        # create a copy of the device's build info
+        msg_data = self.create_notification_data_copy()
+        # increment the build number to trigger an update
+        msg_data.inc_build_number()
+        # create message based on the data
+        msg = self.create_push_message(data=msg_data.json())
+        # send the notification message to the server and check response
+        response = self.send_push_broadcast_notification(msg.json())
+        self._validate_response(response)
+
+        # TODO validate that message is received on client
+
+    def test_expired_broadcast_push_notification(self):
+        """
+        Send an expired broadcast notification message to server
+        """
+        msg = self.create_push_message(expire_after=self.get_past_iso_time())
+        response = self.send_push_broadcast_notification(msg.json())
+        # 400 status is received for an expired message
+        self._validate_response(response, expected_status_code='400')
+
+        # TODO validate that message is not received on client
+
+    def test_near_expiry_broadcast_push_notification(self):
+        """
+        Send a broadcast message with a short validity time
+        """
+        msg = self.create_push_message(
+            expire_after=self.get_near_future_iso_time())
+        response = self.send_push_broadcast_notification(msg.json())
+        self._validate_response(response)
+
+        # TODO validate that message is received on client
+
+    def test_just_expired_broadcast_push_notification(self):
+        """
+        Send a broadcast message which has just expired
+        """
+        msg = self.create_push_message(
+            expire_after=self.get_near_past_iso_time())
+        response = self.send_push_broadcast_notification(msg.json())
+        self._validate_response(response)
+
+        # TODO validate that message is not received on client