Merge lp:autopilot into lp:autopilot/1.5

[Christopher Lee]

Stops attempts to access NoneType instead of bytes''. Considered a workaround, fixed in newer testtools.

Provide a workaround due to older versions of testtools on vivid. Stops attempts to access NoneType instead of bytes''

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:572
http://jenkins.qa.ubuntu.com/job/autopilot-1.5-ci/42/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-1.5-wily-amd64-ci/16
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-1.5-wily-amd64-ci/16/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-1.5-wily-armhf-ci/16
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-1.5-wily-armhf-ci/16/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/autopilot-1.5-wily-i386-ci/16
        deb: http://jenkins.qa.ubuntu.com/job/autopilot-1.5-wily-i386-ci/16/artifact/work/output/*zip*/output.zip
    UNSTABLE: http://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-wily-touch/977
    UNSTABLE: http://jenkins.qa.ubuntu.com/job/generic-deb-autopilot-runner-wily-touch/544
    SUCCESS: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-wily-armhf/977
        deb: http://jenkins.qa.ubuntu.com/job/generic-mediumtests-builder-wily-armhf/977/artifact/work/output/*zip*/output.zip
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/25693

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/autopilot-1.5-ci/42/rebuild

review: Approve (continuous-integration)

[Max Brustkern (nuclearbob)]

This looks good to me, but I haven't run tests yet.

review: Approve

[Selene Scriven (toykeeper)]

Approving since it's the same as the other already-approved MP.

review: Approve

[Max Brustkern (nuclearbob)]

There are sphinx build problems in xenial with docs/api/autopilot.emulators.rst and docs/tutorial/getting_started.rst

I'll look into a resolution for that.

[Max Brustkern (nuclearbob)]

The ".. otto::" lines are creating problems. I suspect maybe something we're using there work differently in the current version of sphinx, so I'll look for API changes.

[Max Brustkern (nuclearbob)]

This line in otto.py is the problem:
image_container.children.append(nodes.image(uri='/images/otto-64.png'))

[Max Brustkern (nuclearbob)]

If I remove that line, sphinx builds fine, but I get 8 test failures around datetime stuff that don't appear on wily. I'll look into that more.

[Max Brustkern (nuclearbob)]

I get these failures running tests under python3.5 on xenial, while they pass if run under 3.4:
/usr/lib/python3/dist-packages/unittest2/loader.py:367: PyGIWarning: UbuntuAppLaunch was imported without specifying a version first. Use gi.require_version('UbuntuAppLaunch', '2') before import to ensure that the right version gets loaded.
  tests = list(self._find_tests(start_dir, pattern))

** (process:27668): WARNING **: Unable to connect to Upstart bus: Error spawning command line 'dbus-launch --autolaunch=435350870788ae3025aa21ed00000538 --binary-syntax --close-stderr': Child process exited with code 1

** (process:27668): WARNING **: Unable to connect to Upstart bus: Error spawning command line 'dbus-launch --autolaunch=435350870788ae3025aa21ed00000538 --binary-syntax --close-stderr': Child process exited with code 1

** (process:27668): WARNING **: Unable to connect to Upstart bus: Error spawning command line 'dbus-launch --autolaunch=435350870788ae3025aa21ed00000538 --binary-syntax --close-stderr': Child process exited with code 1

** (process:27668): WARNING **: Unable to connect to Upstart bus: Error spawning command line 'dbus-launch --autolaunch=435350870788ae3025aa21ed00000538 --binary-syntax --close-stderr': Child process exited with code 1

** (process:27668): WARNING **: Unable to connect to Upstart bus: Error spawning command line 'dbus-launch --autolaunch=435350870788ae3025aa21ed00000538 --binary-syntax --close-stderr': Child process exited with code 1

** (process:27668): WARNING **: Unable to connect to Upstart bus: Error spawning command line 'dbus-launch --autolaunch=435350870788ae3025aa21ed00000538 --binary-syntax --close-stderr': Child process exited with code 1

** (process:27668): WARNING **: Unable to connect to Upstart bus: Error spawning command line 'dbus-launch --autolaunch=435350870788ae3025aa21ed00000538 --binary-syntax --close-stderr': Child process exited with code 1
Followed stream is empty.
Could not add content object 'autopilot.tests.unit.test_content.FileFollowerTests.test_follow_file_does_not_raise_on_IOError-1' due to IO Error: [Errno 13] Permission denied: '/tmp/tmpm6ggk9mx'
Could not add content object 'autopilot.tests.unit.test_content.FileFollowerTests.test_follow_file_returns_empty_content_object_on_error-1' due to IO Error: [Errno 13] Permission denied: '/tmp/tmpgqe4twb0'
Followed stream is empty.
Followed stream is empty.
INFO:root:************************************************************
INFO:root:Starting test Test.id
INFO:autopilot.tests.unit.test_logging:autopilot.tests.unit.test_logging.TestCaseLoggingFixtureTests.test_test_log_is_added-1
INFO:root:Autopilot Source Version: 1.5.0
INFO:root:Autopilot Source Version: 1.5.0
INFO:root:Autopilot Source Version: 1.5.0
INFO:root:Autopilot Source Version: 1.5.0
INFO:root:Autopilot Source Version: 1.5.0
INFO:root:Autopilot Source Version: 1.5.0
INFO:root:Autopilot Source Version: 1.5.0
INFO:root:Autopilot Source Version: 1.5.0
INFO:root:Autopilot Source Version: 1.5.0
INFO:root:Autopilot Source Version: 1.5.0
INFO:root:Autopilot Source Version: 1.5.0
INFO:root:Autopilot Source Version: 1.5.0
INFO:root:Autopilot Source Version: 1.5.0
WARNING:autopil...

[Max Brustkern (nuclearbob)]

The same problems (sphinx and test failures) appear when building lp:autopilot/1.5 under xenial. I think this is an issue with the transition to python3.5

[Max Brustkern (nuclearbob)]

Changing debian/rules to force python3.4 for the testing portion allows that to work, but I can't get sphinx to finish building yet.

[lp:autopilot]

573. By Christopher Lee on 2015-12-08

Fix build issues on Xenial. Incl. simplifying DateTime. Fixes: https://bugs.launchpad.net/bugs/1524000.

Approved by Max Brustkern, PS Jenkins bot.

574. By Christopher Lee on 2015-12-09

Fix missing dep packaging.

Approved by PS Jenkins bot.

[Martin Pitt (pitti)]

(Trivial) packaging change ack, python3-tz is in main and no other changes in the packaging. Thanks!

review: Approve (packaging)

=== modified file 'autopilot/_logging.py'
--- autopilot/_logging.py	2014-07-14 04:07:05 +0000
+++ autopilot/_logging.py	2015-12-09 05:28:20 +0000
@@ -22,10 +22,11 @@
 import logging
 from io import StringIO
 
-from testtools.content import text_content
-
 from autopilot._fixtures import FixtureWithDirectAddDetail
-from autopilot.utilities import LogFormatter
+from autopilot.utilities import (
+    LogFormatter,
+    safe_text_content,
+)
 
 
 class TestCaseLoggingFixture(FixtureWithDirectAddDetail):
@@ -54,7 +55,7 @@
         self._log_buffer.seek(0)
         self.caseAddDetail(
             'test-log',
-            text_content(self._log_buffer.getvalue())
+            safe_text_content(self._log_buffer.getvalue())
         )
         root_logger.removeHandler(self._log_handler)
         self._log_buffer = None

=== modified file 'autopilot/_video.py'
--- autopilot/_video.py	2014-09-05 04:03:57 +0000
+++ autopilot/_video.py	2015-12-09 05:28:20 +0000
@@ -26,10 +26,10 @@
 import signal
 import subprocess
 
-from testtools.content import text_content
 from testtools.matchers import NotEquals
 
 from autopilot.matchers import Eventually
+from autopilot.utilities import safe_text_content
 
 
 logger = logging.getLogger(__name__)
@@ -103,7 +103,7 @@
             if self._capture_process.returncode != 0:
                 test_instance.addDetail(
                     'video capture log',
-                    text_content(self._capture_process.stdout.read()))
+                    safe_text_content(self._capture_process.stdout.read()))
         self._capture_process = None
         self._currently_recording_description = None
 

=== modified file 'autopilot/application/_launcher.py'
--- autopilot/application/_launcher.py	2015-09-07 21:59:05 +0000
+++ autopilot/application/_launcher.py	2015-12-09 05:28:20 +0000
@@ -22,6 +22,8 @@
 import fixtures
 from gi.repository import GLib
 try:
+    from gi import require_version
+    require_version('UbuntuAppLaunch', '2')
     from gi.repository import UbuntuAppLaunch
 except ImportError:
     # Note: the renamed package is not in Trusty.
@@ -32,7 +34,8 @@
 import psutil
 import subprocess
 import signal
-from testtools.content import content_from_file, text_content
+from testtools.content import content_from_file
+from autopilot.utilities import safe_text_content
 
 from autopilot._timeout import Timeout
 from autopilot._fixtures import FixtureWithDirectAddDetail
@@ -428,15 +431,15 @@
         stdout, stderr, return_code = _kill_process(process)
         self.caseAddDetail(
             'process-return-code (%s)' % app_path,
-            text_content(str(return_code))
+            safe_text_content(str(return_code))
         )
         self.caseAddDetail(
             'process-stdout (%s)' % app_path,
-            text_content(stdout)
+            safe_text_content(stdout)
         )
         self.caseAddDetail(
             'process-stderr (%s)' % app_path,
-            text_content(stderr)
+            safe_text_content(stderr)
         )
 
 

=== modified file 'autopilot/content.py'
--- autopilot/content.py	2014-05-23 13:22:56 +0000
+++ autopilot/content.py	2015-12-09 05:28:20 +0000
@@ -22,7 +22,8 @@
 import io
 
 import logging
-from testtools.content import ContentType, content_from_stream, text_content
+from testtools.content import ContentType, content_from_stream
+from autopilot.utilities import safe_text_content
 
 _logger = logging.getLogger(__name__)
 
@@ -46,7 +47,7 @@
             content_name,
             str(e)
         )
-        return text_content('')
+        return safe_text_content('')
     else:
         file_obj.seek(0, io.SEEK_END)
         return follow_stream(
@@ -74,5 +75,13 @@
             ContentType('text', 'plain', {'charset': 'iso8859-1'}),
             buffer_now=True
         )
+
+        # Work around a bug in older testtools where an empty file would result
+        # in None being decoded and exploding.
+        # See: https://bugs.launchpad.net/autopilot/+bug/1517289
+        if list(content_obj.iter_text()) == []:
+            _logger.warning('Followed stream is empty.')
+            content_obj = safe_text_content('Unable to read file data.')
+
         test_case.addDetail(content_name, content_obj)
     test_case.addCleanup(make_content)

=== modified file 'autopilot/introspection/types.py'
--- autopilot/introspection/types.py	2014-10-22 20:43:01 +0000
+++ autopilot/introspection/types.py	2015-12-09 05:28:20 +0000
@@ -37,8 +37,9 @@
 
 """
 
+import pytz
 from datetime import datetime, time, timedelta
-from dateutil.tz import gettz, tzutc
+from dateutil.tz import gettz
 
 import dbus
 import logging
@@ -650,39 +651,15 @@
         super(DateTime, self).__init__(*args, **kwargs)
         # Using timedelta in this manner is a workaround so that we can support
         # timestamps larger than the 32bit time_t limit on 32bit hardware.
-        # We then apply another workaround where timedelta doesn't apply
-        # daylight savings, so we need to work out the offsets for the
-        # localtime manually and apply them to give us the correct local time.
+        # We then apply the timezone information to this to get the correct
+        # datetime.
         #
         # Note. self[0] is a UTC timestamp
-        EPOCH = datetime(1970, 1, 1, tzinfo=tzutc())
+        utc = pytz.timezone('UTC')
+        EPOCH = datetime(1970, 1, 1, tzinfo=utc)
         utc_dt = EPOCH + timedelta(seconds=self[0])
 
-        local_tzinfo = gettz()
-
-        # Get the localtimes timezone offset (known as standard offset) by
-        # subtracting its dst offset (if any) from its utc offset.
-        # We apply this to the utc datetime object to get datetime object in
-        # localtime.
-        # (We will check (once we have a local datetime) if the time is in dst
-        # and make that adjustment then.)
-        utc_offset = local_tzinfo.utcoffset(utc_dt)
-        dst_offset = local_tzinfo.dst(utc_dt)
-        standard_offset = utc_offset - dst_offset
-
-        # Create a local timezone aware datetime object from the utc_dt
-        # (i.e. attaching a timezone to it) and apply the standard offset to
-        # give us the local time.
-        local_dt = utc_dt.replace(tzinfo=local_tzinfo) + standard_offset
-
-        # If the new local time is firmly in std time then the standard offset
-        # will be 0 (i.e. timedelta(0)).
-        # If the delta isn't 0 then we need to use the timezone information to
-        # apply the dst delta to the local time.
-        if standard_offset != timedelta(0):
-            local_dt = local_dt + local_tzinfo.dst(local_dt)
-
-        self._cached_dt = local_dt
+        self._cached_dt = utc_dt.astimezone(gettz())
 
     @property
     def year(self):

=== modified file 'autopilot/tests/unit/test_platform.py'
--- autopilot/tests/unit/test_platform.py	2014-07-23 03:37:24 +0000
+++ autopilot/tests/unit/test_platform.py	2015-12-09 05:28:20 +0000
@@ -39,7 +39,7 @@
     @patch('autopilot.platform._PlatformDetector')
     def test_model_creates_platform_detector(self, mock_detector):
         platform.model()
-        mock_detector.create.assert_called_once()
+        mock_detector.create.assert_called_once_with()
 
     @patch('autopilot.platform._PlatformDetector._cached_detector')
     def test_model_returns_correct_value(self, mock_detector):
@@ -49,7 +49,7 @@
     @patch('autopilot.platform._PlatformDetector')
     def test_image_codename_creates_platform_detector(self, mock_detector):
         platform.image_codename()
-        mock_detector.create.assert_called_once()
+        mock_detector.create.assert_called_once_with()
 
     @patch('autopilot.platform._PlatformDetector._cached_detector')
     def test_image_codename_returns_correct_value(self, mock_detector):

=== modified file 'autopilot/tests/unit/test_utilities.py'
--- autopilot/tests/unit/test_utilities.py	2014-11-10 11:00:21 +0000
+++ autopilot/tests/unit/test_utilities.py	2015-12-09 05:28:20 +0000
@@ -20,6 +20,7 @@
 from unittest.mock import Mock, patch
 import re
 from testtools import TestCase
+from testtools.content import Content
 from testtools.matchers import (
     Equals,
     IsInstance,
@@ -39,6 +40,7 @@
     compatible_repr,
     deprecated,
     EventDelay,
+    safe_text_content,
     sleep,
 )
 
@@ -357,3 +359,17 @@
         wrapped.reset_cache()
         wrapped()
         self.assertThat(inner.call_count, Equals(2))
+
+
+class SafeTextContentTests(TestCase):
+
+    def test_raises_TypeError_on_non_texttype(self):
+        self.assertThat(
+            lambda: safe_text_content(None),
+            raises(TypeError)
+        )
+
+    def test_returns_text_content_object(self):
+        example_string = self.getUniqueString()
+        content_obj = safe_text_content(example_string)
+        self.assertTrue(isinstance(content_obj, Content))

=== modified file 'autopilot/utilities.py'
--- autopilot/utilities.py	2014-11-10 11:00:21 +0000
+++ autopilot/utilities.py	2015-12-09 05:28:20 +0000
@@ -29,6 +29,7 @@
 import os
 import time
 import timeit
+from testtools.content import text_content
 from functools import wraps
 
 from autopilot.exceptions import BackendException
@@ -37,6 +38,25 @@
 logger = logging.getLogger(__name__)
 
 
+def safe_text_content(text):
+    """Return testtools.content.Content object.
+
+    Safe in the sense that it will catch any attempt to attach NoneType
+    objects.
+
+    :raises ValueError: If `text` is not a text-type object.
+
+    """
+    if not isinstance(text, str):
+        raise TypeError(
+            'text argument must be string not {}'.format(
+                type(text).__name__
+            )
+        )
+
+    return text_content(text)
+
+
 def _pick_backend(backends, preferred_backend):
     """Pick a backend and return an instance of it."""
     possible_backends = list(backends.keys())

=== modified file 'debian/control'
--- debian/control	2015-07-22 23:29:11 +0000
+++ debian/control	2015-12-09 05:28:20 +0000
@@ -30,6 +30,7 @@
                python3-subunit,
                python3-testscenarios,
                python3-testtools,
+               python3-tz,
                python3-xlib (>=0.14+20091101-1ubuntu3),
                sphinx-common,
                texlive-latex-extra,
@@ -55,6 +56,7 @@
          python3-subunit,
          python3-testscenarios,
          python3-testtools,
+         python3-tz,
          udev,
          ${misc:Depends},
          ${python3:Depends},

=== modified file 'docs/otto.py'
--- docs/otto.py	2013-07-25 05:47:36 +0000
+++ docs/otto.py	2015-12-09 05:28:20 +0000
@@ -56,12 +56,9 @@
                              self.content, self.lineno, self.content_offset,
                              self.block_text, self.state, self.state_machine)
         image_container = nodes.container()
-        image_container.children.append(nodes.image(uri='/images/otto-64.png'))
+        image_container.append(nodes.image(uri='/images/otto-64.png'))
         image_container['classes'] = ['otto-image-container']
         outer_container = nodes.container()
-        outer_container.children.extend(
-            [image_container]
-            + ad
-        )
+        outer_container.extend([image_container] + ad)
         outer_container['classes'] = ['otto-says-container']
         return [outer_container]