Merge lp:~kissiel/checkbox/loc-svc-test-cc into lp:checkbox

[Maciej Kisielewski]

This MR adds Ubuntu Location Service tests to the 'touch' provider.

It uses template units to generate the tests out of the gtest-framework binaries.

5c91d8f providers:touch: add ULS dependencies to the provider
7e6f9a5 providers:touch: add uls_resource generator script
d74be10 providers:touch: add ULS test definitions
5d2169c checkbox-touch: make build-me less verbose

[Chris Wayne (cwayne)]

Looks like some conflicts

[lp:~kissiel/checkbox/loc-svc-test-cc]

4344. By Paul Larson on 2016-05-17

Remove the deprecation warning for get_session_data_subset

It seems that get_session_data_subset was marked deprecated in favor of
dump_from_session_manager, but dump_from_session_manager actually calls
get_session_data_subset and then just dumps it to a stream. Since I'm
guessing there could be some cases where we might want this as a return
value rather than a stream, I'm proposing that we remove the deprecation
warning.

4345. By Sylvain Pineau on 2016-05-17

checkbox-ng:checkbox-cli: Add the tar exporter to submission_files

Note: The json report is included in the archive.

4346. By Sylvain Pineau on 2016-05-17

plainbox:exporter:tar: Fix the class attribute SUPPORTED_OPTION_LIST

supported_option_list -> SUPPORTED_OPTION_LIST

4347. By Sylvain Pineau on 2016-05-17

plainbox:providers:exporter: Enable all xlsx options by default

4348. By Sylvain Pineau on 2016-05-17

plainbox:providers:exporter: Fix tar extension to be tar.xz

4349. By Maciej Kisielewski on 2016-05-17

providers:touch: add ULS dependencies to the provider

Signed-off-by: Maciej Kisielewski <email address hidden>

4350. By Maciej Kisielewski on 2016-05-17

providers:touch: add uls_resource generator script

Signed-off-by: Maciej Kisielewski <email address hidden>

4351. By Maciej Kisielewski on 2016-05-17

providers:touch: add ULS test definitions

Signed-off-by: Maciej Kisielewski <email address hidden>

4352. By Maciej Kisielewski on 2016-05-17

checkbox-touch: make build-me less verbose

By removing printing of temporary locations from which libs were pulled.

Signed-off-by: Maciej Kisielewski <email address hidden>

[Sylvain Pineau (sylvain-pineau)]

I guess the recent blacklisted items are making TESTS_REQUIRING_ROOT an empty var now? could you confirm it's expected?

review: Needs Information

[Maciej Kisielewski (kissiel)]

> I guess the recent blacklisted items are making TESTS_REQUIRING_ROOT an empty
> var now? could you confirm it's expected?

Yes.
 I spent quite some time researching how to properly run the tests as root, this is why I left it there. Now it should be easy to add next bunch of tests.

(the ones requiring root actually don't work well indoors, this is why they're blacklisted)

[Pierre Equoy (pieq)]

Tested on an Arale device with OTA 10.1.

75 tests were run and passed:

uls/boost_ptree_settings_test_BoostPtreeSettings.correctly_reads_and_writes_values_to_file
uls/boost_ptree_settings_test_BoostPtreeSettings.does_not_throw_for_file_that_cannot_be_read_from
uls/boost_ptree_settings_test_BoostPtreeSettings.never_throws_on_syncing
uls/boost_ptree_settings_test_BoostPtreeSettings.throws_correct_exception_for_missing_value
uls/boost_ptree_settings_test_BoostPtreeSettings.values_are_correctly_stored
uls/connectivity_manager_test_ConnectivityManager.DISABLED_correctly_handles_wifi_devices_and_aps_on_init
uls/connectivity_manager_test_ConnectivityManager.DISABLED_queries_devices_and_modems_when_initialized
uls/connectivity_manager_test_ConnectivityManagerOnDevice.default_implementation_is_queryable_for_wifi_and_wwan_status_requires_hardware
uls/connectivity_manager_test_ConnectivityManagerOnDevice.default_implementation_is_queryable_for_wifis_and_radio_cells_requires_hardware
uls/connectivity_manager_test_ConnectivityManagerOnDevice.repeatedly_querying_the_connected_cell_works_requires_hardware
uls/connectivity_manager_test_ConnectivityManagerOnDevice.repeatedly_requesting_network_scans_works_requires_hardware
uls/controller_test_Controller.controller_starts_and_stops_updates_on_provider
uls/controller_test_Controller.controller_starts_and_stops_updates_on_provider_only_once_requires_hardware
uls/criteria_test_Criteria.AddedUpCriteriaSatisfiesAllIndividualCriteria
uls/criteria_test_Criteria.SatisfiesReturnsFalseForNonSatisfyingCriteria
uls/default_permission_manager_test_DefaultPermissionManager.for_now_only_local_sessions_are_accepted
uls/demultiplexing_reporter_test_DemultiplexingReporter.dispatches_calls_to_all_reporters
uls/engine_test_Engine.adding_a_null_provider_throws
uls/engine_test_Engine.adding_a_provider_creates_connections_to_engine_configuration_properties
uls/engine_test_Engine.adding_and_removing_providers_inserts_and_erases_from_underlying_collection
uls/engine_test_Engine.provider_selection_policy_is_invoked_when_matching_providers_to_criteria
uls/engine_test_Engine.reads_state_from_settings_on_construction
uls/engine_test_Engine.stores_state_from_settings_on_destruction
uls/gps_provider_test_AndroidGpsXtraDownloader.reading_configuration_from_valid_conf_file_works
uls/gps_provider_test_GpsProvider.DISABLED_accessing_starting_and_stopping_gps_provider_works_requires_hardware
uls/gps_provider_test_GpsProvider.injecting_a_reference_position_calls_into_the_hal
uls/gps_provider_test_GpsProvider.starting_updates_on_a_provider_instance_calls_into_the_hal
uls/gps_provider_test_GpsProvider.updates_from_hal_are_passed_on_by_the_provider
uls/gps_provider_test_GpsXtraDownloader.download_attempt_throws_if_timeout_is_reached
uls/gps_provider_test_GpsXtraDownloader.downloading_xtra_data_from_known_host_works
uls/gps_provider_test_GpsXtraDownloader.throws_for_missing_xtra_hosts
uls/gps_provider_test_GpsXtraDownloader.throws_for_unreachable_host
uls/gps_provider_test_HardwareAbstractionLayerFixture.DISABLED_provider_construction_works_requires_hardw...

review: Approve

=== modified file 'checkbox-touch/build-me'
--- checkbox-touch/build-me	2016-05-11 22:13:06 +0000
+++ checkbox-touch/build-me	2016-05-17 07:59:56 +0000
@@ -285,7 +285,6 @@
                 get_package_from_url_and_extract(uris[deb], tmp)
                 bin_path = os.path.join(tmp, 'usr', 'bin')
                 if os.path.exists(bin_path):
-                    print('copying binaries from {}'.format(tmp))
                     executables = find_executables(bin_path)
                     provider_bin = os.path.join(
                             'providers', provider_dir, 'bin')
@@ -303,7 +302,6 @@
                             shutil.copymode(f, target)
                 lib_path = os.path.join(tmp, 'usr', 'lib')
                 if os.path.exists(lib_path):
-                    print('copying libraries from {}'.format(tmp))
                     rsync_tree(lib_path, 'lib', preserve_symlinks=1)
 
 

=== added file 'providers/plainbox-provider-ubuntu-touch/_extra_debs'
--- providers/plainbox-provider-ubuntu-touch/_extra_debs	1970-01-01 00:00:00 +0000
+++ providers/plainbox-provider-ubuntu-touch/_extra_debs	2016-05-17 07:59:56 +0000
@@ -0,0 +1,11 @@
+libboost-filesystem1.58.0
+libboost-iostreams1.58.0
+libboost-program-options1.58.0
+libboost-system1.58.0
+libdbus-cpp5
+libnet-cpp2
+libprocess-cpp3
+libstdc++6
+libtrust-store2
+libubuntu-location-service3
+ubuntu-location-service-tests

=== added directory 'providers/plainbox-provider-ubuntu-touch/bin'
=== added file 'providers/plainbox-provider-ubuntu-touch/bin/uls_resource'
--- providers/plainbox-provider-ubuntu-touch/bin/uls_resource	1970-01-01 00:00:00 +0000
+++ providers/plainbox-provider-ubuntu-touch/bin/uls_resource	2016-05-17 07:59:56 +0000
@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+#
+# This file is part of Checkbox.
+#
+# Copyright 2016 Canonical Ltd.
+#
+# Checkbox is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3,
+# as published by the Free Software Foundation.
+#
+# Checkbox is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Checkbox.  If not, see <http://www.gnu.org/licenses/>.
+"""
+This program generates Ubuntu Location Service tests formatted as plainbox
+jobs. It requires executables ubuntu-location-service-tests package to be
+available in the path.
+
+Number of plainbox jobs generated with this script should match running:
+$ for e in `find . -executable`; do $e --gtest_list_tests 2>/dev/null; done
+|grep '^\ ' |wc -l
+in the ubuntu-location-service-tests bin directory.
+
+Note that for different architectures the binaries names and number of tests
+may vary.
+"""
+
+import os
+import subprocess
+import sys
+
+BINARIES_BLACKLISTED = [
+    'daemon_and_cli_tests', 'acceptance_tests', # all tests from those binaries
+    # require /usr/share/dbus-cpp/session.conf to be present; this file is
+    # normally _installed_ with libdbus-cpp-dev.
+    #'engine_test', # tests from this binary yield 'stack smashing detected'
+    # which is reproducable on desktops
+]
+
+TESTS_BLACKLISTED = [
+    'HardwareAbstractionLayerFixture.time_to_first_fix_cold_start_without_supl_benchmark_requires_hardware',
+    'HardwareAbstractionLayerFixture.time_to_first_fix_cold_start_with_supl_benchmark_requires_hardware',
+    # those two tests when run indoors can run indefinitely
+    'RemoteProviderdTest.AClientReceivesUpdatesFromAnOutOfProcessProvider',
+    # this test also requires /usr/share/dbus-cpp/session.conf
+    'EspooProviderTest.receives_position_updates_requires_daemons',
+    # this test requires X11
+    'AppArmorProfileResolver.libapparmor_profile_resolver_returns_correct_profile_for_unconfined_process',
+    # this test is run as part of checkbox, so apparmor reports app being
+    # checkbox, and not the one that test expects
+]
+
+TESTS_REQUIRING_ROOT = [
+]
+
+def main():
+    uls_binaries = [
+        "acceptance_tests", "boost_ptree_settings_test",
+        "connectivity_manager_test", "controller_test", "criteria_test",
+        "daemon_and_cli_tests", "default_permission_manager_test",
+        "demultiplexing_reporter_test", "engine_test", "espoo_provider_test",
+        "gps_provider_test", "harvester_test", "ichnaea_reporter_test",
+        "position_test", "provider_factory_test",
+        "provider_selection_policy_test", "provider_test",
+        "remote_provider_test", "remote_providerd_test",
+        "time_based_update_policy_test", "trust_store_permission_manager_test",
+        "wgs84_test"]
+    for uls_bin in uls_binaries:
+        if uls_bin in BINARIES_BLACKLISTED:
+            continue
+        output = subprocess.check_output([uls_bin, '--gtest_list_tests'])
+        suite = ''
+        for line in output.decode(sys.stdout.encoding).split('\n'):
+            if not line.startswith(' '):
+                suite = line
+            else:
+                test_name = line.lstrip()
+                full_name = suite + test_name
+                root = 'yes' if full_name in TESTS_REQUIRING_ROOT else 'no'
+                if full_name in TESTS_BLACKLISTED:
+                    continue
+                print('full_name: {}'.format(full_name))
+                print('test_case: {}'.format(test_name))
+                print('binary: {}'.format(uls_bin))
+                print('root: {}'.format(root))
+                if root == 'yes':
+                    print('ld_lib_path: {}'.format(os.environ.get('LD_LIBRARY_PATH') or ''))
+                print()
+
+
+if __name__ == '__main__':
+    main()

=== modified file 'providers/plainbox-provider-ubuntu-touch/units/touch.pxu'
--- providers/plainbox-provider-ubuntu-touch/units/touch.pxu	2016-01-05 02:45:26 +0000
+++ providers/plainbox-provider-ubuntu-touch/units/touch.pxu	2016-05-17 07:59:56 +0000
@@ -5059,3 +5059,4 @@
 _description:
  This test plan selects all the tests required to certify a device with Ubuntu Touch
 include: .*
+bootstrap_include: uls/uls_resource

=== added file 'providers/plainbox-provider-ubuntu-touch/units/uls.pxu'
--- providers/plainbox-provider-ubuntu-touch/units/uls.pxu	1970-01-01 00:00:00 +0000
+++ providers/plainbox-provider-ubuntu-touch/units/uls.pxu	2016-05-17 07:59:56 +0000
@@ -0,0 +1,32 @@
+unit: category
+id: uls
+_name: Ubuntu Location Service tests
+
+id: uls/uls_resource
+category_id: uls
+plugin: resource
+command: uls_resource
+_summary: Generate Ubuntu Location Services tests
+_description: Generates Ubuntu Location Services tests
+
+unit: template
+template-resource: uls/uls_resource
+id: uls/{binary}_{full_name}
+template-imports: from 2014.com.ubuntu import uls/uls_resource as uls_res
+template-filter: uls_res.root == 'yes'
+category_id: uls
+plugin: shell
+command: LD_LIBRARY_PATH={ld_lib_path}:$LD_LIBRARY_PATH {binary} --gtest_filter={full_name}
+user: root
+_description: {test_case} from Ubuntu Location Service tests
+
+unit: template
+template-resource: uls/uls_resource
+id: uls/{binary}_{full_name}
+template-imports: from 2014.com.ubuntu import uls/uls_resource as uls_res
+template-filter: uls_res.root == 'no'
+category_id: uls
+plugin: shell
+command: {binary} --gtest_filter={full_name}
+_description: {test_case} from Ubuntu Location Service tests
+