Merge lp:~kissiel/checkbox/syslog-camera-denial into lp:checkbox

[Maciej Kisielewski]

This MR brings camera-denial test which checks if camera is properly blocked when user doesn't allow access.

It does so by doing two things:
1. Checking if Camera QML component was able to run.
2. Checking whether appropriate entry was written to syslog.

To test it out, build checkbox-converged, by running:

(/checkbox/checkbox-touch/) $ ./get-libs && ./build-me --install

And navigate to apparmor tests and run'em all.

fd2e29b providers:qml-tests: add camera_denial.qml
41e0e5f providers:qml-tests: add definitions for camera-denial test

[Sylvain Pineau (sylvain-pineau)]

Tested on a Nexus7 with OTA10

Only selected the 3 apparmor tests, I always got the popup (http://i.imgur.com/JGgrzb9.png) asking whether or not I want to authorize the app to access the camera (2 times though, I think one per app, cbt and the confined app).

even I if decline the auth, the test fails. The only way for me to pass the test was to go into system settings and manually unset the permissions for cbt and denial app.

review: Needs Fixing

[Sylvain Pineau (sylvain-pineau)]

The syslog I got when refusing auth:

May 2 14:29:09 ubuntu-phablet rsyslogd: [origin software="rsyslogd" swVersion="7.4.4" x-pid="844" x-info="http://www.rsyslog.com"] start
May 2 14:29:09 ubuntu-phablet rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]
May 2 14:29:09 ubuntu-phablet rsyslogd-2307: warning: ~ action is deprecated, consider using the 'stop' statement instead [try http://www.rsyslog.com/e/2307 ]
May 2 14:29:09 ubuntu-phablet rsyslogd: rsyslogd's groupid changed to 103
May 2 14:29:09 ubuntu-phablet rsyslogd: rsyslogd's userid changed to 100
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] Booting Linux on physical CPU 0
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] Initializing cgroup subsys cpu
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] Linux version 3.4.0-5-flo (buildd@kishi14) (gcc version 4.7.4 (Ubuntu/Linaro 4.7.4-3ubuntu2) ) #1-Ubuntu SMP PREEMPT Thu Sep 3 14:21:37 UTC 2015
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] CPU: ARMv7 Processor [511f06f0] revision 0 (ARMv7), cr=10c5387d
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] CPU: PIPT / VIPT nonaliasing data cache, PIPT instruction cache
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] Machine: QCT APQ8064 FLO
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] Truncating memory at 0x90000000 to fit in 32-bit physical address space
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] memory pool 3 (start fe9ff000 size 1600000) initialized
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] Initialized persistent memory from 88d00000-88dfffff
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] cma: CMA: reserved 16 MiB at af800000
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] Memory policy: ECC disabled, Data cache writealloc
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] socinfo_init: v7, id=172, ver=2.0, raw_id=2289, raw_ver=2289, hw_plat=8, hw_plat_ver=65536
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] accessory_chip=0, hw_plat_subtype=0, pmic_model=18, pmic_die_revision=4
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] On node 0 totalpages: 482303
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] Normal zone: 1556 pages used for memmap
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] Normal zone: 0 pages reserved
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] Normal zone: 161772 pages, LIFO batch:31
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] HighMem zone: 2536 pages used for memmap
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] HighMem zone: 316439 pages, LIFO batch:31
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] allocating 44236800 bytes at c2a39000 (82c39000 physical) for fb
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] PERCPU: Embedded 9 pages/cpu @c546c000 s15680 r8192 d12992 u36864
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] pcpu-alloc: s15680 r8192 d12992 u36864 alloc=9*4096
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3
May 2 14:29:09 ubuntu-phablet kernel: [ 0.000000] Built 1 zo...

[Pierre Equoy (pieq)]

A few typos.

I tried to run it on my Arale device, but I don't see any "Apparmor" category in the “Ubuntu Touch certification tests”.

I must have missed something. I'll check with Maciek today.

review: Needs Fixing

[Maciej Kisielewski (kissiel)]

> Tested on a Nexus7 with OTA10
>
> Only selected the 3 apparmor tests, I always got the popup
> (http://i.imgur.com/JGgrzb9.png) asking whether or not I want to authorize the
> app to access the camera (2 times though, I think one per app, cbt and the
> confined app).
>
> even I if decline the auth, the test fails. The only way for me to pass the
> test was to go into system settings and manually unset the permissions for cbt
> and denial app.

The popup was about letting checkbox access to camera. Not for the confined test.

I'll retest it on a freshly flashed device.

[Maciej Kisielewski (kissiel)]

10:38 < ePierre_> kissiel, if this test is here to stay, maybe we should change the wording
10:38 < kissiel> suggestions welcome
10:38 < ePierre_> to explicitely tell the tester to not allow the app to access Camera
10:39 < kissiel> +1
10:39 < kissiel> this is something that's so obvious while implementing stuff it's easily overlooked
10:39 < kissiel> and it is definitely needed
10:39 < kissiel> thanks

review: Needs Fixing

[Pierre Equoy (pieq)]

Re-tested on Arale.

If Checkbox and Checkbox-apparmor apps are not allowed to access the camera, the three tests pass [OK]

If Checkbox and Checkbox-apparmor apps are allowed to access the camera:
app-armor-denial fails
app-armor-denial-check is skipped
app-armor-denial-syslog-save passes
[OK]

Looks good to me!

review: Approve

=== added file 'providers/2015.com.canonical.certification:qml-tests/data/camera_denial.qml'
--- providers/2015.com.canonical.certification:qml-tests/data/camera_denial.qml	1970-01-01 00:00:00 +0000
+++ providers/2015.com.canonical.certification:qml-tests/data/camera_denial.qml	2016-05-17 09:26:34 +0000
@@ -0,0 +1,63 @@
+/*
+ * This file is part of Checkbox.
+ *
+ * Copyright 2016 Canonical Ltd.
+ * Written by:
+ *   Maciej Kisielewski <[e-mail]>
+ *
+ * Checkbox is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 3,
+ * as published by the Free Software Foundation.
+ *
+ * Checkbox is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with Checkbox.  If not, see <http://www.gnu.org/licenses/>.
+ */
+import QtQuick 2.0
+import Ubuntu.Components 1.1
+import QtMultimedia 5.2
+import Plainbox 0.1
+
+/*
+    This test checks whether camera access was blocked by app armor.
+    The test FAILS when the camera was initiated without a problem.
+    This test should be launched as a confined QML job.
+*/
+
+QmlJob {
+    Page {
+        anchors.fill: parent
+        Label {
+            id: label
+            text: i18n.tr("Launching camera")
+        }
+        VideoOutput {
+            id: viewfinder
+            visible: true
+            source: cam
+            anchors.fill: parent
+            orientation: (Screen.primaryOrientation === Qt.PortraitOrientation) ? 270 : 0;
+            fillMode: Image.PreserveAspectCrop
+        }
+    }
+    Timer {
+        id: resultTimer
+        running: true
+        interval: 1000
+        onTriggered: {
+            if (cam.errorCode) {
+                testDone({'outcome': 'pass'});
+            } else {
+                testDone({'outcome': 'fail'});
+            }
+        }
+    }
+    Camera {
+        id: cam
+        captureMode: Camera.CaptureViewfinder
+    }
+}

=== modified file 'providers/2015.com.canonical.certification:qml-tests/units/qml-tests.pxu'
--- providers/2015.com.canonical.certification:qml-tests/units/qml-tests.pxu	2015-11-17 21:05:06 +0000
+++ providers/2015.com.canonical.certification:qml-tests/units/qml-tests.pxu	2016-05-17 09:26:34 +0000
@@ -194,3 +194,37 @@
 estimated_duration: 20
 imports: from 2015.com.canonical.certification import cellularmodem as modem
 requires: modem.ril1_simPresent == "True"
+
+unit: category
+id: Apparmor
+_name: Apparmor tests
+
+id: app-armor-denial-syslog-save
+category_id: Apparmor
+plugin: shell
+command: cp /var/log/syslog $PLAINBOX_SESSION_SHARE/before-app-armor-denial-syslog
+_description: Job that saves syslog
+flags: preserve-locale
+estimated_duration: 1
+
+id: app-armor-denial
+category_id: Apparmor
+plugin: qml
+_summary: Check denial of access to camera
+_description:
+ This test will launch in confined environment. When asked for permission to
+ use the camera, answer NO. Later, Checkbox will check if apparmor really 
+ blocked access to the camera by checking syslog.
+qml_file: camera_denial.qml
+flags: confined
+estimated_duration: 10
+depends: app-armor-denial-syslog-save
+
+id: app-armor-denial-check
+category_id: Apparmor
+plugin: shell
+command: diff /var/log/syslog $PLAINBOX_SESSION_SHARE/before-app-armor-denial-syslog |grep "app-armor-denial.*answer: denied"
+_description: Job that checks if syslog was updated
+flags: preserve-locale
+estimated_duration: 0.1
+depends: app-armor-denial