Merge lp:~mir-team/mir/relative-pointer-events into lp:mir

[Robert Carr]

Provide relative cursor axes as a way to allow Unity8 to draw the pointer.
(LP: #1276322)

Provide relative cursor axes as a way to allow Unity8 to draw the cursor.

This is provided in lieu of the full raw events API we have discussed. The raw events API may still come at a later time other use cases. I wanted to avoid hacking up the android input layer too much when we are intending to replace it soon though.

I have a concern with this branch in that it leaks the location of clients in some cases. If a client receives a stream of input events where dx/dy is changing but x/y is constant then they now know they are on the edge of the screen. I wonder if this is a real problem? If so we could only allow fullscreen surfaces to receive relative coordinates...or brainstorm for more solutions!

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:2747
http://jenkins.qa.ubuntu.com/job/mir-ci/4326/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/3174
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/mir-clang-ts-wily-amd64-build/27
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-clang-wily-amd64-build/699
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/3122
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-wily-amd64-ci/475
        deb: http://jenkins.qa.ubuntu.com/job/mir-wily-amd64-ci/475/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/3122
        deb: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/3122/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-mako/5927
    SUCCESS: http://s-jenkins.ubuntu-ci:8080/job/touch-flash-device/21891

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/4326/rebuild

review: Approve (continuous-integration)

[Andreas Pokorny (andreas-pokorny)]

It would be nice to also have a test case that moves the mouse beyond the border of the screen, and verify that even when - from the pov of the pointer controller - the mouse did not move due to the screen boundaries we still send out relative position changes.

From a short dive through our pointer controller - we seem to override that position related filtering.

review: Approve

[Robert Carr (robertcarr)]

Good idea Andreas I manually tested this but something came up in doing the automated testing! Notably it turns out we are batching pointer events which made verifying large streams of events difficult (e.g. sometimes I would get (0,0) -> (1, 1) -> (2, 2) and sometimes (0, 0) -> (2, 2)).

I've updated the test to feature your suggestion and proposed a dependency branch which ensures only touch events are batched.

=== modified file '3rd_party/android-input/android/frameworks/base/services/input/InputReader.cpp'
--- 3rd_party/android-input/android/frameworks/base/services/input/InputReader.cpp	2015-06-17 05:20:42 +0000
+++ 3rd_party/android-input/android/frameworks/base/services/input/InputReader.cpp	2015-07-16 21:29:50 +0000
@@ -2445,6 +2445,9 @@
 
     mPointerVelocityControl.move(when, &deltaX, &deltaY);
 
+    pointerCoords.setAxisValue(AMOTION_EVENT_AXIS_RX, deltaX);
+    pointerCoords.setAxisValue(AMOTION_EVENT_AXIS_RY, deltaY);
+
     if (mPointerController != NULL) {
         if (moved || scrolled || buttonsChanged) {
             mPointerController->setPresentation(

=== modified file '3rd_party/android-input/android/frameworks/base/services/input/InputTransport.cpp'
--- 3rd_party/android-input/android/frameworks/base/services/input/InputTransport.cpp	2015-04-28 07:54:10 +0000
+++ 3rd_party/android-input/android/frameworks/base/services/input/InputTransport.cpp	2015-07-16 21:29:50 +0000
@@ -61,6 +61,16 @@
     return a + alpha * (b - a);
 }
 
+namespace
+{
+bool source_is_cursor(int32_t source)
+{
+    return source == AINPUT_SOURCE_MOUSE || source == AINPUT_SOURCE_TRACKBALL || source == AINPUT_SOURCE_TOUCHPAD;
+}
+}
+
+
+
 // --- InputMessage ---
 
 InputMessage::InputMessage()
@@ -463,7 +473,8 @@
 
             // Start a new batch if needed.
             if (mMsg.body.motion.action == AMOTION_EVENT_ACTION_MOVE
-                    || mMsg.body.motion.action == AMOTION_EVENT_ACTION_HOVER_MOVE) {
+                    || mMsg.body.motion.action == AMOTION_EVENT_ACTION_HOVER_MOVE
+                && !source_is_cursor(mMsg.body.motion.source)) {
                 mBatches.push();
                 Batch& batch = mBatches.editTop();
                 batch.samples.push(mMsg);

=== modified file 'include/client/mir/events/event_builders.h'
--- include/client/mir/events/event_builders.h	2015-06-18 02:46:16 +0000
+++ include/client/mir/events/event_builders.h	2015-07-16 21:29:50 +0000
@@ -60,12 +60,20 @@
     float pressure_value, float touch_major_value, float touch_minor_value, float size_value);
 
 // Pointer event
+// Deprecated version without relative axis
 EventUPtr make_event(MirInputDeviceId device_id, std::chrono::nanoseconds timestamp,
     MirInputEventModifiers modifiers, MirPointerAction action,
     MirPointerButtons buttons_pressed,
     float x_axis_value, float y_axis_value,
     float hscroll_value, float vscroll_value);
 
+EventUPtr make_event(MirInputDeviceId device_id, std::chrono::nanoseconds timestamp,
+    MirInputEventModifiers modifiers, MirPointerAction action,
+    MirPointerButtons buttons_pressed,
+    float x_axis_value, float y_axis_value,
+    float hscroll_value, float vscroll_value,
+    float relative_x_value, float relative_y_value);
+
 // Input configuration event
 EventUPtr make_event(MirInputConfigurationAction action,
     MirInputDeviceId id, std::chrono::nanoseconds time);

=== modified file 'include/client/mir_toolkit/events/input/pointer_event.h'
--- include/client/mir_toolkit/events/input/pointer_event.h	2015-06-25 11:26:34 +0000
+++ include/client/mir_toolkit/events/input/pointer_event.h	2015-07-16 21:29:50 +0000
@@ -61,7 +61,12 @@
 /* Relative axis containing ticks reported by the vertical scroll wheel */
     mir_pointer_axis_vscroll = 2,
 /* Relative axis containing ticks reported by the horizontal scroll wheel */
-    mir_pointer_axis_hscroll = 3
+    mir_pointer_axis_hscroll = 3,
+/* Relative axis containing the last reported x differential from the pointer */
+    mir_pointer_axis_relative_x = 4,
+/* Relative axis containing the last reported y differential from the pointer */
+    mir_pointer_axis_relative_y = 5
+
 } MirPointerAxis;
 
 /* 

=== modified file 'include/test/mir/test/event_matchers.h'
--- include/test/mir/test/event_matchers.h	2015-06-22 03:03:59 +0000
+++ include/test/mir/test/event_matchers.h	2015-07-16 21:29:50 +0000
@@ -308,6 +308,20 @@
     return true;
 }
 
+MATCHER_P2(PointerEventWithDiff, dx, dy, "")
+{
+    auto pev = maybe_pointer_event(to_address(arg));
+    if (pev == nullptr)
+        return false;
+    if (mir_pointer_event_action(pev) != mir_pointer_action_motion)
+        return false;
+    if (mir_pointer_event_axis_value(pev, mir_pointer_axis_relative_x) != dx)
+        return false;
+    if (mir_pointer_event_axis_value(pev, mir_pointer_axis_relative_y) != dy)
+        return false;
+    return true;
+}
+
 MATCHER_P4(TouchEventInDirection, x0, y0, x1, y1, "")
 {
     auto tev = maybe_touch_event(to_address(arg));

=== modified file 'src/client/event_printer.cpp'
--- src/client/event_printer.cpp	2015-05-19 21:34:34 +0000
+++ src/client/event_printer.cpp	2015-07-16 21:29:50 +0000
@@ -249,6 +249,8 @@
                 << mir_pointer_event_action(pointer_event) << ", button_state=" << button_state
                 << ", x=" << mir_pointer_event_axis_value(pointer_event, mir_pointer_axis_x)
                 << ", y=" << mir_pointer_event_axis_value(pointer_event, mir_pointer_axis_y)
+                << ", dx=" << mir_pointer_event_axis_value(pointer_event, mir_pointer_axis_relative_x)
+                << ", dy=" << mir_pointer_event_axis_value(pointer_event, mir_pointer_axis_relative_y)
                 << ", vscroll=" << mir_pointer_event_axis_value(pointer_event, mir_pointer_axis_vscroll)
                 << ", hscroll=" << mir_pointer_event_axis_value(pointer_event, mir_pointer_axis_hscroll)
                 << ", modifiers=" << mir_pointer_event_modifiers(pointer_event) << ')';

=== modified file 'src/client/events/event_builders.cpp'
--- src/client/events/event_builders.cpp	2015-06-18 02:46:16 +0000
+++ src/client/events/event_builders.cpp	2015-07-16 21:29:50 +0000
@@ -183,7 +183,8 @@
     MirInputEventModifiers modifiers, MirPointerAction action,
     MirPointerButtons buttons_pressed,                               
     float x_axis_value, float y_axis_value,
-    float hscroll_value, float vscroll_value)
+    float hscroll_value, float vscroll_value,
+    float relative_x_value, float relative_y_value)
 {
     auto e = new MirEvent;
     memset(e, 0, sizeof (MirEvent));
@@ -201,12 +202,24 @@
     mev.pointer_coordinates[0].action = action;
     pc.x = x_axis_value;
     pc.y = y_axis_value;
+    pc.dx = relative_x_value;
+    pc.dy = relative_y_value;
     pc.hscroll = hscroll_value;
     pc.vscroll = vscroll_value;
     
     return make_event_uptr(e);
 }
 
+mir::EventUPtr mev::make_event(MirInputDeviceId device_id, std::chrono::nanoseconds timestamp,
+    MirInputEventModifiers modifiers, MirPointerAction action,
+    MirPointerButtons buttons_pressed,                               
+    float x_axis_value, float y_axis_value,
+    float hscroll_value, float vscroll_value)
+{
+    return make_event(device_id, timestamp, modifiers, action, buttons_pressed,
+                      x_axis_value, y_axis_value, hscroll_value, vscroll_value, 0, 0);
+}
+
 mir::EventUPtr mev::make_event(mf::SurfaceId const& surface_id, xkb_rule_names const& rules)
 {
     auto e = new MirEvent;

=== modified file 'src/client/input/android/android_input_lexicon.cpp'
--- src/client/input/android/android_input_lexicon.cpp	2015-06-17 05:20:42 +0000
+++ src/client/input/android/android_input_lexicon.cpp	2015-07-16 21:29:50 +0000
@@ -57,7 +57,9 @@
                                        mia::mir_pointer_buttons_from_android(mev->getButtonState()),
                                        mev->getX(0), mev->getY(0),
                                        mev->getRawAxisValue(AMOTION_EVENT_AXIS_HSCROLL, 0),
-                                       mev->getRawAxisValue(AMOTION_EVENT_AXIS_VSCROLL, 0));
+                                       mev->getRawAxisValue(AMOTION_EVENT_AXIS_VSCROLL, 0),
+                                       mev->getRawAxisValue(AMOTION_EVENT_AXIS_RX, 0),
+                                       mev->getRawAxisValue(AMOTION_EVENT_AXIS_RY, 0));
             }
             else
             {

=== modified file 'src/client/input/input_event.cpp'
--- src/client/input/input_event.cpp	2015-06-29 03:29:31 +0000
+++ src/client/input/input_event.cpp	2015-07-16 21:29:50 +0000
@@ -399,6 +399,10 @@
        return old_mev.pointer_coordinates[0].x;
    case mir_pointer_axis_y:
        return old_mev.pointer_coordinates[0].y;
+   case mir_pointer_axis_relative_x:
+       return old_mev.pointer_coordinates[0].dx;
+   case mir_pointer_axis_relative_y:
+       return old_mev.pointer_coordinates[0].dy;
    case mir_pointer_axis_vscroll:
        return old_mev.pointer_coordinates[0].vscroll;
    case mir_pointer_axis_hscroll:

=== modified file 'src/include/common/mir/events/event_private.h'
--- src/include/common/mir/events/event_private.h	2015-06-18 02:46:16 +0000
+++ src/include/common/mir/events/event_private.h	2015-07-16 21:29:50 +0000
@@ -68,6 +68,8 @@
     int id;
     float x;
     float y;
+    float dx;
+    float dy;
     float touch_major;
     float touch_minor;
     float size;

=== modified file 'src/server/input/android/input_sender.cpp'
--- src/server/input/android/input_sender.cpp	2015-06-26 08:00:59 +0000
+++ src/server/input/android/input_sender.cpp	2015-07-16 21:29:50 +0000
@@ -277,6 +277,8 @@
         // here x, y is used instead of the raw co-ordinates and offset is set to zero
         coords[i].setAxisValue(AMOTION_EVENT_AXIS_X, event.pointer_coordinates[i].x);
         coords[i].setAxisValue(AMOTION_EVENT_AXIS_Y, event.pointer_coordinates[i].y);
+        coords[i].setAxisValue(AMOTION_EVENT_AXIS_RX, event.pointer_coordinates[i].dx);
+        coords[i].setAxisValue(AMOTION_EVENT_AXIS_RY, event.pointer_coordinates[i].dy);
 
         coords[i].setAxisValue(AMOTION_EVENT_AXIS_TOUCH_MAJOR, event.pointer_coordinates[i].touch_major);
         coords[i].setAxisValue(AMOTION_EVENT_AXIS_TOUCH_MINOR, event.pointer_coordinates[i].touch_minor);

=== modified file 'src/server/input/android/input_translator.cpp'
--- src/server/input/android/input_translator.cpp	2015-06-17 05:20:42 +0000
+++ src/server/input/android/input_translator.cpp	2015-07-16 21:29:50 +0000
@@ -112,14 +112,18 @@
 
     if (mia::android_source_id_is_pointer_device(args->source))
     {
+        auto const& pc = args->pointerCoords[0];
         auto mir_event = mev::make_event(MirInputDeviceId(args->deviceId),
-                                    args->eventTime,
-                                    mia::mir_modifiers_from_android(args->metaState),
-                                    mia::mir_pointer_action_from_masked_android(args->action & AMOTION_EVENT_ACTION_MASK),
-                                    mia::mir_pointer_buttons_from_android(args->buttonState),
-                                    args->pointerCoords[0].getX(), args->pointerCoords[0].getY(),
-                                    args->pointerCoords[0].getAxisValue(AMOTION_EVENT_AXIS_HSCROLL),
-                                    args->pointerCoords[0].getAxisValue(AMOTION_EVENT_AXIS_VSCROLL));
+            args->eventTime,
+            mia::mir_modifiers_from_android(args->metaState),
+            mia::mir_pointer_action_from_masked_android(args->action & AMOTION_EVENT_ACTION_MASK),
+            mia::mir_pointer_buttons_from_android(args->buttonState),
+            pc.getX(), pc.getY(),
+            pc.getAxisValue(AMOTION_EVENT_AXIS_HSCROLL),
+            pc.getAxisValue(AMOTION_EVENT_AXIS_VSCROLL),
+            pc.getAxisValue(AMOTION_EVENT_AXIS_RX),
+            pc.getAxisValue(AMOTION_EVENT_AXIS_RY)
+        );
 
         if (!valid_motion_event(mir_event->motion))
             return;

=== modified file 'src/server/input/surface_input_dispatcher.cpp'
--- src/server/input/surface_input_dispatcher.cpp	2015-06-24 20:46:14 +0000
+++ src/server/input/surface_input_dispatcher.cpp	2015-07-16 21:29:50 +0000
@@ -225,7 +225,9 @@
         mir_pointer_event_axis_value(pev,mir_pointer_axis_x),
         mir_pointer_event_axis_value(pev,mir_pointer_axis_y),
         mir_pointer_event_axis_value(pev,mir_pointer_axis_hscroll),
-        mir_pointer_event_axis_value(pev,mir_pointer_axis_vscroll)));
+        mir_pointer_event_axis_value(pev,mir_pointer_axis_vscroll),
+        mir_pointer_event_axis_value(pev, mir_pointer_axis_relative_x),
+        mir_pointer_event_axis_value(pev, mir_pointer_axis_relative_y)));
 }
 
 mi::SurfaceInputDispatcher::PointerInputState& mi::SurfaceInputDispatcher::ensure_pointer_state(MirInputDeviceId id)
@@ -299,7 +301,6 @@
         {
             pointer_state.gesture_owner = target;
         }
-
         deliver(target, pev);
         return true;
     }

=== modified file 'tests/acceptance-tests/test_client_input.cpp'
--- tests/acceptance-tests/test_client_input.cpp	2015-06-26 08:00:59 +0000
+++ tests/acceptance-tests/test_client_input.cpp	2015-07-16 21:29:50 +0000
@@ -239,6 +239,38 @@
     first_client.all_events_received.wait_for_at_most_seconds(120);
 }
 
+TEST_F(TestClientInput, clients_receive_relative_pointer_events)
+{
+    using namespace ::testing;
+    
+    positions[first] = geom::Rectangle{{0,0}, {surface_width, surface_height}};
+    Client first_client(new_connection(), first);
+
+    InSequence seq;
+    EXPECT_CALL(first_client, handle_input(mt::PointerEnterEvent()));
+    EXPECT_CALL(first_client, handle_input(AllOf(mt::PointerEventWithPosition(1, 1), mt::PointerEventWithDiff(1, 1))));
+    EXPECT_CALL(first_client, handle_input(AllOf(mt::PointerEventWithPosition(2, 2), mt::PointerEventWithDiff(1, 1))));
+    EXPECT_CALL(first_client, handle_input(AllOf(mt::PointerEventWithPosition(3, 3), mt::PointerEventWithDiff(1, 1))));
+    EXPECT_CALL(first_client, handle_input(AllOf(mt::PointerEventWithPosition(2, 2), mt::PointerEventWithDiff(-1, -1))));
+    EXPECT_CALL(first_client, handle_input(AllOf(mt::PointerEventWithPosition(1, 1), mt::PointerEventWithDiff(-1, -1))));
+    // Ensure we continue to receive relative moement even when absolute movement is constrained.
+    EXPECT_CALL(first_client, handle_input(AllOf(mt::PointerEventWithPosition(0, 0), mt::PointerEventWithDiff(-1, -1))));
+    EXPECT_CALL(first_client, handle_input(AllOf(mt::PointerEventWithPosition(0, 0), mt::PointerEventWithDiff(-1, -1))));
+    EXPECT_CALL(first_client, handle_input(AllOf(mt::PointerEventWithPosition(0, 0), mt::PointerEventWithDiff(-1, -1))))
+        .WillOnce(mt::WakeUp(&first_client.all_events_received));
+
+    fake_mouse->emit_event(mis::a_pointer_event().with_movement(1, 1));
+    fake_mouse->emit_event(mis::a_pointer_event().with_movement(1, 1));
+    fake_mouse->emit_event(mis::a_pointer_event().with_movement(1, 1));
+    fake_mouse->emit_event(mis::a_pointer_event().with_movement(-1, -1));
+    fake_mouse->emit_event(mis::a_pointer_event().with_movement(-1, -1));
+    fake_mouse->emit_event(mis::a_pointer_event().with_movement(-1, -1));
+    fake_mouse->emit_event(mis::a_pointer_event().with_movement(-1, -1));
+    fake_mouse->emit_event(mis::a_pointer_event().with_movement(-1, -1));
+
+    first_client.all_events_received.wait_for_at_most_seconds(120);
+}
+
 TEST_F(TestClientInput, clients_receive_button_events_inside_window)
 {
     Client first_client(new_connection(), first);

=== modified file 'tests/mir_test_framework/fake_input_device_impl.cpp'
--- tests/mir_test_framework/fake_input_device_impl.cpp	2015-06-17 05:20:42 +0000
+++ tests/mir_test_framework/fake_input_device_impl.cpp	2015-07-16 21:29:50 +0000
@@ -224,7 +224,9 @@
                                                  pos.x.as_float(),
                                                  pos.y.as_float(),
                                                  scroll.x.as_float(),
-                                                 scroll.y.as_float());
+                                                 scroll.y.as_float(),
+                                                 pointer.rel_x,
+                                                 pointer.rel_y);
 
     sink->handle_input(*pointer_event);
 }

=== modified file 'tests/unit-tests/input/android/test_android_input_sender.cpp'
--- tests/unit-tests/input/android/test_android_input_sender.cpp	2015-06-25 03:00:08 +0000
+++ tests/unit-tests/input/android/test_android_input_sender.cpp	2015-07-16 21:29:50 +0000
@@ -103,7 +103,7 @@
     float test_y_coord[2] = {17, 9};
     
     AndroidInputSender()
-        : key_event(mev::make_event(MirInputDeviceId(), std::chrono::nanoseconds(1), mir_keyboard_action_down,
+       : key_event(mev::make_event(MirInputDeviceId(), std::chrono::nanoseconds(1), mir_keyboard_action_down,
                                     7, test_scan_code, mir_input_event_modifier_none)),
           motion_event(mev::make_event(MirInputDeviceId(), std::chrono::nanoseconds(-1), mir_input_event_modifier_none))
     {

=== modified file 'tests/unit-tests/input/android/test_input_translator.cpp'
--- tests/unit-tests/input/android/test_input_translator.cpp	2015-06-26 08:00:59 +0000
+++ tests/unit-tests/input/android/test_input_translator.cpp	2015-07-16 21:29:50 +0000
@@ -167,13 +167,19 @@
 
     const float x_pos = 12.0f;
     const float y_pos = 30.0f;
-    EXPECT_CALL(dispatcher, dispatch(mt::PointerEventWithPosition(x_pos, y_pos))).Times(1);
+    const float dx = 7.0f;
+    const float dy = 9.3f;
+    EXPECT_CALL(dispatcher, dispatch(AllOf(
+        mt::PointerEventWithPosition(x_pos, y_pos),
+        mt::PointerEventWithDiff(dx, dy)))).Times(1);
 
     const uint32_t one_pointer = 1;
 
     properties[0].id = 23;
     coords[0].setAxisValue(AMOTION_EVENT_AXIS_X, x_pos);
     coords[0].setAxisValue(AMOTION_EVENT_AXIS_Y, y_pos);
+    coords[0].setAxisValue(AMOTION_EVENT_AXIS_RX, dx);
+    coords[0].setAxisValue(AMOTION_EVENT_AXIS_RY, dy);
     
     droidinput::NotifyMotionArgs motion(some_time, device_id, AINPUT_SOURCE_MOUSE, 0, motion_action, no_flags,
                                         meta_state, button_state, edge_flags, one_pointer, properties, coords,