Mir

Merge lp:~vanvugt/mir/target into lp:mir

target
Merge into development-branch

Proposed by Daniel van Vugt on 2015-04-01

Status:	Merged
Approved by:	Daniel van Vugt on 2015-04-08
Approved revision:	no longer in the source branch.
Merged at revision:	2464
Proposed branch:	lp:~vanvugt/mir/target
Merge into:	lp:mir
Prerequisite:	lp:~mir-team/mir/fix-1438160
Diff against target:	363 lines (+348/-0) 2 files modified examples/CMakeLists.txt (+6/-0) examples/target.c (+342/-0)
To merge this branch:	bzr merge lp:~vanvugt/mir/target
Related bugs:	Link a bug report

Reviewer	Review Type	Date Requested	Status
PS Jenkins bot (community)	continuous-integration		Approve on 2015-04-08
Robert Carr (community)			Approve on 2015-04-07
Alan Griffiths		2015-04-01	Approve on 2015-04-07
Review via email: mp+254885@code.launchpad.net

Commit message

Introducing mir_demo_client_target, for visual testing of input latency.

This provides a visual scale by which to grade input latency, particularly
when watching the mouse pointer. Such a small scale is required if you are
to reliably see any improvement (or regression) of only a few milliseconds
latency.

mir_demo_client_target also does its own input sampling in place of
Mir's default resampling, so achieves about half a frame lower latency on
average. Even with the default swap interval of one. Although forcing swap
interval to zero (-n option) reduces visible latency even more. Doing its
own sampling also allows mir_demo_client_target to render at the full
native frame rate, rather than being limited by the Mir input resampling
rate.

Tip: If you find it hard to see the cursor against the target then using
the zoom feature (Super+mousewheel) of mir_proving_server helps. Because
that will visually amplify any offset.

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2015-04-01:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/3423/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2015-04-02:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/3436/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

Daniel van Vugt (vanvugt) wrote on 2015-04-02:

The failure seems to be unrelated...

28: C++ exception with description "stop_server() failed to stop server" thrown in TearDown().
Build timed out (after 120 minutes). Marking the build as failed.
Build was aborted
Build was marked for publishing on https://jenkins.qa.ubuntu.com/
Finished: FAILURE

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2015-04-02:

FAILED: Continuous integration, rev:2487
http://jenkins.qa.ubuntu.com/job/mir-ci/3437/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-android-vivid-i386-build/1931
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-clang-vivid-amd64-build/1930/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-vivid-touch/1881/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/1434
        deb: http://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/1434/artifact/work/output/*zip*/output.zip
    FAILURE: http://jenkins.qa.ubuntu.com/job/mir-mediumtests-builder-vivid-armhf/1881/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/3437/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

Daniel van Vugt (vanvugt) wrote on 2015-04-02:

Definitely not related to this proposal...

10: [ RUN ] ClientLibrary.create_simple_normal_surface_from_spec
10: [1427959486.308220] Server: Starting
10: [1427959486.308335] Platform Loader: Selected driver: dummy (version 0.13.0)
10: [1427959486.309135] mirserver: Using software cursor
10: [1427959486.311136] DisplayServer: Mir version 0.13.0
10: /mir/tests/acceptance-tests/test_client_library.cpp:655: Failure
10: Value of: native_buffer->width
10: Expected: is equal to 800
10: Actual: 1 (of type int)
10: /mir/tests/acceptance-tests/test_client_library.cpp:656: Failure
10: Value of: native_buffer->height
10: Expected: is equal to 600
10: Actual: 1 (of type int)
10: [1427959486.318845] Server: Stopping
10: [ FAILED ] ClientLibrary.create_simple_normal_surface_from_spec (13 ms)

Revision history for this message

Robert Carr (robertcarr) wrote on 2015-04-02:

I see no problem with this landing but I'm not sure about the analysis/goal.

>> mir_demo_client_target also does its own input resampling

It seems mir_demo_client_target is just sampling input rather than 'resampling'.

>> in place of
>> Mir's default resampling, so achieves about half a frame lower latency on
>> average.

From my understanding this would be the intentional re sampling latency...we can remove it at the cost of varying the sample rate per frame (which in the case of uniform speed scrolls results in jerkiness).

There are some useful visual depictions of what is going on here at:
http://www.masonchang.com/blog/2014/8/11/smooth-scrolling-frame-uniformity-touch-interpolation-and-touch-responsiveness

Revision history for this message

Daniel van Vugt (vanvugt) wrote on 2015-04-07:

You're right, this is just "sampling" input in the most efficient way. Not "re-sampling". I'll adjust the wording...

Indeed, not all apps can work this way. Those that have not been written to handle input efficiently will need the built-in input resampling. Also those that benefit from touch motion prediction will want that. But in theory any app/toolkit could be given optimal input handling by:

  1. Only wake up on input events.
  2. When woken, schedule a frame if none scheduled already.
  3. At the start of each frame, /get/ the latest input state (i.e. current cursor position).
  4. Use the latest input state to render your frame.

This is all stuff I remember being taught in 1998. It's kind of embarrassing we forget it so often.

Revision history for this message

PS Jenkins bot (ps-jenkins) wrote on 2015-04-07:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/mir-ci/3458/rebuild

review: Needs Fixing (continuous-integration)

Revision history for this message

Alan Griffiths (alan-griffiths) wrote on 2015-04-07:

Looks like the 60's are calling but OK

review: Approve

Revision history for this message

Robert Carr (robertcarr) wrote on 2015-04-07:

>> But in theory any app/toolkit could be given optimal input handling by
>> 1. Only wake up on input events.
>> 2. When woken, schedule a frame if none scheduled already.
>> 3. At the start of each frame, /get/ the latest input state (i.e. current cursor position).
>> 4. Use the latest input state to render your frame.

This seems like it would be true but I don't think it is. What this example fails to take in to account is that the "latest input state" is updated at a varying rate from the frame.

This article:
http://www.masonchang.com/blog/2014/8/11/smooth-scrolling-frame-uniformity-touch-interpolation-and-touch-responsiveness

Has a good visual depiction of unresampled input with a sample rate mismatch (the "current code" version of firefox logo).

Example is ok though

review: Approve

Revision history for this message

Daniel van Vugt (vanvugt) wrote on 2015-04-08:

Absolutely in touch scrolling uniformity needs to be considered. That's why we still have a fixed input resampling interval and it still looks better than our attempts to adjust it dynamically did.

But this demo is mainly for tracking the cursor as quickly as possible. Not animation.

Revision history for this message

PS Jenkins bot (ps-jenkins) on 2015-04-08:

review: Approve (continuous-integration)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk

Subscribers

People subscribed via source and target branches

to all changes:

Daniel van Vugt

Emanuele Antonio Faraone

Gerry Boland

Mir development team

 === modified file 'examples/CMakeLists.txt'
 --- examples/CMakeLists.txt	2015-03-31 12:48:35 +0000
 +++ examples/CMakeLists.txt	2015-04-07 07:01:52 +0000
@@ -50,6 +50,12 @@
  target_link_libraries(mir_demo_client_egltriangle
    eglapp
+ )
++mir_add_wrapped_executable(mir_demo_client_target
++  target.c
++)
++target_link_libraries(mir_demo_client_target
++  eglapp
++)
  mir_add_wrapped_executable(mir_demo_client_eglcounter
    eglcounter.cpp
+ )
 === added file 'examples/target.c'
 --- examples/target.c	1970-01-01 00:00:00 +0000
 +++ examples/target.c	2015-04-07 07:01:52 +0000
@@ -0,0 +1,342 @@
++/*
++ * Copyright © 2015 Canonical Ltd.
++ *
++ * This program is free software: you can redistribute it and/or modify
++ * it under the terms of the GNU General Public License version 3 as
++ * published by the Free Software Foundation.
++ *
++ * This program is distributed in the hope that it will be useful,
++ * but WITHOUT ANY WARRANTY; without even the implied warranty of
++ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
++ * GNU General Public License for more details.
++ *
++ * You should have received a copy of the GNU General Public License
++ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
++ *
++ * Author: Daniel van Vugt <daniel.van.vugt@canonical.com>
++ */
++
++#define _POSIX_C_SOURCE 200112L  // for setenv() from stdlib.h
++#include "eglapp.h"
++#include <assert.h>
++#include <stdio.h>
++#include <math.h>
++#include <GLES2/gl2.h>
++#include <mir_toolkit/mir_surface.h>
++#include <pthread.h>
++#include <stdlib.h>
++#include <signal.h>
++
++enum
++{
++    max_touches = 10
++};
++
++typedef struct
++{
++    float x, y;
++} Vec2;
++
++typedef struct
++{
++    int points;
++    Vec2 point[max_touches];
++} TouchState;
++
++typedef struct
++{
++    pthread_mutex_t mutex;
++    pthread_cond_t change;
++    bool changed;
++
++    bool resized;
++    TouchState touch;
++} State;
++
++static GLuint load_shader(const char *src, GLenum type)
++{
++    GLuint shader = glCreateShader(type);
++    if (shader)
++    {
++        GLint compiled;
++        glShaderSource(shader, 1, &src, NULL);
++        glCompileShader(shader);
++        glGetShaderiv(shader, GL_COMPILE_STATUS, &compiled);
++        if (!compiled)
++        {
++            GLchar log[1024];
++            glGetShaderInfoLog(shader, sizeof log - 1, NULL, log);
++            log[sizeof log - 1] = '\0';
++            printf("load_shader compile failed: %s\n", log);
++            glDeleteShader(shader);
++            shader = 0;
++        }
++    }
++    return shader;
++}
++
++GLuint generate_target_texture()
++{
++    const int width = 512, height = width;
++    typedef struct { GLubyte r, b, g, a; } Texel;
++    Texel image[height][width];
++    // Note the 0.5f to convert from pixel corner (GL) to middle (image)
++    const float centrex = width/2 - 0.5f, centrey = height/2 - 0.5f;
++    const Texel blank = {0, 0, 0, 0};
++    const int radius = centrex - 1;
++    const Texel ring[] =
++    {
++        {  0,   0,   0, 255},
++        {  0,   0, 255, 255},
++        {  0, 255,   0, 255},
++        {255, 255,   0, 255},
++        {255, 128,   0, 255},
++        {255,   0,   0, 255},
++    };
++    const int rings = sizeof(ring) / sizeof(ring[0]);
++
++    for (int y = 0; y < height; ++y)
++    {
++        for (int x = 0; x < width; ++x)
++        {
++            float dx = x - centrex, dy = y - centrey;
++            int layer = rings * sqrtf(dx * dx + dy * dy) / radius;
++            image[y][x] = layer < rings ? ring[layer] : blank;
++        }
++    }
++
++    GLuint tex;
++    glGenTextures(1, &tex);
++    glBindTexture(GL_TEXTURE_2D, tex);
++    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_REPEAT);
++    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_REPEAT);
++    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER,
++                                   GL_LINEAR_MIPMAP_LINEAR);
++    glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
++    glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, width, height, 0, GL_RGBA,
++                 GL_UNSIGNED_BYTE, image);
++    glGenerateMipmap(GL_TEXTURE_2D);
++
++    return tex;
++}
++
++static void get_all_touch_points(const MirInputEvent *ievent, TouchState *touch)
++{
++    if (mir_input_event_get_type(ievent) == mir_input_event_type_pointer)
++    {
++        touch->points = 0;
++        const MirPointerEvent *pevent =
++            mir_input_event_get_pointer_event(ievent);
++        if (mir_pointer_event_action(pevent) != mir_pointer_action_leave)
++        {
++            touch->points = 1;
++            touch->point[0] = (Vec2)
++            {
++                mir_pointer_event_axis_value(pevent, mir_pointer_axis_x),
++                mir_pointer_event_axis_value(pevent, mir_pointer_axis_y)
++            };
++        }
++    }
++    else if (mir_input_event_get_type(ievent) == mir_input_event_type_touch)
++    {
++        const MirTouchEvent *tevent = mir_input_event_get_touch_event(ievent);
++        int n = mir_touch_event_point_count(tevent);
++        if (n > max_touches)
++            n = max_touches;
++        bool all_up = true;
++        for (int p = 0; p < n; ++p)
++        {
++            if (mir_touch_event_action(tevent, p) != mir_touch_action_up)
++                all_up = false;
++            touch->point[p] = (Vec2)
++            {
++                mir_touch_event_axis_value(tevent, p, mir_touch_axis_x),
++                mir_touch_event_axis_value(tevent, p, mir_touch_axis_y)
++            };
++        }
++        touch->points = all_up ? 0 : n;
++    }
++}
++
++static void on_event(MirSurface *surface, const MirEvent *event, void *context)
++{
++    (void)surface;
++    State *state = (State*)context;
++
++    // FIXME: We presently need to know that events come in on a different
++    //        thread to main (LP: #1194384). When that's resolved, simple
++    //        single-threaded apps like this won't need pthread.
++    pthread_mutex_lock(&state->mutex);
++
++    switch (mir_event_get_type(event))
++    {
++    case mir_event_type_input:
++        get_all_touch_points(mir_event_get_input_event(event), &state->touch);
++        break;
++    case mir_event_type_resize:
++        state->resized = true;
++        break;
++    case mir_event_type_close_surface:
++        // TODO: eglapp.h needs a quit() function or different behaviour of
++        //       mir_eglapp_shutdown().
++        raise(SIGTERM);  // handled by eglapp
++        break;
++    default:
++        break;
++    }
++
++    state->changed = true;
++    pthread_cond_signal(&state->change);
++    pthread_mutex_unlock(&state->mutex);
++}
++
++int main(int argc, char *argv[])
++{
++    const char vshadersrc[] =
++        "attribute vec2 position;\n"
++        "attribute vec2 texcoord;\n"
++        "uniform float scale;\n"
++        "uniform vec2 translate;\n"
++        "uniform mat4 projection;\n"
++        "varying vec2 v_texcoord;\n"
++        "\n"
++        "void main()\n"
++        "{\n"
++        "    gl_Position = projection *\n"
++        "                  vec4(position * scale + translate, 0.0, 1.0);\n"
++        "    v_texcoord = texcoord;\n"
++        "}\n";
++
++    const char fshadersrc[] =
++        "precision mediump float;\n"
++        "varying vec2 v_texcoord;\n"
++        "uniform sampler2D texture;\n"
++        "\n"
++        "void main()\n"
++        "{\n"
++        "    gl_FragColor = texture2D(texture, v_texcoord);\n"
++        "}\n";
++
++    // Disable Mir's input resampling. We do our own here, in a way that
++    // has even lower latency than Mir's default algorithm.
++    // TODO: Make a proper client API function for this:
++    setenv("MIR_CLIENT_INPUT_RATE", "0", 0);
++
++    static unsigned int width = 0, height = 0;
++    if (!mir_eglapp_init(argc, argv, &width, &height))
++        return 1;
++
++    GLuint vshader = load_shader(vshadersrc, GL_VERTEX_SHADER);
++    assert(vshader);
++    GLuint fshader = load_shader(fshadersrc, GL_FRAGMENT_SHADER);
++    assert(fshader);
++    GLuint prog = glCreateProgram();
++    assert(prog);
++    glAttachShader(prog, vshader);
++    glAttachShader(prog, fshader);
++    glLinkProgram(prog);
++
++    GLint linked;
++    glGetProgramiv(prog, GL_LINK_STATUS, &linked);
++    if (!linked)
++    {
++        GLchar log[1024];
++        glGetProgramInfoLog(prog, sizeof log - 1, NULL, log);
++        log[sizeof log - 1] = '\0';
++        printf("Link failed: %s\n", log);
++        return 2;
++    }
++
++    glUseProgram(prog);
++
++    const GLfloat square[] =
++    { // position      texcoord
++        -0.5f, +0.5f,  0.0f, 1.0f,
++        +0.5f, +0.5f,  1.0f, 1.0f,
++        +0.5f, -0.5f,  1.0f, 0.0f,
++        -0.5f, -0.5f,  0.0f, 0.0f,
++    };
++    GLint position = glGetAttribLocation(prog, "position");
++    GLint texcoord = glGetAttribLocation(prog, "texcoord");
++    glVertexAttribPointer(position, 2, GL_FLOAT, GL_FALSE, 4*sizeof(GLfloat),
++                          square);
++    glVertexAttribPointer(texcoord, 2, GL_FLOAT, GL_FALSE, 4*sizeof(GLfloat),
++                          square+2);
++    glEnableVertexAttribArray(position);
++    glEnableVertexAttribArray(texcoord);
++
++    GLint scale = glGetUniformLocation(prog, "scale");
++    glUniform1f(scale, 128.0f);
++
++    GLint translate = glGetUniformLocation(prog, "translate");
++    glUniform2f(translate, 0.0f, 0.0f);
++
++    GLint projection = glGetUniformLocation(prog, "projection");
++
++    GLuint tex = generate_target_texture();
++    glBindTexture(GL_TEXTURE_2D, tex);
++
++    glClearColor(0.0f, 0.0f, 0.0f, 1.0f);
++    glViewport(0, 0, width, height);
++
++    glEnable(GL_BLEND);
++    glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
++
++    State state =
++    {
++        PTHREAD_MUTEX_INITIALIZER,
++        PTHREAD_COND_INITIALIZER,
++        true,
++        true,
++        {0, {{0,0}}}
++    };
++    MirSurface *surface = mir_eglapp_native_surface();
++    mir_surface_set_event_handler(surface, on_event, &state);
++
++    while (mir_eglapp_running())
++    {
++        pthread_mutex_lock(&state.mutex);
++
++        while (mir_eglapp_running() && !state.changed)
++            pthread_cond_wait(&state.change, &state.mutex);
++
++        if (state.resized)
++        {
++            // mir_eglapp_swap_buffers updates the viewport for us...
++            GLint viewport[4];
++            glGetIntegerv(GL_VIEWPORT, viewport);
++            int w = viewport[2], h = viewport[3];
++
++            // TRANSPOSED projection matrix to convert from the Mir input
++            // rectangle {{0,0},{w,h}} to GL screen rectangle {{-1,1},{2,2}}.
++            GLfloat matrix[16] = {2.0f/w, 0.0f,   0.0f, 0.0f,
++                                  0.0f,  -2.0f/h, 0.0f, 0.0f,
++                                  0.0f,   0.0f,   1.0f, 0.0f,
++                                 -1.0f,   1.0f,   0.0f, 1.0f};
++            // Note GL_FALSE: GLES does not support the transpose option
++            glUniformMatrix4fv(projection, 1, GL_FALSE, matrix);
++            state.resized = false;
++        }
++
++        glClear(GL_COLOR_BUFFER_BIT);
++
++        for (int p = 0; p < state.touch.points; ++p)
++        {
++            // Note the 0.5f to convert from pixel middle to corner (GL)
++            glUniform2f(translate, state.touch.point[p].x + 0.5f,
++                                   state.touch.point[p].y + 0.5f);
++            glDrawArrays(GL_TRIANGLE_FAN, 0, 4);
++        }
++
++        // Put the event loop back to sleep:
++        state.changed = false;
++        pthread_mutex_unlock(&state.mutex);
++
++        mir_eglapp_swap_buffers();
++    }
++
++    mir_surface_set_event_handler(surface, NULL, NULL);
++    mir_eglapp_shutdown();
++
++    return 0;
++}

Mir

Merge lp:~vanvugt/mir/target into lp:mir

Commit message

Description of the change

Preview Diff

Subscribers