> Sam, gw->priv->clip only seems to get updated after glPaintOutputs which looks
> too late. How does one use clip() normally?

The clip region is updated after the occlusion pass. You are right, this happens within paintOutputRegion and not paintOutputs like I thought.

I feel like there are two options here:

1. Framebuffer object rebinds are not condition on whether or not the shell is repainting, but just as it is now with BackgroundEffectHelper::HasDirtyHelpers (). That leaves the problem of windows on top of the shell. At the moment, the damage collection code examines all damage events, but we don't care about stuff underneath the shell. As such, you could probably ignore any damage events coming in between PaintDisplay and glPaintOutput (damageRegionSetEnabled (false)). I don't think this will handle the case of fullscreen windows on top of the shell very well however. In compiz at least, we'd need a concept of layered damage, which is not trivial as it requires all damage to be tied to a particular object.

At this point, you can determine whether or not to paint the shell when you actually hit a nux window. This again, depends on being able to see nux windows in the paint stack (see lp:~smspillaz/unity/unity.less-paint-insanity). I guess at the moment, you could use the heuristic, and then Window -> CompWindow the nux window and check the clip region there to see if it is empty.

2. Second option is a bit trickier, but might also work. You'll need to effectively make unity do the occlusion pass. What you would do here is loop all the windows and do something like:

class UnityWindow :
....
{
....
private:

    CompRegion clip_;
};

void DetectOcclusionsOnUnityWindows (const GLMatrix   &matrix,
                                     CompOutput       *output)
{
GLWindowPaintAttrib a;
CompRegion          tmpRegion (*output);

/* Top to bottom */
for (CompWindow *w : screen->reverseWindows ())
{
    UnityWindow *uw = UnityWindow::get (w);

    /* We only care about windows we actually clipped */
    clip = infiniteRegion;

    if (window->alpha ())
        continue;

    if (!window->isViewable ())
        continue;

    /* GLWindow::glPaint will return true if this window should be regarded
     * as occluding other windows. PAINT_WINDOW_OCCLUSION_DETECTION_MASK will
     * not actually paint the window, but will allow plugins to modify the state
     * of the window as if it was being painted, so we will know whether it should
     * occlude other windows */
    if (gw->glPaint (a, matrix, tmpRegion, PAINT_WINDOW_OCCLUSION_DETECTION_MASK))
        tmpRegion -= w->region (); // Not inputRect () or outputRect (): Decorations are ARGB windows

    uw->clip_ = tmpRegion;
}
}

Now, you can loop the list of nux windows (probably safe to store that internally, a nux window is a nux window for life, just detect it on the UnityWindow ctor) and see if they all have empty clip regions. If so, don't paint the shell.

You can also use the region detection to determine what parts of the shell to "damage" as well.

iirc, this is something similar to what the blur plugin does. Have a look at that for other ideas. This will also fix problems to do with windows being transformed and still "occluding" the shell even if they aren't. For example, scale with fullscreen windows.

> 
> shellIsHidden is designed to be fast. And it is AFAIK. Yes, I did already add
> a comment about transparent windows. But we're only talking about fullscreen
> windows here. And I don't think the case of transparent or alpha fullscreen
> windows is one that we should need to waste time handling. If you make a
> fullscreen window transparent then you won't see the shell behind it. Does
> that matter?

Fullscreen argb-glx windows are not too far away from reality. In addition, there are things like fullscreen transparent terminals (I hate the terminal usecase). Fullscreen windows could become transparent thanks to other running effects, and I don't think we should design ourselves into a corner by sacrificing the consistency of how the UI works.

There are no games with alpha channels, so where the window isn't transparent, then there will be no impact anyways. (Using glPaint to do occlusions handles all that for you).