Looks pretty good now :)

427	+  if (isNuxWindow(window))
428	+  {
429	+    if (mask & PAINT_WINDOW_OCCLUSION_DETECTION_MASK)
430	+    {
431	+      uScreen->nuxRegion += window->geometry();
432	+      uScreen->nuxRegion -= uScreen->overShell;
433	+    }
434	+    return false;
435	+  }
436	+  else if (mask & PAINT_WINDOW_OCCLUSION_DETECTION_MASK &&
437	+           !(mask & PAINT_WINDOW_TRANSLUCENT_MASK) &&
438	+           window->state() & CompWindowStateFullscreenMask)
439	+           // && !window->alpha()  <-- doesn't work. False positives.
440	+  {
441	+    uScreen->overShell += window->geometry();
442	+    uScreen->overShell -= uScreen->nuxRegion;
443	+  }

I can't quite tell based on the context of the diff, but its probably safer to put this after the glPaint call, eg

bool retval = gWindow->glPaint (attrib, transform, region, mask);

And then something like:

bool nux_window = isNuxWindow (window);

if (mask & PAINT_WINDOW_OCCLUSION_DETECTION_MASK)
{
  if (nux_window)
  {
    uScreen->nuxRegion += window->geometry();
    uScreen->nuxRegion -= uScreen->overShell;
  }
  else
  {
    if (window->alpha () && attrib.opacity != opaque)
      mask |= PAINT_WINDOW_TRANSLUCENT_MASK;

    unsigned int noOcclusionMask = PAINT_WINDOW_TRANSLUCENT_MASK | PAINT_WINDOW_TRANSFORMED_MASK | PAINT_WINDOW_NO_CORE_INSTANCE_MASK;

    if (!(mask & noOcclusionMask == noOcclusionMask))
    {
      uScreen->overShell += window->geometry();
      uScreen->overShell -= uScreen->nuxRegion;
    }
  }
}

if (nux_window)
  return false;

return retval;

In that sample, I didn't check to see if the window was a fullscreen window before adding it to the occlusion region. Reason being that it is possible that a nonfullscreen window can be validly stacked above the shell (think override redirect windows, onboard, some other corner cases involving fullscreen windows and nonfullscreen windows). Note also the flags for NO_CORE_INSTANCE_MASK and TRANSFORMED_MASK.

I think the code here for determining whether or not something is over the shell using the occlusion detection is the wrong way around. The occlusion pass happens from bottom to top, so what seems to happen is that for windows stacked top to bottom as such (nux windows prefixed with N)

N1 > N2 > N3 > 4 > 5 > 6

You'll end up with:

overShell = (6 + 5 + 4)
nuxRegion = ((N3 - overShell) + (N2 - overShell) + (N1 - overShell))

Since when we hit the isNuxWindow case at N3, we add N3 to nuxRegion, and then subtract overShell from it (even though those windows aren't actually over the shell).

If you have no windows underneath the shell and a fullscreen window on top like this:

F1 > N2 > N3 > N4 > 5 > 6 > 7

You'll end up with:

nuxRegion = N4 + N3 + N2
overShell = (7 + 6 + 5 + F1) - nuxRegion

So in that case, you'll have a fullscreen window over the shell, and the shell will paint. If you don't have a fullscreen window over the shell, then the parts underneath it will obscure the shell.

The solution in this case is probably to only start to collect the shell region, and clear the "overShell" region when you hit the first nux window in PAINT_WINDOW_OCCLUSION_DETECTION_MASK. Then once you hit another window, you can start to collect the overShell region again.

453	+  /*
454	+   * Paint the shell in *roughly* the compiz stacking order. This is only
455	+   * approximate because we're painting all the nux windows as soon as we find
456	+   * the bottom-most nux window (from bottom to top).
457	+   * But remember to avoid painting the shell if it's within the overShell
458	+   * region.
459	+   */
460	+  if (uScreen->doShellRepaint &&
461	+      !uScreen->forcePaintOnTop () &&
462	+      !uScreen->overShell.contains(window->geometry())
463	+     )

Dunno if that's in glPaint or glDraw. If its the former, make sure we're not doing an occlusion detection pass.

216	+  overShell = CompRegion();
217	+  nuxRegion = CompRegion();

I think this can be optimized by using overShell -= infiniteRegion. Not sure.