Merge lp:~vanvugt/compiz-core/fix-731685 into lp:compiz-core/0.9.5

[Daniel van Vugt]

Fix dock shadow artifacts with rounded window decorations. (LP: #731685)

Removes the assumption that all window decorations are rectangular when overlapping the shadow of a dock like gnome-panel.

[Sam Spilsbury (smspillaz)]

Indeed this assumption (window is not transformed) is in place.

However, I'd prefer not to render shadows on top of the desktop window. I think that there's a better way to handle this and ensure that we're not rendering the decoration twice. It might be worth clipping the shadow based on the frame region rather than the border rect itself.

[Daniel van Vugt (vanvugt)]

The code is not fresh in my mind, however I think you will find that:

* This fix is much smaller, simpler to maintain and probably faster than the clipping/stencilling you talk about.
* This fix potentially supports smoothed window decorations, but clipping/stencilling would not.
* This fix works right now. No other working fix exists AFAIK :)

[Daniel van Vugt (vanvugt)]

Also, I don't think this proposal is "rendering the decoration twice".

It avoids rendering all dock shadows until it draws the desktop window. Only then does it render the dock shadow for the first time. You might have to apply the patch and look at the resulting DecorWindow::glDraw() to see why.

[Sam Spilsbury (smspillaz)]

On Thu, Sep 1, 2011 at 12:09 PM, Daniel van Vugt <email address hidden> wrote:
> The code is not fresh in my mind, however I think you will find that:
>
> * This fix is much smaller, simpler to maintain and probably faster than the clipping/stencilling you talk about.
> * This fix potentially supports smoothed window decorations, but clipping/stencilling would not.
> * This fix works right now. No other working fix exists AFAIK :)

Yeah, I agree with all this. In that case, I've changed my mind about
this and will be happy to take it, but I'll note here that decorations
really ought to be moved into their own independently paintable
objects in the paint stack, rather than hooking the paint function for
the desktop window (the primary problem with this approach is that a
plugin can inhibit the painting of a desktop window and then the panel
will have no shadow, which doesn't make any sense to me). However,
since we don't have the infrastructure in core to do that yet, we'll
have to do it this way for now.

>
> --
> https://code.launchpad.net/~vanvugt/compiz/fix-731685-trunk/+merge/70836
> You are requested to review the proposed merge of lp:~vanvugt/compiz/fix-731685-trunk into lp:compiz.
>

--
Sam Spilsbury

[Daniel van Vugt (vanvugt)]

Trying to resubmit the proposal
from: ~vanvugt/compiz-core/fix-731685
to: lp:compiz-core

But Launchpad is just timing out every time. Please wait :(

[Sam Spilsbury (smspillaz)]

+1 as per previous discussion

=== modified file 'plugins/decor/src/decor.cpp'
--- plugins/decor/src/decor.cpp	2011-08-19 14:25:11 +0000
+++ plugins/decor/src/decor.cpp	2011-09-01 07:24:26 +0000
@@ -93,34 +93,8 @@
 {
     shadowRegion = CompRegion (window->outputRect ());
 
-    if (window->type () == CompWindowTypeDockMask)
-    {
-        /* windows above this one in the stack should
-         * clip the shadow */
-
-        CompWindowList::iterator it = std::find (screen->windows ().begin (),
-                                                 screen->windows ().end (),
-                                                 window);
-
-        for (it--; it != screen->windows ().end (); it--)
-        {
-            CompRegion inter;
-
-            if (!(*it)->isViewable ())
-                continue;
-
-            if ((*it)->type () & CompWindowTypeDesktopMask)
-                continue;
-
-	    inter = shadowRegion.intersected ((*it)->borderRect ());
-
-            if (!inter.isEmpty ())
-		shadowRegion = shadowRegion.subtracted (inter);
-
-        }
-    }
-    else if (window->type () == CompWindowTypeDropdownMenuMask ||
-             window->type () == CompWindowTypePopupMenuMask)
+    if (window->type () == CompWindowTypeDropdownMenuMask ||
+        window->type () == CompWindowTypePopupMenuMask)
     {
         /* Other transient menus should clip
          * this menu's shadows, also the panel
@@ -245,6 +219,34 @@
 
     status = gWindow->glDraw (transform, attrib, region, mask);
 
+    /* Don't render dock decorations (shadows) on just any old window */
+    if (!(window->type () & CompWindowTypeDockMask))
+    {
+	glDecorate (transform, attrib, region, mask);
+	/* Render dock decorations (shadows) on desktop windows only */
+	if (window->type () & CompWindowTypeDesktopMask)
+	{
+	    foreach (CompWindow *w, dScreen->cScreen->getWindowPaintList ())
+	    {
+		if ((w->type () & CompWindowTypeDockMask) &&
+		    !(w->destroyed () || w->invisible ()))
+		{
+		    DecorWindow *d = DecorWindow::get (w);
+		    d->glDecorate (transform, attrib, region, mask);
+		}
+	    }
+	}
+    }
+
+    return status;
+}
+
+void
+DecorWindow::glDecorate (const GLMatrix     &transform,
+		         GLFragment::Attrib &attrib,
+		         const CompRegion   &region,
+		         unsigned int       mask)
+{
     const CompRegion reg = (mask & PAINT_WINDOW_TRANSFORMED_MASK) ?
 			   infiniteRegion : shadowRegion.intersected (region);
 
@@ -283,7 +285,7 @@
 	if (gWindow->textures ().empty ())
 	    gWindow->bind ();
 	if (gWindow->textures ().empty ())
-	    return status;
+	    return;
 
 	if (gWindow->textures ().size () == 1)
 	{
@@ -310,8 +312,6 @@
 	    }
 	}
     }
-
-    return status;
 }
 
 static bool bindFailed;

=== modified file 'plugins/decor/src/decor.h'
--- plugins/decor/src/decor.h	2011-05-08 13:44:52 +0000
+++ plugins/decor/src/decor.h	2011-09-01 07:24:26 +0000
@@ -205,6 +205,8 @@
 
 	bool glDraw (const GLMatrix &, GLFragment::Attrib &,
 		     const CompRegion &, unsigned int);
+	void glDecorate (const GLMatrix &, GLFragment::Attrib &,
+		         const CompRegion &, unsigned int);
 
 	void windowNotify (CompWindowNotify n);