Merge lp:~jeremywootten/screenshot-tool/fix-1438250-include-child-windows into lp:~elementary-apps/screenshot-tool/trunk

[Jeremy Wootten]

Include menus, tooltips and other child windows as well as non-CSD decorations in screenshot.

This branch provides a solution to the problem of including popups and menus in a snapshot of a current window. Instead of using pixbuf_get_from_window, the same method as used for grabbing selected areas is used - that is obtaining a subpixbuf from the whole window - after calculating a suitable selection rectangle.

This is complicated by the fact that window.get_frame_extents includes any border around the main window area used to show shadows etc. The function that discounts this relies on it having a low opacity compared with the rest of the window. The method therefore relies on the pixbufs obtained will have four channels per pixel with one byte per channel. It is not known whether changing to Wayland will affect this.

The final screenshot is a composite of both methods in order that the window shadow and non-CSD decorations are included (if present). In order to show the shadow more clearly it is drawn on a white background.

It is recognised that this solution is "hacky" so if a simpler solution is available it should be preferred.

If the current window is partially off the current screen any menus or popups extending outside the main window will be truncated (as they are not drawn by the current window).

While rewriting grab_save (), the method of capturing the cursor position was changed in order to place it more accurately. (lp:1627704)

In addition, due to problems experienced during development where the program crashed while system text was redacted, code was included that recovered the normal text when the program was run again.

Testing on a hi-DPI as well as a standard display is desirable.

[Sergey Kislyakov (defman21)]

You can't capture the Terminal window anymore - it crashes with segfault.

review: Needs Fixing

[Sergey Kislyakov (defman21)]

There are no shadows when making a screenshot of a window (no matter CSD or not). They were there before.

review: Needs Fixing

[Jeremy Wootten (jeremywootten)]

The r317 should fix the problems with capturing terminal (which is translucent) and adds the shadow to the final result. However, it no longer fixes bug 1678520 so I am putting it back to "in progress" while seeking a solution.

[Jeremy Wootten (jeremywootten)]

The last revision provides a solution to the non-CSD window bug by using Cairo rather than Gdk to composite the final screenshot. Some care was required to ensure reasonable handling of partially off-screen windows in each case.

[lp:~jeremywootten/screenshot-tool/fix-1438250-include-child-windows]

319. By Jeremy Wootten on 2017-04-08

Composite using Cairo

[Sergey Kislyakov (defman21)]

Well it works fine, but non CSD windows lacks shadow. Regardless of that, it works fine. Nice job!

review: Needs Fixing

[Jeremy Wootten (jeremywootten)]

I do not think there is a way of capturing shadows on non CSD windows without capturing the underlying screen which could be objected to on security grounds. This was the problem with an earlier version of this fix. At the end of the day, it is always possible to include whatever you want by using the "grab selected area" option.

[Sergey Kislyakov (defman21)]

I thought shadows are being applied after taking a screenshot of a window. I can workaround it by applying shadows using some external tools like ImageMagick though.

[Jeremy Wootten (jeremywootten)]

That is the case for CSD windows, where "clean" shadows are present in the image obtained from Gdk.pixbuf_get_from_window () and are composited into the final result. However, this does not work for non-CSD windows - I do not know how to get "clean" shadows for those.

[Sergey Kislyakov (defman21)]

I see. Well at least I can now capture non-CSD windows with their titles.

review: Approve

[Danielle Foré (danrabbit)]

This branch breaks transparent screenshots. There is now a white background

review: Needs Fixing

[Jeremy Wootten (jeremywootten)]

The background of the shadow region is intentionally white in order to make the shadow visible but that could be changed easily. However, using terminal with the background alpha dialled right down is a test, the desktop is visible through the transparent parts so is whatever colour(s) the desktop is. This is inevitable if the associated child windows are to be captured using this method.

One possibility would be to have another RadioButton choice "Capture active window with child windows" and keep the old method as well for when that is more appropriate.

[Jeremy Wootten (jeremywootten)]

Actually, there may be a way of dealing with transparent windows without showing the underlying if it is assumed that any child windows (menus, tooltips etc) are opaque - at the cost of increased complexity in the compositing.

[Jeremy Wootten (jeremywootten)]

Now working on github branch.

[Jeremy Wootten (jeremywootten)]

The possible refinement mentioned above did not work out - sorry :(

Unmerged revisions

319. By Jeremy Wootten on 2017-04-08

Composite using Cairo

318. By Jeremy Wootten on 2017-04-06

Merge trunk to r318 and resolve conflict

317. By Jeremy Wootten on 2017-04-06

Fix detect shadow; composite shadow in screenshot

316. By Jeremy Wootten on 2017-03-23

Fix placement of cursor

315. By Jeremy Wootten on 2017-03-21

Include popups and menus in current window snapshot; Recover from redacted system after crash

=== modified file 'src/ScreenshotWindow.vala'
--- src/ScreenshotWindow.vala	2017-03-21 17:13:44 +0000
+++ src/ScreenshotWindow.vala	2017-04-08 18:20:47 +0000
@@ -30,9 +30,9 @@
         private Settings settings = new Settings ("net.launchpad.screenshot");
 
         private CaptureType capture_mode;
-        private string prev_font_regular;
-        private string prev_font_document;
-        private string prev_font_mono;
+        private string? prev_font_regular = null;
+        private string? prev_font_document = null;
+        private string? prev_font_mono = null;
 
         private bool mouse_pointer;
         private bool from_command;
@@ -55,6 +55,8 @@
         }
 
         construct {
+            redact_text (false); /* Ensure system is not redacted */
+
             if (from_command) {
                 return;
             }
@@ -224,67 +226,44 @@
                 return false;
             });
 
-            var win_rect = Gdk.Rectangle ();
             var root = Gdk.get_default_root_window ();
 
             if (win == null) {
                 win = root;
             }
 
-            Gdk.Pixbuf? screenshot;
-            int scale_factor = win.get_scale_factor ();
-
-            if (capture_mode == CaptureType.AREA) {
-                Gdk.Rectangle selection_rect;
-                win.get_frame_extents (out selection_rect);
-
-                screenshot = new Gdk.Pixbuf.subpixbuf (Gdk.pixbuf_get_from_window (root, 0, 0, root.get_width (), root.get_height ()),
-                                                    selection_rect.x, selection_rect.y, selection_rect.width, selection_rect.height);
-
-                win_rect.x = selection_rect.x;
-                win_rect.y = selection_rect.y;
-                win_rect.width = selection_rect.width;
-                win_rect.height = selection_rect.height;
-            } else {
-                int width = win.get_width ();
-                int height = win.get_height ();
-
-                // Check the scaling factor in use, and if greater than 1 scale the image. (for HiDPI displays)
-                if (scale_factor > 1 && capture_mode == CaptureType.SCREEN) {
-                    screenshot = Gdk.pixbuf_get_from_window (win, 0, 0, width / scale_factor, height / scale_factor);
-                    screenshot.scale (screenshot, width, height, width, height, 0, 0, scale_factor, scale_factor, Gdk.InterpType.BILINEAR);
-                } else {
-                    screenshot = Gdk.pixbuf_get_from_window (win, 0, 0, width, height);
-                }
-
-                win_rect.x = 0;
-                win_rect.y = 0;
-                win_rect.width = width;
-                win_rect.height = height;
-            }
-
-            if (screenshot == null) {
+            var root_width = root.get_width ();
+            var root_height = root.get_height ();
+            int scale_factor = root.get_scale_factor ();
+
+            Gdk.Pixbuf? root_pix = get_scaled_pixbuf_from_window (root, capture_mode, scale_factor);
+            /* Ensure redacted text restored asap */
+            redact_text (false);
+
+            if (root_pix == null) {
                 show_error_dialog ();
                 return false;
             }
 
             if (mouse_pointer) {
+                /* Getting actual cursor for current window fails for some reason so we construct a default cursor */
                 var cursor = new Gdk.Cursor.for_display (Gdk.Display.get_default (), Gdk.CursorType.LEFT_PTR);
                 var cursor_pixbuf = cursor.get_image ();
 
+                /* It is easier to accurately place cursor by compositing cursor image with the root screen */
+                /* It may or may not appear in the final screenshot */
                 if (cursor_pixbuf != null) {
                     var manager = Gdk.Display.get_default ().get_device_manager ();
                     var device = manager.get_client_pointer ();
 
                     int cx, cy, xhot, yhot;
-                    win.get_device_position (device, out cx, out cy, null);
+                    root.get_device_position (device, out cx, out cy, null);
                     xhot = int.parse (cursor_pixbuf.get_option ("x_hot")); // Left padding in cursor_pixbuf between the margin and the actual pointer
                     yhot = int.parse (cursor_pixbuf.get_option ("y_hot")); // Top padding in cursor_pixbuf between the margin and the actual pointer
 
                     var cursor_rect = Gdk.Rectangle ();
-
-                    cursor_rect.x = cx + win_rect.x - xhot;
-                    cursor_rect.y = cy + win_rect.y - yhot;
+                    cursor_rect.x = cx - xhot;
+                    cursor_rect.y = cy - yhot;
                     cursor_rect.width = cursor_pixbuf.get_width ();
                     cursor_rect.height = cursor_pixbuf.get_height ();
 
@@ -295,14 +274,91 @@
                         cursor_rect.height *= scale_factor;
                     }
 
-                    if (win_rect.intersect (cursor_rect, out cursor_rect)) {
-                        cursor_pixbuf.composite (screenshot, cursor_rect.x, cursor_rect.y, cursor_rect.width, cursor_rect.height, cursor_rect.x, cursor_rect.y, scale_factor, scale_factor, Gdk.InterpType.BILINEAR, 255);
-                    }
-                }
-            }
-
-            if (redact) {
-                redact_text (false);
+                    cursor_pixbuf.composite (root_pix,
+                                             cursor_rect.x, cursor_rect.y,
+                                             cursor_rect.width, cursor_rect.height,
+                                             cursor_rect.x, cursor_rect.y,
+                                             scale_factor,
+                                             scale_factor,
+                                             Gdk.InterpType.BILINEAR,
+                                             255);
+                }
+            }
+
+            Gdk.Pixbuf? screenshot = null;
+
+            if (capture_mode == CaptureType.AREA || capture_mode == CaptureType.CURRENT_WINDOW) {
+                /* We use a subpixbuf of the root screen for capturing the current window because this will include
+                 * menus, popups, tooltips etc whereas pixbuf_get_from_window will not */
+                Gdk.Rectangle selection_rect;
+                win.get_frame_extents (out selection_rect); /* Includes non-CSD decorations and regions offscreen */
+                Gdk.Pixbuf? window_pix = null;
+                int offset_x, offset_y;
+                offset_x = offset_y = 0;
+                Gdk.Rectangle subpix_rect = {selection_rect.x, selection_rect.y ,selection_rect.width ,selection_rect.height};
+
+                if (capture_mode == CaptureType.CURRENT_WINDOW) {
+                    /* frame_extents may include shadow region as a translucent border which we
+                     * do not want when creating the subpixbuf because it will cause selection of
+                     * parts of the background screen.*/
+
+                    /* Need to scale frame extents when not selected manually */
+                    if (scale_factor > 1) {
+                        selection_rect.x *= scale_factor;
+                        selection_rect.y *= scale_factor;
+                        selection_rect.width *= scale_factor;
+                        selection_rect.height *= scale_factor;
+                    }
+
+                    window_pix = get_scaled_pixbuf_from_window (win, capture_mode, scale_factor);
+                    /* window_pix does not include non-CSD decorations but includes regions offscreen */
+                    /* For non-CSD windows window_pix has no shadow and selection_rect will be unaltered and larger than window_pix;
+                     * the offsets will be zero.  For CSD windows, selection_rect starts the same size as window_pix
+                     * and will be shrunk to exclude the shadows; the offsets will reflect the thickness of the shadows. */
+                    selection_rect = remove_translucent_border (window_pix, selection_rect, out offset_x, out offset_y);
+                    subpix_rect = {selection_rect.x, selection_rect.y ,selection_rect.width ,selection_rect.height};
+
+                    /* Do not try to select region outside the root window */
+                    if (selection_rect.x < 0) {
+                        subpix_rect.width += selection_rect.x;
+                        subpix_rect.x = 0;
+                    }
+
+                    if (selection_rect.x + selection_rect.width > root_width) {
+                        subpix_rect.width = root_width - selection_rect.x;
+                    }
+
+                    if (selection_rect.y < 0) {
+                        subpix_rect.height += selection_rect.y;
+                        subpix_rect.y = 0;
+                    }
+
+                    if (selection_rect.y + selection_rect.height > root_height) {
+                        subpix_rect.height = root_height - selection_rect.y;
+                    }
+                }
+
+                Gdk.Pixbuf subpix = new Gdk.Pixbuf.subpixbuf (root_pix,
+                                                              subpix_rect.x,
+                                                              subpix_rect.y,
+                                                              subpix_rect.width,
+                                                              subpix_rect.height);
+
+                if (capture_mode == CaptureType.CURRENT_WINDOW) {
+                    /* For whole window grab, construct a composite of selection_pix and window_pix
+                     * (including shadow if there is one - absent with non-CSD windows.  Use Cairo since
+                     * we do not know which of the pixbufs is larger.  */
+                    screenshot = composite_pix (subpix, window_pix, selection_rect, offset_x, offset_y);
+                } else {
+                    screenshot = subpix;
+                }
+            } else {
+                screenshot = root_pix;
+            }
+
+            if (screenshot == null) {
+                show_error_dialog ();
+                return false;
             }
 
             play_shutter_sound ("screen-capture", _("Screenshot taken"));
@@ -349,6 +405,7 @@
                     /// TRANSLATORS: %s represents a timestamp here
                     string file_name = _("Screenshot from %s").printf (date_time);
                     string format = settings.get_string ("format");
+
                     try {
                         save_file (file_name, format, "", screenshot);
                     } catch (GLib.Error e) {
@@ -356,12 +413,180 @@
                         debug (e.message);
                     }
                 }
+
                 this.destroy ();
             }
 
             return false;
         }
 
+        private Gdk.Pixbuf? get_scaled_pixbuf_from_window (Gdk.Window win, CaptureType mode, int scale_factor) {
+            int width = win.get_width ();
+            int height = win.get_height ();
+            // Check the scaling factor in use, and if greater than 1 scale the image. (for HiDPI displays)
+            Gdk.Pixbuf? pix;
+            if (scale_factor > 1 && mode == CaptureType.SCREEN) {
+                pix = Gdk.pixbuf_get_from_window (win, 0, 0, width / scale_factor, height / scale_factor);
+                pix.scale (pix, width, height, width, height, 0, 0, scale_factor, scale_factor, Gdk.InterpType.BILINEAR);
+            } else {
+                pix = Gdk.pixbuf_get_from_window (win, 0, 0, width, height);
+            }
+
+            return pix;
+        }
+
+
+        private Gdk.Rectangle remove_translucent_border (Gdk.Pixbuf? pix, Gdk.Rectangle rect, out int offset_x, out int offset_y) {
+            const int MIN_ALPHA = 128;
+            const int MIN_COLOR = 10;
+            offset_x = offset_y = 0;
+
+            if (pix == null) {
+                rect.x = rect.y = rect.width = rect.height = 0;
+            } else {
+                /* Measure any dark translucent shadow around pix */
+                var height = pix.get_height ();
+                var width = pix.get_width ();
+                uint8[] bytes = pix.read_pixel_bytes ().get_data ();
+                var length = pix.get_byte_length ();
+                var rowstride = pix.rowstride;
+                var half_row = (width / 2) * 4;
+
+                /* Measure top margin */
+                int count = 0;
+                uint index = half_row + 3;
+                while (index < length &&
+                       bytes[index] < MIN_ALPHA &&
+                       bytes[index - 1] < MIN_COLOR &&
+                       bytes[index - 2] < MIN_COLOR &&
+                       bytes[index - 3] < MIN_COLOR) {
+
+
+                    index += rowstride;
+                    count++;
+                }
+
+                rect.height -= count;
+                rect.y += count;
+                offset_y = count;
+
+                /* Measure bottom margin */
+                count = 0;
+                index = (uint)(length - half_row + 3);
+
+                while (index < length &&
+                       bytes[index] < MIN_ALPHA &&
+                       bytes[index - 1] < MIN_COLOR &&
+                       bytes[index - 2] < MIN_COLOR &&
+                       bytes[index - 3] < MIN_COLOR) {
+
+                    index -= rowstride;
+                    count++;
+                }
+
+                rect.height -= count;
+
+                /* Measure left margin */
+                count = 0;
+                index = (uint)(rowstride * (height / 2) + 3);
+                while (index < length &&
+                       bytes[index] < MIN_ALPHA &&
+                       bytes[index - 1] < MIN_COLOR &&
+                       bytes[index - 2] < MIN_COLOR &&
+                       bytes[index - 3] < MIN_COLOR) {
+
+                    index += 4;
+                    count++;
+                }
+
+                rect.width -= count;
+                rect.x += count;
+                offset_x = count;
+
+                /* Measure right margin */
+                count = 0;
+                index = rowstride * (height / 2) - 1;
+                while (index < length &&
+                       bytes[index] < MIN_ALPHA &&
+                       bytes[index - 1] < MIN_COLOR &&
+                       bytes[index - 2] < MIN_COLOR &&
+                       bytes[index - 3] < MIN_COLOR) {
+
+                    index -= 4;
+                    count++;
+                }
+
+                rect.width -= count;
+            }
+
+            return rect;
+        }
+        /*** @pix1 is the subpix of the root screen.
+           * @pix2 is the pixbuf of the whole window with CSD decorations (if any) including shadows.
+           * @selection_rect is the area of the whole window, including non-CSD decorations (if any) but excluding shadows.
+           * @offset_x and @offset_y are zero for non-CSD window since @pix2 is smaller than @selection_rect.
+           * @offset_x and @offset_y reflect the left and top shadow widths for CSD windows.
+           * @pix1 will be composited on top of @pix2. The final result
+           * will contain the whole of each pix on a white background.
+         ***/   
+        private Gdk.Pixbuf composite_pix (Gdk.Pixbuf pix1, Gdk.Pixbuf pix2, Gdk.Rectangle selection_rect,
+                                          int offset_x, int offset_y) {
+
+            double offset_x1 = 0.0;
+            double offset_y1 = 0.0;
+            double offset_x2 = 0.0;
+            double offset_y2 = 0.0;
+            int width1 = pix1.get_width ();
+            int width2 = pix2.get_width ();
+            int height1 = pix1.get_height ();
+            int height2 = pix2.get_height ();
+
+            offset_x1 = offset_x;
+            offset_y1 = offset_y;
+
+            bool left_truncation = (selection_rect.width > width1 && selection_rect.x < 0);
+            bool top_truncation = (selection_rect.height > height1 && selection_rect.y < 0);
+            int top_truncation_amount = top_truncation ? selection_rect.height - height1 : 0;
+            bool non_CSD = (selection_rect.height > height2);
+            int non_CSD_amount = non_CSD ? selection_rect.height - height2 : 0;
+            int non_CSD_not_showing_amount = non_CSD ? int.min (top_truncation_amount, non_CSD_amount) : 0;
+
+            if (left_truncation) {
+                offset_x1 -= selection_rect.x;
+            }
+
+            if (top_truncation) {
+                if (!non_CSD) {
+                    offset_y1 -= selection_rect.y;
+                } else {
+                    offset_y1 -= (selection_rect.y + non_CSD_amount);
+                }
+            } else {
+                offset_y2 = non_CSD_amount;
+            }
+
+            var cairo_width = int.max (selection_rect.width, width2);
+            var cairo_height = int.max (selection_rect.height - non_CSD_not_showing_amount, height2);
+
+            /* Create an Image surface large enough to hold composite image*/
+            var cs = new Cairo.ImageSurface (Cairo.Format.ARGB32, cairo_width, cairo_height);
+            var cr = new Cairo.Context (cs);
+            cr.set_operator (Cairo.Operator.OVER);
+
+            cr.set_source_rgba (255, 255, 255, 255); // White background to show shadow
+            cr.paint ();
+
+            cr.translate (offset_x2, offset_y2);
+            Gdk.cairo_set_source_pixbuf (cr, pix2, 0.0, 0.0);
+            cr.paint ();
+
+            cr.translate (offset_x1 - offset_x2, offset_y1 - offset_y2);
+            Gdk.cairo_set_source_pixbuf (cr, pix1, 0.0, 0.0);
+            cr.paint ();
+
+            return Gdk.pixbuf_get_from_surface (cs, 0, 0, cairo_width, cairo_height);
+        }
+
         private void save_file (string file_name, string format, string folder_dir, Gdk.Pixbuf screenshot) throws GLib.Error {
             string full_file_name = "";
 
@@ -479,9 +704,9 @@
             var win = selection_area.get_window ();
 
             selection_area.captured.connect (() => {
-            	if (delay == 0) {
-            		selection_area.set_opacity (0);
-            	}
+                if (delay == 0) {
+                    selection_area.set_opacity (0);
+                }
                 selection_area.close ();
                 Timeout.add_seconds (delay - (redact ? 1 : 0), () => {
                     if (from_command == false) {
@@ -511,10 +736,14 @@
                 settings.set_string ("font-name", "Redacted Script Regular 9");
                 settings.set_string ("monospace-font-name", "Redacted Script Light 10");
                 settings.set_string ("document-font-name", "Redacted Script Regular 10");
-            } else {
+            } else if (prev_font_regular != null) {
                 settings.set_string ("font-name", prev_font_regular);
                 settings.set_string ("monospace-font-name", prev_font_mono);
                 settings.set_string ("document-font-name", prev_font_document);
+            } else if (settings.get_string ("font-name").contains ("Redacted")) { /* Fallback in case a program crash leaves the system redacted */
+                settings.reset ("font-name");
+                settings.reset ("monospace-font-name");
+                settings.reset ("document-font-name");
             }
         }