Merge lp:~ev/apport/grouped_reports into lp:~apport-hackers/apport/trunk

[Evan]

I would like to clean up the tests a little more (and have more of a think about how we can avoid loading all the reports so many times), but I think this is basically ready for review.

This branch makes two major changes to apport.

The first introduces the concept of grouped or silent error reports. To quote Matthew,

"When an error occurs that's unlikely to affect you noticably, you won't be notified separately. Instead, there will be an extra "Report previous internal errors too" checkbox in the alert for the next error that you *do* need to know about. This will substantially reduce the number of alerts shown."

https://wiki.ubuntu.com/ErrorTracker#error

The second change does away with the error dialogs when encountering malformed reports or unhandled errors in the apport code. It replaces these with a new pair of fields, InvalidReason and InvalidMachineReason, that will be sent to errors.ubuntu.com. If either of these fields are present, the report will not be sent to Launchpad. We will count the occurrences of the permutations of these fields to get an understanding of what kinds of errors users are encountering when apport appears (and hopefully remedy the non-hardware ones). We will also use it to better inform the "average errors per calendar day" graph, filling in this otherwise missing error counts. Finally, it presents this information to the user just in the "Show Details" window in a less technical string, as mentioned in the specification:

"If no details are available (for example, the crash file is unreadable), below the timestamp and (if available) the process name should appear the paragraph “No details were recorded for this error.”"

Thanks

[Martin Pitt (pitti)]

Hello Evan,

sorry for the late answer! Nexus sprint, FOSDEM and all that..

> + grouped_reports = {}
> + for f in reports:
> + self.load_report(f)
> + if not self.load_report(f) or not self.get_desktop_entry():

It seems we load the application reports twice here, and further down.
I guess it does it that way instead of calling run_crash(f)
immediately to avoid spreading the internal ones over several
application report popups. This should eventually be made more clever
by either saving the application reports in a separate map, or using
load() with binary=False, or just accepting that in the rare case of
multiple apps crashing at the same time the internal reports might get
spread out over two runs (which, from the POV of avoiding too many
reports in memory at the same time might not actually a bad thing?)

> + # We may not be able to load the report, but we should still
> + # show that it occurred and send the reason for statistical
> + # purposes.
> + grouped_reports[f] = self.report
> + self.packages[f] = self.cur_package

So grouped_reports() has both the "failed to load" as well as the
"system" ones. If load_report() fails we need to be *much* more
careful, though. The file might not be readable, have disappeared from
disk, or be totally corrupted, so you cannot assume anything about
self.report or self.cur_package. self.report will be None for
many failure conditions, while self.cur_package could just be the
previous successful one.

I don't think we should do anything if the first set of checks in
load_reports() fails. For the "ensure that the crashed program is
still installed" case, if that fails we should either do nothing, or
continue to show the error message, but what's the point of telling
errors.u.c. about that?

In any case, this means that grouped_reports/self.package has some
reliable and some bogus data, and after this code we don't kno any
more which is which. Can we perhaps continue to ignore the broken ones
for now, and perhaps handle these in a separate MP?

> + # Make sure we don't report these again. Note that these
> + # grouped reports will not be marked as seen if we do not
> + # have a regular application error report to present.
> + for f in grouped_reports.keys():
> + apport.fileutils.mark_report_seen(f)

So we will always mark them as seen even if the user declines to send
them? I don't have a better idea about these either, but it kind of
seems to be contradictory to your usual approach of "I want to get
all reports, no matter how broken they are" :-) But this seems ok to
me.

> def run_crash(self, report_file, confirm=True):
> [...]
> + for path, report in self.get_reports():
> + # check for absent CoreDumps (removed if they exceed size limit)
> + if report.get('ProblemType') == 'Crash' and 'Signal' in report and 'CoreDump' not in report and 'Stacktrace' not in report:
> + reason = _('This problem report is damaged and cannot be processe...

[lp:~ev/apport/grouped_reports]

2596. By Evan on 2013-04-27

Merge with trunk.

Unmerged revisions

2596. By Evan on 2013-04-27

Merge with trunk.

2595. By Evan on 2013-01-23

Split does not take keyword arguments.

2594. By Evan on 2013-01-23

Fix failing test from new collect_info interface

2593. By Evan on 2013-01-23

Fix the kernel oops test GTK test.

2592. By Evan on 2013-01-23

Fix UI tests.

2591. By Evan on 2013-01-22

Collect packages for grouped reports. Correctly pass in the current package to collect_info.

2590. By Evan on 2013-01-22

Pass in the current package.

2589. By Evan on 2013-01-21

Update the tests to handle the new handle_duplicate function signature.

2588. By Evan on 2013-01-21

Fix the UI tests to match the new collect_info function signature. Correctly initialise the response from ui_present_report_details.

2587. By Evan on 2013-01-21

Properly initialise the grouped_reports dictionary.

=== modified file 'apport/ui.py'
--- apport/ui.py	2013-03-19 10:38:05 +0000
+++ apport/ui.py	2013-04-27 08:52:25 +0000
@@ -15,8 +15,8 @@
 
 __version__ = '2.9.2'
 
-import glob, sys, os.path, optparse, traceback, locale, gettext, re
-import errno, zlib
+import glob, sys, os.path, optparse, traceback, locale, gettext, re, itertools
+import errno, zlib, struct
 import subprocess, threading, webbrowser
 import signal
 import time
@@ -150,6 +150,11 @@
     #if report.get('Signal') == '6' and 'AssertionMessage' not in report:
     #    report['UnreportableReason'] = _('The program crashed on an assertion failure, but the message could not be retrieved. Apport does not support reporting these crashes.')
 
+    update_report(report, reportfile)
+
+
+def update_report(report, reportfile):
+    '''Write the report to disk.'''
     if reportfile:
         try:
             with open(reportfile, 'ab') as f:
@@ -166,6 +171,16 @@
         os.chmod(reportfile, 0o640)
 
 
+def mark_invalid(report, reportfile, reason, machine_reason):
+    '''Mark the report as invalid while continuing to process further reports.
+    The invalid report will appear in the UI with the 'reason' argument as its
+    text and will be uploaded to daisy.ubuntu.com for statistical purposes.
+    '''
+    report['InvalidReason'] = reason
+    report['InvalidMachineReason'] = machine_reason
+    update_report(report, reportfile)
+
+
 class UserInterface:
     '''Apport user interface API.
 
@@ -180,7 +195,9 @@
         self.gettext_domain = 'apport'
         self.report = None
         self.report_file = None
+        self.grouped_reports = {}
         self.cur_package = None
+        self.packages = {}
 
         try:
             self.crashdb = apport.crashdb.get_crashdb(None)
@@ -211,17 +228,58 @@
             reports = apport.fileutils.get_new_system_reports()
         else:
             reports = apport.fileutils.get_new_reports()
-        for f in reports:
-            if not self.load_report(f):
+
+        # Determine which reports should be grouped with the first regular
+        # crash rather than shown in a dialog on their own. These are referred
+        # to as 'silent' errors.
+        grouped_reports = {}
+        for f in reports:
+            self.load_report(f)
+            if not self.load_report(f) or not self.get_desktop_entry():
+                # We may not be able to load the report, but we should still
+                # show that it occurred and send the reason for statistical
+                # purposes.
+                grouped_reports[f] = self.report
+                self.packages[f] = self.cur_package
+
+        grouped_reports_reported = False
+        for f in reports:
+            if f in grouped_reports or not self.load_report(f):
                 continue
+
             if self.report['ProblemType'] == 'Hang':
                 self.finish_hang(f)
             else:
-                self.run_crash(f)
+                if not grouped_reports_reported:
+                    grouped_reports_reported = True
+                    self.grouped_reports = grouped_reports
+                    self.run_crash(f)
+
+                    # Make sure we don't report these again. Note that these
+                    # grouped reports will not be marked as seen if we do not
+                    # have a regular application error report to present.
+                    for f in grouped_reports.keys():
+                        apport.fileutils.mark_report_seen(f)
+
+                    self.grouped_reports = None
+                else:
+                    self.run_crash(f)
             result = True
 
         return result
 
+    def get_reports(self):
+        '''Return an iterator over all the reports and their respective
+        files.
+        '''
+        r = [(self.report_file, self.report)]
+
+        if self.grouped_reports:
+            grouped = self.grouped_reports.items()
+            return itertools.chain(r, grouped)
+        else:
+            return r
+
     def run_crash(self, report_file, confirm=True):
         '''Present and report a particular crash.
 
@@ -246,44 +304,17 @@
             if 'Ignore' in self.report:
                 return
 
-            # check for absent CoreDumps (removed if they exceed size limit)
-            if self.report.get('ProblemType') == 'Crash' and 'Signal' in self.report and 'CoreDump' not in self.report and 'Stacktrace' not in self.report:
-                subject = os.path.basename(self.report.get('ExecutablePath', _('unknown program')))
-                heading = _('Sorry, the program "%s" closed unexpectedly') % subject
-                self.ui_error_message(
-                    _('Problem in %s') % subject,
-                    '%s\n\n%s' % (heading, _('Your computer does not have '
-                    'enough free memory to automatically analyze the problem '
-                    'and send a report to the developers.')))
-                return
+            for path, report in self.get_reports():
+                # check for absent CoreDumps (removed if they exceed size limit)
+                if report.get('ProblemType') == 'Crash' and 'Signal' in report and 'CoreDump' not in report and 'Stacktrace' not in report:
+                    reason = _('This problem report is damaged and cannot be processed.')
+                    machine_reason = 'No core dump'
+                    mark_invalid(report, path, reason, machine_reason)
 
             allowed_to_report = apport.fileutils.allowed_to_report()
             response = self.ui_present_report_details(allowed_to_report)
             if response['report'] or response['examine']:
-                try:
-                    if 'Dependencies' not in self.report:
-                        self.collect_info()
-                except (IOError, zlib.error) as e:
-                    # can happen with broken core dumps
-                    self.report = None
-                    self.ui_error_message(
-                        _('Invalid problem report'), '%s\n\n%s' % (
-                            _('This problem report is damaged and cannot be processed.'),
-                            repr(e)))
-                    self.ui_shutdown()
-                    return
-                except ValueError:  # package does not exist
-                    self.ui_error_message(_('Invalid problem report'),
-                                          _('The report belongs to a package that is not installed.'))
-                    self.ui_shutdown()
-                    return
-                except Exception as e:
-                    apport.error(repr(e))
-                    self.ui_error_message(_('Invalid problem report'),
-                                          _('An error occurred while attempting to process this'
-                                            ' problem report:') + '\n\n' + str(e))
-                    self.ui_shutdown()
-                    return
+                self.collect_all_info()
 
             if self.report is None:
                 # collect() does that on invalid reports
@@ -299,21 +330,22 @@
             if not response['report']:
                 return
 
-            apport.fileutils.mark_report_upload(report_file)
-            # We check for duplicates and unreportable crashes here, rather
-            # than before we show the dialog, as we want to submit these to the
-            # crash database, but not Launchpad.
-            if self.crashdb.accepts(self.report):
-                # FIXME: This behaviour is not really correct, but necessary as
-                # long as we only support a single crashdb and have whoopsie
-                # hardcoded. Once we have multiple crash dbs, we need to check
-                # accepts() earlier, and not even present the data if none of
-                # the DBs wants the report. See LP#957177 for details.
-                if self.handle_duplicate():
-                    return
-                if self.check_unreportable():
-                    return
-                self.file_report()
+            for path, report in self.get_reports():
+                apport.fileutils.mark_report_upload(path)
+                # We check for duplicates and unreportable crashes here, rather
+                # than before we show the dialog, as we want to submit these to the
+                # crash database, but not Launchpad.
+                if self.crashdb.accepts(report):
+                    # FIXME: This behaviour is not really correct, but necessary as
+                    # long as we only support a single crashdb and have whoopsie
+                    # hardcoded. Once we have multiple crash dbs, we need to check
+                    # accepts() earlier, and not even present the data if none of
+                    # the DBs wants the report. See LP#957177 for details.
+                    if self.handle_duplicate(report):
+                        return
+                    if self.check_unreportable(report):
+                        return
+                    self.file_report(report)
         except IOError as e:
             # fail gracefully if file is not readable for us
             if e.errno in (errno.EPERM, errno.EACCES):
@@ -450,7 +482,8 @@
             self.cur_package = self.options.package
 
         try:
-            self.collect_info(symptom_script)
+            self.collect_info(self.report, self.report_file, self.cur_package,
+                              symptom_script)
         except ValueError as e:
             if 'package' in str(e) and 'does not exist' in str(e):
                 if not self.cur_package:
@@ -463,12 +496,12 @@
             else:
                 raise
 
-        if self.check_unreportable():
+        if self.check_unreportable(self.report):
             return
 
         self.add_extra_tags()
 
-        if self.handle_duplicate():
+        if self.handle_duplicate(self.report):
             return True
 
         # not useful for bug reports, and has potentially sensitive information
@@ -488,7 +521,7 @@
             allowed_to_report = apport.fileutils.allowed_to_report()
             response = self.ui_present_report_details(allowed_to_report)
             if response['report']:
-                self.file_report()
+                self.file_report(self.report)
 
         return True
 
@@ -539,7 +572,8 @@
                 if not os.path.exists(os.path.join(apport.report._hook_dir, 'source_%s.py' % p)):
                     print('Package %s not installed and no hook available, ignoring' % p)
                     continue
-            self.collect_info(ignore_uninstalled=True)
+            self.collect_info(self.report, self.report_file, self.cur_package,
+                              ignore_uninstalled=True)
             info_collected = True
 
         if not info_collected:
@@ -935,7 +969,45 @@
 
         return True
 
-    def collect_info(self, symptom_script=None, ignore_uninstalled=False,
+    def exception_wrapped_collect_info(self, report, report_file=None,
+                                       cur_package=None, symptom_script=None,
+                                       ignore_uninstalled=False,
+                                       on_finished=None):
+        try:
+            self.collect_info(report, report_file, cur_package, symptom_script,
+                              ignore_uninstalled, on_finished)
+        except (IOError, zlib.error, struct.error) as e:
+            # can happen with broken core dumps
+            reason = _('This problem report is damaged and cannot be processed.')
+            machine_reason = excstr(e)
+            mark_invalid(report, report_file, reason, machine_reason)
+        except ValueError as e:  # package does not exist
+            reason = _('The report belongs to a package that is not installed.')
+            machine_reason = excstr(e)
+            mark_invalid(report, report_file, reason, machine_reason)
+        except Exception as e:
+            reason = _('This problem report is damaged and cannot be processed.')
+            machine_reason = excstr(e)
+            mark_invalid(report, report_file, reason, machine_reason)
+
+    def collect_all_info(self, on_finished=None):
+        '''Collect additional information from an application report and from
+        all grouped reports.'''
+        if 'Dependencies' not in self.report:
+            self.exception_wrapped_collect_info(self.report,
+                                                self.report_file,
+                                                self.cur_package)
+
+        for path in self.grouped_reports:
+            report = self.grouped_reports[path]
+            if 'Dependencies' not in report:
+                self.exception_wrapped_collect_info(report, path, self.packages[path])
+
+        if on_finished:
+            on_finished()
+
+    def collect_info(self, report, report_file=None, cur_package=None,
+                     symptom_script=None, ignore_uninstalled=False,
                      on_finished=None):
         '''Collect additional information.
 
@@ -949,25 +1021,25 @@
         run_symptom()).
         '''
         # check if binary changed since the crash happened
-        if 'ExecutablePath' in self.report and 'ExecutableTimestamp' in self.report:
-            orig_time = int(self.report['ExecutableTimestamp'])
-            del self.report['ExecutableTimestamp']
-            cur_time = int(os.stat(self.report['ExecutablePath']).st_mtime)
+        if 'ExecutablePath' in report and 'ExecutableTimestamp' in report:
+            orig_time = int(report['ExecutableTimestamp'])
+            del report['ExecutableTimestamp']
+            cur_time = int(os.stat(report['ExecutablePath']).st_mtime)
 
             if orig_time != cur_time:
-                self.report['UnreportableReason'] = (
+                report['UnreportableReason'] = (
                     _('The problem happened with the program %s which changed '
-                      'since the crash occurred.') % self.report['ExecutablePath'])
+                      'since the crash occurred.') % report['ExecutablePath'])
                 return
 
-        if not self.cur_package and 'ExecutablePath' not in self.report \
+        if not cur_package and 'ExecutablePath' not in report \
                 and not symptom_script:
             # this happens if we file a bug without specifying a PID or a
             # package
-            self.report.add_os_info()
+            report.add_os_info()
         else:
             # check if we already ran, skip if so
-            if (self.report.get('ProblemType') == 'Crash' and 'Stacktrace' in self.report) or (self.report.get('ProblemType') != 'Crash' and 'Dependencies' in self.report):
+            if (report.get('ProblemType') == 'Crash' and 'Stacktrace' in report) or (report.get('ProblemType') != 'Crash' and 'Dependencies' in report):
                 if on_finished:
                     on_finished()
                 return
@@ -978,12 +1050,12 @@
 
             hookui = HookUI(self)
 
-            if 'Stacktrace' not in self.report:
+            if 'Stacktrace' not in report:
                 # save original environment, in case hooks change it
                 orig_env = os.environ.copy()
                 icthread = apport.REThread.REThread(target=thread_collect_info,
                                                     name='thread_collect_info',
-                                                    args=(self.report, self.report_file, self.cur_package,
+                                                    args=(report, report_file, cur_package,
                                                           hookui, symptom_script, ignore_uninstalled))
                 icthread.start()
                 while icthread.isAlive():
@@ -1007,8 +1079,8 @@
                 return
 
             # check bug patterns
-            if self.report['ProblemType'] == 'KernelCrash' or self.report['ProblemType'] == 'KernelOops' or 'Package' in self.report:
-                bpthread = apport.REThread.REThread(target=self.report.search_bug_patterns,
+            if report['ProblemType'] == 'KernelCrash' or report['ProblemType'] == 'KernelOops' or 'Package' in report:
+                bpthread = apport.REThread.REThread(target=report.search_bug_patterns,
                                                     args=(self.crashdb.get_bugpattern_baseurl(),))
                 bpthread.start()
                 while bpthread.isAlive():
@@ -1019,12 +1091,12 @@
                         sys.exit(1)
                 bpthread.exc_raise()
                 if bpthread.return_value():
-                    self.report['KnownReport'] = bpthread.return_value()
+                    report['KnownReport'] = bpthread.return_value()
 
             # check crash database if problem is known
-            if self.report['ProblemType'] != 'Bug':
+            if report['ProblemType'] != 'Bug':
                 known_thread = apport.REThread.REThread(target=self.crashdb.known,
-                                                        args=(self.report,))
+                                                        args=(report,))
                 known_thread.start()
                 while known_thread.isAlive():
                     self.ui_pulse_info_collection_progress()
@@ -1036,13 +1108,13 @@
                 val = known_thread.return_value()
                 if val is not None:
                     if val is True:
-                        self.report['KnownReport'] = '1'
+                        report['KnownReport'] = '1'
                     else:
-                        self.report['KnownReport'] = val
+                        report['KnownReport'] = val
 
             # anonymize; needs to happen after duplicate checking, otherwise we
             # might damage the stack trace
-            anonymize_thread = apport.REThread.REThread(target=self.report.anonymize)
+            anonymize_thread = apport.REThread.REThread(target=report.anonymize)
             anonymize_thread.start()
             while anonymize_thread.isAlive():
                 self.ui_pulse_info_collection_progress()
@@ -1055,10 +1127,10 @@
             self.ui_stop_info_collection_progress()
 
             # check that we were able to determine package names
-            if 'UnreportableReason' not in self.report:
-                if ('SourcePackage' not in self.report or
-                    (not self.report['ProblemType'].startswith('Kernel')
-                     and 'Package' not in self.report)):
+            if 'UnreportableReason' not in report:
+                if ('SourcePackage' not in report or
+                    (not report['ProblemType'].startswith('Kernel')
+                     and 'Package' not in report)):
                     self.ui_error_message(_('Invalid problem report'),
                                           _('Could not determine the package or source package name.'))
                     # TODO This is not called consistently, is it really needed?
@@ -1112,26 +1184,26 @@
             os.write(w, str(e))
             sys.exit(1)
 
-    def file_report(self):
+    def file_report(self, report):
         '''Upload the current report and guide the user to the reporting web page.'''
         # FIXME: This behaviour is not really correct, but necessary as
         # long as we only support a single crashdb and have whoopsie
         # hardcoded. Once we have multiple crash dbs, we need to check
         # accepts() earlier, and not even present the data if none of
         # the DBs wants the report. See LP#957177 for details.
-        if not self.crashdb.accepts(self.report):
+        if not self.crashdb.accepts(report):
             return
         # drop PackageArchitecture if equal to Architecture
-        if self.report.get('PackageArchitecture') == self.report.get('Architecture'):
+        if report.get('PackageArchitecture') == report.get('Architecture'):
             try:
-                del self.report['PackageArchitecture']
+                del report['PackageArchitecture']
             except KeyError:
                 pass
 
         # StacktraceAddressSignature is redundant and does not need to clutter
         # the database
         try:
-            del self.report['StacktraceAddressSignature']
+            del report['StacktraceAddressSignature']
         except KeyError:
             pass
 
@@ -1144,7 +1216,7 @@
 
         self.ui_start_upload_progress()
         upthread = apport.REThread.REThread(target=self.crashdb.upload,
-                                            args=(self.report, progress_callback))
+                                            args=(report, progress_callback))
         upthread.start()
         while upthread.isAlive():
             self.ui_set_upload_progress(__upload_progress)
@@ -1161,7 +1233,7 @@
                     user, password = data
                     self.crashdb.set_credentials(user, password)
                     upthread = apport.REThread.REThread(target=self.crashdb.upload,
-                                                        args=(self.report, progress_callback))
+                                                        args=(report, progress_callback))
                     upthread.start()
             except (TypeError, SyntaxError, ValueError):
                 raise
@@ -1176,7 +1248,7 @@
         ticket = upthread.return_value()
         self.ui_stop_upload_progress()
 
-        url = self.crashdb.get_comment_url(self.report, ticket)
+        url = self.crashdb.get_comment_url(report, ticket)
         if url:
             self.open_url(url)
 
@@ -1187,27 +1259,26 @@
         be processed, otherwise self.report is initialized and True is
         returned.
         '''
+        self.report = apport.Report()
         try:
-            self.report = apport.Report()
             with open(path, 'rb') as f:
                 self.report.load(f, binary='compressed')
             if 'ProblemType' not in self.report:
                 raise ValueError('Report does not contain "ProblemType" field')
-        except MemoryError:
-            self.report = None
-            self.ui_error_message(_('Memory exhaustion'),
-                                  _('Your system does not have enough memory to process this crash report.'))
+        except MemoryError as e:
+            reason = _('Your system does not have enough memory to process this crash report.')
+            machine_reason = excstr(e)
+            mark_invalid(self.report, path, reason, machine_reason)
             return False
         except IOError as e:
-            self.report = None
-            self.ui_error_message(_('Invalid problem report'), e.strerror)
+            reason = _('Invalid problem report')
+            machine_reason = excstr(e)
+            mark_invalid(self.report, path, reason, machine_reason)
             return False
         except (TypeError, ValueError, AssertionError, zlib.error) as e:
-            self.report = None
-            self.ui_error_message(_('Invalid problem report'),
-                                  '%s\n\n%s' % (
-                                      _('This problem report is damaged and cannot be processed.'),
-                                      repr(e)))
+            reason = _('This problem report is damaged and cannot be processed.')
+            machine_reason = excstr(e)
+            mark_invalid(self.report, path, reason, machine_reason)
             return False
 
         if 'Package' in self.report:
@@ -1219,38 +1290,38 @@
         if self.report['ProblemType'] == 'Crash':
             exe_path = self.report.get('ExecutablePath', '')
             if not os.path.exists(exe_path):
-                msg = _('This problem report applies to a program which is not installed any more.')
-                if exe_path:
-                    msg = '%s (%s)' % (msg, self.report['ExecutablePath'])
-                self.report = None
-                self.ui_info_message(_('Invalid problem report'), msg)
+                reason = _('This problem report applies to a program which is not installed any more.')
+                machine_reason = "ENOENT: '%s'" % exe_path
+                mark_invalid(self.report, path, reason, machine_reason)
                 return False
 
             if 'InterpreterPath' in self.report:
                 if not os.path.exists(self.report['InterpreterPath']):
-                    msg = _('This problem report applies to a program which is not installed any more.')
-                    self.ui_info_message(_('Invalid problem report'), '%s (%s)'
-                                         % (msg, self.report['InterpreterPath']))
+                    reason = _('This problem report applies to a program which is not installed any more.')
+                    machine_reason = "ENOENT: '%s'" % self.report['InterpreterPath']
+                    mark_invalid(self.report, path, reason, machine_reason)
                     return False
 
         return True
 
-    def check_unreportable(self):
+    def check_unreportable(self, report):
         '''Check if the current report is unreportable.
 
         If so, display an info message and return True.
         '''
-        if not self.crashdb.accepts(self.report):
+        if not self.crashdb.accepts(report):
             return False
-        if 'UnreportableReason' in self.report:
-            if type(self.report['UnreportableReason']) == bytes:
-                self.report['UnreportableReason'] = self.report['UnreportableReason'].decode('UTF-8')
-            if 'Package' in self.report:
-                title = _('Problem in %s') % self.report['Package'].split()[0]
+        if 'InvalidReason' in report:
+            return True
+        if 'UnreportableReason' in report:
+            if type(report['UnreportableReason']) == bytes:
+                report['UnreportableReason'] = report['UnreportableReason'].decode('UTF-8')
+            if 'Package' in report:
+                title = _('Problem in %s') % report['Package'].split()[0]
             else:
                 title = ''
             self.ui_info_message(title, _('The problem cannot be reported:\n\n%s') %
-                                 self.report['UnreportableReason'])
+                                 report['UnreportableReason'])
             return True
         return False
 
@@ -1291,26 +1362,26 @@
             return None
         return result
 
-    def handle_duplicate(self):
+    def handle_duplicate(self, report):
         '''Check if current report matches a bug pattern.
 
         If so, tell the user about it, open the existing bug in a browser, and
         return True.
         '''
-        if not self.crashdb.accepts(self.report):
+        if not self.crashdb.accepts(report):
             return False
-        if 'KnownReport' not in self.report:
+        if 'KnownReport' not in report:
             return False
 
         # if we have an URL, open it; otherwise this is just a marker that we
         # know about it
-        if self.report['KnownReport'].startswith('http'):
+        if report['KnownReport'].startswith('http'):
             self.ui_info_message(_('Problem already known'),
                                  _('This problem was already reported in the bug report displayed \
 in the web browser. Please check if you can add any further information that \
 might be helpful for the developers.'))
 
-            self.open_url(self.report['KnownReport'])
+            self.open_url(report['KnownReport'])
         else:
             self.ui_info_message(_('Problem already known'),
                                  _('This problem was already reported to developers. Thank you!'))
 
=== modified file 'gtk/apport-gtk'
--- gtk/apport-gtk	2012-11-23 14:23:46 +0000
+++ gtk/apport-gtk	2013-04-27 08:52:25 +0000
@@ -11,7 +11,7 @@
 # option) any later version.  See http://www.gnu.org/copyleft/gpl.html for
 # the full text of the license.
 
-import os.path, sys, subprocess, os, re, errno
+import os.path, sys, subprocess, os, re, errno, itertools
 
 from gi.repository import GObject, GLib, Wnck, GdkX11, Gdk
 try:
@@ -63,11 +63,13 @@
         column = Gtk.TreeViewColumn('Report', Gtk.CellRendererText(), text=0)
         self.w('details_treeview').append_column(column)
         self.spinner = self.add_spinner_over_treeview(self.w('details_treeview'))
+        self.w('details_treeview').set_row_separator_func(self.on_row_separator, None)
 
         self.md = None
 
         self.desktop_info = None
 
+
     #
     # ui_* implementation of abstract UserInterface classes
     #
@@ -100,42 +102,56 @@
         if not self.w('details_treeview').get_property('visible'):
             return
 
-        if shown_keys:
-            keys = set(self.report.keys()) & set(shown_keys)
-        else:
-            keys = self.report.keys()
-        # show the most interesting items on top
-        keys = sorted(keys)
-        for k in ('Traceback', 'StackTrace', 'Title', 'ProblemType', 'Package', 'ExecutablePath'):
-            if k in keys:
-                keys.remove(k)
-                keys.insert(0, k)
-
         self.tree_model.clear()
-        for key in keys:
-            keyiter = self.tree_model.insert_before(None, None)
-            self.tree_model.set_value(keyiter, 0, key)
-
-            valiter = self.tree_model.insert_before(keyiter, None)
-            if not hasattr(self.report[key], 'gzipvalue') and \
-                hasattr(self.report[key], 'isspace') and \
-                not self.report._is_binary(self.report[key]):
-                v = self.report[key]
-                if len(v) > 4000:
-                    v = v[:4000]
+
+        for report in itertools.chain([self.report], self.grouped_reports.values()):
+            if shown_keys:
+                keys = set(report.keys()) & set(shown_keys)
+            else:
+                keys = report.keys()
+            # show the most interesting items on top
+            keys = sorted(keys)
+            for k in ('Traceback', 'StackTrace', 'Title', 'ProblemType', 'Package', 'ExecutablePath'):
+                if k in keys:
+                    keys.remove(k)
+                    keys.insert(0, k)
+
+            if report != self.report:
+                # insert separator
+                keyiter = self.tree_model.insert_before(None, None)
+                self.tree_model.set_value(keyiter, 0, '')
+
+            if 'InvalidReason' in keys:
+                # This report is invalid, but we need to tell the user that
+                # we're going to send it for statistical purposes.
+                keyiter = self.tree_model.insert_before(None, None)
+                self.tree_model.set_value(keyiter, 0, report['InvalidReason'])
+                continue
+
+            for key in keys:
+                keyiter = self.tree_model.insert_before(None, None)
+                self.tree_model.set_value(keyiter, 0, key)
+
+                valiter = self.tree_model.insert_before(keyiter, None)
+                if not hasattr(report[key], 'gzipvalue') and \
+                    hasattr(report[key], 'isspace') and \
+                    not report._is_binary(report[key]):
+                    v = report[key]
+                    if len(v) > 4000:
+                        v = v[:4000]
+                        if type(v) == bytes:
+                            v += b'\n[...]'
+                        else:
+                            v += '\n[...]'
                     if type(v) == bytes:
-                        v += b'\n[...]'
-                    else:
-                        v += '\n[...]'
-                if type(v) == bytes:
-                    v = v.decode('UTF-8', errors='replace')
-                self.tree_model.set_value(valiter, 0, v)
-                # expand the row if the value has less than 5 lines
-                if len(list(filter(lambda c: c == '\n', self.report[key]))) < 4:
-                    self.w('details_treeview').expand_row(
-                        self.tree_model.get_path(keyiter), False)
-            else:
-                self.tree_model.set_value(valiter, 0, _('(binary data)'))
+                        v = v.decode('UTF-8', errors='replace')
+                    self.tree_model.set_value(valiter, 0, v)
+                    # expand the row if the value has less than 5 lines
+                    if len(list(filter(lambda c: c == '\n', report[key]))) < 4:
+                        self.w('details_treeview').expand_row(
+                            self.tree_model.get_path(keyiter), False)
+                else:
+                    self.tree_model.set_value(valiter, 0, _('(binary data)'))
 
     def get_system_application_title(self):
         '''Get dialog title for a non-.desktop application.
@@ -207,6 +223,14 @@
             self.w('send_error_report').set_active(False)
             self.w('send_error_report').hide()
 
+        if self.grouped_reports and allowed_to_report:
+            self.w('previous_internal_errors').set_active(True)
+            self.w('previous_internal_errors').show()
+            self.w('ignore_future_problems').hide()
+        else:
+            self.w('previous_internal_errors').set_active(False)
+            self.w('previous_internal_errors').hide()
+
         self.w('examine').set_visible(self.can_examine_locally())
 
         self.w('continue_button').set_label(_('Continue'))
@@ -346,8 +370,18 @@
         if len(sys.argv) == 1:
             d.set_focus_on_map(False)
 
-        return_value = { 'report' : False, 'blacklist' : False,
-                         'restart' : False, 'examine' : False }
+        return_value = { 
+            # Should the problem be reported?
+            'report' : False,
+            # Should future instances of this type be ignored?
+            'blacklist' : False,
+            # Should the application be restarted?
+            'restart' : False,
+            # Has an interactive debugging session been requested?
+            'examine' : False,
+            # Should grouped reports also be reported?
+            'grouped': False
+        }
         def dialog_crash_dismissed(widget):
             self.w('dialog_crash_new').hide()
             if widget is self.w('dialog_crash_new'):
@@ -364,6 +398,8 @@
                 return_value['blacklist'] = True
             if widget == self.w('continue_button') and self.desktop_info:
                 return_value['restart'] = True
+            if self.w('previous_internal_errors').get_active():
+                return_value['grouped'] = True
             Gtk.main_quit()
 
         self.w('dialog_crash_new').connect('destroy', dialog_crash_dismissed)
@@ -561,6 +597,11 @@
     # Event handlers
     #
 
+    def on_row_separator(self, model, iterator, data):
+        if model[iterator][0] == '':
+            return True
+        return False
+
     def on_show_details_clicked(self, widget):
         sw = self.w('details_scrolledwindow')
         if sw.get_property('visible'):
@@ -574,7 +615,7 @@
             if not self.collect_called:
                 self.collect_called = True
                 self.ui_update_view(['ExecutablePath'])
-                GLib.idle_add(lambda: self.collect_info(on_finished=self.ui_update_view))
+                GLib.idle_add(lambda: self.collect_all_info(on_finished=self.ui_update_view))
         return True
 
     def on_progress_window_close_event(self, widget, event=None):
 
=== modified file 'gtk/apport-gtk.ui'
--- gtk/apport-gtk.ui	2012-03-02 12:03:31 +0000
+++ gtk/apport-gtk.ui	2013-04-27 08:52:25 +0000
@@ -27,11 +27,9 @@
             <child>
               <object class="GtkButton" id="button7">
                 <property name="label">gtk-cancel</property>
-                <property name="use_action_appearance">False</property>
                 <property name="visible">True</property>
                 <property name="can_focus">True</property>
                 <property name="receives_default">True</property>
-                <property name="use_action_appearance">False</property>
                 <property name="use_stock">True</property>
               </object>
               <packing>
@@ -43,11 +41,9 @@
             <child>
               <object class="GtkButton" id="button13">
                 <property name="label">gtk-ok</property>
-                <property name="use_action_appearance">False</property>
                 <property name="visible">True</property>
                 <property name="can_focus">True</property>
                 <property name="receives_default">True</property>
-                <property name="use_action_appearance">False</property>
                 <property name="use_stock">True</property>
               </object>
               <packing>
@@ -213,11 +209,9 @@
                     <child>
                       <object class="GtkCheckButton" id="send_error_report">
                         <property name="label" translatable="yes">Send an error report to help fix this problem</property>
-                        <property name="use_action_appearance">False</property>
                         <property name="visible">True</property>
                         <property name="can_focus">True</property>
                         <property name="receives_default">False</property>
-                        <property name="use_action_appearance">False</property>
                         <property name="xalign">0</property>
                         <property name="active">True</property>
                         <property name="draw_indicator">True</property>
@@ -231,11 +225,9 @@
                     <child>
                       <object class="GtkCheckButton" id="ignore_future_problems">
                         <property name="label" translatable="yes">Ignore future problems of this program version</property>
-                        <property name="use_action_appearance">False</property>
                         <property name="visible">True</property>
                         <property name="can_focus">True</property>
                         <property name="receives_default">False</property>
-                        <property name="use_action_appearance">False</property>
                         <property name="xalign">0</property>
                         <property name="draw_indicator">True</property>
                       </object>
@@ -245,6 +237,22 @@
                         <property name="position">1</property>
                       </packing>
                     </child>
+                    <child>
+                      <object class="GtkCheckButton" id="previous_internal_errors">
+                        <property name="label" translatable="yes">Report previous internal errors too</property>
+                        <property name="visible">True</property>
+                        <property name="can_focus">True</property>
+                        <property name="receives_default">False</property>
+                        <property name="xalign">0.0099999997764825821</property>
+                        <property name="yalign">0.51999998092651367</property>
+                        <property name="draw_indicator">True</property>
+                      </object>
+                      <packing>
+                        <property name="expand">False</property>
+                        <property name="fill">True</property>
+                        <property name="position">2</property>
+                      </packing>
+                    </child>
                   </object>
                   <packing>
                     <property name="expand">False</property>
@@ -285,11 +293,9 @@
                     <child>
                       <object class="GtkButton" id="show_details">
                         <property name="label" translatable="yes">Show Details</property>
-                        <property name="use_action_appearance">False</property>
                         <property name="visible">True</property>
                         <property name="can_focus">True</property>
                         <property name="receives_default">True</property>
-                        <property name="use_action_appearance">False</property>
                         <signal name="clicked" handler="on_show_details_clicked" swapped="no"/>
                       </object>
                       <packing>
@@ -301,11 +307,9 @@
                     <child>
                       <object class="GtkButton" id="examine">
                         <property name="label" translatable="yes">_Examine locally</property>
-                        <property name="use_action_appearance">False</property>
                         <property name="visible">True</property>
                         <property name="can_focus">True</property>
                         <property name="receives_default">True</property>
-                        <property name="use_action_appearance">False</property>
                         <property name="use_underline">True</property>
                       </object>
                       <packing>
@@ -317,10 +321,8 @@
                     <child>
                       <object class="GtkButton" id="cancel_button">
                         <property name="label">gtk-cancel</property>
-                        <property name="use_action_appearance">False</property>
                         <property name="can_focus">True</property>
                         <property name="receives_default">True</property>
-                        <property name="use_action_appearance">False</property>
                         <property name="use_stock">True</property>
                       </object>
                       <packing>
@@ -346,11 +348,9 @@
                 <child>
                   <object class="GtkButton" id="closed_button">
                     <property name="label" translatable="yes">Leave Closed</property>
-                    <property name="use_action_appearance">False</property>
                     <property name="visible">True</property>
                     <property name="can_focus">True</property>
                     <property name="receives_default">True</property>
-                    <property name="use_action_appearance">False</property>
                   </object>
                   <packing>
                     <property name="expand">False</property>
@@ -361,11 +361,9 @@
                 <child>
                   <object class="GtkButton" id="continue_button">
                     <property name="label" translatable="yes">Continue</property>
-                    <property name="use_action_appearance">False</property>
                     <property name="visible">True</property>
                     <property name="can_focus">True</property>
                     <property name="receives_default">True</property>
-                    <property name="use_action_appearance">False</property>
                   </object>
                   <packing>
                     <property name="expand">False</property>
@@ -466,12 +464,10 @@
             <child>
               <object class="GtkButton" id="button_cancel_collecting">
                 <property name="label">gtk-cancel</property>
-                <property name="use_action_appearance">False</property>
                 <property name="visible">True</property>
                 <property name="can_focus">True</property>
                 <property name="can_default">True</property>
                 <property name="receives_default">False</property>
-                <property name="use_action_appearance">False</property>
                 <property name="use_stock">True</property>
                 <signal name="clicked" handler="on_progress_window_close_event" swapped="no"/>
               </object>
@@ -569,12 +565,10 @@
             <child>
               <object class="GtkButton" id="button_cancel_upload">
                 <property name="label">gtk-cancel</property>
-                <property name="use_action_appearance">False</property>
                 <property name="visible">True</property>
                 <property name="can_focus">True</property>
                 <property name="can_default">True</property>
                 <property name="receives_default">False</property>
-                <property name="use_action_appearance">False</property>
                 <property name="use_stock">True</property>
                 <signal name="clicked" handler="on_progress_window_close_event" swapped="no"/>
               </object>
 
=== modified file 'kde/apport-kde'
--- kde/apport-kde	2012-11-23 14:23:46 +0000
+++ kde/apport-kde	2013-04-27 08:52:25 +0000
@@ -278,7 +278,7 @@
         self.treeview.setVisible(visible)
         if visible and not self.collect_called:
             self.ui.ui_update_view(self, ['ExecutablePath'])
-            QTimer.singleShot(0, lambda: self.ui.collect_info(on_finished=self.collect_done))
+            QTimer.singleShot(0, lambda: self.ui.collect_info(self.ui.report, on_finished=self.collect_done))
             self.collect_called = True
         if visible:
             self.setMaximumSize(16777215, 16777215)
 
=== modified file 'test/test_backend_apt_dpkg.py'
--- test/test_backend_apt_dpkg.py	2012-12-10 08:35:28 +0000
+++ test/test_backend_apt_dpkg.py	2013-04-27 08:52:25 +0000
@@ -762,7 +762,7 @@
         assert readelf.returncode == 0
         for line in out.splitlines():
             if line.startswith('  Machine:'):
-                machine = line.split(maxsplit=1)[1]
+                machine = line.split(' ', 1)[1]
                 break
         else:
             self.fail('could not fine Machine: in readelf output')
 
=== modified file 'test/test_ui.py'
--- test/test_ui.py	2013-04-03 09:05:41 +0000
+++ test/test_ui.py	2013-04-27 08:52:25 +0000
@@ -10,11 +10,11 @@
 from io import BytesIO
 
 import apport.ui
-from apport.ui import _
 import apport.report
 import problem_report
 import apport.crashdb_impl.memory
 import stat
+from mock import patch
 
 
 class TestSuiteUserInterface(apport.ui.UserInterface):
@@ -53,7 +53,7 @@
         # these store the choices the ui_present_* calls do
         self.present_package_error_response = None
         self.present_kernel_error_response = None
-        self.present_details_response = None
+        self.present_details_response = {}
         self.question_yesno_response = None
         self.question_choice_response = None
         self.question_file_response = None
@@ -264,11 +264,7 @@
         self.update_report_file()
         self.ui.load_report(self.report_file.name)
 
-        self.assertTrue(self.ui.report is None)
-        self.assertEqual(self.ui.msg_title, _('Invalid problem report'))
-        self.assertEqual(self.ui.msg_severity, 'info')
-
-        self.ui.clear_msg()
+        self.assertTrue('ENOENT' in self.ui.report['InvalidMachineReason'])
 
         # invalid base64 encoding
         self.report_file.seek(0)
@@ -281,9 +277,7 @@
         self.report_file.flush()
 
         self.ui.load_report(self.report_file.name)
-        self.assertTrue(self.ui.report is None)
-        self.assertEqual(self.ui.msg_title, _('Invalid problem report'))
-        self.assertEqual(self.ui.msg_severity, 'error')
+        self.assertTrue('damaged' in self.ui.report['InvalidReason'])
 
     def test_restart(self):
         '''restart()'''
@@ -323,7 +317,7 @@
 
         # report without any information (distro bug)
         self.ui.report = apport.Report()
-        self.ui.collect_info()
+        self.ui.collect_info(self.ui.report)
         self.assertTrue(set(['Date', 'Uname', 'DistroRelease', 'ProblemType']).issubset(
             set(self.ui.report.keys())))
         self.assertEqual(self.ui.ic_progress_pulses, 0,
@@ -341,7 +335,7 @@
         # apport hooks)
         self.ui.report['Fstab'] = ('/etc/fstab', True)
         self.ui.report['CompressedValue'] = problem_report.CompressedValue(b'Test')
-        self.ui.collect_info()
+        self.ui.collect_info(self.ui.report)
         self.assertTrue(set(['SourcePackage', 'Package', 'ProblemType',
                              'Uname', 'Dependencies', 'DistroRelease', 'Date',
                              'ExecutablePath']).issubset(set(self.ui.report.keys())))
@@ -356,7 +350,7 @@
         # report with only package information
         self.ui.report = apport.Report('Bug')
         self.ui.cur_package = 'bash'
-        self.ui.collect_info()
+        self.ui.collect_info(self.ui.report, cur_package=self.ui.cur_package)
         self.assertTrue(set(['SourcePackage', 'Package', 'ProblemType',
                              'Uname', 'Dependencies', 'DistroRelease',
                              'Date']).issubset(set(self.ui.report.keys())))
@@ -371,7 +365,7 @@
         self.ui.report = apport.Report('Bug')
         self.ui.cur_package = 'bash'
         self.ui.report_file = self.report_file.name
-        self.ui.collect_info()
+        self.ui.collect_info(self.ui.report, self.ui.report_file, cur_package=self.ui.cur_package)
         self.assertTrue(os.stat(self.report_file.name).st_mode & stat.S_IRGRP)
 
     def test_collect_info_crashdb_spec(self):
@@ -386,7 +380,7 @@
 
         self.ui.report = apport.Report('Bug')
         self.ui.cur_package = 'bash'
-        self.ui.collect_info()
+        self.ui.collect_info(self.ui.report, cur_package=self.ui.cur_package)
         self.assertTrue('CrashDB' in self.ui.report)
         self.assertFalse('UnreportableReason' in self.ui.report,
                          self.ui.report.get('UnreportableReason'))
@@ -405,7 +399,7 @@
 
         self.ui.report = apport.Report('Bug')
         self.ui.cur_package = 'bash'
-        self.ui.collect_info()
+        self.ui.collect_info(self.ui.report, cur_package=self.ui.cur_package)
         self.assertFalse('UnreportableReason' in self.ui.report,
                          self.ui.report.get('UnreportableReason'))
         self.assertEqual(self.ui.report['BashHook'], 'Moo')
@@ -422,7 +416,7 @@
 
         self.ui.report = apport.Report('Bug')
         self.ui.cur_package = 'bash'
-        self.ui.collect_info()
+        self.ui.collect_info(self.ui.report, cur_package=self.ui.cur_package)
         self.assertTrue('nonexisting' in self.ui.report['UnreportableReason'],
                         self.ui.report.get('UnreportableReason', '<not set>'))
 
@@ -434,7 +428,7 @@
 
         self.ui.report = apport.Report('Bug')
         self.ui.cur_package = 'bash'
-        self.ui.collect_info()
+        self.ui.collect_info(self.ui.report, cur_package=self.ui.cur_package)
         self.assertTrue('package hook' in self.ui.report['UnreportableReason'],
                         self.ui.report.get('UnreportableReason', '<not set>'))
 
@@ -446,7 +440,7 @@
 
         self.ui.report = apport.Report('Bug')
         self.ui.cur_package = 'bash'
-        self.ui.collect_info()
+        self.ui.collect_info(self.ui.report, cur_package=self.ui.cur_package)
         self.assertTrue('nonexisting' in self.ui.report['UnreportableReason'],
                         self.ui.report.get('UnreportableReason', '<not set>'))
 
@@ -454,7 +448,7 @@
         '''handle_duplicate()'''
 
         self.ui.load_report(self.report_file.name)
-        self.assertEqual(self.ui.handle_duplicate(), False)
+        self.assertEqual(self.ui.handle_duplicate(self.ui.report), False)
         self.assertEqual(self.ui.msg_title, None)
         self.assertEqual(self.ui.opened_url, None)
 
@@ -462,7 +456,7 @@
         self.report['KnownReport'] = demo_url
         self.update_report_file()
         self.ui.load_report(self.report_file.name)
-        self.assertEqual(self.ui.handle_duplicate(), True)
+        self.assertEqual(self.ui.handle_duplicate(self.ui.report), True)
         self.assertEqual(self.ui.msg_severity, 'info')
         self.assertEqual(self.ui.opened_url, demo_url)
 
@@ -471,7 +465,7 @@
         self.report['KnownReport'] = '1'
         self.update_report_file()
         self.ui.load_report(self.report_file.name)
-        self.assertEqual(self.ui.handle_duplicate(), True)
+        self.assertEqual(self.ui.handle_duplicate(self.ui.report), True)
         self.assertEqual(self.ui.msg_severity, 'info')
         self.assertEqual(self.ui.opened_url, None)
 
@@ -742,6 +736,76 @@
 
         return r
 
+    @patch.object(apport.fileutils, 'get_new_reports')
+    def test_run_crashes(self, patched_fileutils):
+        '''run_crashes()'''
+        r = self._gen_test_crash()
+        self.ui = TestSuiteUserInterface()
+
+        files = [os.path.join(apport.fileutils.report_dir, '%s%i.crash' % t)
+                 for t in zip(['test'] * 3, range(3))]
+        patched_fileutils.return_value = files
+
+        mtimes = []
+        for report_file in files:
+            with open(report_file, 'wb') as f:
+                r.write(f)
+            mtimes.append(os.stat(report_file).st_mtime)
+
+        # Only system internal reports. None should be processed at this time.
+        with patch.object(self.ui, 'run_crash') as patched_run_crash:
+            self.ui.run_crashes()
+        self.assertEqual(patched_run_crash.call_count, 0)
+
+        # They shouldn't be marked as seen.
+        new_mtimes = []
+        for report_file in files:
+            new_mtimes.append(os.stat(report_file).st_mtime)
+        self.assertEqual(mtimes, new_mtimes)
+
+        # Three system internal reports and one application report.
+        path = os.path.join(apport.fileutils.report_dir, 'test3.crash')
+        r['DesktopFile'] = glob.glob('/usr/share/applications/*.desktop')[0]
+        with open(path, 'wb') as f:
+            r.write(f)
+        files.append(path)
+        patched_fileutils.return_value = files
+
+        with patch.object(apport.fileutils, 'mark_report_upload') as p:
+            self.ui.present_details_response = {'report': True,
+                                                'blacklist': False,
+                                                'examine': False,
+                                                'restart': False}
+            self.ui.run_crashes()
+
+            # All reports should be uploaded.
+            self.assertEqual(p.call_count, 4)
+
+        # They should be marked as seen.
+        for i in range(len(files) - 1):
+            self.assertNotEqual(os.stat(files[i]).st_mtime, mtimes[i])
+
+        r = apport.Report()
+        with open(files[0], 'rb') as f:
+            r.load(f)
+        with open(files[0], 'wb') as f:
+            r['ExecutablePath'] = '/nonexistant'
+            r.write(f)
+
+        with patch.object(apport.fileutils, 'seen_report') as seen:
+            # Look at all the reports again.
+            seen.return_value = False
+            with patch.object(apport.fileutils, 'mark_report_upload') as p:
+                self.ui.present_details_response = {'report': True,
+                                                    'blacklist': False,
+                                                    'examine': False,
+                                                    'restart': False}
+                self.ui.run_crashes()
+                with open(files[0], 'rb') as f:
+                    r.load(f)
+                self.assertEqual(r['InvalidReason'],
+                                 'This problem report applies to a program which is not installed any more.')
+
     def test_run_crash(self):
         '''run_crash()'''
 
@@ -859,8 +923,7 @@
                                             'examine': False,
                                             'restart': False}
         self.ui.run_crash(report_file)
-        self.assertEqual(self.ui.msg_severity, 'error', self.ui.msg_text)
-        self.assertTrue('decompress' in self.ui.msg_text)
+        self.assertTrue('decompress' in self.ui.report['InvalidMachineReason'])
         self.assertTrue(self.ui.present_details_shown)
 
     def test_run_crash_argv_file(self):
@@ -967,10 +1030,12 @@
 
         # run
         self.ui = TestSuiteUserInterface()
+        self.ui.present_details_response = {'report': False,
+                                            'blacklist': False,
+                                            'examine': False,
+                                            'restart': False}
         self.ui.run_crash(report_file)
-        self.assertEqual(self.ui.msg_severity, 'error')
-        self.assertTrue('memory' in self.ui.msg_text, '%s: %s' %
-                        (self.ui.msg_title, self.ui.msg_text))
+        self.assertEqual(self.ui.report['InvalidMachineReason'], 'No core dump')
 
     def test_run_crash_preretraced(self):
         '''run_crash() pre-retraced reports.
@@ -1008,7 +1073,7 @@
         copying a .crash file.
         '''
         self.ui.report = self._gen_test_crash()
-        self.ui.collect_info()
+        self.ui.collect_info(self.ui.report)
 
         # now pretend to move it to a machine where the package is not
         # installed
@@ -1048,9 +1113,7 @@
                                             'examine': False,
                                             'restart': False}
         self.ui.run_crash(report_file)
-
-        self.assertEqual(self.ui.msg_title, _('Invalid problem report'))
-        self.assertEqual(self.ui.msg_severity, 'error')
+        self.assertTrue('does not exist' in self.ui.report['InvalidMachineReason'])
 
     def test_run_crash_uninstalled(self):
         '''run_crash() on reports with subsequently uninstalled packages'''
@@ -1069,8 +1132,10 @@
                                             'restart': False}
         self.ui.run_crash(report_file)
 
-        self.assertEqual(self.ui.msg_title, _('Invalid problem report'))
-        self.assertEqual(self.ui.msg_severity, 'info')
+        self.assertTrue('not installed' in self.ui.report['InvalidReason'])
+        self.assertTrue('ENOENT' in self.ui.report['InvalidMachineReason'])
+
+        self.ui.report = None
 
         # interpreted program got uninstalled between crash and report
         r = apport.Report()
@@ -1080,8 +1145,10 @@
 
         self.ui.run_crash(report_file)
 
-        self.assertEqual(self.ui.msg_title, _('Invalid problem report'))
-        self.assertEqual(self.ui.msg_severity, 'info')
+        self.assertTrue('not installed' in self.ui.report['InvalidReason'])
+        self.assertTrue('ENOENT' in self.ui.report['InvalidMachineReason'])
+
+        self.ui.report = None
 
         # interpreter got uninstalled between crash and report
         r = apport.Report()
@@ -1091,8 +1158,8 @@
 
         self.ui.run_crash(report_file)
 
-        self.assertEqual(self.ui.msg_title, _('Invalid problem report'))
-        self.assertEqual(self.ui.msg_severity, 'info')
+        self.assertTrue('not installed' in self.ui.report['InvalidReason'])
+        self.assertTrue('ENOENT' in self.ui.report['InvalidMachineReason'])
 
     def test_run_crash_updated_binary(self):
         '''run_crash() on binary that got updated in the meantime'''
 
=== modified file 'test/test_ui_gtk.py'
--- test/test_ui_gtk.py	2012-11-23 13:43:15 +0000
+++ test/test_ui_gtk.py	2013-04-27 08:52:25 +0000
@@ -18,6 +18,7 @@
 import apport
 import shutil
 import subprocess
+import glob
 from gi.repository import GLib, Gtk
 from apport import unicode_gettext as _
 from mock import patch
@@ -606,8 +607,10 @@
             self.app.w('continue_button').clicked()
             return False
 
-        # remove the crash from setUp() and create a kernel oops
-        os.remove(self.app.report_file)
+        self.app.report['DesktopFile'] = glob.glob('/usr/share/applications/*.desktop')[0]
+        self.app.report['ProcCmdline'] = 'apport-bug apport'
+        with open(self.app.report_file, 'wb') as f:
+            self.app.report.write(f)
         kernel_oops = subprocess.Popen([kernel_oops_path], stdin=subprocess.PIPE)
         kernel_oops.communicate(b'Plasma conduit phase misalignment')
         self.assertEqual(kernel_oops.returncode, 0)
@@ -616,8 +619,8 @@
         self.app.run_crashes()
 
         # we should have reported one crash
-        self.assertEqual(self.app.crashdb.latest_id(), 0)
-        r = self.app.crashdb.download(0)
+        self.assertEqual(self.app.crashdb.latest_id(), 1)
+        r = self.app.crashdb.download(1)
         self.assertEqual(r['ProblemType'], 'KernelOops')
         self.assertEqual(r['OopsText'], 'Plasma conduit phase misalignment')
 
@@ -627,7 +630,7 @@
         self.assertTrue('Plasma conduit' in r['Title'])
 
         # URL was opened
-        self.assertEqual(self.app.open_url.call_count, 1)
+        self.assertEqual(self.app.open_url.call_count, 2)
 
     def test_bug_report_installed_package(self):
         '''Bug report for installed package'''