Merge lp:~ev/apport/disable-core-removal into lp:~apport-hackers/apport/trunk

[Evan]

Martin presumably needs a dataset of core files to verify his work on the crash signature. I need this same data to test the load on various parts of the crash database implementation.

The proposed branch disables removal of core files from bug reports.

[Martin Pitt (pitti)]

For the record, if we do this, we need to keep all bugs private, even the duplicates. Please note that I don't need the core dumps for implementing the new duplication algorithm, just the original Stacktrace.txt attachments. So we should not remove those, the core dumps can go for my sake. If you need them, we should just let this run for a few weeks, and I'd just make the change locally in the data center. It's not something I actually want to commit to trunk.

Do you need the actual core dumps for anything?

[Martin Pitt (pitti)]

review: Needs Information

[Martin Pitt (pitti)]

I applied this to the data center retracer:

--- apport/crashdb_impl/launchpad.py 2011-11-16 14:29:24 +0000
+++ apport/crashdb_impl/launchpad.py 2011-11-16 14:29:57 +0000
@@ -636,7 +636,7 @@
                     return

             for a in bug.attachments:
- if a.title in ('CoreDump.gz', 'Stacktrace.txt',
+ if a.title in ('CoreDump.gz',
                     'ThreadStacktrace.txt', 'Dependencies.txt', 'ProcMaps.txt',
                     'ProcStatus.txt', 'Registers.txt', 'Disassembly.txt'):
                     try:

I. e. we'll retain the original Stacktrace.txt as it was produced on the client machines. This will give me the sample data for checking the "client side duplicate signature" idea we discussed. Is this also enough for you?

[Evan (ev)]

It would be good to have a large sample set of data to work with before we see how things pan out when this is in the archive. Ideally, this would be the raw .crash files, but I'm happy to reconstruct them from the constituent pieces.

I can see a few uses for this:

- Profiling the performance and overhead of the crash reporting daemon.
- Ensuring the retracing system in Cassandra and its message queues work.
- Having a set of real world data for testing the crash format parsing.

Equally, I think we can get by in just mocking this data up. So, if you're concerned about the disk space, security implications, or anything else: by all means, leave the data center retracer as you've modified it.

Cheers, and apologies for the delayed reply.

[Martin Pitt (pitti)]

Right, then we don't need this. We can get 833 reports which have a core dump:

  https://bugs.launchpad.net/ubuntu/+bugs?field.tag=apport-failed-retrace

using python-apport it's really easy to download them from launchpad and save them as a .crash file.

I'll just leave the Stacktrace.txt for a while then, to get some data for the duplication detection.

Unmerged revisions

2067. By Evan on 2011-11-11

Disable removing core files from bug reports to give us a data set to work with for generating a crash signature, as well as profiling the crash database.

=== modified file 'apport/crashdb_impl/launchpad.py'
--- apport/crashdb_impl/launchpad.py	2011-11-01 19:55:36 +0000
+++ apport/crashdb_impl/launchpad.py	2011-11-11 12:19:31 +0000
@@ -410,14 +410,7 @@
                     bug = self.launchpad.bugs[id]
                     break
 
-        # remove core dump if stack trace is usable
         if report.has_useful_stacktrace():
-            for a in bug.attachments:
-                if a.title == 'CoreDump.gz':
-                    try:
-                        a.removeFromBug()
-                    except HTTPError:
-                        pass # LP#249950 workaround
             try:
                 task = self._get_distro_tasks(bug.bug_tasks).next()
                 task.importance = 'Medium'
@@ -636,9 +629,9 @@
                     return
             
             for a in bug.attachments:
-                if a.title in ('CoreDump.gz', 'Stacktrace.txt',
-                    'ThreadStacktrace.txt', 'Dependencies.txt', 'ProcMaps.txt',
-                    'ProcStatus.txt', 'Registers.txt', 'Disassembly.txt'):
+                if a.title in ('Stacktrace.txt', 'ThreadStacktrace.txt',
+                    'Dependencies.txt', 'ProcMaps.txt', 'ProcStatus.txt',
+                    'Registers.txt', 'Disassembly.txt'):
                     try:
                         a.removeFromBug()
                     except HTTPError:
@@ -721,13 +714,6 @@
             task.lp_save()
             bug.newMessage(content=invalid_msg,
                     subject='Crash report cannot be processed')
-            
-            for a in bug.attachments:
-                if a.title == 'CoreDump.gz':
-                    try:
-                        a.removeFromBug()
-                    except HTTPError:
-                        pass # LP#249950 workaround
         else:
             if 'apport-failed-retrace' not in bug.tags:
                 bug.tags = bug.tags + ['apport-failed-retrace'] # LP#254901 workaround