Merge lp:~ev/apport/disable-core-removal into lp:apport

Proposed by Evan on 2011-11-11
Status: Rejected
Rejected by: Martin Pitt on 2011-11-17
Proposed branch: lp:~ev/apport/disable-core-removal
Merge into: lp:apport
Diff against target: 45 lines (+3/-17)
1 file modified
apport/crashdb_impl/launchpad.py (+3/-17)
To merge this branch: bzr merge lp:~ev/apport/disable-core-removal
Reviewer Review Type Date Requested Status
Martin Pitt 2011-11-11 Needs Information on 2011-11-11
Review via email: mp+81974@code.launchpad.net

Description of the change

Martin presumably needs a dataset of core files to verify his work on the crash signature. I need this same data to test the load on various parts of the crash database implementation.

The proposed branch disables removal of core files from bug reports.

To post a comment you must log in.
Martin Pitt (pitti) wrote :

For the record, if we do this, we need to keep all bugs private, even the duplicates. Please note that I don't need the core dumps for implementing the new duplication algorithm, just the original Stacktrace.txt attachments. So we should not remove those, the core dumps can go for my sake. If you need them, we should just let this run for a few weeks, and I'd just make the change locally in the data center. It's not something I actually want to commit to trunk.

Do you need the actual core dumps for anything?

Martin Pitt (pitti) :
review: Needs Information
Martin Pitt (pitti) wrote :

I applied this to the data center retracer:

--- apport/crashdb_impl/launchpad.py 2011-11-16 14:29:24 +0000
+++ apport/crashdb_impl/launchpad.py 2011-11-16 14:29:57 +0000
@@ -636,7 +636,7 @@
                     return

             for a in bug.attachments:
- if a.title in ('CoreDump.gz', 'Stacktrace.txt',
+ if a.title in ('CoreDump.gz',
                     'ThreadStacktrace.txt', 'Dependencies.txt', 'ProcMaps.txt',
                     'ProcStatus.txt', 'Registers.txt', 'Disassembly.txt'):
                     try:

I. e. we'll retain the original Stacktrace.txt as it was produced on the client machines. This will give me the sample data for checking the "client side duplicate signature" idea we discussed. Is this also enough for you?

Evan (ev) wrote :

It would be good to have a large sample set of data to work with before we see how things pan out when this is in the archive. Ideally, this would be the raw .crash files, but I'm happy to reconstruct them from the constituent pieces.

I can see a few uses for this:

- Profiling the performance and overhead of the crash reporting daemon.
- Ensuring the retracing system in Cassandra and its message queues work.
- Having a set of real world data for testing the crash format parsing.

Equally, I think we can get by in just mocking this data up. So, if you're concerned about the disk space, security implications, or anything else: by all means, leave the data center retracer as you've modified it.

Cheers, and apologies for the delayed reply.

Martin Pitt (pitti) wrote :

Right, then we don't need this. We can get 833 reports which have a core dump:

  https://bugs.launchpad.net/ubuntu/+bugs?field.tag=apport-failed-retrace

using python-apport it's really easy to download them from launchpad and save them as a .crash file.

I'll just leave the Stacktrace.txt for a while then, to get some data for the duplication detection.

Unmerged revisions

2067. By Evan on 2011-11-11

Disable removing core files from bug reports to give us a data set to work with for generating a crash signature, as well as profiling the crash database.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'apport/crashdb_impl/launchpad.py'
--- apport/crashdb_impl/launchpad.py 2011-11-01 19:55:36 +0000
+++ apport/crashdb_impl/launchpad.py 2011-11-11 12:19:31 +0000
@@ -410,14 +410,7 @@
410 bug = self.launchpad.bugs[id]410 bug = self.launchpad.bugs[id]
411 break411 break
412412
413 # remove core dump if stack trace is usable
414 if report.has_useful_stacktrace():413 if report.has_useful_stacktrace():
415 for a in bug.attachments:
416 if a.title == 'CoreDump.gz':
417 try:
418 a.removeFromBug()
419 except HTTPError:
420 pass # LP#249950 workaround
421 try:414 try:
422 task = self._get_distro_tasks(bug.bug_tasks).next()415 task = self._get_distro_tasks(bug.bug_tasks).next()
423 task.importance = 'Medium'416 task.importance = 'Medium'
@@ -636,9 +629,9 @@
636 return629 return
637 630
638 for a in bug.attachments:631 for a in bug.attachments:
639 if a.title in ('CoreDump.gz', 'Stacktrace.txt',632 if a.title in ('Stacktrace.txt', 'ThreadStacktrace.txt',
640 'ThreadStacktrace.txt', 'Dependencies.txt', 'ProcMaps.txt',633 'Dependencies.txt', 'ProcMaps.txt', 'ProcStatus.txt',
641 'ProcStatus.txt', 'Registers.txt', 'Disassembly.txt'):634 'Registers.txt', 'Disassembly.txt'):
642 try:635 try:
643 a.removeFromBug()636 a.removeFromBug()
644 except HTTPError:637 except HTTPError:
@@ -721,13 +714,6 @@
721 task.lp_save()714 task.lp_save()
722 bug.newMessage(content=invalid_msg,715 bug.newMessage(content=invalid_msg,
723 subject='Crash report cannot be processed')716 subject='Crash report cannot be processed')
724
725 for a in bug.attachments:
726 if a.title == 'CoreDump.gz':
727 try:
728 a.removeFromBug()
729 except HTTPError:
730 pass # LP#249950 workaround
731 else:717 else:
732 if 'apport-failed-retrace' not in bug.tags:718 if 'apport-failed-retrace' not in bug.tags:
733 bug.tags = bug.tags + ['apport-failed-retrace'] # LP#254901 workaround719 bug.tags = bug.tags + ['apport-failed-retrace'] # LP#254901 workaround

Subscribers

People subscribed via source and target branches