Merge lp:~brian-murray/daisy/remove-sandbox-dir-after-retrace into lp:daisy

Proposed by Brian Murray on 2014-03-21
Status: Merged
Merge reported by: Brian Murray
Merged at revision: not available
Proposed branch: lp:~brian-murray/daisy/remove-sandbox-dir-after-retrace
Merge into: lp:daisy
Diff against target: 15 lines (+5/-0)
1 file modified
daisy/retracer.py (+5/-0)
To merge this branch: bzr merge lp:~brian-murray/daisy/remove-sandbox-dir-after-retrace
Reviewer Review Type Date Requested Status
Evan (community) 2014-03-21 Approve on 2014-04-04
Review via email: mp+212187@code.launchpad.net

Description of the change

This will help reduce the amount of storage that the retracer processes require by cleaning up the sandbox dir in $release-name/cache-$tmpdir/ after every retrace. The cache folder could probably use cleaning up on a regular basis (for the development release at least) as new package versions are a common thing.

To post a comment you must log in.
Evan (ev) wrote :

This was originally done for efficiency, if memory serves. In fact, leaving off the --sandbox-dir option would give you much the same effect in a cleaner way.

Could we not instead have a process watching the disk usage on the retracer? When it gets to a charm-defined percentage limit, this process could send SIGINT to the retracer process, which could be taught to finish up the current retrace and then terminate. The watchdog process would then clean up the retracer cache dirs and restart the retracer.

Alternatively, you could have the retracer do the cleanup on behalf of the watchdog, or just flat out kill the retracer process and put our ability to deal with retracers that die mid-retrace to the test. :)

It *should* throw the work back on the queue, since we didn't ack the message in that case, but I've never thoroughly tested it.

review: Needs Information
Evan (ev) wrote :

Alternatively, if you're keen on the more expedient solution, I'd suggest just dropping the --sandbox-dir option, knowing that we can horizontally scale out the retracers now with juju add-unit.

If you're going down that road, +1.

review: Approve
Brian Murray (brian-murray) wrote :

I think removing --sandbox-dir does more than we actually want. I'm suggesting that we stop using apport-retrace's "--sandbox-dir" option but continuing to use "--cache".

"--sandbox-dir" for retracer.py is a combination of both of these as setup_cache returns the directory for "--sandbox-dir" for apport-retrace and the directory for "--cache". So by not using retracer.py's --sandbox-dir we'll lose the benefit of cacheing deb files.

Evan (ev) wrote :

Could we not teach retracer.py to just use apport's --cache and not
--sandbox-dir? It seems like a bug that --sandbox-dir is an optional
argument in retracer.py despite it very much not being an optional
thing. :)

Either way you want to implement it, though.

On 21 March 2014 13:33, Brian Murray <email address hidden> wrote:
> I think removing --sandbox-dir does more than we actually want. I'm suggesting that we stop using apport-retrace's "--sandbox-dir" option but continuing to use "--cache".
>
> "--sandbox-dir" for retracer.py is a combination of both of these as setup_cache returns the directory for "--sandbox-dir" for apport-retrace and the directory for "--cache". So by not using retracer.py's --sandbox-dir we'll lose the benefit of cacheing deb files.
> --
> https://code.launchpad.net/~brian-murray/daisy/remove-sandbox-dir-after-retrace/+merge/212187
> You are reviewing the proposed merge of lp:~brian-murray/daisy/remove-sandbox-dir-after-retrace into lp:daisy.

Evan (ev) :
review: Approve
Brian Murray (brian-murray) wrote :

A variation of this was merged:

------------------------------------------------------------
revno: 594
committer: Brian Murray <email address hidden>
branch nick: trunk
timestamp: Tue 2014-12-16 14:40:34 -0800
message:
  daisy/retracer.py: add options to have sandbox directory and ddebs cache wiped after retracing.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'daisy/retracer.py'
2--- daisy/retracer.py 2014-03-13 22:53:52 +0000
3+++ daisy/retracer.py 2014-03-21 16:10:29 +0000
4@@ -563,6 +563,11 @@
5 log(traceback.format_exc())
6 raise
7 finally:
8+ if sandbox:
9+ # remove the sandbox as it consumes the most space
10+ # but recreate the dir as setup_cache does
11+ shutil.rmtree(sandbox)
12+ os.mkdir(sandbox)
13 rm_eff(report_path)
14
15 try:

Subscribers

People subscribed via source and target branches

to all changes: