Launchpad itself

Librarian garbage collector does not cope with symlinks

Bug #562828 reported by Stuart Bishop
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Gary Poster
Launchpad itself 10.04

Bug Description

The Librarian garbage collector traverses the file store using os.walk(). We have recently split this over multiple disks using symlinks. Unfortunately, os.walk() does not follow symlinks. This generates a lot of ERROR messages, as the garbage collector does not find files on disk that exist in the database.

A followlinks argument was added to os.walk() in Python 2.6. If we can switch to 2.6 quickly, using this option is the best solution.

Otherwise, we need an os.walk() replacement that follows symlinks.

Related branches

lp:~gary/launchpad/bug562828
Aaron Bentley (community): Approve
Diff: 170 lines (+129/-2)
2 files modified
lib/canonical/librarian/ftests/test_gc.py (+31/-0)
lib/canonical/librarian/librariangc.py (+98/-2)
Tom Haddon (mthaddon)
Changed in launchpad-foundations:
importance: Undecided → High
Revision history for this message
Steve McInerney (spm) wrote :

<stub> Can you run it again? I need to know the first id reported
<stub> If it is 67108865 then it is safe to leave it enabled

2010-04-14 08:24:11 ERROR LibraryFileContent 67108865 exists in the database but was not found on disk.

1st entry off the manual re-run

Revision history for this message
Stuart Bishop (stub) wrote :

So we can leave the script enabled and it will keep doing its job. If we do this, we should filter these errors from the output as we can expect them to grow by tens of thousands each day.

Revision history for this message
Max Bowsher (maxb) wrote :

Would it be practical to (temporarily) use bindmounts instead of symlinks?

Gary Poster (gary)
Changed in launchpad-foundations:
status: New → Triaged
Revision history for this message
Gary Poster (gary) wrote :

Because we will need to carefully dance to both Lucid and Py 2.6 in the data center at the same time, we will not be able to fully switch to Python 2.6 until at least 1.5 cycles from now. We'll want this addressed sooner.

The LOSAs and James regarded hacking os.walk into our scripts as a superior hack than bindmounts--the os.walk copy-n-paste-n-hack should be much easier for us to implement than bindmounts on their side.

I do not remember clearly why filtering the errors was rejected as a work-around, but it was, for the time of 1.5 cycles.

Therefore, we need to copy the 2.6 os.walk code and hack it to work in 2.5 if necessary.

I'm pushing a branch with this change and a test.

Changed in launchpad-foundations:
assignee: nobody → Gary Poster (gary)
status: Triaged → In Progress
milestone: none → 10.04
Revision history for this message
Ursula Junque (ursinha) wrote : Bug fixed by a commit

Fixed in stable r10785 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/10785>

Changed in launchpad-foundations:
status: In Progress → Fix Committed
tags: added: qa-needstesting
Gary Poster (gary)
tags: added: qa-untestable
removed: qa-needstesting
Gary Poster (gary)
Changed in launchpad-foundations:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.