Launchpad itself

librarian-feed-swift needs to upload faster (parallel) at times to keep up

Bug #1599001 reported by Michael Foley
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Colin Watson

Bug Description

When librarian-feed-swift falls behind as has been the case for cRT#93197, we have had to run it in parallel for specific ranges using the --start and --end flags. It would be good if this process could scale itself up to do multiple parallel uploads to swift (with some max number of parallel uploads to limit load).

Tags: librarian

Related branches

~cjwatson/launchpad:parallel-feed-swift
Ioana Lasc (community): Approve
Diff: 168 lines (+90/-4)
3 files modified
cronscripts/librarian-feed-swift.py (+35/-3)
lib/lp/services/librarianserver/swift.py (+14/-1)
lib/lp/services/librarianserver/tests/test_swift.py (+41/-0)
Revision history for this message
Stuart Bishop (stub) wrote :

The simplest approach is to add --instance-id and --num-instances arguments, and have feed swift skip files unless (LibraryFileContent.id % num_instances==instance_id). This and updating the script locking to allow parallel runs is the least invasive approach, allowing us to easily schedule n tasks in parallel.

IIRC feed-swift will eventually disappear in favour of direct uploads to Swift, removing a SPOF.

Colin Watson (cjwatson)
tags: added: librarian
Changed in launchpad:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Colin Watson (cjwatson)
Colin Watson (cjwatson)
Changed in launchpad:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.