Merge lp:~vorlon/ubuntu-archive-tools/queue-item-scaling into lp:ubuntu-archive-tools

[Steve Langasek]

"if not item in list [...] list.append(item)" doesn't scale so good. Use a
set instead so the queue command will scale to handling the 800+ matches for
'queue accept language-pack'.

[Colin Watson (cjwatson)]

I think this will break ordering in 'queue info'. How about using
collections.OrderedDict instead? (Or the collections documentation has
a link to a recipe for OrderedSet.)

[Steve Langasek (vorlon)]

On Fri, Aug 03, 2012 at 07:13:19PM -0000, Colin Watson wrote:
> I think this will break ordering in 'queue info'.

Confirmed.

> How about using collections.OrderedDict instead? (Or the collections
> documentation has a link to a recipe for OrderedSet.)

Looks like this works without too much trouble. I don't see any easy way to
test the performance currently, now that the language packs have been
flushed from the queue, but it at least seems to be working reliably.
Pushed an update.

[lp:~vorlon/ubuntu-archive-tools/queue-item-scaling]

566. By Steve Langasek on 2012-08-03

use collections.OrderedDict() instead of a set, so we can still display items in the queue in order

[Colin Watson (cjwatson)]

review: Approve

=== modified file 'queue'
--- queue	2012-07-18 15:32:38 +0000
+++ queue	2012-08-03 22:02:18 +0000
@@ -23,6 +23,7 @@
 from operator import attrgetter
 from optparse import OptionParser
 import sys
+import collections
 try:
     from urllib.parse import unquote, urlsplit
     from urllib.request import urlretrieve
@@ -71,7 +72,7 @@
     if not args:
         args = ['']
 
-    items = []
+    items = collections.OrderedDict()
     for arg in args:
         arg = arg.strip()
         if arg.isdigit():
@@ -94,7 +95,7 @@
                                  item_suite, options.distribution.name,
                                  options.suite))
             if queue_item_allowed(options, item):
-                items.append(item)
+                items[item] = 1
         else:
             kwargs = {}
             if "/" in arg:
@@ -105,8 +106,8 @@
                 pocket=options.pocket, status=options.queue,
                 exact_match=options.exact_match, **kwargs)
             for item in new_items:
-                if item not in items and queue_item_allowed(options, item):
-                    items.append(item)
+                if queue_item_allowed(options, item):
+                    items[item] = 1
 
     return items