Merge lp:~jjed/archive-crawler/use-aptfile into lp:~mvo/archive-crawler/mvo

[Jjed]

Apologies for the huge patch. I abstracted IconFinder out, and that
ballooned the line count of this solution quite a bit.

SUMMARY

This branch adds a new component, `IconFinder`, which should solve the
issue of finding icons in an archive. It is a frontend for `ArchiveCache`,
a general solution for searching archive files via Contents-<ARCH>.gz.

In the future, `ArchiveCache` might replace `ArchiveCrawler` as a cleaner,
quicker, and more modularity-friendly solution. As it stands, only `IconFinder` uses it.

DESCRIPTION

`ArchiveCache` loads all lines from a Content.gz file and either writes them
to files or stores them in RAM depending on memory constraints. It provides
methods to search them for files and paths.

Its major feature over a simple grep is the separate caching of index prefixes,
allowing a frontend to quickly search only the files in a certain directory
(eg /usr/share/icons). `IconFinder` is a simple wrapper for this functionality,
searching a sequence of prefixes for a match.

`IconFinder` integration replaces most of my work in `DesktopDataExtractor`.
`ArchiveCrawler` now finds all deb paths before crawling, so as to allow
`IconFinder` access to any deb from the start.

`DesktopDataExtractor` now takes `archive_dir` rather than `pool_dir`, the
pool directory being found instead at `archive_dir`/pool, and the contents
file at `archive_dir`/Contents-<ARCH>.gz.

TESTING

Out of 361 desktop entries I tested (an old main[a-k] + universe[2-d]) with
343 icon requests, 13 failed. Of these, 11 failed because they requested
non-existent icons, 1 failed because it requested an icon too large ('arista'),
and 1 failed because it stored its icon in /usr/lib without a symlink in
/usr/share or and absolute path request ('batmon.app').

The unit tests that pass as of r123 pass now, as well as new ones for
`IconFinder` and `ArchiveCache`.

FUTURE

Hopefully this branch -- together with a solution for extracting large icons
-- will solve #599535. With the exception of arista, all icons that fail to
extract should now be the fault of the package. Hopefully. We'll see if the
full archive has an interesting edge cases.

[Michael Vogt (mvo)]

On Tue, May 31, 2011 at 05:55:57AM -0000, Jacob Johan Edwards wrote:
> Jacob Johan Edwards has proposed merging lp:~j-johan-edwards/archive-crawler/use-aptfile into lp:~mvo/archive-crawler/mvo.
>
> Requested reviews:
> Michael Vogt (mvo)
>
> For more details, see:
> https://code.launchpad.net/~j-johan-edwards/archive-crawler/use-aptfile/+merge/62944
>
> Apologies for the huge patch. I abstracted IconFinder out, and that
> ballooned the line count of this solution quite a bit.

That is fine, the content is absolutely worth it!

> SUMMARY
>
> This branch adds a new component, `IconFinder`, which should solve the
> issue of finding icons in an archive. It is a frontend for `ArchiveCache`,
> a general solution for searching archive files via Contents-<ARCH>.gz.
>
> In the future, `ArchiveCache` might replace `ArchiveCrawler` as a cleaner,
> quicker, and more modularity-friendly solution. As it stands, only `IconFinder` uses it.

Awsome, this is great work already and has even more potential!

[..]
> TESTING
>
> Out of 361 desktop entries I tested (an old main[a-k] + universe[2-d]) with
> 343 icon requests, 13 failed. Of these, 11 failed because they requested
> non-existent icons, 1 failed because it requested an icon too large ('arista'),
> and 1 failed because it stored its icon in /usr/lib without a symlink in
> /usr/share or and absolute path request ('batmon.app').
>
> The unit tests that pass as of r123 pass now, as well as new ones for
> `IconFinder` and `ArchiveCache`.

Thanks for updating the tests as well (and adding more!), it looks
really well done.

> FUTURE
>
> Hopefully this branch -- together with a solution for extracting large icons
> -- will solve #599535. With the exception of arista, all icons that fail to
> extract should now be the fault of the package. Hopefully. We'll see if the
> full archive has an interesting edge cases.

I'm running a full oneiric extraction now, it will be interessting to
see what it outputs and I will compare to the previous runs to find
regressions (but I really doubt there will be any).

Thanks,
 Michael

=== added directory 'ArchiveCache'
=== added file 'ArchiveCache/__init__.py'
--- ArchiveCache/__init__.py	1970-01-01 00:00:00 +0000
+++ ArchiveCache/__init__.py	2011-05-31 05:55:54 +0000
@@ -0,0 +1,313 @@
+import gzip
+import hashlib
+import logging
+import os.path
+import re
+import tempfile
+
+
+class ArchiveCache(object):
+    """
+    This class caches the file paths in a deb archive--as described by
+    that archive's Contents-<ARCH>.gz--to make search of them efficient.
+    It can also index subsets of the archives (eg all paths beginning
+    with /usr/share/icons/) for quicker informed searches.
+
+    Indexes are created at instantiation via the prefixes_to_index
+    argument. They can then be searched simply by using the `prefix`
+    argument in search methods.
+
+    In the event of ArchiveCache runs out of memory to allocate, it
+    will prioritize the indexes in order, and write all excess data
+    to cache files.
+    """
+
+    def __init__(self, contents_file, prefixes_to_index=[], memory=500):
+        """
+        @contents_file: The path to a Contents-<ARCH>.gz file.
+        @prefixes_to_index: A list of path prefixes (eg /usr/share is a
+                            prefix of /usr/share/*) to separately cache
+                            and make searchable for better speeds.
+        @memory: The maximum amount of data in the contents_file (in MB)
+                 that the cache will load into memory.
+        """
+        self._contents_file = contents_file
+        self._memory, self._max_memory = 0, memory * 1024 * 1024
+        self._num_lines = 0
+        self._checksum = None
+        self._create_indexes(prefixes_to_index)
+        self._load_cache()
+
+    def search(self, regex, prefix="", first_only=False):
+        """
+        Returns a list of (package name, path) pairs, where "path" is a
+        file location in the archive that the search term matches, and
+        "package name" is the name of a package that contains it.
+
+        @regex: A string regular expression to filter paths with.
+        @prefix: A string path from prefix_indexes. Only paths starting
+                 with the prefix will be searched. Speeds up queries.
+        @first_only: Set True to search only until finding one hit.
+        """
+        prefix = prefix.strip("/")
+        if prefix:
+            index = self._indexes[prefix]
+            logging.debug("Searching cache for all files '%s' with prefix %s" %
+                         (regex, prefix))
+        else:
+            index = self._contents_index
+            logging.debug("Searching cache for all files '%s'" % regex)
+        results = index.search(regex, first_only)
+        if results and first_only:
+            return results
+        for p in self._prefixes:
+            if p.startswith(prefix) and p != prefix:
+                child_index = self._indexes[p]
+                results.extend(child_index.search(regex, first_only))
+                if results and first_only:
+                    return results
+        return results
+
+    def search_exact(self, path):
+        """
+        Returns the name of a package that contains the exact `path`
+        location, or None if `path` does not exist in the archive.
+        """
+        path = path.strip("/")
+        logging.debug("Searching cache for '%s'" % path)
+        prefix = self._find_index(path)
+        if prefix:
+            return self._indexes[prefix].search_exact(path)
+        return self._contents_index.search_exact(path)
+
+    def _create_indexes(self, prefixes):
+        self._indexes = {}
+        self._prefixes = tuple([p.strip("/") for p in prefixes])
+        # indexes for user-defined paths like '/usr/share/icons'
+        for prefix in self._prefixes:
+            self._indexes[prefix] = _Index()
+        # an index for lines of all other paths
+        self._contents_index = _Index()
+
+    def _load_cache(self):
+        logging.info("Loading archive paths from %s into cache" %
+                     os.path.basename(self._contents_file))
+        contents = gzip.open(self._contents_file)
+        while not re.match("^FILE\W+LOCATION", contents.readline()):
+            continue
+        for line in contents:
+            self._memory += self._place_in_index(line)
+            if self._memory > self._max_memory:
+                self._reduce_memory_use()
+            self._num_lines += 1
+        contents.close()
+        logging.info("Done loading cache. %iMB in active memory." %
+                     (self._memory / 1048576))
+
+    def _place_in_index(self, line):
+        # put a line in its right index and return the amount of memory it took
+        path = line.split()[0]
+        prefix = self._find_index(path)
+        if prefix:
+            return self._indexes[prefix].add_line(line)
+        else:
+            return self._contents_index.add_line(line)
+
+    def _find_index(self, path):
+        # get the longest index path prefixing the given path
+        candidate_index = ""
+        for prefix in self._prefixes:
+            if path.startswith(prefix) and len(prefix) > len(candidate_index):
+                candidate_index = prefix
+        return candidate_index
+
+    def _reduce_memory_use(self):
+        if self._contents_index.bytes_loaded:
+            logging.info("* Max cache load exceeded. Removing non-indexed"
+                         "paths from memory")
+            self._contents_index.flush_memory()
+            return
+        for prefix in reversed(self._prefixes):
+            if self._indexes[prefix].bytes_loaded:
+                logging.info("Max cache load exceeded. Removing '%s' index "
+                             "from memory" % prefix)
+                self._indexes[prefix].flush_memory()
+                return
+
+    def __len__(self, prefix=""):
+        """
+        Returns the number of paths searchable in this cache. Assign
+        `prefix` to count the paths in a particular index.
+        """
+        if not prefix:
+            return self._num_lines
+        else:
+            # the length of the prefix and its children (eg /usr/ -> /usr/share)
+            prefix = prefix.strip("/")
+            num_lines = self._indexes[prefix].num_lines
+            for p in self._prefixes:
+                if p.startswith(prefix + "/"):
+                    num_lines += self._indexes[p].num_lines
+            return num_lines
+
+    def __contains__(self, prefix):
+        return prefix.strip("/") in self._prefixes
+
+    def _get_checksum(self):
+        if not self._checksum:
+            contents = open(self._contents_file, 'r')
+            data = contents.read()
+            contents.close()
+            self._checksum = hashlib.md5(data).hexdigest()
+        return self._checksum
+
+    checksum = property(_get_checksum, doc="""
+        An MD5 hash of this cache's Contents.gz file""")
+
+    prefixes = property(lambda self: self._prefixes, doc="""
+        The path prefixes that the cache has indexed for searching""")
+
+
+class _Index:
+    # a helper implementation class for indexes: cached lines all beginning
+    # with the same prefix
+    def __init__(self):
+        self.bytes_loaded = 0
+        self.num_lines = 0
+        self._file = tempfile.TemporaryFile()
+        self._lines = []
+        self._using_memory = True
+
+    def add_line(self, line):
+        # add a line to the cache file, and memory if possible
+        self.num_lines += 1
+        self._file.write(line)
+        if self._using_memory:
+            to_store = line.strip()
+            self._lines.append(to_store)
+            bytes_added = len(to_store)
+            self.bytes_loaded += bytes_added
+            return bytes_added
+        return 0
+
+    def flush_memory(self):
+        # stop caching lines in memory
+        self._lines = None
+        self.bytes_loaded = 0
+        self._using_memory = False
+
+    def search_exact(self, path):
+        if self._using_memory:
+            line = _search_lines_exact(path, self._lines)
+        else:
+            line = _search_file_exact(path, self._file)
+        if not line:
+            return None
+        # the right-most part of a Contents file is a package for the path
+        parts = line.rsplit("/", 1)
+        if len(parts) != 2:
+            logging.warning("Invalid contents line: %s" % line)
+            return None
+        return parts[1].strip()
+
+    def search(self, regex, first_only=False):
+        lines = _search_pattern(regex, self._file, self._lines, first_only)
+        results = []
+        for line in lines:
+            parts = line.split()
+            if len(parts) != 2:
+                logging.warning("Invalid contents line: %s" % line)
+                continue
+            package_data = parts[1].rsplit("/", 1)
+            if len(package_data) != 2:
+                logging.warning("Invalid contents line: %s" % line)
+                continue
+            path, package_name = parts[0], package_data[1]
+            results.append((package_name, path))
+        return results
+
+
+# Utility functions for quickly searching files according to the
+# following line format used by Contents-<ARCH>:
+#
+#    absolute/path/to/file          section/cat/pkgname
+
+def _search_file_exact(search_term, file):
+    # Searches the file at `file` for a line starting with
+    # `search_term`. Assumes lines are in alphabetical order. Returns
+    # None if there is no match.
+    file.seek(0, 2)
+    low, high = 0, file.tell()
+
+    # perform a binary search on the file
+    while high > low:
+        mid = (low + high) / 2
+        file.seek(mid)
+
+        # read current line from the beginning
+        while file.tell() != 0 and file.read(1) != '\n':
+            file.seek(-2, os.SEEK_CUR)
+        line_begin = file.tell()
+        line = file.readline()
+        line_end = file.tell()
+
+        first_term = line.split()[0]
+        if search_term.startswith(first_term):
+            # return if first term matches search term
+            if len(search_term) == len(first_term):
+                return line
+            # or if the first term looks like a directory (symlink)
+            elif (len(search_term) > len(first_term) and
+                  search_term[len(first_term)] == "/"):
+                return line
+        # otherwise continue the binary search
+        match = cmp(search_term, first_term)
+        if match < 0:
+            high = line_begin
+        elif match > 0:
+            low = line_end
+        else:
+            return line
+
+
+def _search_lines_exact(search_term, lines):
+    # Searches `lines` for a line starting with `search_term`. Assumes
+    # lines are in alphabetical order. Returns None if there is no match.
+    low, high = 0, len(lines)
+    while high > low:
+        mid = (low + high) / 2
+        first_term = lines[mid].split()[0]
+        if search_term.startswith(first_term):
+            # return if first term matches search term
+            if len(search_term) == len(first_term):
+                return lines[mid]
+            # or if the first term looks like a directory (symlink)
+            elif (len(search_term) > len(first_term) and
+                  search_term[len(first_term)] == "/"):
+                return lines[mid]
+        # otherwise continue the binary search
+        match = cmp(search_term, first_term)
+        if match < 0:
+            high = mid
+        elif match > 0:
+            low = mid + 1
+        else:
+            return lines[mid]
+
+
+def _search_pattern(search_pattern, file, lines=None, first_only=False):
+    # Searches the file at `file` or the `line_list` for all lines where
+    # the first term (whitespace delimited) matches the `search_pattern`
+    # regex string. Use `lines` if they are already loaded.
+    pattern = re.compile(search_pattern.replace('+', '\+'))
+    results = []
+    if lines == None:
+        lines = file
+        lines.seek(0)
+    for line in lines:
+        first_term = line.split()[0]
+        if pattern.match(first_term):
+            results.append(line)
+            if first_only:
+                return tuple(results)
+    return tuple(results)

=== modified file 'ArchiveCrawler/__init__.py'
--- ArchiveCrawler/__init__.py	2011-04-13 13:15:17 +0000
+++ ArchiveCrawler/__init__.py	2011-05-31 05:55:54 +0000
@@ -48,6 +48,7 @@
         self._loadDebFilesDone()
         self.callbacks = set()
         self.pkgs_to_debpath = {}
+        self.pkgs_to_debinfo = {}
 
     def registerCallback(self, c):
         if not callable(c):
@@ -93,8 +94,8 @@
         logging.debug("adding '%s' to debfiles_done" % debfile)
         self.debfiles_done.add(debfile)
 
-    def inspectDeb(self, debfile):
-        #logging.debug("inspectDeb %s" % debfile)
+    def indexDeb(self, debfile):
+        # add debfile to self.pkgs_to_debpath if it's valid for this crawler
         m = re.match(".*/(.*)_(.*)_(.*).deb", debfile)
         pkgname = m.group(1)
         pkgver = urllib.unquote(m.group(2))
@@ -130,16 +131,20 @@
             logging.debug("skipping, compoent does not match (expected '%s' got '%s' "% (component, debfile))
             return False
         # add to mapping of name and deb
-        # it may be needed if it contains an application's icon
         self.pkgs_to_debpath[pkgname] = debfile
-        # ... then filter if done already
+        self.pkgs_to_debinfo[pkgname] = (candVer, pkgarch, component)
+        return True
+
+    def inspectDeb(self, pkgname):
+        debfile = self.pkgs_to_debpath[pkgname]
+        candVer, pkgarch, component = self.pkgs_to_debinfo[pkgname]
+        # ... filter if done already
         if debfile in self.debfiles_done:
             logging.debug("skipping, already in debfiles_done '%s'" % debfile)
             return False
 
         # looks like we have a valid ver
         logging.debug("found valid deb: '%s'" % debfile)
-
         
         # pass the epoch here too, this information is not encoded in the
         # filename
@@ -152,7 +157,9 @@
         for (root, dirs, files) in os.walk(self.pooldir):
             for f in files:
                 if f.endswith(".deb"):
-                    self.inspectDeb(os.path.join(root,f))
+                    self.indexDeb(os.path.join(root, f))
+        for pkg in self.pkgs_to_debpath:
+            self.inspectDeb(pkg)
         self._saveDebFilesDone()
 
     def findOrphanedFiles(self):

=== modified file 'DesktopDataExtractor/__init__.py'
--- DesktopDataExtractor/__init__.py	2011-04-19 13:54:07 +0000
+++ DesktopDataExtractor/__init__.py	2011-05-31 05:55:54 +0000
@@ -19,6 +19,8 @@
 import time
 
 import ArchiveCrawler
+import ArchiveCache
+import IconFinder
 
 try:
     # only available on 2.7 so we provide a very simple backport
@@ -30,16 +32,19 @@
 
     SUPPORTED_ARCHES = ("i386","amd64")
 
-    def __init__(self, aptroot, pooldir, datadir):
+    def __init__(self, aptroot, archivedir, datadir, dist):
         # init dirs
         self.aptroot = aptroot
-        self.pooldir = pooldir
+        self.archivedir = archivedir
+        self.pooldir = os.path.join(archivedir, 'pool')
         self.datadir = datadir
+        self.dist = dist
         self.tmpdir  = tempfile.mkdtemp()
         self.menu_data = os.path.join(self.datadir, "menu-data")
         self.menu_data_codecs = os.path.join(self.datadir, "menu-data-codecs")
         if not os.path.exists(self.menu_data_codecs):
             raise Exception, "no menu-data-codecs/ dir in %s" % datadir
+
             
         self.codecs_foradditional = { }
         # packages we have already seen
@@ -62,12 +67,6 @@
         # available in certain arches
         self.pkgs_per_arch = {}
         self.pkgs_per_arch["all"] = set()
-        # a mapping of package names to wanted application icons their
-        # packages don't contain
-        self.pkgs_to_missing_icons = {}
-        # regular expressions for finding packages that might contain
-        # wanted icons
-        self.iconsearch_regex = []
         # now read the config
         self._readConfig()
 
@@ -89,7 +88,6 @@
         blacklist_desktop = os.path.join(self.datadir,"blacklist_desktop.cfg")
         renamecfg = os.path.join(self.datadir,"rename.cfg")
         annotatecfg = os.path.join(self.datadir,"annotate.cfg")
-        iconsearchcfg = os.path.join(self.datadir,"icon_search.cfg")
         if os.path.exists(blacklist):
             logging.info("using blacklist: '%s'" % blacklist)
             for line in open(blacklist).readlines():
@@ -124,14 +122,6 @@
                     annotations = annotations_str.split(",")
                     logging.debug("annotations: '%s': %s" % (desktopfile,annotations))
                     self.desktop_annotate[desktopfile] = annotations
-        if os.path.exists(iconsearchcfg):
-            logging.info("using icon search: '%s'" % iconsearchcfg)
-            for line in open(iconsearchcfg):
-                line = line.strip()
-                if line != "" and not line.startswith("#"):
-                    logging.debug("icon search regex: '%s'" % line)
-                    self.iconsearch_regex.append(line)
-
 
 
     def extract(self):
@@ -144,8 +134,19 @@
         self._cleanOrphans()
 
         logging.info("Starting extraction in %s" % self.pooldir)
+        contents = {}
+        for arch in self.SUPPORTED_ARCHES:
+            contents_file = "Contents-%s.gz" % arch
+            contents_path = os.path.join(self.archivedir, "dists",
+                                          self.dist, contents_file)
+            contents[arch] = contents_path
+            assert os.path.exists(contents_path), "Cannot find Contents at %s" % contents_path
+        prefixes = IconFinder.IconFinder.prefixes
         for arch in self.SUPPORTED_ARCHES:
             logging.debug("looking at '%s'" % arch)
+            self.archiveCache = ArchiveCache.ArchiveCache(contents[arch],
+                                                          prefixes)
+            self.iconFinder = IconFinder.IconFinder(self.archiveCache)
             self.pkgs_per_arch[arch] = set()
             self.crawler = ArchiveCrawler.ArchiveCrawler(self.aptroot,
                                                          self.pooldir,
@@ -156,65 +157,10 @@
             self.crawler.updateCache()
             self.crawler.registerCallback(self.inspectDeb)
             self.crawler.crawl()
-            self._findMissingIcons()
         self._calcArchSpecific()
         self._addCodecInformation()
         pickle.dump(self.deb_to_files,open(self.deb_to_files_f,"w"))
         logging.info("extract() finished")
-
-    def _findMissingIcons(self):
-        """ search for missing desktop icons in using the crawl cache """
-        for (pkgname, icons) in self.pkgs_to_missing_icons.items():
-            logging.debug("Searching for missing '%s' icons" % pkgname)
-            # get an ordered set from most likely to least likely package
-            to_search = OrderedDict()
-            # add (in order of importance) all cached packages matching regex
-            for regex in self.iconsearch_regex:
-                # FIXME: use {0} here once the extraction host moves from
-                #        py2.5 to 2.6
-                if r'%(first_term)s' in regex:
-                    first_term = re.split('-|_', pkgname, 1)[0]
-                    regex = regex % { 'first_term' : first_term }
-                try:
-                    matches = filter(re.compile(regex).match, self.pkgs_seen)
-                    for match in matches:
-                        to_search[match] = None
-                except Exception as e:
-                    print "ERROR: %s" % e
-            # queue all non-library dependencies of the package
-            deps = self.crawler.cache[pkgname].candidate.dependencies
-            for dep in deps:
-                for dep_candidate in dep.or_dependencies:
-                    depname = dep_candidate.name
-                    if not depname.startswith('lib'):
-                        to_search[depname] = None
-
-            # finally, search the set of likely packages
-            for name in to_search:
-                # get cached tarfile
-                logging.debug("* Looking in %s" % name)
-                if name not in self.crawler.pkgs_to_debpath:
-                    logging.debug("  Deb for %s not found!" % name)
-                    continue
-                try:
-                    debPath = self.crawler.pkgs_to_debpath[name]
-                    datafile = self._extractDebData(debPath)
-                    tar = tarfile.open(datafile)
-                except:
-                    logging.debug("  Deb for %s could not be opened!" % name)
-                    continue
-                found = set()
-                for icon in icons:
-                    (res, n) = self.search_icon(tar, icon, self.menu_data)
-                    if res == True:
-                        logging.debug("  Icon %s found!" % icon)
-                        found.add(icon)
-                # stop searching for any icons we find
-                icons = icons.difference(found)
-                if len(icons) == 0:
-                    del self.pkgs_to_missing_icons[pkgname]
-                    break
-        logging.info("missing icons left: '%s'" % self.pkgs_to_missing_icons)
  
     def _calcArchSpecific(self):
         # now add the architecture information
@@ -358,54 +304,44 @@
         return False
 
     def search_icon(self, tarfile, iconName, outputdir):
-        if iconName == None:
-            logging.warning("search_icon() called with no icon name")
-            return (False, None)
-
-        # a iconName can be a single name or a full path
-        # if it is a single name, look into a icon-theme path (usr/share/icons/hicolor) and then into usr/share/pixmaps
-        # if it is a full path just look for this
-
-        # this is the "full-path" case
-        # FIXME: there are (some) icons that are not full pathes like "/usr/.../"
-        #        but "zapping/zapping.png"
-        if "/" in iconName:
-            newIconName = iconName.replace("/", "_")
-            outpath = os.path.join(outputdir,"icons",newIconName)
-            # prevent wasted disk read
-            if os.path.exists(outpath):
-                return (True, newIconName)
-            res = self.extract_icon(tarfile, iconName, outpath)
-            return (res, newIconName)
-
-        # this is the "get-it-from-a-icontheme" case, look into icon-theme hicolor and usr/share/pixmaps
-
-        # search path (ordered by importance)
-        search_dirs = [
-            "usr/share/icons/hicolor/64x64",
-            "usr/share/icons/hicolor/48x48",
-            "usr/share/icons/hicolor/128x128",
-            "usr/share/pixmaps",
-            "usr/share/icons/hicolor/32x32",
-            "usr/share/icons/hicolor/22x22",
-            "usr/share/icons/hicolor/16x16",
-            "usr/share/icons"
-            ]
-        # extensions (ordered by importance)
-        pixmaps_ext = ["", ".png",".xpm",".svg"]
-
-        # prevent wasted disk read
-        if os.path.exists(os.path.join(outputdir,"icons",iconName)):
-            return (True, None)
-        for d in search_dirs:
-            for name in tarfile.getnames():
-                if d in name:
-                    for ext in pixmaps_ext:
-                        if name.endswith(iconName+ext):
-                            res = self.extract_icon(tarfile, name, os.path.join(outputdir,"icons", os.path.basename(name)))
-                            return (res, None)
-        logging.warning("no icon: '%s' could be found" % iconName)
-        return (False, None)
+        # ask IconFinder for an icon and its package
+        results = self.iconFinder.search(iconName)
+        if not results:
+            logging.warning("Could not find package with icon '%s'" % iconName)
+            return None
+        logging.debug("Found icon and package %s" % str(results))
+        pkgname, iconPath = results
+
+        # retrieve the package deb from the archive crawler
+        if pkgname not in self.crawler.pkgs_to_debpath:
+            logging.warning("Could not find deb '%s' for '%s'" %
+                            (pkgname, iconName))
+            return None
+        try:
+            debPath = self.crawler.pkgs_to_debpath[pkgname]
+            datafile = self._extractDebData(debPath)
+            tar = tarfile.open(datafile)
+        except Exception as e:
+            logging.warning("Could not open deb '%s' for '%s'" %
+                            (pkgname, iconName, str(e)))
+            return None
+
+        # extract the icon
+        newIconName = iconName.replace("/", "_")
+        outpath = os.path.join(outputdir, "icons", newIconName)
+        success = self.extract_icon(tar, os.path.join(".", iconPath), outpath)
+        os.remove(datafile)
+        if not success:
+            logging.warning("Could not extract '%s' from '%s' tarfile" %
+                            (iconPath, pkgname))
+            return None
+
+        # validate it is an icon
+        filetype = subprocess.check_output(["file", "-b", outpath]).strip()
+        if filetype == "ASCII text":
+            logging.warning("'%s' is not an icon" % iconPath)
+            return None
+        return newIconName
     
     def tarfile_extract_orlog(self, dataFile, path):
         try:
@@ -493,12 +429,16 @@
                 line = string.strip(line)
                 if line.startswith("Icon="):
                     iconName = line[line.index("=")+1:]
-                    logging.debug("Package '%s' needs icon '%s'" % (pkgname, iconName))
-            (res, newIconName) = self.search_icon(dataFile, iconName, outputdir)
-            if res == False:
-                if not pkgname in self.pkgs_to_missing_icons:
-                    self.pkgs_to_missing_icons[pkgname] = set()
-                self.pkgs_to_missing_icons[pkgname].add(iconName)
+            if not iconName:
+                logging.debug("No icon needed for '%s'" %
+                              os.path.basename(path))
+                newIconName = None
+            else:
+                logging.debug("Package '%s' needs icon '%s'" % (pkgname, iconName))
+                newIconName = self.search_icon(dataFile, iconName, outputdir)
+                if newIconName == None:
+                    logging.warning("Could not retrieve icon for '%s'" %
+                                     os.path.basename(path))
 
             # now check for supicious pkgnames (FIXME: make this not hardcoded)
             if "-common" in pkgname or "-data" in pkgname:
@@ -580,7 +520,14 @@
                     # extract it here, python tarfile does not support lzma
                     subprocess.call(["lzma","-d",datafile])
                     datafile = os.path.splitext(datafile)[0]
-        return datafile
+        # make name unique
+        datafile_new = datafile.replace("data", repr(time.time()))
+        try:
+            os.rename(datafile, datafile_new)
+        except:
+            logging.warning("Renaming tarball to %s failed" % datafile_new)
+            return
+        return datafile_new
 
     def inspectDeb(self, crawler, filename, pkgname, ver, pkgarch, component):
         """ check if the deb is interessting for us (not blacklisted) """

=== added directory 'IconFinder'
=== added file 'IconFinder/__init__.py'
--- IconFinder/__init__.py	1970-01-01 00:00:00 +0000
+++ IconFinder/__init__.py	2011-05-31 05:55:54 +0000
@@ -0,0 +1,90 @@
+import logging
+
+# Policies for searching icon directories.
+BEST_ICON_POLICY, SMALL_ICON_POLICY = range(2)
+
+_SEARCH_DIRS = {
+    # Finds better, but often extremely large icons. Favors SVG.
+    BEST_ICON_POLICY: ["usr/share/icons/hicolor/scalable",
+                       "usr/share/icons/hicolor/128x128",
+                       "usr/share/icons/hicolor/256x256",
+                       "usr/share/icons/hicolor/64x64",
+                       "usr/share/icons/hicolor/48x48",
+                       "usr/share/pixmaps",
+                       "usr/share/icons/hicolor/32x32",
+                       "usr/share/icons/hicolor/22x22",
+                       "usr/share/icons/hicolor/16x16",
+                       "usr/share/icons/Humanity",
+                       "usr/share/icons",
+                       "usr/share"],
+
+    # Provides smaller icons. Favors PNG.
+    SMALL_ICON_POLICY: ["usr/share/icons/hicolor/64x64",
+                        "usr/share/icons/hicolor/48x48",
+                        "usr/share/icons/hicolor/128x128",
+                        "usr/share/pixmaps",
+                        "usr/share/icons/hicolor/32x32",
+                        "usr/share/icons/hicolor/22x22",
+                        "usr/share/icons/hicolor/16x16",
+                        "usr/share/icons/Humanity",
+                        "usr/share/icons",
+                        "usr/share"]}
+
+
+class IconFinder:
+    """Searches for application icons according to a given policy."""
+
+    # When called to search for an icon, IconFinder looks in indexed
+    # path prefixes in an order decided by policy.
+
+    def __init__(self, cache, policy=SMALL_ICON_POLICY):
+        """
+        @cache: An ArchiveCache instance to search for icons with.
+        @policy: the priority policy to use when searching for icons
+                 Either BEST_ICON_POLICY or SMALL_ICON_POLICY (default)
+        """
+        self._set_cache(cache)
+        self._set_policy(policy)
+
+    def search(self, icon_name):
+        """
+        Finds, according to policy, the best package name and icon
+        path for `icon_name`.
+        @icon_name: A valid Icon string for an XDG Desktop Entry.
+        @returns ('packagename', '/path/to/icon') or None
+        """
+        if icon_name.startswith("/"):
+            result = self._cache.search_exact(icon_name)
+            if result:
+                return (result, icon_name.strip("/"))
+        else:
+            # match any path ending with the name and an extension
+            pattern = ".*/%s(.png|.xpm|.svg|)$" % icon_name
+            for prefix in self._search_dirs:
+                results = self._cache.search(pattern, prefix, first_only=True)
+                if results:
+                    return results[0]
+
+    def _set_cache(self, cache):
+        if not self.prefixes.issubset(set(cache.prefixes)):
+            logging.warning("Using an ArchiveCache that doesn't index "
+                            "IconFinder's search prefixes")
+        self._cache = cache
+
+    def _set_policy(self, policy):
+        # Ensure the policy exists before assigning
+        assert policy in (BEST_ICON_POLICY, SMALL_ICON_POLICY)
+        self._policy = policy
+        self._search_dirs = _SEARCH_DIRS[policy]
+
+    # A set of every path in the search directories
+    prefixes = set(sum(_SEARCH_DIRS.values(), []))
+
+    cache = property(lambda self: self._cache, _set_cache, doc="""
+        An ArchiveCache to search for icons with. It should index all
+        file paths in `IconFinder.prefixes` for good performance.""")
+
+    policy = property(lambda self: self._policy, _set_policy, doc="""
+        Determines what type and size of icon the IconFinder will
+        search for first. Uses SMALL_ICON_POLICY by default. Switch to
+        BEST_ICON_POLICY to favor large and scalable icons.""")

=== removed file 'data/icon_search.cfg'
--- data/icon_search.cfg	2011-04-14 09:16:47 +0000
+++ data/icon_search.cfg	1970-01-01 00:00:00 +0000
@@ -1,11 +0,0 @@
-# these are regular expressions for finding packages with icons for a
-# *.desktop file, when icons are not in the application package. Any
-# string "{0}" will be formatted to the first hyphen-delimited term of
-# an application package (eg "foo-bar-baz" -> "foo")
-
-# for cases like wesnoth/wesnoth-data 
-^%(first_term)s.+(data|common)$
-
-# for when environment applications use environmental icons
-gnome-icon-theme
-oxygen-icon-theme

=== modified file 'getMenuData.py'
--- getMenuData.py	2011-04-11 13:12:12 +0000
+++ getMenuData.py	2011-05-31 05:55:54 +0000
@@ -39,8 +39,10 @@
     parser.add_option("--actionsdir", "--actionsdir", dest="actionsdir",
                       help="actionsdir",
                       default=apt_pkg.Config.Find("APT::Architecture"))
-    parser.add_option("-p", "--pooldir", dest="pooldir",
-                      help="pooldir", default="/srv/archive.ubuntu.com/ubuntu/pool")
+    parser.add_option("-a", "--archivedir", dest="archivedir",
+                      help="archivedir", default="/srv/archive.ubuntu.com/ubuntu")
+    parser.add_option("--dist", dest="dist",
+                      help="dist", default="natty")
     parser.add_option("-d", "--datadir", dest="datadir",
                       help="datadir", default="data/")
     (options, args) = parser.parse_args()
@@ -48,8 +50,9 @@
 
     # now run it
     desktop_extractor = DesktopDataExtractor(options.aptroot,
-                                             options.pooldir,
-                                             options.datadir)
+                                             options.archivedir,
+                                             options.datadir,
+                                             options.dist)
     desktop_extractor.extract()
 
     logging.info("extraction finished")

=== added directory 'tests/archive'
=== added directory 'tests/archive/dists'
=== added directory 'tests/archive/dists/testing'
=== added file 'tests/archive/dists/testing/Contents_amd64'
--- tests/archive/dists/testing/Contents_amd64	1970-01-01 00:00:00 +0000
+++ tests/archive/dists/testing/Contents_amd64	2011-05-31 05:55:54 +0000
@@ -0,0 +1,5 @@
+
+FILE    LOCATION
+usr/share/icons/hicolor/48x48/apps/audacious.png    foo/audacious
+usr/share/pixmaps/synaptic.png    foo/synaptic
+usr/lib/GNUstep/Applications/Cynthiune.app/Resources/Cynthiune.tiff    foo/cynthiune.app

=== added file 'tests/archive/dists/testing/Contents_i386'
--- tests/archive/dists/testing/Contents_i386	1970-01-01 00:00:00 +0000
+++ tests/archive/dists/testing/Contents_i386	2011-05-31 05:55:54 +0000
@@ -0,0 +1,3 @@
+FILE    LOCATION
+usr/share/icons/hicolor/48x48/apps/cheese.png    foo/cheese-common
+usr/share/pixmaps/python2.5.xpm    foo/python2.5

=== renamed directory 'tests/pool' => 'tests/archive/pool'
=== modified file 'tests/archive/pool/get_pkgs.sh'
--- tests/pool/get_pkgs.sh	2011-04-16 18:10:30 +0000
+++ tests/archive/pool/get_pkgs.sh	2011-05-31 05:55:54 +0000
@@ -57,3 +57,13 @@
 cd universe/c/cynthiune.app
 wget -c https://launchpad.net/ubuntu/+source/cynthiune.app/0.9.5-11ubuntu1/+buildjob/1962087/+files/cynthiune.app_0.9.5-11ubuntu1_amd64.deb
 cd ../../..
+
+# zip contents files
+cd ../dists/testing
+if [ ! -e Contents-i386.gz ] || [ ! -e Contents-amd64.gz ]; then
+  rm -f Contents-*
+  cp Contents_i386 Contents-i386
+  cp Contents_amd64 Contents-amd64
+  gzip Contents-*
+fi
+cd ../../pool

=== added file 'tests/data/Contents-i386.gz'
Binary files tests/data/Contents-i386.gz	1970-01-01 00:00:00 +0000 and tests/data/Contents-i386.gz	2011-05-31 05:55:54 +0000 differ
=== removed file 'tests/pool/main/h/hello/hello_2.1.1-4_i386.deb'
Binary files tests/pool/main/h/hello/hello_2.1.1-4_i386.deb	2007-07-24 10:17:39 +0000 and tests/pool/main/h/hello/hello_2.1.1-4_i386.deb	1970-01-01 00:00:00 +0000 differ
=== added file 'tests/test_archive_cache.py'
--- tests/test_archive_cache.py	1970-01-01 00:00:00 +0000
+++ tests/test_archive_cache.py	2011-05-31 05:55:54 +0000
@@ -0,0 +1,96 @@
+#!/usr/bin/env python
+
+import sys
+import unittest
+import subprocess
+sys.path.insert(0, "../")
+
+from ArchiveCache import ArchiveCache
+
+# This data will need to be refreshed if Contents-i386.gz changes
+CONTENTS_PATH = "data/Contents-i386.gz"
+MD5_CHECKSUM = subprocess.check_output(["md5sum", CONTENTS_PATH]).split()[0]
+INDEX_PATHS = ("usr/share/applications", "/usr/share/icons", "/usr/share/")
+CONTENTS_LINES, INDEX_LINES = 258, (1, 12, 255)
+
+SEARCH_EXACT_PAIRS = {
+  "/etc/dbus-1/system.d/com.ubuntu.SoftwareCenter.conf": "software-center",
+  "/usr/bin/software-center": "software-center",
+  "usr/share/icons/hicolor/32x32/apps/softwarecenter.png": "software-center",
+  "/usr/share/xubuntu-docs/ubuntu-software-center.html": "xubuntu-docs",
+  "/red/herring/path": None}
+
+SEARCH_PAIRS = {
+  (".*SoftwareCenter.conf", ""): [("software-center",
+   "etc/dbus-1/system.d/com.ubuntu.SoftwareCenter.conf")],
+  (".*softwarecenter.svg", "usr/share"): [("software-center",
+   "usr/share/icons/hicolor/scalable/apps/softwarecenter.svg")],
+  (".*\.desktop$", "/usr/share/applications"): [("software-center",
+   "usr/share/applications/ubuntu-software-center.desktop")],
+  (".*software-center.html", "usr/share/"): [("xubuntu-docs",
+   "usr/share/xubuntu-docs/ubuntu-software-center.html")]}
+
+
+class ArchiveCacheTest(unittest.TestCase):
+    """
+    Tests the proper operation of ArchiveCache.
+    """
+    @classmethod
+    def setUp(self):
+        self.cache = ArchiveCache(CONTENTS_PATH, INDEX_PATHS)
+
+    def test_contents_checksum(self):
+        """archive cache loads contents.gz files correctly"""
+        self.assertEqual(self.cache.checksum, MD5_CHECKSUM)
+
+    def test_contents_length(self):
+        """archive cache stores its length in lines"""
+        self.assertTrue(hasattr(self.cache, "__len__"))
+        self.assertEqual(len(self.cache), CONTENTS_LINES)
+
+    def test_index_length(self):
+        """archive cache indexes store their length in lines"""
+        for i, path in enumerate(INDEX_PATHS):
+            self.assertEqual(self.cache.__len__(path), INDEX_LINES[i])
+
+    def test_index_checking(self):
+        """archive cache allows idiomatic checking of index existance"""
+        self.assertTrue(hasattr(self.cache, "__contains__"))
+        self.assertTrue(hasattr(self.cache, "prefixes"))
+        for path in INDEX_PATHS:
+            self.assertTrue(path.strip("/") in self.cache.prefixes)
+
+    def test_search_exact(self):
+        """archive cache can find an exact path from contents"""
+        for arg, result in SEARCH_EXACT_PAIRS.items():
+            self.assertEqual(self.cache.search_exact(arg), result)
+
+    def test_search_regex(self):
+        """archive cache can find a regex result from contents"""
+        for args, results in SEARCH_PAIRS.items():
+            self.assertEqual(set(self.cache.search(*args)), set(results))
+
+    def test_first_only_search(self):
+        """archive cache can stop at first result in search"""
+        results = self.cache.search(".*softwarecenter(.png|.svg)$",
+                                    "/usr/share/icons", first_only=True)
+        self.assertEqual(len(results), 1)
+
+    def test_search_exact_symlinked(self):
+        """archive cache should return partial path matches for symlinks"""
+        # Contents.gz interprets symlinks as files. If we search for an exact
+        # file via a symlinked directory, it should still work.
+        result = self.cache.search_exact("usr/share/symlinked_dir/linked_file")
+        self.assertEqual(result, "software-center")
+
+    def test_search_no_memory(self):
+        """archive cache search should work without memory caching"""
+        old_cache = self.cache
+        self.cache = ArchiveCache(CONTENTS_PATH, INDEX_PATHS, memory=0)
+        self.test_search_exact()
+        self.test_search_regex()
+        self.test_search_exact_symlinked()
+        self.cache = old_cache
+
+if __name__ == "__main__":
+    unittest.main()

=== modified file 'tests/test_cmd_data_extractor.py'
--- tests/test_cmd_data_extractor.py	2008-06-30 10:47:06 +0000
+++ tests/test_cmd_data_extractor.py	2011-05-31 05:55:54 +0000
@@ -26,7 +26,7 @@
 
     def testExtractorSimple(self):
         extractor = CommandDataExtractor("./aptroot",
-                                         "./pool",
+                                         "./archive/pool",
                                          "./data")
         extractor.extract()
         self.assert_(os.path.exists("./data/scan.data"))
@@ -34,13 +34,13 @@
 
     def testExtractorOrphan(self):
         extractor = CommandDataExtractor("./aptroot",
-                                         "./pool",
+                                         "./archive/pool",
                                          "./data")
         extractor.extract()
         self.assert_(len(open("./data/scan.data").readlines()) > 2)
         self.assert_("gnome-session-remove" in open(extractor.cmd_data_f).read())
         # now simulate archive removal of a pkg
-        b = "./pool/main/g/gnome-session/gnome-session_2.14.1-0ubuntu11_i386.deb"
+        b = "./archive/pool/main/g/gnome-session/gnome-session_2.22.1.1-0ubuntu2_i386.deb"
         os.rename(b, b+".xxx")
         # clean orphaned desktop files and make sure that the file really
         # got removed
@@ -51,7 +51,7 @@
 
     def testExtractorCleanPkg(self):
         extractor = CommandDataExtractor("./aptroot",
-                                         "./pool",
+                                         "./archive/pool",
                                          "./data")
         # simulate superseeding
         open("./data/scan.data","w").write("i386|main|gnome-session|gnome-session-remove,gnome-session-save,gnome-session-properties,x-session-manager\n")
@@ -60,7 +60,7 @@
         self.assert_("gnome-wm" in open(extractor.cmd_data_f).read())
 
 if __name__ == "__main__":
-    subprocess.call(["(cd pool; ./get_pkgs.sh)"],shell=True)
+    subprocess.call(["(cd archive/pool; ./get_pkgs.sh)"],shell=True)
     logging.basicConfig(level=logging.DEBUG)
     apt_pkg.init()
     #unittest.main(defaultTest="testCommandDataExtractor.testExtractorOrphan")

=== modified file 'tests/test_cralwer.py'
--- tests/test_cralwer.py	2007-11-28 09:47:20 +0000
+++ tests/test_cralwer.py	2011-05-31 05:55:54 +0000
@@ -21,7 +21,7 @@
         
     def setUp(self):
         # update the cache only once
-        crawler = ArchiveCrawler("./aptroot", "./pool", "./actions", "./data", "i386")
+        crawler = ArchiveCrawler("./aptroot", "./archive/pool", "./actions", "./data", "i386")
         crawler.updateCache()
         # remove pkgs-found file to make tests meaningful
         self.rm(self.pkgs_found_file)
@@ -31,7 +31,7 @@
         def callback_helper(crawler, debfile, pkg, ver, arch, component):
             self._i += 1
             return True
-        crawler = ArchiveCrawler("./aptroot", "./pool", "./actions", "./data", "i386")
+        crawler = ArchiveCrawler("./aptroot", "./archive/pool", "./actions", "./data", "i386")
         # test if we allow only callables
         try:
             crawler.registerCallback(self._i)
@@ -49,7 +49,7 @@
         del self._i
 
     def testCrawlerSkipDone(self):
-        crawler = ArchiveCrawler("./aptroot", "./pool", "./actions", "./data", "i386")
+        crawler = ArchiveCrawler("./aptroot", "./archive/pool", "./actions", "./data", "i386")
         crawler.crawl()
         self.assert_(len(crawler.debfiles_done) > 0)
         # delete the found-pkgs marker file, crawl again
@@ -59,41 +59,41 @@
         self.rm(self.debfiles_done)
         
     def testCrawlerSkipPersistant(self):
-        crawler = ArchiveCrawler("./aptroot", "./pool", "./actions", "./data", "i386")
+        crawler = ArchiveCrawler("./aptroot", "./archive/pool", "./actions", "./data", "i386")
         crawler.crawl()
         # delete the found-pkgs marker file, create new crawler, crawl again,
         # see if we skip the done files
         self.rm(self.pkgs_found_file)
-        crawler = ArchiveCrawler("./aptroot", "./pool", "./actions", "./data", "i386")
+        crawler = ArchiveCrawler("./aptroot", "./archive/pool", "./actions", "./data", "i386")
         crawler.crawl()
         self.assert_(not os.path.exists(self.pkgs_found_file))
 
     def testFindOrphanedFiles(self):
-        crawler = ArchiveCrawler("./aptroot", "./pool", "./actions", "./data", "i386")
+        crawler = ArchiveCrawler("./aptroot", "./archive/pool", "./actions", "./data", "i386")
         crawler.crawl()
-        crawler.debfiles_done.add("./pool/main/h/hello/hello_2.1.1-3_i386.deb")
+        crawler.debfiles_done.add("./archive/pool/main/h/hello/hello_2.1.1-3_i386.deb")
         print crawler.findOrphanedFiles()
         self.assert_(len(crawler.findOrphanedFiles()) == 1)
 
     def testFindOrphanedPackages(self):
-        crawler = ArchiveCrawler("./aptroot", "./pool", "./actions", "./data", "i386")
+        crawler = ArchiveCrawler("./aptroot", "./archive/pool", "./actions", "./data", "i386")
         crawler.crawl()
         print crawler.findOrphanedPackages()
         self.assert_(len(crawler.findOrphanedPackages()) == 0)
-        crawler.debfiles_done.add("./pool/main/h/hello/hello_2.1.1-3_i386.deb")
+        crawler.debfiles_done.add("./archive/pool/main/h/hello/hello_2.1.1-3_i386.deb")
         print crawler.findOrphanedPackages()
         self.assert_(len(crawler.findOrphanedPackages()) == 1)
 
     def testFindObsoletedPackages(self):
         # FIXME: this test is pretty useless
-        crawler = ArchiveCrawler("./aptroot", "./pool", "./actions", "./data", "i386")
+        crawler = ArchiveCrawler("./aptroot", "./archive/pool", "./actions", "./data", "i386")
         crawler.crawl()
         #print "obsolete: ", crawler.findObsoletedPackages()
         self.assert_(len(crawler.findObsoletedPackages()) > 0)
 
     def testArches(self):
         for arch in ('amd64', 'i386'):
-            crawler = ArchiveCrawler("./aptroot", "./pool", "./actions", "./data", arch)
+            crawler = ArchiveCrawler("./aptroot", "./archive/pool", "./actions", "./data", arch)
             crawler.updateCache()
             self.assert_(len(glob.glob(os.path.join(crawler.aptroot,"var/lib/apt/lists")+"/*%s*" % arch)) > 0)
 

=== modified file 'tests/test_deb_package.py'
--- tests/test_deb_package.py	2008-03-04 12:45:12 +0000
+++ tests/test_deb_package.py	2011-05-31 05:55:54 +0000
@@ -11,7 +11,7 @@
 
 class testDebPackage(unittest.TestCase):
     def testDebPackage(self):
-        name = "pool/main/g/git-core/git-core_1.5.4.3-1ubuntu1_i386.deb"
+        name = "archive/pool/main/g/git-core/git-core_1.5.4.3-1ubuntu2_i386.deb"
         pkg = CommandDataExtractor.data_extractor.load(name)
         print pkg
         print pkg.name

=== modified file 'tests/test_desktop_data_extractor.py'
--- tests/test_desktop_data_extractor.py	2011-04-16 18:10:30 +0000
+++ tests/test_desktop_data_extractor.py	2011-05-31 05:55:54 +0000
@@ -13,74 +13,69 @@
 
 from DesktopDataExtractor import DesktopDataExtractor
 
+
 class TestDesktopDataExtractor(unittest.TestCase):
 
     debfiles_done = "./data/desktop.p"
 
-    def rm(self, f):
-        if os.path.exists(f):
-            os.unlink(f)
-    def setUp(self):
-        self.rm(self.debfiles_done)
+    @classmethod
+    def setUpClass(self):
+        if os.path.exists(self.debfiles_done):
+            os.unlink(self.debfiles_done)
         try:
             shutil.rmtree("./data/menu-data")
         except:
             pass
+        self.extractor = DesktopDataExtractor("./aptroot",
+                                              "./archive",
+                                              "./data",
+                                             "testing")
+        self.extractor.extract()
 
     def test_extractor_cheese_common(self):
-        extractor = DesktopDataExtractor("./aptroot-lucid",
-                                         "./pool",
-                                         "./data")
-        extractor.extract()
         # check if icon extraction works
-        self.assertTrue(os.path.exists(os.path.join(extractor.menu_data,"icons", "cheese.png")))
-        
+        self.extractor = DesktopDataExtractor("./aptroot-lucid",
+                                              "./archive",
+                                              "./data",
+                                              "testing")
+        self.extractor.extract()
+        self.assertTrue(os.path.exists(os.path.join(self.extractor.menu_data,"icons", "cheese")))
+
     def test_symlinked_icons(self):
-        extractor = DesktopDataExtractor("./aptroot-maverick",
-                                         "./pool",
-                                         "./data")
-        extractor.extract()
         # check if symlinked (a) icon (b) directory extraction work
-        self.assertTrue(os.path.exists(os.path.join(extractor.menu_data,"icons", "audacious.png")))
-        self.assertTrue(os.path.exists(os.path.join(extractor.menu_data,"icons", "_usr_lib_GNUstep_Applications_Cynthiune.app_Resources_Cynthiune.tiff")))
+        self.assertTrue(os.path.exists(os.path.join(self.extractor.menu_data,"icons", "audacious")))
+        self.assertTrue(os.path.exists(os.path.join(self.extractor.menu_data,"icons", "_usr_lib_GNUstep_Applications_Cynthiune.app_Resources_Cynthiune.tiff")))
 
     def test_extractor_simple(self):
-        extractor = DesktopDataExtractor("./aptroot",
-                                         "./pool",
-                                         "./data")
-        extractor.extract()
         # see if extraction works
-        self.assertTrue(os.path.exists(os.path.join(extractor.menu_data,"synaptic.desktop")))
+        self.assertTrue(os.path.exists(os.path.join(self.extractor.menu_data,"synaptic.desktop")))
         # see if lzma works
-        self.assertTrue(os.path.exists(os.path.join(extractor.menu_data,"ooo-math.desktop")))
+        self.assertTrue(os.path.exists(os.path.join(self.extractor.menu_data,"ooo-math.desktop")))
         # gnome-about is blacklisted
-        self.assertTrue(not os.path.exists(os.path.join(extractor.menu_data,"gnome-about.desktop")))
+        self.assertTrue(not os.path.exists(os.path.join(self.extractor.menu_data,"gnome-about.desktop")))
         # check if icon extraction works
-        self.assertTrue(os.path.exists(os.path.join(extractor.menu_data,"icons", "_usr_share_pixmaps_python2.5.xpm")))
+        self.assertTrue(os.path.exists(os.path.join(self.extractor.menu_data,"icons", "_usr_share_pixmaps_python2.5.xpm")))
 
     def test_extractor_simple_maverick(self):
         extractor = DesktopDataExtractor("./aptroot-maverick",
-                                         "./pool",
-                                         "./data")
+                                         "./archive",
+                                         "./data",
+                                         "testing")
         extractor.extract()
         # check if we have baobab
-        self.assertTrue(os.path.exists(os.path.join(extractor.menu_data,"baobab.desktop")))
+        self.assertTrue(os.path.exists(os.path.join(self.extractor.menu_data,"baobab.desktop")))
 
 
     def test_extractor_orphan(self):
-        extractor = DesktopDataExtractor("./aptroot",
-                                         "./pool",
-                                         "./data")
-        extractor.extract()
-        self.assert_(os.path.exists(os.path.join(extractor.menu_data,"session-properties.desktop")))
+        self.assert_(os.path.exists(os.path.join(self.extractor.menu_data,"session-properties.desktop")))
 
         # now simulate archive removal of a pkg
-        b = "./pool/main/g/gnome-session/gnome-session_2.22.1.1-0ubuntu2_i386.deb"
+        b = "./archive/pool/main/g/gnome-session/gnome-session_2.22.1.1-0ubuntu2_i386.deb"
         os.rename(b, b+".xxx")
         # clean orphaned desktop files and make sure that the file really
         # got removed
-        extractor.extract()
-        self.assert_(not os.path.exists(os.path.join(extractor.menu_data,"session-properties.desktop")))
+        self.extractor.extract()
+        self.assert_(not os.path.exists(os.path.join(self.extractor.menu_data,"session-properties.desktop")))
         os.rename(b+".xxx", b)
 
 if __name__ == "__main__":
@@ -88,6 +83,6 @@
         logging.basicConfig(level=logging.DEBUG)
     else:
         logging.basicConfig(level=logging.INFO)        
-    subprocess.call(["(cd pool; ./get_pkgs.sh)"],shell=True)
+    subprocess.call(["(cd archive/pool; ./get_pkgs.sh)"],shell=True)
     apt_pkg.init()
     unittest.main()

=== added file 'tests/test_icon_finder.py'
--- tests/test_icon_finder.py	1970-01-01 00:00:00 +0000
+++ tests/test_icon_finder.py	2011-05-31 05:55:54 +0000
@@ -0,0 +1,44 @@
+#!/usr/bin/env python
+
+import sys
+import unittest
+sys.path.insert(0, "../")
+
+from ArchiveCache import ArchiveCache
+from IconFinder import IconFinder
+
+# This data will need to be refreshed as Contents-i386.gz changes
+CONTENTS_PATH = "data/Contents-i386.gz"
+ICONS = (
+  "usr/share/icons/hicolor/128x128/apps/softwarecenter.png",
+  "usr/share/icons/hicolor/16x16/apps/softwarecenter.png",
+  "usr/share/icons/hicolor/22x22/apps/softwarecenter.png",
+  "usr/share/icons/hicolor/24x24/apps/ppa.svg",
+  "usr/share/icons/hicolor/24x24/apps/softwarecenter.png",
+  "usr/share/icons/hicolor/24x24/apps/unknown-channel.svg",
+  "usr/share/icons/hicolor/32x32/apps/softwarecenter.png",
+  "usr/share/icons/hicolor/48x48/apps/softwarecenter.png",
+  "usr/share/icons/hicolor/64x64/apps/softwarecenter.png",
+  "usr/share/icons/hicolor/scalable/apps/category-show-all.svg",
+  "usr/share/icons/hicolor/scalable/apps/partner.svg",
+  "usr/share/icons/hicolor/scalable/apps/softwarecenter.svg")
+
+
+class IconFinderTest(unittest.TestCase):
+    """
+    Tests the proper operation of IconFinder.
+    """
+    def setUp(self):
+        self.cache = ArchiveCache(CONTENTS_PATH, IconFinder.prefixes)
+
+    def test_search(self):
+        finder = IconFinder(self.cache)
+        # results for faulty search
+        self.assertEqual(finder.search("foobarbaz"), None)
+        # results for good search
+        package, path = finder.search("softwarecenter")
+        self.assertEqual(package, "software-center")
+        self.assertTrue(path in ICONS)
+
+if __name__ == "__main__":
+    unittest.main()