Merge lp:~jjed/archive-crawler/foreign-icons into lp:~mvo/archive-crawler/mvo

[Jjed]

This branch adds support for fetching icons for a number of different
storage patterns. For example:

    audacity -> audacity-data
    emacs23 -> emacs23-common
    gnome-terminal -> gnome-icon-theme
    bluedevil-audio -> oxygen-icon-theme
    icedtea-netx-javaws -> openjdk-6 [direct dependency]

It introduces a non-trivial performance overhead due to the number extra
debs it has to search, changing the DesktopDataExtractor's workflow to
this:

    *Nearly all DesktopDataExtractor activities before this branch*
    Search for missing icons for `x` in any package x`-(common|data)
    Search for missing icons for `x` in common icon themes
    Search for missing icons for `x` in `x`s direct, nonlib dependencies
    *calcArchSpecific and addCodecInformation*

More patterns, if needed, can be added to a file "icon_search.cfg"

TESTING

I've tested this on main[a-j] and universe[2-d] (to avoid downloading
all of archive.ubuntu.com). A few icon fetches failed, but there were no
regressions, and all of them make sense as either:

  (b) icons from package dependencies outside the pool I tested
  (c) symbolic links (tarfile doesn't work well with them)
  (a) broken packages

This should be a safe merge: the bulk of my added code executes after
what the script already does is finished.

FUTURE CONSIDERATIONS

* Symlinked icons still fail. This is affecting some GNUstep apps.
* A blacklist of frequent, non-lib dependencies that never have icons
* A hardcoded list for edge-case apps/long, fruitless icon searches
* More regular expressions to filter likely icon packages

[Michael Vogt (mvo)]

Thanks a bunch! this looks excellent. I am currently running it on a full mirror and I can't wait to see what the results.

review: Approve

[Michael Vogt (mvo)]

I added a small test (based on cheese/cheese-common) and a logging additon to print out the missing icons after the post-processing finder has done its work. It will be interessting to see what is left and if we could utilize something like apt-file to search for the remaining icons.

Btw, I noticed another branch for the crawler, is that still work-in-progress or should I take a look as well :)

[Jjed (jjed)]

Thanks for the quick merge, hope it works out well.

As for my other branch, I have better-icons mostly finished locally, testing still pending. The branch will default to preferring higher quality icons and scaling them down to a reasonable size (for example, `arista` goes from a 1.8M svg to a 3K png). In using it so far, icons are significantly higher quality with smaller average disk size: pretty nice.

The big question is whether you're willing to add an optional `rsvg` dependency to the script. The preincluded python batteries (PIL) don't work with scalable graphics.

=== modified file 'ArchiveCrawler/__init__.py'
--- ArchiveCrawler/__init__.py	2010-09-08 08:26:06 +0000
+++ ArchiveCrawler/__init__.py	2011-04-11 13:39:24 +0000
@@ -47,6 +47,7 @@
         self.debfiles_done_database = debfiles_done_database
         self._loadDebFilesDone()
         self.callbacks = set()
+        self.pkgs_to_debpath = {}
 
     def registerCallback(self, c):
         if not callable(c):
@@ -103,10 +104,6 @@
         if pkgarch != "all" and pkgarch != self.arch:
             logging.debug("skipping, wrong arch: '%s' (expected '%s')" % (pkgarch, self.arch))
             return False
-        # ... done already
-        if debfile in self.debfiles_done:
-            logging.debug("skipping, already in debfiles_done '%s'" % debfile)
-            return False
         # ... by name
         if not self.cache.has_key(pkgname):
             logging.debug("skipping, not in cache: '%s'" % pkgname)
@@ -132,6 +129,13 @@
         if not debfile.startswith("%s/%s" % (self.pooldir, component)):
             logging.debug("skipping, compoent does not match (expected '%s' got '%s' "% (component, debfile))
             return False
+        # add to mapping of name and deb
+        # it may be needed if it contains an application's icon
+        self.pkgs_to_debpath[pkgname] = debfile
+        # ... then filter if done already
+        if debfile in self.debfiles_done:
+            logging.debug("skipping, already in debfiles_done '%s'" % debfile)
+            return False
 
         # looks like we have a valid ver
         logging.debug("found valid deb: '%s'" % debfile)

=== modified file 'DesktopDataExtractor/__init__.py'
--- DesktopDataExtractor/__init__.py	2011-03-17 15:53:26 +0000
+++ DesktopDataExtractor/__init__.py	2011-04-11 13:39:24 +0000
@@ -5,6 +5,7 @@
 import apt
 import apt_pkg
 import apt_inst
+import collections
 import os.path
 import re
 import tempfile
@@ -55,6 +56,12 @@
         # available in certain arches
         self.pkgs_per_arch = {}
         self.pkgs_per_arch["all"] = set()
+        # a mapping of package names to wanted application icons their
+        # packages don't contain
+        self.pkgs_to_missing_icons = {}
+        # regular expressions for finding packages that might contain
+        # wanted icons
+        self.iconsearch_regex = []
         # now read the config
         self._readConfig()
 
@@ -76,6 +83,7 @@
         blacklist_desktop = os.path.join(self.datadir,"blacklist_desktop.cfg")
         renamecfg = os.path.join(self.datadir,"rename.cfg")
         annotatecfg = os.path.join(self.datadir,"annotate.cfg")
+        iconsearchcfg = os.path.join(self.datadir,"icon_search.cfg")
         if os.path.exists(blacklist):
             logging.info("using blacklist: '%s'" % blacklist)
             for line in open(blacklist).readlines():
@@ -110,6 +118,13 @@
                     annotations = annotations_str.split(",")
                     logging.debug("annotations: '%s': %s" % (desktopfile,annotations))
                     self.desktop_annotate[desktopfile] = annotations
+        if os.path.exists(iconsearchcfg):
+            logging.info("using icon search: '%s'" % iconsearchcfg)
+            for line in open(iconsearchcfg):
+                line = line.strip()
+                if line != "" and not line.startswith("#"):
+                    logging.debug("icon search regex: '%s'" % line)
+                    self.iconsearch_regex.append(line)
 
 
 
@@ -134,11 +149,59 @@
             self.crawler.updateCache()
             self.crawler.registerCallback(self.inspectDeb)
             self.crawler.crawl()
+            self._findMissingIcons()
         self._calcArchSpecific()
         self._addCodecInformation()
         pickle.dump(self.deb_to_files,open(self.deb_to_files_f,"w"))
         logging.info("extract() finished")
-        
+
+    def _findMissingIcons(self):
+        """ search for missing desktop icons in using the crawl cache """
+        for (pkgname, icons) in self.pkgs_to_missing_icons.items():
+            logging.debug("Searching for missing '%s' icons" % pkgname)
+            # get an ordered set from most likely to least likely package
+            to_search = collections.OrderedDict()
+            # add (in order of importance) all cached packages matching regex
+            for regex in self.iconsearch_regex:
+                if '{0}' in regex:
+                    first_term = re.split('-|_', pkgname, 1)[0]
+                    regex = regex.format(first_term)
+                matches = filter(re.compile(regex).match, self.pkgs_seen)
+                for match in matches:
+                    to_search[match] = None
+            # queue all non-library dependencies of the package
+            deps = self.crawler.cache[pkgname].candidate.dependencies
+            for dep in deps:
+                for dep_candidate in dep.or_dependencies:
+                    depname = dep_candidate.name
+                    if not depname.startswith('lib'):
+                        to_search[depname] = None
+
+            # finally, search the set of likely packages
+            for name in to_search:
+                # get cached tarfile
+                logging.debug("* Looking in %s" % name)
+                if name not in self.crawler.pkgs_to_debpath:
+                    logging.debug("  Deb for %s not found!" % name)
+                    continue
+                try:
+                    debPath = self.crawler.pkgs_to_debpath[name]
+                    datafile = self._extractDebData(debPath)
+                    tar = tarfile.open(datafile)
+                except:
+                    logging.debug("  Deb for %s could not be opened!" % name)
+                    continue
+                found = set()
+                for icon in icons:
+                    (res, n) = self.search_icon(tar, icon, self.menu_data)
+                    if res == True:
+                        logging.debug("  Icon %s found!" % icon)
+                        found.add(icon)
+                # stop searching for any icons we find
+                icons = icons.difference(found)
+                if len(icons) == 0:
+                    break
+ 
     def _calcArchSpecific(self):
         # now add the architecture information
         arch_specific = set()
@@ -258,7 +321,11 @@
         #        but "zapping/zapping.png"
         if "/" in iconName:
             newIconName = iconName.replace("/", "_")
-            res = self.extract_icon(tarfile, iconName, os.path.join(outputdir,"icons",newIconName))
+            outpath = os.path.join(outputdir,"icons",newIconName)
+            # prevent wasted disk read
+            if os.path.exists(outpath):
+                return (True, newIconName)
+            res = self.extract_icon(tarfile, iconName, outpath)
             return (res, newIconName)
 
         # this is the "get-it-from-a-icontheme" case, look into icon-theme hicolor and usr/share/pixmaps
@@ -277,6 +344,9 @@
         # extensions (ordered by importance)
         pixmaps_ext = ["", ".png",".xpm",".svg"]
 
+        # prevent wasted disk read
+        if os.path.exists(os.path.join(outputdir,"icons",iconName)):
+            return (True, None)
         for d in search_dirs:
             for name in tarfile.getnames():
                 if d in name:
@@ -375,6 +445,10 @@
                     iconName = line[line.index("=")+1:]
                     logging.debug("Package '%s' needs icon '%s'" % (pkgname, iconName))
             (res, newIconName) = self.search_icon(dataFile, iconName, outputdir)
+            if res == False:
+                if not pkgname in self.pkgs_to_missing_icons:
+                    self.pkgs_to_missing_icons[pkgname] = set()
+                self.pkgs_to_missing_icons[pkgname].add(iconName)
 
             # now check for supicious pkgnames (FIXME: make this not hardcoded)
             if "-common" in pkgname or "-data" in pkgname:
@@ -438,6 +512,12 @@
         """ extract the desktop file and the icons from a deb """
         outputdir=self.menu_data
         logging.debug("processing: %s" % debPath)
+        datafile = self._extractDebData(debPath)
+        if datafile:
+            self.getFiles(datafile, pkgname, section, debPath)
+            os.remove(datafile)
+
+    def _extractDebData(self, debPath):
         datafile = self._getMemberFromAr(debPath, "data.tar.gz")
         if datafile == None:
             datafile = self._getMemberFromAr(debPath, "data.tar.bz2")
@@ -450,8 +530,7 @@
                     # extract it here, python tarfile does not support lzma
                     subprocess.call(["lzma","-d",datafile])
                     datafile = os.path.splitext(datafile)[0]
-        self.getFiles(datafile, pkgname, section, debPath)
-        os.remove(datafile)
+        return datafile
 
     def inspectDeb(self, crawler, filename, pkgname, ver, pkgarch, component):
         """ check if the deb is interessting for us (not blacklisted) """

=== added file 'data/icon_search.cfg'
--- data/icon_search.cfg	1970-01-01 00:00:00 +0000
+++ data/icon_search.cfg	2011-04-11 13:39:24 +0000
@@ -0,0 +1,11 @@
+# these are regular expressions for finding packages with icons for a
+# *.desktop file, when icons are not in the application package. Any
+# string "{0}" will be formatted to the first hyphen-delimited term of
+# an application package (eg "foo-bar-baz" -> "foo")
+
+# for cases like wesnoth/wesnoth-data 
+^{0}.+(data|common)$
+
+# for when environment applications use environmental icons
+gnome-icon-theme
+oxygen-icon-theme

=== modified file 'getMenuData.py'
--- getMenuData.py	2008-06-30 12:00:04 +0000
+++ getMenuData.py	2011-04-11 13:39:24 +0000
@@ -8,9 +8,7 @@
 # FIXME: strip "TryExec" from the extracted menu files (and noDisplay)
 #        
 # TODO:
-# - emacs21 ships it's icon in emacs-data, deal with this
 # - some stuff needs to be blacklisted (e.g. gnome-about)
-# - lots of packages have there desktop file in "-data", "-comon" (e.g. anjuta)
 # - lots of packages have multiple desktop files for the same application
 #   abiword, abiword-gnome, abiword-gtk
 #