Merge lp:~evfool/update-manager/changelogs into lp:update-manager

[Robert Roth]

Implemented changelog and NEWS.Debian caching using httplib2. The httplib2 dependency is already pulled in anyway (ubuntu-desktop depends on software-center depends on python-lazr.restfulclient depends on python-httplib2).
So the branch does the following: when a changelog/news is downloaded, it caches it using httplib2's file-based cache in $XDG_CACHE_DIR/update-manager-core/changelogs (if $XDG_CACHE_DIR is set, else it will be $HOME/.cache, based on the XDG directories specification [1]), so it will be loaded much faster if it should be downloaded for other packages too. The changelog cache is cleared whenever the Check button is pressed (to avoid showing old changelogs for new packages) or Update-Manager is closed (to free up the disk space used for the people in need of disk space). Fixes bug #582432.
This speeds up update-manager when reading through the changelogs (usually there are many packages updatable from the same source package, so having the same changelog), and reduces internet usage.

[1] http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

[Michael Vogt (mvo)]

Many thanks for this branch! I merged it with a tiny tweak that explains that httplib2 sets headers.status to 200 even if its a 304 not modified response in the http header (which confused me initially as I thought that we would have to handle both headers.status != 200 and != 304).

Thanks!
 Michael

review: Approve

[Michael Vogt (mvo)]

I need to re-check this with the release team. While ubuntu-desktop has python-httplib2 as part of its dependencies already the server and other derivatives do not have it. With this change httplib2 would be pulled into ubuntu-standard. Which should be fine, but at this point (after beta-1) I need to confirm first that its ok.

[Michael Vogt (mvo)]

I pushed the branch to:
lp:~mvo/update-manager/evfool-changelogs-httplib2
(with my small comment tweak).

[Robert Roth (evfool)]

Michael, what's the status with this? What did the release team say? I didn't see this merged.

=== modified file 'DistUpgrade/DistUpgradeViewGtk3.py'
--- DistUpgrade/DistUpgradeViewGtk3.py	2011-08-10 07:37:28 +0000
+++ DistUpgrade/DistUpgradeViewGtk3.py	2011-09-07 03:07:50 +0000
@@ -644,8 +644,7 @@
                                                 [parent_text % len(details_list)])
                 for pkg in details_list:
                     self.details_list.append(node, ["<b>%s</b> - %s" % (
-                          pkg.name, GLib.markup_escape_text(pkg.summary,
-                                                            len(pkg.summary)))])
+                          pkg.name, GLib.markup_escape_text(pkg.summary))])
         # prepare dialog
         self.dialog_changes.realize()
         self.dialog_changes.set_transient_for(self.window_main)

=== modified file 'UpdateManager/Core/MyCache.py'
--- UpdateManager/Core/MyCache.py	2011-07-25 07:54:47 +0000
+++ UpdateManager/Core/MyCache.py	2011-09-07 03:07:50 +0000
@@ -28,6 +28,8 @@
 import string
 import urllib2
 import httplib
+import httplib2
+import shutil
 import socket
 import re
 import DistUpgrade.DistUpgradeCache
@@ -50,6 +52,11 @@
         # too because libapt will only do it when it tries to lock
         # the packaging system)
         assert(not self._dpkgJournalDirty())
+        # init the httplib2 flatfile cache for changelog caching
+        cache_dir = os.getenv(
+            "XDG_CACHE_HOME", os.path.expanduser("~/.cache"))
+        self.cache_path = os.path.join(cache_dir, 'update-manager-core','changelogs')
+        self.httplib = httplib2.Http(cache=self.cache_path)
         # init the regular cache
         self._initDepCache()
         self.all_changes = {}
@@ -59,7 +66,6 @@
             self.saveDistUpgrade()
         assert (self._depcache.broken_count == 0 and 
                 self._depcache.del_count == 0)
-
     def _dpkgJournalDirty(self):
         """
         test if the dpkg journal is dirty
@@ -72,6 +78,12 @@
                 return True
         return False
 
+    # empty the changelog cache on clear/update
+    def _clearChangelogAndNewsCache(self):
+        if os.path.exists(self.cache_path):
+            shutil.rmtree(self.cache_path)
+        os.mkdir(self.cache_path)
+        
     def _initDepCache(self):
         #apt_pkg.Config.Set("Debug::pkgPolicy","1")
         #self.depcache = apt_pkg.GetDepCache(self.cache)
@@ -80,6 +92,7 @@
         if os.path.exists(SYNAPTIC_PINFILE):
             self._depcache.read_pinfile(SYNAPTIC_PINFILE)
         self._depcache.init()
+        self._clearChangelogAndNewsCache()
     def clear(self):
         self._initDepCache()
     @property
@@ -161,6 +174,7 @@
     def _get_changelog_or_news(self, name, fname, strict_versioning=False, changelogs_uri=None):
         " helper that fetches the file in question "
         # don't touch the gui in this function, it needs to be thread-safe
+        # total = time.time()
         pkg = self[name]
 
         # get the src package name
@@ -194,16 +208,14 @@
             uri = CHANGELOGS_URI % (src_section,prefix,srcpkg,srcpkg, srcver, fname)
 
         # print "Trying: %s " % uri
-        changelog = urllib2.urlopen(uri)
-        #print changelog.read()
+        headers, changelog = self.httplib.request(uri)
+        if headers.status != 200 :
+            changelog = ""
         # do only get the lines that are new
         alllines = ""
         regexp = "^%s \((.*)\)(.*)$" % (re.escape(srcpkg))
         
-        while True:
-            line = changelog.readline()
-            if line == "":
-                break
+        for line in changelog.split('\n'):
             match = re.match(regexp,line)
             if match:
                 # strip epoch from installed version
@@ -229,7 +241,11 @@
                     if (installed and 
                         apt_pkg.VersionCompare(changelogver,installed)==0):
                         break
-            alllines = alllines + line
+            alllines = alllines +  line + "\n"
+        if alllines =="\n":
+            alllines = ""
+        #total = time.time() - total
+        #print "changelog from %s for %s retrieved in %.2f"%("/".join(uri.split('/')[5:]), name, total)
         return alllines
 
     def _guess_third_party_changelogs_uri_by_source(self, name):

=== modified file 'debian/control'
--- debian/control	2011-08-12 15:45:42 +0000
+++ debian/control	2011-09-07 03:07:50 +0000
@@ -20,7 +20,8 @@
 Section: admin
 Depends: ${python:Depends}, 
          ${misc:Depends}, 
-         python-apt (>= 0.7.13.4ubuntu3),  
+         python-apt (>= 0.7.13.4ubuntu3),
+         python-httplib2, 
          lsb-release,  
          python-gnupginterface
 Recommends: libpam-modules (>= 1.0.1-9ubuntu3)