Merge lp:~evfool/update-manager/changelogs into lp:update-manager

Proposed by Robert Roth
Status: Merged
Merged at revision: 2228
Proposed branch: lp:~evfool/update-manager/changelogs
Merge into: lp:update-manager
Diff against target: 123 lines (+27/-11)
3 files modified
DistUpgrade/DistUpgradeViewGtk3.py (+1/-2)
UpdateManager/Core/MyCache.py (+24/-8)
debian/control (+2/-1)
To merge this branch: bzr merge lp:~evfool/update-manager/changelogs
Reviewer Review Type Date Requested Status
Michael Vogt Approve
Review via email: mp+74328@code.launchpad.net

Description of the change

Implemented changelog and NEWS.Debian caching using httplib2. The httplib2 dependency is already pulled in anyway (ubuntu-desktop depends on software-center depends on python-lazr.restfulclient depends on python-httplib2).
So the branch does the following: when a changelog/news is downloaded, it caches it using httplib2's file-based cache in $XDG_CACHE_DIR/update-manager-core/changelogs (if $XDG_CACHE_DIR is set, else it will be $HOME/.cache, based on the XDG directories specification [1]), so it will be loaded much faster if it should be downloaded for other packages too. The changelog cache is cleared whenever the Check button is pressed (to avoid showing old changelogs for new packages) or Update-Manager is closed (to free up the disk space used for the people in need of disk space). Fixes bug #582432.
This speeds up update-manager when reading through the changelogs (usually there are many packages updatable from the same source package, so having the same changelog), and reduces internet usage.

[1] http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

To post a comment you must log in.
Revision history for this message
Michael Vogt (mvo) wrote :

Many thanks for this branch! I merged it with a tiny tweak that explains that httplib2 sets headers.status to 200 even if its a 304 not modified response in the http header (which confused me initially as I thought that we would have to handle both headers.status != 200 and != 304).

Thanks!
 Michael

review: Approve
Revision history for this message
Michael Vogt (mvo) wrote :

I need to re-check this with the release team. While ubuntu-desktop has python-httplib2 as part of its dependencies already the server and other derivatives do not have it. With this change httplib2 would be pulled into ubuntu-standard. Which should be fine, but at this point (after beta-1) I need to confirm first that its ok.

Revision history for this message
Michael Vogt (mvo) wrote :

I pushed the branch to:
lp:~mvo/update-manager/evfool-changelogs-httplib2
(with my small comment tweak).

Revision history for this message
Robert Roth (evfool) wrote :

Michael, what's the status with this? What did the release team say? I didn't see this merged.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'DistUpgrade/DistUpgradeViewGtk3.py'
2--- DistUpgrade/DistUpgradeViewGtk3.py 2011-08-10 07:37:28 +0000
3+++ DistUpgrade/DistUpgradeViewGtk3.py 2011-09-07 03:07:50 +0000
4@@ -644,8 +644,7 @@
5 [parent_text % len(details_list)])
6 for pkg in details_list:
7 self.details_list.append(node, ["<b>%s</b> - %s" % (
8- pkg.name, GLib.markup_escape_text(pkg.summary,
9- len(pkg.summary)))])
10+ pkg.name, GLib.markup_escape_text(pkg.summary))])
11 # prepare dialog
12 self.dialog_changes.realize()
13 self.dialog_changes.set_transient_for(self.window_main)
14
15=== modified file 'UpdateManager/Core/MyCache.py'
16--- UpdateManager/Core/MyCache.py 2011-07-25 07:54:47 +0000
17+++ UpdateManager/Core/MyCache.py 2011-09-07 03:07:50 +0000
18@@ -28,6 +28,8 @@
19 import string
20 import urllib2
21 import httplib
22+import httplib2
23+import shutil
24 import socket
25 import re
26 import DistUpgrade.DistUpgradeCache
27@@ -50,6 +52,11 @@
28 # too because libapt will only do it when it tries to lock
29 # the packaging system)
30 assert(not self._dpkgJournalDirty())
31+ # init the httplib2 flatfile cache for changelog caching
32+ cache_dir = os.getenv(
33+ "XDG_CACHE_HOME", os.path.expanduser("~/.cache"))
34+ self.cache_path = os.path.join(cache_dir, 'update-manager-core','changelogs')
35+ self.httplib = httplib2.Http(cache=self.cache_path)
36 # init the regular cache
37 self._initDepCache()
38 self.all_changes = {}
39@@ -59,7 +66,6 @@
40 self.saveDistUpgrade()
41 assert (self._depcache.broken_count == 0 and
42 self._depcache.del_count == 0)
43-
44 def _dpkgJournalDirty(self):
45 """
46 test if the dpkg journal is dirty
47@@ -72,6 +78,12 @@
48 return True
49 return False
50
51+ # empty the changelog cache on clear/update
52+ def _clearChangelogAndNewsCache(self):
53+ if os.path.exists(self.cache_path):
54+ shutil.rmtree(self.cache_path)
55+ os.mkdir(self.cache_path)
56+
57 def _initDepCache(self):
58 #apt_pkg.Config.Set("Debug::pkgPolicy","1")
59 #self.depcache = apt_pkg.GetDepCache(self.cache)
60@@ -80,6 +92,7 @@
61 if os.path.exists(SYNAPTIC_PINFILE):
62 self._depcache.read_pinfile(SYNAPTIC_PINFILE)
63 self._depcache.init()
64+ self._clearChangelogAndNewsCache()
65 def clear(self):
66 self._initDepCache()
67 @property
68@@ -161,6 +174,7 @@
69 def _get_changelog_or_news(self, name, fname, strict_versioning=False, changelogs_uri=None):
70 " helper that fetches the file in question "
71 # don't touch the gui in this function, it needs to be thread-safe
72+ # total = time.time()
73 pkg = self[name]
74
75 # get the src package name
76@@ -194,16 +208,14 @@
77 uri = CHANGELOGS_URI % (src_section,prefix,srcpkg,srcpkg, srcver, fname)
78
79 # print "Trying: %s " % uri
80- changelog = urllib2.urlopen(uri)
81- #print changelog.read()
82+ headers, changelog = self.httplib.request(uri)
83+ if headers.status != 200 :
84+ changelog = ""
85 # do only get the lines that are new
86 alllines = ""
87 regexp = "^%s \((.*)\)(.*)$" % (re.escape(srcpkg))
88
89- while True:
90- line = changelog.readline()
91- if line == "":
92- break
93+ for line in changelog.split('\n'):
94 match = re.match(regexp,line)
95 if match:
96 # strip epoch from installed version
97@@ -229,7 +241,11 @@
98 if (installed and
99 apt_pkg.VersionCompare(changelogver,installed)==0):
100 break
101- alllines = alllines + line
102+ alllines = alllines + line + "\n"
103+ if alllines =="\n":
104+ alllines = ""
105+ #total = time.time() - total
106+ #print "changelog from %s for %s retrieved in %.2f"%("/".join(uri.split('/')[5:]), name, total)
107 return alllines
108
109 def _guess_third_party_changelogs_uri_by_source(self, name):
110
111=== modified file 'debian/control'
112--- debian/control 2011-08-12 15:45:42 +0000
113+++ debian/control 2011-09-07 03:07:50 +0000
114@@ -20,7 +20,8 @@
115 Section: admin
116 Depends: ${python:Depends},
117 ${misc:Depends},
118- python-apt (>= 0.7.13.4ubuntu3),
119+ python-apt (>= 0.7.13.4ubuntu3),
120+ python-httplib2,
121 lsb-release,
122 python-gnupginterface
123 Recommends: libpam-modules (>= 1.0.1-9ubuntu3)

Subscribers

People subscribed via source and target branches

to status/vote changes: