Merge lp:~mvo/click/acquire into lp:click/devel

Proposed by Michael Vogt
Status: Needs review
Proposed branch: lp:~mvo/click/acquire
Merge into: lp:click/devel
Diff against target: 740 lines (+631/-3)
8 files modified
acquire/http-udm (+28/-0)
acquire/pycurl (+30/-0)
click/acquire.py (+369/-0)
click/commands/install.py (+17/-1)
click/paths.py.in (+3/-0)
click/tests/test_acquire.py (+174/-0)
debian/control (+2/-2)
setup.py.in (+8/-0)
To merge this branch: bzr merge lp:~mvo/click/acquire
Reviewer Review Type Date Requested Status
PS Jenkins bot (community) continuous-integration Approve
Colin Watson Needs Fixing
Review via email: mp+235753@code.launchpad.net

Commit message

Provide support to download click packages.

Description of the change

This branch implements "acquire" support in click to support "click install http://example.com/foo_1.0.click".

The model is similar with what apt is using and click can reuse the apt methods to download packages. A pycurl based implementation is also provided. Feedback welcome especially suggestsions if the low-level read_messages() code can be done in a more elegant way.

To post a comment you must log in.
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:541
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~mvo/click/acquire/+merge/235753/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/click-devel-ci/71/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/73/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/71/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/71/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/71/rebuild

review: Needs Fixing (continuous-integration)
lp:~mvo/click/acquire updated
542. By Michael Vogt

click/acquire.py: make dbus optional

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:542
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~mvo/click/acquire/+merge/235753/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/click-devel-ci/73/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/75/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/73/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/73/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/73/rebuild

review: Needs Fixing (continuous-integration)
lp:~mvo/click/acquire updated
543. By Michael Vogt

add missing python3-pycurl depdendency

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:543
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~mvo/click/acquire/+merge/235753/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/click-devel-ci/74/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/76/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/74/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/74/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/74/rebuild

review: Needs Fixing (continuous-integration)
lp:~mvo/click/acquire updated
544. By Michael Vogt

add python3-pycurl to b-d (needed during the tests

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:544
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~mvo/click/acquire/+merge/235753/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/click-devel-ci/75/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/77
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/75
        deb: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/75/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/75

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/75/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
Revision history for this message
Colin Watson (cjwatson) wrote :

Overall this looks like nice work, but I think you could productively simplify some things.

review: Needs Fixing
lp:~mvo/click/acquire updated
545. By Michael Vogt

first round of addressing review points from Colin

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
lp:~mvo/click/acquire updated
546. By Michael Vogt

click/acquire.py: add note about ssl options for curl

547. By Michael Vogt

rename read_messages() -> _read_messages()

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
lp:~mvo/click/acquire updated
548. By Michael Vogt

add missing python3-dbus build-dependency

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)
Revision history for this message
Michael Vogt (mvo) wrote :

> Overall this looks like nice work, but I think you could productively simplify
> some things.

Thanks a lot for your detailed review. I addressed all the points you raised now.

The only one left is the blocking/non-blocking issue. The reason I made this non-blocking is that we can easily report progress about the download from the main click app this way. I created a lp:~mvo/click/acquire-asyncio that still uses a non-blocking io model but is much simpler to follow due to the use of the new python3 asyncio module.

Alternatively we can ditch some of the compatibility with the apt methods and move the progress into the method. The downside is that if/when we support parallel downloads its more difficult to present a overall progress when its done in the individual downloaders.

lp:~mvo/click/acquire updated
549. By Michael Vogt

improve error erporting

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Needs Fixing (continuous-integration)
lp:~mvo/click/acquire updated
550. By Michael Vogt

merged lp:click/devel

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :
review: Approve (continuous-integration)

Unmerged revisions

550. By Michael Vogt

merged lp:click/devel

549. By Michael Vogt

improve error erporting

548. By Michael Vogt

add missing python3-dbus build-dependency

547. By Michael Vogt

rename read_messages() -> _read_messages()

546. By Michael Vogt

click/acquire.py: add note about ssl options for curl

545. By Michael Vogt

first round of addressing review points from Colin

544. By Michael Vogt

add python3-pycurl to b-d (needed during the tests

543. By Michael Vogt

add missing python3-pycurl depdendency

542. By Michael Vogt

click/acquire.py: make dbus optional

541. By Michael Vogt

remove debug message

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== added directory 'acquire'
=== added symlink 'acquire/file'
=== target is u'pycurl'
=== added symlink 'acquire/ftp'
=== target is u'pycurl'
=== added symlink 'acquire/http'
=== target is u'pycurl'
=== added file 'acquire/http-udm'
--- acquire/http-udm 1970-01-01 00:00:00 +0000
+++ acquire/http-udm 2014-10-14 13:55:44 +0000
@@ -0,0 +1,28 @@
1#!/usr/bin/python3
2
3# Copyright (C) 2014 Canonical Ltd.
4# Author: Michael Vogt <michael.vogt@ubuntu.com>
5
6# This program is free software: you can redistribute it and/or modify
7# it under the terms of the GNU General Public License as published by
8# the Free Software Foundation; version 3 of the License.
9#
10# This program is distributed in the hope that it will be useful,
11# but WITHOUT ANY WARRANTY; without even the implied warranty of
12# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
13# GNU General Public License for more details.
14#
15# You should have received a copy of the GNU General Public License
16# along with this program. If not, see <http://www.gnu.org/licenses/>.
17
18"""Acquire method for ubuntu-download-manager."""
19
20from click.acquire import ClickAcquireMethodUbuntuDownloadManager
21
22from dbus.mainloop.glib import DBusGMainLoop
23DBusGMainLoop(set_as_default=True)
24
25
26if __name__ == "__main__":
27 m = ClickAcquireMethodUbuntuDownloadManager()
28 m.run()
029
=== added symlink 'acquire/https'
=== target is u'pycurl'
=== added file 'acquire/pycurl'
--- acquire/pycurl 1970-01-01 00:00:00 +0000
+++ acquire/pycurl 2014-10-14 13:55:44 +0000
@@ -0,0 +1,30 @@
1#!/usr/bin/python3
2
3# Copyright (C) 2014 Canonical Ltd.
4# Author: Michael Vogt <michael.vogt@ubuntu.com>
5
6# This program is free software: you can redistribute it and/or modify
7# it under the terms of the GNU General Public License as published by
8# the Free Software Foundation; version 3 of the License.
9#
10# This program is distributed in the hope that it will be useful,
11# but WITHOUT ANY WARRANTY; without even the implied warranty of
12# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
13# GNU General Public License for more details.
14#
15# You should have received a copy of the GNU General Public License
16# along with this program. If not, see <http://www.gnu.org/licenses/>.
17
18"""Acquire pycurl method click."""
19
20import signal
21import sys
22
23from click.acquire import ClickAcquireMethodPycurl
24
25
26if __name__ == "__main__":
27 # apt will send a SIGINT to all methods that do not send "Need-Cleanup: 1"
28 signal.signal(signal.SIGINT, lambda *args: sys.exit())
29 m = ClickAcquireMethodPycurl()
30 m.run()
031
=== added file 'click/acquire.py'
--- click/acquire.py 1970-01-01 00:00:00 +0000
+++ click/acquire.py 2014-10-14 13:55:44 +0000
@@ -0,0 +1,369 @@
1# Copyright (C) 2014 Canonical Ltd.
2# Author: Michael Vogt <michael.vogt@ubuntu.com>
3
4# This program is free software: you can redistribute it and/or modify
5# it under the terms of the GNU General Public License as published by
6# the Free Software Foundation; version 3 of the License.
7#
8# This program is distributed in the hope that it will be useful,
9# but WITHOUT ANY WARRANTY; without even the implied warranty of
10# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11# GNU General Public License for more details.
12#
13# You should have received a copy of the GNU General Public License
14# along with this program. If not, see <http://www.gnu.org/licenses/>.
15
16"""Acquire of Click packages."""
17
18from __future__ import print_function
19
20__metaclass__ = type
21__all__ = [
22 'ClickAcquire',
23 'ClickAcquireError',
24 'ClickAcquireMethod',
25 'ClickAcquireMethodPycurl',
26 'ClickAcquireMethodUbuntuDownloadManager',
27 'ClickAcquireStatus',
28 'ClickAcquireStatusText',
29 ]
30
31import os
32import select
33import subprocess
34import sys
35from textwrap import dedent
36
37import apt_pkg
38from debian.deb822 import Deb822
39import dbus
40from gi.repository import GLib
41import pycurl
42from six.moves.urllib.parse import urlparse
43
44from click.paths import acquire_methods_dir
45
46
47class ClickAcquireError(Exception):
48 """Error during acquire"""
49 pass
50
51
52def extract_one_message(lines):
53 msg = []
54 while True:
55 try:
56 line = lines.pop(0)
57 except IndexError:
58 break
59 if os.environ.get("CLICK_DEBUG_ACQUIRE", ""):
60 sys.stderr.write("[%s] raw_line: '%s'\n" % (
61 os.getpid(), line.replace("\n", "\\n")))
62 msg.append(line)
63 if len(msg) > 0 and line == "":
64 # we are done, collect all remaining "\n" and stop
65 while len(lines) > 0 and lines[0] == "":
66 lines.pop(0)
67 break
68 if len(msg) < 2:
69 return -1 , ""
70 if os.environ.get("CLICK_DEBUG_ACQUIRE", ""):
71 sys.stderr.write("[%s] msg: '%s'\n" % (os.getpid(), str(msg)))
72 number = int(msg[0].split()[0])
73 return number, "\n".join(msg)
74
75
76def _read_messages(input_file, timeout=None):
77 infd = input_file.fileno()
78 msgs = []
79 rl, wl, xl = select.select([infd], [], [], timeout)
80 if rl:
81 # FIXME: 16k message limit is arbitrary
82 buf = os.read(infd, 16*1024).decode("utf-8")
83 if os.environ.get("CLICK_DEBUG_ACQUIRE", ""):
84 sys.stderr.write("[%s] read buf: '%s'\n" % (
85 os.getpid(), buf))
86 lines = buf.split("\n")
87 while True:
88 number, msg = extract_one_message(lines)
89 if number < 0:
90 break
91 msgs.append( (number, msg) )
92 return msgs
93
94
95class ClickAcquireStatus:
96 """Base class for the status reporting """
97
98 def __init__(self):
99 self.fetched_bytes = 0
100 self.total_bytes = 1.0
101 self.uri = ""
102
103 def pulse(self):
104 pass
105
106 def done(self):
107 pass
108
109
110class ClickAcquireStatusText(ClickAcquireStatus):
111 """Text based progress reporting for the acquire progress"""
112
113 def pulse(self):
114 sys.stdout.write("\r")
115 sys.stdout.write("[%3.2f %%] Fetching %s" % (
116 (self.fetched_bytes/self.total_bytes)*100.0,
117 os.path.basename(self.uri)))
118 sys.stdout.flush()
119
120 def done(self):
121 self.pulse()
122 sys.stdout.write("\n")
123 sys.stdout.write("Done fetching %s\n" % self.uri)
124
125
126# similar to Acquire/AcquireWorker
127class ClickAcquire:
128 """Acquire from remote locations"""
129
130 # default status reporting timeout
131 TIMEOUT = 0.05
132
133 # acquire method status codecs
134 M_CAPABILITIES = 100
135 M_STATUS = 102
136 M_REDIRECT = 103
137 URI_START = 200
138 URI_SUCCESS = 201
139 URI_FAILURE = 400
140
141 def __init__(self, status=ClickAcquireStatus()):
142 self._fetch_queue = []
143 self._status = status
144
145 def _uri_acquire(self, pipe, uri, destfile):
146 cmd = dedent("""\
147 600 URI Acquire
148 URI: {uri}
149 Filename: {destfile}
150
151 """).format(uri=uri, destfile=destfile)
152 pipe.write(cmd)
153 pipe.flush()
154
155 def _redirect(self, pipe, new_uri, destfile):
156 self._uri_acquire(pipe, new_uri, destfile)
157
158 def _run_acquire_method(self, uri, destfile):
159 parsed_uri = urlparse(uri)
160 cmd = os.path.join(acquire_methods_dir, parsed_uri.scheme)
161 p = subprocess.Popen([cmd],
162 stdout=subprocess.PIPE, stdin=subprocess.PIPE,
163 universal_newlines=True)
164 while True:
165 for number, raw_message in _read_messages(p.stdout, self.TIMEOUT):
166 message = Deb822(raw_message)
167 if number == self.M_CAPABILITIES:
168 self._uri_acquire(p.stdin, uri, destfile)
169 elif number == self.M_STATUS:
170 pass
171 elif number == self.M_REDIRECT:
172 self._redirect(p.stdin, message.get("New-URI"), destfile)
173 elif number == self.URI_START:
174 self._status.uri = message.get("URI", 0)
175 self._status.total_bytes = int(message.get("Size", 0))
176 elif number == self.URI_SUCCESS:
177 self._status.fetched_bytes = int(message.get("Size"))
178 self._status.done()
179 p.stdout.close()
180 p.stdin.close()
181 return True
182 elif number == self.URI_FAILURE:
183 p.stdout.close()
184 p.stdin.close()
185 raise ClickAcquireError("Uri failure for %s: %s" % (
186 message.get("uri"), message.get("Message")))
187 # update progress
188 if os.path.exists(destfile):
189 self._status.fetched_bytes = os.path.getsize(destfile)
190 self._status.pulse()
191 return False
192
193 def fetch(self, uri, destfile):
194 self._run_acquire_method(uri, destfile)
195
196
197# similar to the apt AcquireMethod
198class ClickAcquireMethod:
199
200 M_CONFIGURATION = 601
201 M_FETCH = 600
202
203 VERSION = 0.1
204
205 def __init__(self):
206 s = dedent("""\
207 100 Capabilities
208 Version: {version}
209 Single-Instance: true
210
211 """).format(version=self.VERSION)
212 sys.stdout.write(s)
213 sys.stdout.flush()
214
215 def run(self):
216 while True:
217 msgs = _read_messages(sys.stdin)
218 if not msgs:
219 break
220 for number, raw_message in msgs:
221 message = Deb822(raw_message)
222 if number == self.M_CONFIGURATION:
223 pass
224 elif number == self.M_FETCH:
225 self.fetch(message.get("URI"), message.get("FileName"))
226
227 def uri_start(self, uri, filename, size):
228 # note that apt itself does not use "Filename" here because it
229 # will set a filename and expects the method to use it, however
230 # this will not work with UbuntuDownloadManager as it downloads
231 # to its own location
232 sys.stdout.write(dedent("""\
233 200 URI Start
234 URI: {uri}
235 Filename: {filename}
236 Size: {size}
237
238 """).format(uri=uri, filename=filename, size=size))
239 sys.stdout.flush()
240
241 def uri_done(self, uri, filename):
242 # bug in python-apt
243 hashes = apt_pkg.Hashes(filename.encode("utf-8"))
244 sys.stdout.write(dedent("""\
245 201 URI Done
246 URI: {uri}
247 Filename: {filename}
248 Size: {size}
249 Sha256-Hash: {sha256hash}
250
251 """).format(uri=uri, filename=filename, size=os.path.getsize(filename),
252 sha256hash=hashes.sha256))
253 sys.stdout.flush()
254
255 def fail(self, uri, err="unknown error"):
256 sys.stdout.write(dedent("""\
257 400 URI Failure
258 URI: {uri}
259 Message: {err}
260
261 """).format(uri=uri, err=err))
262 sys.stdout.flush()
263
264 def fetch(self, uri, destfile):
265 pass
266
267
268class ClickAcquireMethodPycurl(ClickAcquireMethod):
269
270 def _write_to_file_callback(self, data):
271 self._destfile_fp.write(data)
272 return len(data)
273
274 def _progress_info(self, dltotal, dlnow, ultotal, ulnow):
275 if dltotal > 0 and not self._uri_start_reported:
276 self.uri_start(
277 self.uri, self._destfile_fp.name, int(dltotal))
278 self._uri_start_reported = True
279
280 def fetch(self, uri, destfile):
281 self._uri_start_reported = False
282 self.uri = uri
283 self._destfile_fp = open(destfile, "wb")
284 curl = pycurl.Curl()
285 curl.setopt(pycurl.URL, uri)
286 curl.setopt(pycurl.WRITEFUNCTION, self._write_to_file_callback)
287 curl.setopt(pycurl.NOPROGRESS, 0)
288 curl.setopt(pycurl.PROGRESSFUNCTION, self._progress_info)
289 curl.setopt(pycurl.FOLLOWLOCATION, 1)
290 curl.setopt(pycurl.MAXREDIRS, 5)
291 curl.setopt(pycurl.FAILONERROR, 1)
292 # timeout 120s for conenction
293 curl.setopt(pycurl.CONNECTTIMEOUT, 120)
294 # timeout if the speed is 120s below 10 bytes/sec
295 curl.setopt(pycurl.LOW_SPEED_LIMIT, 10);
296 curl.setopt(pycurl.LOW_SPEED_TIME, 120);
297 # ssl: no need to set any option here,
298 # SSL_VERIFYPEER=1, SSL_VERIFYHOST=2,
299 # CAINFO=/etc/ssl/certs/ca-certificates.crt
300 # by default in libcurl3 these days
301 try:
302 curl.perform()
303 self._destfile_fp.close()
304 self.uri_done(uri, destfile)
305 except pycurl.error as e:
306 self.fail(self.uri, e.args)
307 self._destfile_fp.close()
308
309
310class ClickAcquireMethodUbuntuDownloadManager(ClickAcquireMethod):
311
312 MANAGER_PATH = '/'
313 MANAGER_IFACE = 'com.canonical.applications.DownloadManager'
314 DOWNLOAD_IFACE = 'com.canonical.applications.Download'
315
316 def __init__(self):
317 super(ClickAcquireMethodUbuntuDownloadManager, self).__init__()
318 self.bus = dbus.SessionBus()
319 self.loop = GLib.MainLoop()
320
321 def _created_callback(self, dbus_path):
322 pass
323
324 def _finished_callback(self, path, loop):
325 self.down_path = path
326 loop.quit()
327
328 def _progress_callback(self, total, progress):
329 #print('Progress is %s/%s' % (progress, total))
330 if not self._started:
331 # FIXME: we need the tmpfile path from udm
332 tmpfile = "meep"
333 self.uri_start(self.uri, tmpfile, total)
334 self._started = True
335
336 def fetch(self, uri, destfile):
337 self.uri = uri
338 self._started = False
339 manager = self.bus.get_object(
340 'com.canonical.applications.Downloader', self.MANAGER_PATH)
341 manager_dev_iface = dbus.Interface(
342 manager, dbus_interface=self.MANAGER_IFACE)
343 manager_dev_iface.connect_to_signal(
344 'downloadCreated', self._created_callback)
345 down_path = manager_dev_iface.createDownload(
346 (uri, "", "", dbus.Dictionary({}, signature="sv"),
347 dbus.Dictionary({}, signature="ss")))
348 download1 = self.bus.get_object('com.canonical.applications.Downloader',
349 down_path)
350 download_dev_iface1 = dbus.Interface(
351 download1, dbus_interface=self.DOWNLOAD_IFACE)
352 download_dev_iface1.connect_to_signal(
353 'progress', self._progress_callback)
354 download_dev_iface1.connect_to_signal(
355 'finished',
356 lambda path: self._finished_callback(path, self.loop))
357 download_dev_iface1.start()
358 self.loop.run()
359 # FIXME error handling, i.e. send self.fail()
360 self.uri_done(uri, self.down_path)
361
362
363if __name__ == "__main__":
364 log = ClickAcquireStatusText()
365 acq = ClickAcquire(log)
366 uri = sys.argv[1]
367 if not acq.fetch(uri, os.path.basename(uri)):
368 sys.exit(1)
369 sys.exit(0)
0370
=== modified file 'click/commands/install.py'
--- click/commands/install.py 2014-09-10 12:28:49 +0000
+++ click/commands/install.py 2014-10-14 13:55:44 +0000
@@ -19,10 +19,17 @@
1919
20from optparse import OptionParser20from optparse import OptionParser
21import sys21import sys
22import tempfile
22from textwrap import dedent23from textwrap import dedent
2324
25from six.moves.urllib.parse import urlparse
26
24from gi.repository import Click27from gi.repository import Click
2528
29from click.acquire import (
30 ClickAcquire,
31 ClickAcquireStatusText,
32)
26from click.install import ClickInstaller, ClickInstallerError33from click.install import ClickInstaller, ClickInstallerError
2734
2835
@@ -53,10 +60,19 @@
53 db.read(db_dir=None)60 db.read(db_dir=None)
54 if options.root is not None:61 if options.root is not None:
55 db.add(options.root)62 db.add(options.root)
56 package_path = args[0]63 package_uri = args[0]
57 installer = ClickInstaller(64 installer = ClickInstaller(
58 db=db, force_missing_framework=options.force_missing_framework,65 db=db, force_missing_framework=options.force_missing_framework,
59 allow_unauthenticated=options.allow_unauthenticated)66 allow_unauthenticated=options.allow_unauthenticated)
67 parsed_uri = urlparse(package_uri)
68 if parsed_uri.scheme != "":
69 t = tempfile.NamedTemporaryFile()
70 package_path = t.name
71 log = ClickAcquireStatusText()
72 acq = ClickAcquire(log)
73 acq.fetch(package_uri, package_path)
74 else:
75 package_path = package_uri
60 try:76 try:
61 installer.install(77 installer.install(
62 package_path, user=options.user, all_users=options.all_users)78 package_path, user=options.user, all_users=options.all_users)
6379
=== modified file 'click/paths.py.in'
--- click/paths.py.in 2014-05-08 15:48:01 +0000
+++ click/paths.py.in 2014-10-14 13:55:44 +0000
@@ -14,6 +14,9 @@
14# along with this program. If not, see <http://www.gnu.org/licenses/>.14# along with this program. If not, see <http://www.gnu.org/licenses/>.
1515
16"""Click paths."""16"""Click paths."""
17import os
1718
18preload_path = "@pkglibdir@/libclickpreload.so"19preload_path = "@pkglibdir@/libclickpreload.so"
19frameworks_dir = "@pkgdatadir@/frameworks"20frameworks_dir = "@pkgdatadir@/frameworks"
21acquire_methods_dir = (os.environ.get("CLICK_ACQUIRE_METHODS_DIR", "") or
22 "@pkglibdir@/acquire")
2023
=== added file 'click/tests/test_acquire.py'
--- click/tests/test_acquire.py 1970-01-01 00:00:00 +0000
+++ click/tests/test_acquire.py 2014-10-14 13:55:44 +0000
@@ -0,0 +1,174 @@
1# Copyright (C) 2014 Canonical Ltd.
2# Author: Michael Vogt <michael.vogt@ubuntu.com>
3
4# This program is free software: you can redistribute it and/or modify
5# it under the terms of the GNU General Public License as published by
6# the Free Software Foundation; version 3 of the License.
7#
8# This program is distributed in the hope that it will be useful,
9# but WITHOUT ANY WARRANTY; without even the implied warranty of
10# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
11# GNU General Public License for more details.
12#
13# You should have received a copy of the GNU General Public License
14# along with this program. If not, see <http://www.gnu.org/licenses/>.
15
16"""Unit tests for click.acquire."""
17
18from __future__ import print_function
19
20__metaclass__ = type
21__all__ = [
22 'TestClickAcquire',
23 ]
24
25import multiprocessing
26import os.path
27import time
28
29from six.moves import (
30 SimpleHTTPServer,
31 socketserver,
32)
33
34from click.acquire import (
35 ClickAcquire,
36 ClickAcquireError,
37 _read_messages,
38)
39
40from click.tests.helpers import (
41 TestCase,
42)
43
44import click.acquire
45click.acquire.acquire_methods_dir = os.path.join(
46 os.path.dirname(__file__), "..", "..", "acquire")
47os.environ["PYTHONPATH"] = os.path.abspath(
48 os.path.join(os.path.dirname(__file__), "..", ".."))
49
50# local httpd
51LOCALHOST = "localhost"
52PORT = 8128
53
54
55class MyHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
56 def log_message(self, format, *args):
57 pass
58
59
60class MySocketServer(socketserver.TCPServer):
61 allow_reuse_address = True
62
63
64class Httpd(multiprocessing.Process):
65
66 def __init__(self, basedir):
67 super(Httpd, self).__init__()
68 self.basedir = basedir
69
70 def run(self):
71 os.chdir(self.basedir)
72 server = MySocketServer((LOCALHOST, PORT), MyHandler)
73 while True:
74 server.handle_request()
75 server.shutdown()
76
77 def stop(self):
78 self.terminate()
79 self.join()
80
81
82class TestClickAcquire(TestCase):
83
84 def setUp(self):
85 super(TestClickAcquire, self).setUp()
86 self.use_temp_dir()
87 self.httpd = Httpd(self.temp_dir)
88 self.httpd.start()
89
90 def tearDown(self):
91 self.httpd.stop()
92
93 def test_acquire_fail(self):
94 acq = ClickAcquire()
95 destfile = os.path.join(self.temp_dir, "meep")
96 with self.assertRaises(ClickAcquireError):
97 acq.fetch("http://%s:%s/not-here" % (LOCALHOST, PORT), destfile)
98
99 def test_acquire_good(self):
100 acq = ClickAcquire()
101 destfile = os.path.join(self.temp_dir, "meep")
102 canary_str = "hello"
103 with open(os.path.join(self.temp_dir, "i-am-here"), "w") as f:
104 f.write(canary_str)
105 acq.fetch("http://%s:%s/i-am-here" % (LOCALHOST, PORT), destfile)
106 with open(destfile) as f:
107 data = f.read()
108 self.assertEqual(canary_str, data)
109
110
111class TestClickAcquireReadMessages(TestCase):
112
113 def test_forked_read_message(self):
114 read_end, write_end = os.pipe()
115 pid = os.fork()
116 if pid == 0:
117 os.close(write_end)
118 with os.fdopen(read_end) as f:
119 number, msg = _read_messages(f)[0]
120 self.assertEqual(msg, "102 Status\nFoo: Bar\n")
121 self.assertEqual(number, 102)
122 os._exit(0)
123 os.close(read_end)
124 os.write(write_end, "102 Status\nFoo: Bar\n\n".encode("utf-8"))
125 os.waitpid(pid, 0)
126 os.close(write_end)
127
128 def test_forked_read_multiple_messages(self):
129 read_end, write_end = os.pipe()
130 pid = os.fork()
131 if pid == 0:
132 os.close(write_end)
133 with os.fdopen(read_end) as f:
134 msgs = _read_messages(f)
135 self.assertEqual(len(msgs), 2)
136 self.assertEqual(msgs[0][0], 100)
137 self.assertEqual(msgs[1][0], 200)
138 os._exit(0)
139 os.close(read_end)
140 os.write(write_end, "100 Status\n\n200 meep\n\n".encode("utf-8"))
141 os.waitpid(pid, 0)
142 os.close(write_end)
143
144 def test_forked_read_message_broken_pipe(self):
145 read_end, write_end = os.pipe()
146 pid = os.fork()
147 if pid == 0:
148 os.close(write_end)
149 with os.fdopen(read_end) as f:
150 self.assertEqual(_read_messages(f), [])
151 os._exit(0)
152 os.close(read_end)
153 os.write(write_end, "102 close-before-msg".encode("utf-8"))
154 os.close(write_end)
155 os.waitpid(pid, 0)
156
157 def test_forked_read_message_with_timeout(self):
158 read_end, write_end = os.pipe()
159 pid = os.fork()
160 if pid == 0:
161 os.close(write_end)
162 with os.fdopen(read_end) as f:
163 self.assertEqual(_read_messages(f, 0.1), [])
164 msgs = _read_messages(f)
165 self.assertEqual(len(msgs), 2)
166 self.assertEqual(msgs[0][1], "100 aaa\n")
167 self.assertEqual(msgs[1][1], "200 bbb\n")
168 os._exit(0)
169 os.close(read_end)
170 # simulate slow method
171 time.sleep(0.2)
172 os.write(write_end, "100 aaa\n\n".encode("utf-8"))
173 os.write(write_end, "200 bbb\n\n".encode("utf-8"))
174 os.waitpid(pid, 0)
0175
=== modified file 'debian/control'
--- debian/control 2014-10-10 07:10:23 +0000
+++ debian/control 2014-10-14 13:55:44 +0000
@@ -3,7 +3,7 @@
3Priority: optional3Priority: optional
4Maintainer: Colin Watson <cjwatson@ubuntu.com>4Maintainer: Colin Watson <cjwatson@ubuntu.com>
5Standards-Version: 3.9.55Standards-Version: 3.9.5
6Build-Depends: debhelper (>= 9~), dh-autoreconf, intltool, python3:any (>= 3.2), python3-all:any, python3-setuptools, python3-apt, python3-debian, python3-gi, python3:any (>= 3.3) | python3-mock, pep8, python3-pep8, pyflakes, python3-sphinx, pkg-config, valac, gobject-introspection (>= 0.6.7), libgirepository1.0-dev (>= 0.6.7), libglib2.0-dev (>= 2.34), gir1.2-glib-2.0, libjson-glib-dev (>= 0.10), libgee-0.8-dev, libpackagekit-glib2-dev (>= 0.7.2), python3-coverage, python3-six, dh-systemd (>= 1.3)6Build-Depends: debhelper (>= 9~), dh-autoreconf, intltool, python3:any (>= 3.2), python3-all:any, python3-setuptools, python3-apt, python3-debian, python3-gi, python3:any (>= 3.3) | python3-mock, pep8, python3-pep8, pyflakes, python3-sphinx, pkg-config, valac, gobject-introspection (>= 0.6.7), libgirepository1.0-dev (>= 0.6.7), libglib2.0-dev (>= 2.34), gir1.2-glib-2.0, libjson-glib-dev (>= 0.10), libgee-0.8-dev, libpackagekit-glib2-dev (>= 0.7.2), python3-coverage, python3-six, dh-systemd (>= 1.3), python3-pycurl, python3-dbus
7Vcs-Bzr: https://code.launchpad.net/~ubuntu-managed-branches/click/click7Vcs-Bzr: https://code.launchpad.net/~ubuntu-managed-branches/click/click
8Vcs-Browser: http://bazaar.launchpad.net/~ubuntu-managed-branches/click/click/files8Vcs-Browser: http://bazaar.launchpad.net/~ubuntu-managed-branches/click/click/files
9X-Auto-Uploader: no-rewrite-version9X-Auto-Uploader: no-rewrite-version
@@ -14,7 +14,7 @@
14Package: click14Package: click
15Architecture: any15Architecture: any
16Pre-Depends: ${misc:Pre-Depends}16Pre-Depends: ${misc:Pre-Depends}
17Depends: ${shlibs:Depends}, ${misc:Depends}, ${python3:Depends}, python3-click (= ${binary:Version}), adduser17Depends: ${shlibs:Depends}, ${misc:Depends}, ${python3:Depends}, python3-click (= ${binary:Version}), adduser, python3-pycurl, python3-dbus
18Recommends: click-apparmor18Recommends: click-apparmor
19Suggests: click-reviewers-tools (>= 0.9), ubuntu-app-launch-tools | upstart-app-launch-tools19Suggests: click-reviewers-tools (>= 0.9), ubuntu-app-launch-tools | upstart-app-launch-tools
20Conflicts: click-package20Conflicts: click-package
2121
=== modified file 'setup.py.in'
--- setup.py.in 2014-06-30 16:48:31 +0000
+++ setup.py.in 2014-10-14 13:55:44 +0000
@@ -1,5 +1,6 @@
1#! /usr/bin/env python31#! /usr/bin/env python3
22
3import glob
3import sys4import sys
45
5from setuptools import find_packages, setup6from setuptools import find_packages, setup
@@ -18,6 +19,8 @@
18if sys.version < "3.3":19if sys.version < "3.3":
19 require('mock')20 require('mock')
20require('chardet')21require('chardet')
22require('click.paths')
23import click.paths
2124
22if "@GCOVR@":25if "@GCOVR@":
23 require('coverage')26 require('coverage')
@@ -47,6 +50,11 @@
47 license="GNU GPL",50 license="GNU GPL",
48 packages=find_packages(),51 packages=find_packages(),
49 scripts=['bin/click'],52 scripts=['bin/click'],
53 data_files=[
54 # we need to remove the prefix here or we add it twice
55 (click.paths.acquire_methods_dir[1:].lstrip(sys.prefix),
56 glob.glob("acquire/*")),
57 ],
50 install_requires=requirements,58 install_requires=requirements,
51 cmdclass={"test": test_extra},59 cmdclass={"test": test_extra},
52 test_suite="click.tests",60 test_suite="click.tests",

Subscribers

People subscribed via source and target branches

to all changes: