click

Merge lp:~mvo/click/acquire into lp:click/devel

  1. acquire
  2. Merge into devel
Proposed by Michael Vogt
Status: Needs review
Proposed branch: lp:~mvo/click/acquire
Merge into: lp:click/devel
Diff against target: 740 lines (+631/-3)
8 files modified
acquire/http-udm (+28/-0)
acquire/pycurl (+30/-0)
click/acquire.py (+369/-0)
click/commands/install.py (+17/-1)
click/paths.py.in (+3/-0)
click/tests/test_acquire.py (+174/-0)
debian/control (+2/-2)
setup.py.in (+8/-0)
To merge this branch: bzr merge lp:~mvo/click/acquire
Reviewer Review Type Date Requested Status
PS Jenkins bot (community) continuous-integration Approve
Colin Watson Needs Fixing
Review via email: mp+235753@code.launchpad.net

Commit message

Provide support to download click packages.

Description of the change

This branch implements "acquire" support in click to support "click install http://example.com/foo_1.0.click".

The model is similar with what apt is using and click can reuse the apt methods to download packages. A pycurl based implementation is also provided. Feedback welcome especially suggestsions if the low-level read_messages() code can be done in a more elegant way.

To post a comment you must log in.
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:541
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~mvo/click/acquire/+merge/235753/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/click-devel-ci/71/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/73/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/71/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/71/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/71/rebuild

review: Needs Fixing (continuous-integration)
Reply
lp:~mvo/click/acquire updated
542. By Michael Vogt

click/acquire.py: make dbus optional

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:542
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~mvo/click/acquire/+merge/235753/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/click-devel-ci/73/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/75/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/73/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/73/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/73/rebuild

review: Needs Fixing (continuous-integration)
Reply
lp:~mvo/click/acquire updated
543. By Michael Vogt

add missing python3-pycurl depdendency

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:543
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~mvo/click/acquire/+merge/235753/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/click-devel-ci/74/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/76/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/74/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/74/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/74/rebuild

review: Needs Fixing (continuous-integration)
Reply
lp:~mvo/click/acquire updated
544. By Michael Vogt

add python3-pycurl to b-d (needed during the tests

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:544
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~mvo/click/acquire/+merge/235753/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/click-devel-ci/75/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/77
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/75
        deb: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/75/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/75

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/75/rebuild

review: Needs Fixing (continuous-integration)
Reply
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

PASSED: Continuous integration, rev:544
http://jenkins.qa.ubuntu.com/job/click-devel-ci/78/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/80
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/78
        deb: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/78/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/78

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/78/rebuild

review: Approve (continuous-integration)
Reply
Revision history for this message
Colin Watson (cjwatson) wrote :

Overall this looks like nice work, but I think you could productively simplify some things.

review: Needs Fixing
Reply
lp:~mvo/click/acquire updated
545. By Michael Vogt

first round of addressing review points from Colin

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:545
http://jenkins.qa.ubuntu.com/job/click-devel-ci/86/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/88/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/86/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/86/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/86/rebuild

review: Needs Fixing (continuous-integration)
Reply
lp:~mvo/click/acquire updated
546. By Michael Vogt

click/acquire.py: add note about ssl options for curl

547. By Michael Vogt

rename read_messages() -> _read_messages()

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:546
http://jenkins.qa.ubuntu.com/job/click-devel-ci/87/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/89/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/87/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/87/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/87/rebuild

review: Needs Fixing (continuous-integration)
Reply
lp:~mvo/click/acquire updated
548. By Michael Vogt

add missing python3-dbus build-dependency

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

PASSED: Continuous integration, rev:548
http://jenkins.qa.ubuntu.com/job/click-devel-ci/88/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/90
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/88
        deb: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/88/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/88

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/88/rebuild

review: Approve (continuous-integration)
Reply
Revision history for this message
Michael Vogt (mvo) wrote :

> Overall this looks like nice work, but I think you could productively simplify
> some things.

Thanks a lot for your detailed review. I addressed all the points you raised now.

The only one left is the blocking/non-blocking issue. The reason I made this non-blocking is that we can easily report progress about the download from the main click app this way. I created a lp:~mvo/click/acquire-asyncio that still uses a non-blocking io model but is much simpler to follow due to the use of the new python3 asyncio module.

Alternatively we can ditch some of the compatibility with the apt methods and move the progress into the method. The downside is that if/when we support parallel downloads its more difficult to present a overall progress when its done in the individual downloaders.

Reply
lp:~mvo/click/acquire updated
549. By Michael Vogt

improve error erporting

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:549
http://jenkins.qa.ubuntu.com/job/click-devel-ci/107/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/110/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/107/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/107/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/107/rebuild

review: Needs Fixing (continuous-integration)
Reply
lp:~mvo/click/acquire updated
550. By Michael Vogt

merged lp:click/devel

Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

PASSED: Continuous integration, rev:550
http://jenkins.qa.ubuntu.com/job/click-devel-ci/109/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-amd64-ci/112
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/109
        deb: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-armhf-ci/109/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/click-devel-utopic-i386-ci/109

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/click-devel-ci/109/rebuild

review: Approve (continuous-integration)
Reply

Unmerged revisions

550. By Michael Vogt

merged lp:click/devel

549. By Michael Vogt

improve error erporting

548. By Michael Vogt

add missing python3-dbus build-dependency

547. By Michael Vogt

rename read_messages() -> _read_messages()

546. By Michael Vogt

click/acquire.py: add note about ssl options for curl

545. By Michael Vogt

first round of addressing review points from Colin

544. By Michael Vogt

add python3-pycurl to b-d (needed during the tests

543. By Michael Vogt

add missing python3-pycurl depdendency

542. By Michael Vogt

click/acquire.py: make dbus optional

541. By Michael Vogt

remove debug message

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added directory 'acquire'
2=== added symlink 'acquire/file'
3=== target is u'pycurl'
4=== added symlink 'acquire/ftp'
5=== target is u'pycurl'
6=== added symlink 'acquire/http'
7=== target is u'pycurl'
8=== added file 'acquire/http-udm'
9--- acquire/http-udm 1970-01-01 00:00:00 +0000
10+++ acquire/http-udm 2014-10-14 13:55:44 +0000
11@@ -0,0 +1,28 @@
12+#!/usr/bin/python3
13+
14+# Copyright (C) 2014 Canonical Ltd.
15+# Author: Michael Vogt <michael.vogt@ubuntu.com>
16+
17+# This program is free software: you can redistribute it and/or modify
18+# it under the terms of the GNU General Public License as published by
19+# the Free Software Foundation; version 3 of the License.
20+#
21+# This program is distributed in the hope that it will be useful,
22+# but WITHOUT ANY WARRANTY; without even the implied warranty of
23+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
24+# GNU General Public License for more details.
25+#
26+# You should have received a copy of the GNU General Public License
27+# along with this program. If not, see <http://www.gnu.org/licenses/>.
28+
29+"""Acquire method for ubuntu-download-manager."""
30+
31+from click.acquire import ClickAcquireMethodUbuntuDownloadManager
32+
33+from dbus.mainloop.glib import DBusGMainLoop
34+DBusGMainLoop(set_as_default=True)
35+
36+
37+if __name__ == "__main__":
38+ m = ClickAcquireMethodUbuntuDownloadManager()
39+ m.run()
40
41=== added symlink 'acquire/https'
42=== target is u'pycurl'
43=== added file 'acquire/pycurl'
44--- acquire/pycurl 1970-01-01 00:00:00 +0000
45+++ acquire/pycurl 2014-10-14 13:55:44 +0000
46@@ -0,0 +1,30 @@
47+#!/usr/bin/python3
48+
49+# Copyright (C) 2014 Canonical Ltd.
50+# Author: Michael Vogt <michael.vogt@ubuntu.com>
51+
52+# This program is free software: you can redistribute it and/or modify
53+# it under the terms of the GNU General Public License as published by
54+# the Free Software Foundation; version 3 of the License.
55+#
56+# This program is distributed in the hope that it will be useful,
57+# but WITHOUT ANY WARRANTY; without even the implied warranty of
58+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
59+# GNU General Public License for more details.
60+#
61+# You should have received a copy of the GNU General Public License
62+# along with this program. If not, see <http://www.gnu.org/licenses/>.
63+
64+"""Acquire pycurl method click."""
65+
66+import signal
67+import sys
68+
69+from click.acquire import ClickAcquireMethodPycurl
70+
71+
72+if __name__ == "__main__":
73+ # apt will send a SIGINT to all methods that do not send "Need-Cleanup: 1"
74+ signal.signal(signal.SIGINT, lambda *args: sys.exit())
75+ m = ClickAcquireMethodPycurl()
76+ m.run()
77
78=== added file 'click/acquire.py'
79--- click/acquire.py 1970-01-01 00:00:00 +0000
80+++ click/acquire.py 2014-10-14 13:55:44 +0000
81@@ -0,0 +1,369 @@
82+# Copyright (C) 2014 Canonical Ltd.
83+# Author: Michael Vogt <michael.vogt@ubuntu.com>
84+
85+# This program is free software: you can redistribute it and/or modify
86+# it under the terms of the GNU General Public License as published by
87+# the Free Software Foundation; version 3 of the License.
88+#
89+# This program is distributed in the hope that it will be useful,
90+# but WITHOUT ANY WARRANTY; without even the implied warranty of
91+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
92+# GNU General Public License for more details.
93+#
94+# You should have received a copy of the GNU General Public License
95+# along with this program. If not, see <http://www.gnu.org/licenses/>.
96+
97+"""Acquire of Click packages."""
98+
99+from __future__ import print_function
100+
101+__metaclass__ = type
102+__all__ = [
103+ 'ClickAcquire',
104+ 'ClickAcquireError',
105+ 'ClickAcquireMethod',
106+ 'ClickAcquireMethodPycurl',
107+ 'ClickAcquireMethodUbuntuDownloadManager',
108+ 'ClickAcquireStatus',
109+ 'ClickAcquireStatusText',
110+ ]
111+
112+import os
113+import select
114+import subprocess
115+import sys
116+from textwrap import dedent
117+
118+import apt_pkg
119+from debian.deb822 import Deb822
120+import dbus
121+from gi.repository import GLib
122+import pycurl
123+from six.moves.urllib.parse import urlparse
124+
125+from click.paths import acquire_methods_dir
126+
127+
128+class ClickAcquireError(Exception):
129+ """Error during acquire"""
130+ pass
131+
132+
133+def extract_one_message(lines):
134+ msg = []
135+ while True:
136+ try:
137+ line = lines.pop(0)
138+ except IndexError:
139+ break
140+ if os.environ.get("CLICK_DEBUG_ACQUIRE", ""):
141+ sys.stderr.write("[%s] raw_line: '%s'\n" % (
142+ os.getpid(), line.replace("\n", "\\n")))
143+ msg.append(line)
144+ if len(msg) > 0 and line == "":
145+ # we are done, collect all remaining "\n" and stop
146+ while len(lines) > 0 and lines[0] == "":
147+ lines.pop(0)
148+ break
149+ if len(msg) < 2:
150+ return -1 , ""
151+ if os.environ.get("CLICK_DEBUG_ACQUIRE", ""):
152+ sys.stderr.write("[%s] msg: '%s'\n" % (os.getpid(), str(msg)))
153+ number = int(msg[0].split()[0])
154+ return number, "\n".join(msg)
155+
156+
157+def _read_messages(input_file, timeout=None):
158+ infd = input_file.fileno()
159+ msgs = []
160+ rl, wl, xl = select.select([infd], [], [], timeout)
161+ if rl:
162+ # FIXME: 16k message limit is arbitrary
163+ buf = os.read(infd, 16*1024).decode("utf-8")
164+ if os.environ.get("CLICK_DEBUG_ACQUIRE", ""):
165+ sys.stderr.write("[%s] read buf: '%s'\n" % (
166+ os.getpid(), buf))
167+ lines = buf.split("\n")
168+ while True:
169+ number, msg = extract_one_message(lines)
170+ if number < 0:
171+ break
172+ msgs.append( (number, msg) )
173+ return msgs
174+
175+
176+class ClickAcquireStatus:
177+ """Base class for the status reporting """
178+
179+ def __init__(self):
180+ self.fetched_bytes = 0
181+ self.total_bytes = 1.0
182+ self.uri = ""
183+
184+ def pulse(self):
185+ pass
186+
187+ def done(self):
188+ pass
189+
190+
191+class ClickAcquireStatusText(ClickAcquireStatus):
192+ """Text based progress reporting for the acquire progress"""
193+
194+ def pulse(self):
195+ sys.stdout.write("\r")
196+ sys.stdout.write("[%3.2f %%] Fetching %s" % (
197+ (self.fetched_bytes/self.total_bytes)*100.0,
198+ os.path.basename(self.uri)))
199+ sys.stdout.flush()
200+
201+ def done(self):
202+ self.pulse()
203+ sys.stdout.write("\n")
204+ sys.stdout.write("Done fetching %s\n" % self.uri)
205+
206+
207+# similar to Acquire/AcquireWorker
208+class ClickAcquire:
209+ """Acquire from remote locations"""
210+
211+ # default status reporting timeout
212+ TIMEOUT = 0.05
213+
214+ # acquire method status codecs
215+ M_CAPABILITIES = 100
216+ M_STATUS = 102
217+ M_REDIRECT = 103
218+ URI_START = 200
219+ URI_SUCCESS = 201
220+ URI_FAILURE = 400
221+
222+ def __init__(self, status=ClickAcquireStatus()):
223+ self._fetch_queue = []
224+ self._status = status
225+
226+ def _uri_acquire(self, pipe, uri, destfile):
227+ cmd = dedent("""\
228+ 600 URI Acquire
229+ URI: {uri}
230+ Filename: {destfile}
231+
232+ """).format(uri=uri, destfile=destfile)
233+ pipe.write(cmd)
234+ pipe.flush()
235+
236+ def _redirect(self, pipe, new_uri, destfile):
237+ self._uri_acquire(pipe, new_uri, destfile)
238+
239+ def _run_acquire_method(self, uri, destfile):
240+ parsed_uri = urlparse(uri)
241+ cmd = os.path.join(acquire_methods_dir, parsed_uri.scheme)
242+ p = subprocess.Popen([cmd],
243+ stdout=subprocess.PIPE, stdin=subprocess.PIPE,
244+ universal_newlines=True)
245+ while True:
246+ for number, raw_message in _read_messages(p.stdout, self.TIMEOUT):
247+ message = Deb822(raw_message)
248+ if number == self.M_CAPABILITIES:
249+ self._uri_acquire(p.stdin, uri, destfile)
250+ elif number == self.M_STATUS:
251+ pass
252+ elif number == self.M_REDIRECT:
253+ self._redirect(p.stdin, message.get("New-URI"), destfile)
254+ elif number == self.URI_START:
255+ self._status.uri = message.get("URI", 0)
256+ self._status.total_bytes = int(message.get("Size", 0))
257+ elif number == self.URI_SUCCESS:
258+ self._status.fetched_bytes = int(message.get("Size"))
259+ self._status.done()
260+ p.stdout.close()
261+ p.stdin.close()
262+ return True
263+ elif number == self.URI_FAILURE:
264+ p.stdout.close()
265+ p.stdin.close()
266+ raise ClickAcquireError("Uri failure for %s: %s" % (
267+ message.get("uri"), message.get("Message")))
268+ # update progress
269+ if os.path.exists(destfile):
270+ self._status.fetched_bytes = os.path.getsize(destfile)
271+ self._status.pulse()
272+ return False
273+
274+ def fetch(self, uri, destfile):
275+ self._run_acquire_method(uri, destfile)
276+
277+
278+# similar to the apt AcquireMethod
279+class ClickAcquireMethod:
280+
281+ M_CONFIGURATION = 601
282+ M_FETCH = 600
283+
284+ VERSION = 0.1
285+
286+ def __init__(self):
287+ s = dedent("""\
288+ 100 Capabilities
289+ Version: {version}
290+ Single-Instance: true
291+
292+ """).format(version=self.VERSION)
293+ sys.stdout.write(s)
294+ sys.stdout.flush()
295+
296+ def run(self):
297+ while True:
298+ msgs = _read_messages(sys.stdin)
299+ if not msgs:
300+ break
301+ for number, raw_message in msgs:
302+ message = Deb822(raw_message)
303+ if number == self.M_CONFIGURATION:
304+ pass
305+ elif number == self.M_FETCH:
306+ self.fetch(message.get("URI"), message.get("FileName"))
307+
308+ def uri_start(self, uri, filename, size):
309+ # note that apt itself does not use "Filename" here because it
310+ # will set a filename and expects the method to use it, however
311+ # this will not work with UbuntuDownloadManager as it downloads
312+ # to its own location
313+ sys.stdout.write(dedent("""\
314+ 200 URI Start
315+ URI: {uri}
316+ Filename: {filename}
317+ Size: {size}
318+
319+ """).format(uri=uri, filename=filename, size=size))
320+ sys.stdout.flush()
321+
322+ def uri_done(self, uri, filename):
323+ # bug in python-apt
324+ hashes = apt_pkg.Hashes(filename.encode("utf-8"))
325+ sys.stdout.write(dedent("""\
326+ 201 URI Done
327+ URI: {uri}
328+ Filename: {filename}
329+ Size: {size}
330+ Sha256-Hash: {sha256hash}
331+
332+ """).format(uri=uri, filename=filename, size=os.path.getsize(filename),
333+ sha256hash=hashes.sha256))
334+ sys.stdout.flush()
335+
336+ def fail(self, uri, err="unknown error"):
337+ sys.stdout.write(dedent("""\
338+ 400 URI Failure
339+ URI: {uri}
340+ Message: {err}
341+
342+ """).format(uri=uri, err=err))
343+ sys.stdout.flush()
344+
345+ def fetch(self, uri, destfile):
346+ pass
347+
348+
349+class ClickAcquireMethodPycurl(ClickAcquireMethod):
350+
351+ def _write_to_file_callback(self, data):
352+ self._destfile_fp.write(data)
353+ return len(data)
354+
355+ def _progress_info(self, dltotal, dlnow, ultotal, ulnow):
356+ if dltotal > 0 and not self._uri_start_reported:
357+ self.uri_start(
358+ self.uri, self._destfile_fp.name, int(dltotal))
359+ self._uri_start_reported = True
360+
361+ def fetch(self, uri, destfile):
362+ self._uri_start_reported = False
363+ self.uri = uri
364+ self._destfile_fp = open(destfile, "wb")
365+ curl = pycurl.Curl()
366+ curl.setopt(pycurl.URL, uri)
367+ curl.setopt(pycurl.WRITEFUNCTION, self._write_to_file_callback)
368+ curl.setopt(pycurl.NOPROGRESS, 0)
369+ curl.setopt(pycurl.PROGRESSFUNCTION, self._progress_info)
370+ curl.setopt(pycurl.FOLLOWLOCATION, 1)
371+ curl.setopt(pycurl.MAXREDIRS, 5)
372+ curl.setopt(pycurl.FAILONERROR, 1)
373+ # timeout 120s for conenction
374+ curl.setopt(pycurl.CONNECTTIMEOUT, 120)
375+ # timeout if the speed is 120s below 10 bytes/sec
376+ curl.setopt(pycurl.LOW_SPEED_LIMIT, 10);
377+ curl.setopt(pycurl.LOW_SPEED_TIME, 120);
378+ # ssl: no need to set any option here,
379+ # SSL_VERIFYPEER=1, SSL_VERIFYHOST=2,
380+ # CAINFO=/etc/ssl/certs/ca-certificates.crt
381+ # by default in libcurl3 these days
382+ try:
383+ curl.perform()
384+ self._destfile_fp.close()
385+ self.uri_done(uri, destfile)
386+ except pycurl.error as e:
387+ self.fail(self.uri, e.args)
388+ self._destfile_fp.close()
389+
390+
391+class ClickAcquireMethodUbuntuDownloadManager(ClickAcquireMethod):
392+
393+ MANAGER_PATH = '/'
394+ MANAGER_IFACE = 'com.canonical.applications.DownloadManager'
395+ DOWNLOAD_IFACE = 'com.canonical.applications.Download'
396+
397+ def __init__(self):
398+ super(ClickAcquireMethodUbuntuDownloadManager, self).__init__()
399+ self.bus = dbus.SessionBus()
400+ self.loop = GLib.MainLoop()
401+
402+ def _created_callback(self, dbus_path):
403+ pass
404+
405+ def _finished_callback(self, path, loop):
406+ self.down_path = path
407+ loop.quit()
408+
409+ def _progress_callback(self, total, progress):
410+ #print('Progress is %s/%s' % (progress, total))
411+ if not self._started:
412+ # FIXME: we need the tmpfile path from udm
413+ tmpfile = "meep"
414+ self.uri_start(self.uri, tmpfile, total)
415+ self._started = True
416+
417+ def fetch(self, uri, destfile):
418+ self.uri = uri
419+ self._started = False
420+ manager = self.bus.get_object(
421+ 'com.canonical.applications.Downloader', self.MANAGER_PATH)
422+ manager_dev_iface = dbus.Interface(
423+ manager, dbus_interface=self.MANAGER_IFACE)
424+ manager_dev_iface.connect_to_signal(
425+ 'downloadCreated', self._created_callback)
426+ down_path = manager_dev_iface.createDownload(
427+ (uri, "", "", dbus.Dictionary({}, signature="sv"),
428+ dbus.Dictionary({}, signature="ss")))
429+ download1 = self.bus.get_object('com.canonical.applications.Downloader',
430+ down_path)
431+ download_dev_iface1 = dbus.Interface(
432+ download1, dbus_interface=self.DOWNLOAD_IFACE)
433+ download_dev_iface1.connect_to_signal(
434+ 'progress', self._progress_callback)
435+ download_dev_iface1.connect_to_signal(
436+ 'finished',
437+ lambda path: self._finished_callback(path, self.loop))
438+ download_dev_iface1.start()
439+ self.loop.run()
440+ # FIXME error handling, i.e. send self.fail()
441+ self.uri_done(uri, self.down_path)
442+
443+
444+if __name__ == "__main__":
445+ log = ClickAcquireStatusText()
446+ acq = ClickAcquire(log)
447+ uri = sys.argv[1]
448+ if not acq.fetch(uri, os.path.basename(uri)):
449+ sys.exit(1)
450+ sys.exit(0)
451
452=== modified file 'click/commands/install.py'
453--- click/commands/install.py 2014-09-10 12:28:49 +0000
454+++ click/commands/install.py 2014-10-14 13:55:44 +0000
455@@ -19,10 +19,17 @@
456
457 from optparse import OptionParser
458 import sys
459+import tempfile
460 from textwrap import dedent
461
462+from six.moves.urllib.parse import urlparse
463+
464 from gi.repository import Click
465
466+from click.acquire import (
467+ ClickAcquire,
468+ ClickAcquireStatusText,
469+)
470 from click.install import ClickInstaller, ClickInstallerError
471
472
473@@ -53,10 +60,19 @@
474 db.read(db_dir=None)
475 if options.root is not None:
476 db.add(options.root)
477- package_path = args[0]
478+ package_uri = args[0]
479 installer = ClickInstaller(
480 db=db, force_missing_framework=options.force_missing_framework,
481 allow_unauthenticated=options.allow_unauthenticated)
482+ parsed_uri = urlparse(package_uri)
483+ if parsed_uri.scheme != "":
484+ t = tempfile.NamedTemporaryFile()
485+ package_path = t.name
486+ log = ClickAcquireStatusText()
487+ acq = ClickAcquire(log)
488+ acq.fetch(package_uri, package_path)
489+ else:
490+ package_path = package_uri
491 try:
492 installer.install(
493 package_path, user=options.user, all_users=options.all_users)
494
495=== modified file 'click/paths.py.in'
496--- click/paths.py.in 2014-05-08 15:48:01 +0000
497+++ click/paths.py.in 2014-10-14 13:55:44 +0000
498@@ -14,6 +14,9 @@
499 # along with this program. If not, see <http://www.gnu.org/licenses/>.
500
501 """Click paths."""
502+import os
503
504 preload_path = "@pkglibdir@/libclickpreload.so"
505 frameworks_dir = "@pkgdatadir@/frameworks"
506+acquire_methods_dir = (os.environ.get("CLICK_ACQUIRE_METHODS_DIR", "") or
507+ "@pkglibdir@/acquire")
508
509=== added file 'click/tests/test_acquire.py'
510--- click/tests/test_acquire.py 1970-01-01 00:00:00 +0000
511+++ click/tests/test_acquire.py 2014-10-14 13:55:44 +0000
512@@ -0,0 +1,174 @@
513+# Copyright (C) 2014 Canonical Ltd.
514+# Author: Michael Vogt <michael.vogt@ubuntu.com>
515+
516+# This program is free software: you can redistribute it and/or modify
517+# it under the terms of the GNU General Public License as published by
518+# the Free Software Foundation; version 3 of the License.
519+#
520+# This program is distributed in the hope that it will be useful,
521+# but WITHOUT ANY WARRANTY; without even the implied warranty of
522+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
523+# GNU General Public License for more details.
524+#
525+# You should have received a copy of the GNU General Public License
526+# along with this program. If not, see <http://www.gnu.org/licenses/>.
527+
528+"""Unit tests for click.acquire."""
529+
530+from __future__ import print_function
531+
532+__metaclass__ = type
533+__all__ = [
534+ 'TestClickAcquire',
535+ ]
536+
537+import multiprocessing
538+import os.path
539+import time
540+
541+from six.moves import (
542+ SimpleHTTPServer,
543+ socketserver,
544+)
545+
546+from click.acquire import (
547+ ClickAcquire,
548+ ClickAcquireError,
549+ _read_messages,
550+)
551+
552+from click.tests.helpers import (
553+ TestCase,
554+)
555+
556+import click.acquire
557+click.acquire.acquire_methods_dir = os.path.join(
558+ os.path.dirname(__file__), "..", "..", "acquire")
559+os.environ["PYTHONPATH"] = os.path.abspath(
560+ os.path.join(os.path.dirname(__file__), "..", ".."))
561+
562+# local httpd
563+LOCALHOST = "localhost"
564+PORT = 8128
565+
566+
567+class MyHandler(SimpleHTTPServer.SimpleHTTPRequestHandler):
568+ def log_message(self, format, *args):
569+ pass
570+
571+
572+class MySocketServer(socketserver.TCPServer):
573+ allow_reuse_address = True
574+
575+
576+class Httpd(multiprocessing.Process):
577+
578+ def __init__(self, basedir):
579+ super(Httpd, self).__init__()
580+ self.basedir = basedir
581+
582+ def run(self):
583+ os.chdir(self.basedir)
584+ server = MySocketServer((LOCALHOST, PORT), MyHandler)
585+ while True:
586+ server.handle_request()
587+ server.shutdown()
588+
589+ def stop(self):
590+ self.terminate()
591+ self.join()
592+
593+
594+class TestClickAcquire(TestCase):
595+
596+ def setUp(self):
597+ super(TestClickAcquire, self).setUp()
598+ self.use_temp_dir()
599+ self.httpd = Httpd(self.temp_dir)
600+ self.httpd.start()
601+
602+ def tearDown(self):
603+ self.httpd.stop()
604+
605+ def test_acquire_fail(self):
606+ acq = ClickAcquire()
607+ destfile = os.path.join(self.temp_dir, "meep")
608+ with self.assertRaises(ClickAcquireError):
609+ acq.fetch("http://%s:%s/not-here" % (LOCALHOST, PORT), destfile)
610+
611+ def test_acquire_good(self):
612+ acq = ClickAcquire()
613+ destfile = os.path.join(self.temp_dir, "meep")
614+ canary_str = "hello"
615+ with open(os.path.join(self.temp_dir, "i-am-here"), "w") as f:
616+ f.write(canary_str)
617+ acq.fetch("http://%s:%s/i-am-here" % (LOCALHOST, PORT), destfile)
618+ with open(destfile) as f:
619+ data = f.read()
620+ self.assertEqual(canary_str, data)
621+
622+
623+class TestClickAcquireReadMessages(TestCase):
624+
625+ def test_forked_read_message(self):
626+ read_end, write_end = os.pipe()
627+ pid = os.fork()
628+ if pid == 0:
629+ os.close(write_end)
630+ with os.fdopen(read_end) as f:
631+ number, msg = _read_messages(f)[0]
632+ self.assertEqual(msg, "102 Status\nFoo: Bar\n")
633+ self.assertEqual(number, 102)
634+ os._exit(0)
635+ os.close(read_end)
636+ os.write(write_end, "102 Status\nFoo: Bar\n\n".encode("utf-8"))
637+ os.waitpid(pid, 0)
638+ os.close(write_end)
639+
640+ def test_forked_read_multiple_messages(self):
641+ read_end, write_end = os.pipe()
642+ pid = os.fork()
643+ if pid == 0:
644+ os.close(write_end)
645+ with os.fdopen(read_end) as f:
646+ msgs = _read_messages(f)
647+ self.assertEqual(len(msgs), 2)
648+ self.assertEqual(msgs[0][0], 100)
649+ self.assertEqual(msgs[1][0], 200)
650+ os._exit(0)
651+ os.close(read_end)
652+ os.write(write_end, "100 Status\n\n200 meep\n\n".encode("utf-8"))
653+ os.waitpid(pid, 0)
654+ os.close(write_end)
655+
656+ def test_forked_read_message_broken_pipe(self):
657+ read_end, write_end = os.pipe()
658+ pid = os.fork()
659+ if pid == 0:
660+ os.close(write_end)
661+ with os.fdopen(read_end) as f:
662+ self.assertEqual(_read_messages(f), [])
663+ os._exit(0)
664+ os.close(read_end)
665+ os.write(write_end, "102 close-before-msg".encode("utf-8"))
666+ os.close(write_end)
667+ os.waitpid(pid, 0)
668+
669+ def test_forked_read_message_with_timeout(self):
670+ read_end, write_end = os.pipe()
671+ pid = os.fork()
672+ if pid == 0:
673+ os.close(write_end)
674+ with os.fdopen(read_end) as f:
675+ self.assertEqual(_read_messages(f, 0.1), [])
676+ msgs = _read_messages(f)
677+ self.assertEqual(len(msgs), 2)
678+ self.assertEqual(msgs[0][1], "100 aaa\n")
679+ self.assertEqual(msgs[1][1], "200 bbb\n")
680+ os._exit(0)
681+ os.close(read_end)
682+ # simulate slow method
683+ time.sleep(0.2)
684+ os.write(write_end, "100 aaa\n\n".encode("utf-8"))
685+ os.write(write_end, "200 bbb\n\n".encode("utf-8"))
686+ os.waitpid(pid, 0)
687
688=== modified file 'debian/control'
689--- debian/control 2014-10-10 07:10:23 +0000
690+++ debian/control 2014-10-14 13:55:44 +0000
691@@ -3,7 +3,7 @@
692 Priority: optional
693 Maintainer: Colin Watson <cjwatson@ubuntu.com>
694 Standards-Version: 3.9.5
695-Build-Depends: debhelper (>= 9~), dh-autoreconf, intltool, python3:any (>= 3.2), python3-all:any, python3-setuptools, python3-apt, python3-debian, python3-gi, python3:any (>= 3.3) | python3-mock, pep8, python3-pep8, pyflakes, python3-sphinx, pkg-config, valac, gobject-introspection (>= 0.6.7), libgirepository1.0-dev (>= 0.6.7), libglib2.0-dev (>= 2.34), gir1.2-glib-2.0, libjson-glib-dev (>= 0.10), libgee-0.8-dev, libpackagekit-glib2-dev (>= 0.7.2), python3-coverage, python3-six, dh-systemd (>= 1.3)
696+Build-Depends: debhelper (>= 9~), dh-autoreconf, intltool, python3:any (>= 3.2), python3-all:any, python3-setuptools, python3-apt, python3-debian, python3-gi, python3:any (>= 3.3) | python3-mock, pep8, python3-pep8, pyflakes, python3-sphinx, pkg-config, valac, gobject-introspection (>= 0.6.7), libgirepository1.0-dev (>= 0.6.7), libglib2.0-dev (>= 2.34), gir1.2-glib-2.0, libjson-glib-dev (>= 0.10), libgee-0.8-dev, libpackagekit-glib2-dev (>= 0.7.2), python3-coverage, python3-six, dh-systemd (>= 1.3), python3-pycurl, python3-dbus
697 Vcs-Bzr: https://code.launchpad.net/~ubuntu-managed-branches/click/click
698 Vcs-Browser: http://bazaar.launchpad.net/~ubuntu-managed-branches/click/click/files
699 X-Auto-Uploader: no-rewrite-version
700@@ -14,7 +14,7 @@
701 Package: click
702 Architecture: any
703 Pre-Depends: ${misc:Pre-Depends}
704-Depends: ${shlibs:Depends}, ${misc:Depends}, ${python3:Depends}, python3-click (= ${binary:Version}), adduser
705+Depends: ${shlibs:Depends}, ${misc:Depends}, ${python3:Depends}, python3-click (= ${binary:Version}), adduser, python3-pycurl, python3-dbus
706 Recommends: click-apparmor
707 Suggests: click-reviewers-tools (>= 0.9), ubuntu-app-launch-tools | upstart-app-launch-tools
708 Conflicts: click-package
709
710=== modified file 'setup.py.in'
711--- setup.py.in 2014-06-30 16:48:31 +0000
712+++ setup.py.in 2014-10-14 13:55:44 +0000
713@@ -1,5 +1,6 @@
714 #! /usr/bin/env python3
715
716+import glob
717 import sys
718
719 from setuptools import find_packages, setup
720@@ -18,6 +19,8 @@
721 if sys.version < "3.3":
722 require('mock')
723 require('chardet')
724+require('click.paths')
725+import click.paths
726
727 if "@GCOVR@":
728 require('coverage')
729@@ -47,6 +50,11 @@
730 license="GNU GPL",
731 packages=find_packages(),
732 scripts=['bin/click'],
733+ data_files=[
734+ # we need to remove the prefix here or we add it twice
735+ (click.paths.acquire_methods_dir[1:].lstrip(sys.prefix),
736+ glob.glob("acquire/*")),
737+ ],
738 install_requires=requirements,
739 cmdclass={"test": test_extra},
740 test_suite="click.tests",

Subscribers

People subscribed via source and target branches

to all changes: