Merge lp:~cjwatson/launchpad-buildd/local-snap-proxy into lp:launchpad-buildd
- local-snap-proxy
- Merge into trunk
Status: | Merged | ||||||||
---|---|---|---|---|---|---|---|---|---|
Merged at revision: | 341 | ||||||||
Proposed branch: | lp:~cjwatson/launchpad-buildd/local-snap-proxy | ||||||||
Merge into: | lp:launchpad-buildd | ||||||||
Diff against target: |
598 lines (+370/-41) 10 files modified
buildd-genconfig (+6/-1) debian/changelog (+4/-0) debian/control (+1/-0) debian/postinst (+1/-1) debian/upgrade-config (+13/-0) lpbuildd/buildd-slave.tac (+1/-0) lpbuildd/snap.py (+262/-2) lpbuildd/target/build_snap.py (+0/-37) lpbuildd/tests/test_snap.py (+79/-0) template-buildd-slave.conf (+3/-0) |
||||||||
To merge this branch: | bzr merge lp:~cjwatson/launchpad-buildd/local-snap-proxy | ||||||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
William Grant (community) | code | Approve | |
Review via email:
|
Commit message
Add a local unauthenticated proxy on port 8222, which proxies through to
the remote authenticated proxy. This should allow running a wider range
of network clients, since some of them apparently don't support
authenticated proxies very well.
Description of the change
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Colin Watson (cjwatson) wrote : | # |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Adam Collard (adam-collard) wrote : | # |
drive by review
- 217. By Colin Watson
-
Remove dead code.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Colin Watson (cjwatson) : | # |
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Kevin W Monroe (kwmonroe) wrote : | # |
Aside from the merge conflicts, the spirit of this gets a big +1 from me. I'm running into all sorts of trouble with java projects (mvn, ant, ivy, gradle) and their poor support for auth'd proxies. I think an unauth'd alternative would be super useful.
- 218. By Colin Watson
-
Merge trunk.
- 219. By Colin Watson
-
Drop now-unnecessary compatibility with Twisted < 14.0.0.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Brett Sutton (bsutton) wrote : | # |
Just to add my voice.
This is currently blocking a number of apps I'm trying to build in the store plus anyone that uses a remote part for Apache Tomcat with SSL that I've built.
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
William Grant (wgrant) wrote : | # |
This seems to work fine with some manual testing.
However, this won't directly work for snap builds any more: that --proxy-url is going to point to the container rather than the host.
- 220. By Colin Watson
-
Merge trunk.
- 221. By Colin Watson
-
Build-depend on curl for proxy tests.
- 222. By Colin Watson
-
Fix version in upgrade-config.
- 223. By Colin Watson
-
Remove duplicate --proxy-url option.
- 224. By Colin Watson
-
Point --proxy-url to the LXD host.
- 225. By Colin Watson
-
Close LP #1690834 and #1753340.
Preview Diff
1 | === modified file 'buildd-genconfig' |
2 | --- buildd-genconfig 2016-12-09 18:04:00 +0000 |
3 | +++ buildd-genconfig 2018-06-10 12:47:59 +0000 |
4 | @@ -37,6 +37,11 @@ |
5 | metavar="FILE", |
6 | default="/usr/share/launchpad-buildd/template-buildd-slave.conf") |
7 | |
8 | +parser.add_option("--snap-proxy-port", dest="SNAPPROXYPORT", |
9 | + help="the port the local snap proxy binds to", |
10 | + metavar="PORT", |
11 | + default="8222") |
12 | + |
13 | (options, args) = parser.parse_args() |
14 | |
15 | template = open(options.TEMPLATE, "r").read() |
16 | @@ -46,6 +51,7 @@ |
17 | "@BINDHOST@": options.BINDHOST, |
18 | "@ARCHTAG@": options.ARCHTAG, |
19 | "@BINDPORT@": options.BINDPORT, |
20 | + "@SNAPPROXYPORT@": options.SNAPPROXYPORT, |
21 | } |
22 | |
23 | for replacement_key in replacements: |
24 | @@ -53,4 +59,3 @@ |
25 | replacements[replacement_key]) |
26 | |
27 | print(template.strip()) |
28 | - |
29 | |
30 | === modified file 'debian/changelog' |
31 | --- debian/changelog 2018-06-05 02:06:04 +0000 |
32 | +++ debian/changelog 2018-06-10 12:47:59 +0000 |
33 | @@ -10,6 +10,10 @@ |
34 | * Refactor VCS operations from lpbuildd.target.build_snap out to a module |
35 | that can be used by other targets. |
36 | * Allow checking out a git tag rather than a branch (LP: #1687078). |
37 | + * Add a local unauthenticated proxy on port 8222, which proxies through to |
38 | + the remote authenticated proxy. This should allow running a wider range |
39 | + of network clients, since some of them apparently don't support |
40 | + authenticated proxies very well (LP: #1690834, #1753340). |
41 | |
42 | -- Colin Watson <cjwatson@ubuntu.com> Tue, 08 May 2018 10:36:22 +0100 |
43 | |
44 | |
45 | === modified file 'debian/control' |
46 | --- debian/control 2017-11-27 17:14:19 +0000 |
47 | +++ debian/control 2018-06-10 12:47:59 +0000 |
48 | @@ -5,6 +5,7 @@ |
49 | Standards-Version: 3.9.5 |
50 | Build-Depends: apt-utils, |
51 | bzr, |
52 | + curl, |
53 | debhelper (>= 9~), |
54 | dh-exec, |
55 | dh-python, |
56 | |
57 | === modified file 'debian/postinst' |
58 | --- debian/postinst 2017-08-29 13:19:35 +0000 |
59 | +++ debian/postinst 2018-06-10 12:47:59 +0000 |
60 | @@ -14,7 +14,7 @@ |
61 | |
62 | make_buildd() |
63 | { |
64 | - /usr/share/launchpad-buildd/buildd-genconfig --name=default --host=0.0.0.0 --port=8221 > \ |
65 | + /usr/share/launchpad-buildd/buildd-genconfig --name=default --host=0.0.0.0 --port=8221 --snap-proxy-port=8222 > \ |
66 | /etc/launchpad-buildd/default |
67 | echo Default buildd created. |
68 | } |
69 | |
70 | === modified file 'debian/upgrade-config' |
71 | --- debian/upgrade-config 2016-12-09 18:05:21 +0000 |
72 | +++ debian/upgrade-config 2018-06-10 12:47:59 +0000 |
73 | @@ -197,6 +197,17 @@ |
74 | in_file.close() |
75 | out_file.close() |
76 | |
77 | +def upgrade_to_162(): |
78 | + print("Upgrading %s to version 162" % conf_file) |
79 | + os.rename(conf_file, conf_file + "-prev162~") |
80 | + |
81 | + with open(conf_file + "-prev162~", "r") as in_file: |
82 | + with open(conf_file, "w") as out_file: |
83 | + out_file.write(in_file.read()) |
84 | + out_file.write( |
85 | + "\n[snapmanager]\n" |
86 | + "proxyport = 8222\n") |
87 | + |
88 | if __name__ == "__main__": |
89 | old_version = re.sub(r'[~-].*', '', old_version) |
90 | if apt_pkg.version_compare(old_version, "12") < 0: |
91 | @@ -223,3 +234,5 @@ |
92 | upgrade_to_126() |
93 | if apt_pkg.version_compare(old_version, "127") < 0: |
94 | upgrade_to_127() |
95 | + if apt_pkg.version_compare(old_version, "162") < 0: |
96 | + upgrade_to_162() |
97 | |
98 | === modified file 'lpbuildd/buildd-slave.tac' |
99 | --- lpbuildd/buildd-slave.tac 2018-02-27 13:25:36 +0000 |
100 | +++ lpbuildd/buildd-slave.tac 2018-06-10 12:47:59 +0000 |
101 | @@ -51,6 +51,7 @@ |
102 | application.addComponent( |
103 | RotatableFileLogObserver(options.get('logfile')), ignoreClass=1) |
104 | builddslaveService = service.IServiceCollection(application) |
105 | +slave.slave.service = builddslaveService |
106 | |
107 | root = resource.Resource() |
108 | root.putChild('rpc', slave) |
109 | |
110 | === modified file 'lpbuildd/snap.py' |
111 | --- lpbuildd/snap.py 2018-04-21 10:04:37 +0000 |
112 | +++ lpbuildd/snap.py 2018-06-10 12:47:59 +0000 |
113 | @@ -5,9 +5,39 @@ |
114 | |
115 | __metaclass__ = type |
116 | |
117 | +import base64 |
118 | +import io |
119 | import json |
120 | import os |
121 | import sys |
122 | +try: |
123 | + from urllib.error import ( |
124 | + HTTPError, |
125 | + URLError, |
126 | + ) |
127 | + from urllib.parse import urlparse |
128 | + from urllib.request import ( |
129 | + Request, |
130 | + urlopen, |
131 | + ) |
132 | +except ImportError: |
133 | + from urllib2 import ( |
134 | + HTTPError, |
135 | + Request, |
136 | + URLError, |
137 | + urlopen, |
138 | + ) |
139 | + from urlparse import urlparse |
140 | + |
141 | +from twisted.application import strports |
142 | +from twisted.internet import reactor |
143 | +from twisted.internet.interfaces import IHalfCloseableProtocol |
144 | +from twisted.python.compat import intToBytes |
145 | +from twisted.web import ( |
146 | + http, |
147 | + proxy, |
148 | + ) |
149 | +from zope.interface import implementer |
150 | |
151 | from lpbuildd.debian import ( |
152 | DebianBuildManager, |
153 | @@ -21,6 +51,195 @@ |
154 | RETCODE_FAILURE_BUILD = 201 |
155 | |
156 | |
157 | +class SnapProxyClient(proxy.ProxyClient): |
158 | + |
159 | + def __init__(self, command, rest, version, headers, data, father): |
160 | + proxy.ProxyClient.__init__( |
161 | + self, command, rest, version, headers, data, father) |
162 | + # Why doesn't ProxyClient at least store this? |
163 | + self.version = version |
164 | + # We must avoid calling self.father.finish in the event that its |
165 | + # connection was already lost, i.e. if the original client |
166 | + # disconnects first (which is particularly likely in the case of |
167 | + # CONNECT). |
168 | + d = self.father.notifyFinish() |
169 | + d.addBoth(self.requestFinished) |
170 | + |
171 | + def connectionMade(self): |
172 | + proxy.ProxyClient.connectionMade(self) |
173 | + self.father.setChildClient(self) |
174 | + |
175 | + def sendCommand(self, command, path): |
176 | + # For some reason, HTTPClient.sendCommand doesn't preserve the |
177 | + # protocol version. |
178 | + self.transport.writeSequence( |
179 | + [command, b' ', path, b' ', self.version, b'\r\n']) |
180 | + |
181 | + def handleEndHeaders(self): |
182 | + self.father.handleEndHeaders() |
183 | + |
184 | + def sendData(self, data): |
185 | + self.transport.write(data) |
186 | + |
187 | + def endData(self): |
188 | + if self.transport is not None: |
189 | + self.transport.loseWriteConnection() |
190 | + |
191 | + def requestFinished(self, result): |
192 | + self._finished = True |
193 | + self.transport.loseConnection() |
194 | + |
195 | + |
196 | +class SnapProxyClientFactory(proxy.ProxyClientFactory): |
197 | + |
198 | + protocol = SnapProxyClient |
199 | + |
200 | + |
201 | +class SnapProxyRequest(http.Request): |
202 | + |
203 | + child_client = None |
204 | + _request_buffer = None |
205 | + _request_data_done = False |
206 | + |
207 | + def setChildClient(self, child_client): |
208 | + self.child_client = child_client |
209 | + if self._request_buffer is not None: |
210 | + self.child_client.sendData(self._request_buffer.getvalue()) |
211 | + self._request_buffer = None |
212 | + if self._request_data_done: |
213 | + self.child_client.endData() |
214 | + |
215 | + def allHeadersReceived(self, command, path, version): |
216 | + # Normally done in `requestReceived`, but we disable that since it |
217 | + # does other things we don't want. |
218 | + self.method, self.uri, self.clientproto = command, path, version |
219 | + self.client = self.channel.transport.getPeer() |
220 | + self.host = self.channel.transport.getHost() |
221 | + |
222 | + remote_parsed = urlparse(self.channel.factory.remote_url) |
223 | + request_parsed = urlparse(path) |
224 | + headers = self.getAllHeaders().copy() |
225 | + if b"host" not in headers and request_parsed.netloc: |
226 | + headers[b"host"] = request_parsed.netloc |
227 | + if remote_parsed.username: |
228 | + auth = (remote_parsed.username + ":" + |
229 | + remote_parsed.password).encode("ASCII") |
230 | + authHeader = b"Basic " + base64.b64encode(auth) |
231 | + headers[b"proxy-authorization"] = authHeader |
232 | + self.client_factory = SnapProxyClientFactory( |
233 | + command, path, version, headers, b"", self) |
234 | + reactor.connectTCP( |
235 | + remote_parsed.hostname, remote_parsed.port, self.client_factory) |
236 | + |
237 | + def requestReceived(self, command, path, version): |
238 | + # We do most of our work in `allHeadersReceived` instead. |
239 | + pass |
240 | + |
241 | + def rawDataReceived(self, data): |
242 | + if self.child_client is not None: |
243 | + if not self._request_data_done: |
244 | + self.child_client.sendData(data) |
245 | + else: |
246 | + if self._request_buffer is None: |
247 | + self._request_buffer = io.BytesIO() |
248 | + self._request_buffer.write(data) |
249 | + |
250 | + def handleEndHeaders(self): |
251 | + # Cut-down version of Request.write. We must avoid switching to |
252 | + # chunked encoding for the sake of CONNECT; since our actual |
253 | + # response data comes from another proxy, we can cut some corners. |
254 | + if self.startedWriting: |
255 | + return |
256 | + self.startedWriting = 1 |
257 | + l = [] |
258 | + l.append( |
259 | + self.clientproto + b" " + intToBytes(self.code) + b" " + |
260 | + self.code_message + b"\r\n") |
261 | + for name, values in self.responseHeaders.getAllRawHeaders(): |
262 | + for value in values: |
263 | + l.extend([name, b": ", value, b"\r\n"]) |
264 | + l.append(b"\r\n") |
265 | + self.transport.writeSequence(l) |
266 | + |
267 | + def write(self, data): |
268 | + if self.channel is not None: |
269 | + self.channel.resetTimeout() |
270 | + http.Request.write(self, data) |
271 | + |
272 | + def endData(self): |
273 | + if self.child_client is not None: |
274 | + self.child_client.endData() |
275 | + self._request_data_done = True |
276 | + |
277 | + |
278 | +@implementer(IHalfCloseableProtocol) |
279 | +class SnapProxy(http.HTTPChannel): |
280 | + """A channel that streams request data. |
281 | + |
282 | + The stock HTTPChannel isn't quite suitable for our needs, because it |
283 | + expects to read the entire request data before passing control to the |
284 | + request. This doesn't work well for CONNECT. |
285 | + """ |
286 | + |
287 | + requestFactory = SnapProxyRequest |
288 | + |
289 | + def checkPersistence(self, request, version): |
290 | + # ProxyClient.__init__ forces "Connection: close". |
291 | + return False |
292 | + |
293 | + def allHeadersReceived(self): |
294 | + http.HTTPChannel.allHeadersReceived(self) |
295 | + self.requests[-1].allHeadersReceived( |
296 | + self._command, self._path, self._version) |
297 | + if self._command == b"CONNECT": |
298 | + # This is a lie, but we don't want HTTPChannel to decide that |
299 | + # the request is finished just because a CONNECT request |
300 | + # (naturally) has no Content-Length. |
301 | + self.length = -1 |
302 | + |
303 | + def rawDataReceived(self, data): |
304 | + self.resetTimeout() |
305 | + if self.requests: |
306 | + self.requests[-1].rawDataReceived(data) |
307 | + |
308 | + def readConnectionLost(self): |
309 | + for request in self.requests: |
310 | + request.endData() |
311 | + |
312 | + def writeConnectionLost(self): |
313 | + pass |
314 | + |
315 | + |
316 | +class SnapProxyFactory(http.HTTPFactory): |
317 | + |
318 | + protocol = SnapProxy |
319 | + |
320 | + def __init__(self, manager, remote_url, *args, **kwargs): |
321 | + http.HTTPFactory.__init__(self, *args, **kwargs) |
322 | + self.manager = manager |
323 | + self.remote_url = remote_url |
324 | + |
325 | + def log(self, request): |
326 | + # Log requests to the build log rather than to Twisted. |
327 | + # Reimplement log formatting because there's no point logging the IP |
328 | + # here. |
329 | + referrer = http._escape(request.getHeader(b"referer") or b"-") |
330 | + agent = http._escape(request.getHeader(b"user-agent") or b"-") |
331 | + line = ( |
332 | + u'%(timestamp)s "%(method)s %(uri)s %(protocol)s" ' |
333 | + u'%(code)d %(length)s "%(referrer)s" "%(agent)s"\n' % { |
334 | + 'timestamp': self._logDateTime, |
335 | + 'method': http._escape(request.method), |
336 | + 'uri': http._escape(request.uri), |
337 | + 'protocol': http._escape(request.clientproto), |
338 | + 'code': request.code, |
339 | + 'length': request.sentLength or "-", |
340 | + 'referrer': referrer, |
341 | + 'agent': agent, |
342 | + }) |
343 | + self.manager._slave.log(line.encode("UTF-8")) |
344 | + |
345 | + |
346 | class SnapBuildState(DebianBuildState): |
347 | BUILD_SNAP = "BUILD_SNAP" |
348 | |
349 | @@ -47,9 +266,49 @@ |
350 | self.revocation_endpoint = extra_args.get("revocation_endpoint") |
351 | self.build_source_tarball = extra_args.get( |
352 | "build_source_tarball", False) |
353 | + self.proxy_service = None |
354 | |
355 | super(SnapBuildManager, self).initiate(files, chroot, extra_args) |
356 | |
357 | + def startProxy(self): |
358 | + """Start the local snap proxy, if necessary.""" |
359 | + if not self.proxy_url: |
360 | + return [] |
361 | + proxy_port = self._slave._config.get("snapmanager", "proxyport") |
362 | + proxy_factory = SnapProxyFactory(self, self.proxy_url, timeout=60) |
363 | + self.proxy_service = strports.service(proxy_port, proxy_factory) |
364 | + self.proxy_service.setServiceParent(self._slave.service) |
365 | + if self.backend_name == "lxd": |
366 | + proxy_host = self.backend.ipv4_network.ip |
367 | + else: |
368 | + proxy_host = "localhost" |
369 | + return ["--proxy-url", "http://{}:{}/".format(proxy_host, proxy_port)] |
370 | + |
371 | + def stopProxy(self): |
372 | + """Stop the local snap proxy, if necessary.""" |
373 | + if self.proxy_service is None: |
374 | + return |
375 | + self.proxy_service.disownServiceParent() |
376 | + self.proxy_service = None |
377 | + |
378 | + def revokeProxyToken(self): |
379 | + """Revoke builder proxy token.""" |
380 | + if not self.revocation_endpoint: |
381 | + return |
382 | + self._slave.log("Revoking proxy token...\n") |
383 | + url = urlparse(self.proxy_url) |
384 | + auth = "{}:{}".format(url.username, url.password) |
385 | + headers = { |
386 | + "Authorization": "Basic {}".format(base64.b64encode(auth)) |
387 | + } |
388 | + req = Request(self.revocation_endpoint, None, headers) |
389 | + req.get_method = lambda: "DELETE" |
390 | + try: |
391 | + urlopen(req) |
392 | + except (HTTPError, URLError) as e: |
393 | + self._slave.log( |
394 | + "Unable to revoke token for %s: %s" % (url.username, e)) |
395 | + |
396 | def status(self): |
397 | status_path = get_build_path(self.home, self._buildid, "status") |
398 | try: |
399 | @@ -78,8 +337,7 @@ |
400 | file=sys.stderr) |
401 | if self.build_url: |
402 | args.extend(["--build-url", self.build_url]) |
403 | - if self.proxy_url: |
404 | - args.extend(["--proxy-url", self.proxy_url]) |
405 | + args.extend(self.startProxy()) |
406 | if self.revocation_endpoint: |
407 | args.extend(["--revocation-endpoint", self.revocation_endpoint]) |
408 | if self.branch is not None: |
409 | @@ -95,6 +353,8 @@ |
410 | |
411 | def iterate_BUILD_SNAP(self, retcode): |
412 | """Finished building the snap.""" |
413 | + self.stopProxy() |
414 | + self.revokeProxyToken() |
415 | if retcode == RETCODE_SUCCESS: |
416 | self.gatherResults() |
417 | print("Returning build status: OK") |
418 | |
419 | === modified file 'lpbuildd/target/build_snap.py' |
420 | --- lpbuildd/target/build_snap.py 2018-06-05 02:06:04 +0000 |
421 | +++ lpbuildd/target/build_snap.py 2018-06-10 12:47:59 +0000 |
422 | @@ -5,30 +5,11 @@ |
423 | |
424 | __metaclass__ = type |
425 | |
426 | -import base64 |
427 | from collections import OrderedDict |
428 | import json |
429 | import logging |
430 | import os.path |
431 | import sys |
432 | -try: |
433 | - from urllib.error import ( |
434 | - HTTPError, |
435 | - URLError, |
436 | - ) |
437 | - from urllib.request import ( |
438 | - Request, |
439 | - urlopen, |
440 | - ) |
441 | - from urllib.parse import urlparse |
442 | -except ImportError: |
443 | - from urllib2 import ( |
444 | - HTTPError, |
445 | - Request, |
446 | - URLError, |
447 | - urlopen, |
448 | - ) |
449 | - from urlparse import urlparse |
450 | |
451 | from lpbuildd.target.operation import Operation |
452 | from lpbuildd.target.vcs import VCSOperationMixin |
453 | @@ -206,21 +187,6 @@ |
454 | cwd=os.path.join("/build", self.args.name), |
455 | env=env) |
456 | |
457 | - def revoke_token(self): |
458 | - """Revoke builder proxy token.""" |
459 | - logger.info("Revoking proxy token...") |
460 | - url = urlparse(self.args.proxy_url) |
461 | - auth = '{}:{}'.format(url.username, url.password) |
462 | - headers = { |
463 | - 'Authorization': 'Basic {}'.format(base64.b64encode(auth)) |
464 | - } |
465 | - req = Request(self.args.revocation_endpoint, None, headers) |
466 | - req.get_method = lambda: 'DELETE' |
467 | - try: |
468 | - urlopen(req) |
469 | - except (HTTPError, URLError): |
470 | - logger.exception('Unable to revoke token for %s', url.username) |
471 | - |
472 | def run(self): |
473 | try: |
474 | self.install() |
475 | @@ -234,7 +200,4 @@ |
476 | except Exception: |
477 | logger.exception('Build failed') |
478 | return RETCODE_FAILURE_BUILD |
479 | - finally: |
480 | - if self.args.revocation_endpoint is not None: |
481 | - self.revoke_token() |
482 | return 0 |
483 | |
484 | === modified file 'lpbuildd/tests/test_snap.py' |
485 | --- lpbuildd/tests/test_snap.py 2018-04-21 10:04:00 +0000 |
486 | +++ lpbuildd/tests/test_snap.py 2018-06-10 12:47:59 +0000 |
487 | @@ -10,10 +10,25 @@ |
488 | TempDir, |
489 | ) |
490 | from testtools import TestCase |
491 | +from testtools.content import text_content |
492 | +from testtools.deferredruntest import AsynchronousDeferredRunTest |
493 | +from twisted.internet import ( |
494 | + defer, |
495 | + reactor, |
496 | + utils, |
497 | + ) |
498 | +from twisted.web import ( |
499 | + http, |
500 | + proxy, |
501 | + resource, |
502 | + server, |
503 | + static, |
504 | + ) |
505 | |
506 | from lpbuildd.snap import ( |
507 | SnapBuildManager, |
508 | SnapBuildState, |
509 | + SnapProxyFactory, |
510 | ) |
511 | from lpbuildd.tests.fakeslave import FakeSlave |
512 | from lpbuildd.tests.matchers import HasWaitingFiles |
513 | @@ -35,6 +50,9 @@ |
514 | |
515 | class TestSnapBuildManagerIteration(TestCase): |
516 | """Run SnapBuildManager through its iteration steps.""" |
517 | + |
518 | + run_tests_with = AsynchronousDeferredRunTest.make_factory(timeout=5) |
519 | + |
520 | def setUp(self): |
521 | super(TestSnapBuildManagerIteration, self).setUp() |
522 | self.working_dir = self.useFixture(TempDir()).path |
523 | @@ -229,3 +247,64 @@ |
524 | self.assertEqual( |
525 | self.buildmanager.iterate, self.buildmanager.iterators[-1]) |
526 | self.assertFalse(self.slave.wasCalled("buildFail")) |
527 | + |
528 | + def getListenerURL(self, listener): |
529 | + port = listener.getHost().port |
530 | + return b"http://localhost:%d/" % port |
531 | + |
532 | + def startFakeRemoteEndpoint(self): |
533 | + remote_endpoint = resource.Resource() |
534 | + remote_endpoint.putChild("a", static.Data("a" * 1024, "text/plain")) |
535 | + remote_endpoint.putChild("b", static.Data("b" * 65536, "text/plain")) |
536 | + remote_endpoint_listener = reactor.listenTCP( |
537 | + 0, server.Site(remote_endpoint)) |
538 | + self.addCleanup(remote_endpoint_listener.stopListening) |
539 | + return remote_endpoint_listener |
540 | + |
541 | + def startFakeRemoteProxy(self): |
542 | + remote_proxy_factory = http.HTTPFactory() |
543 | + remote_proxy_factory.protocol = proxy.Proxy |
544 | + remote_proxy_listener = reactor.listenTCP(0, remote_proxy_factory) |
545 | + self.addCleanup(remote_proxy_listener.stopListening) |
546 | + return remote_proxy_listener |
547 | + |
548 | + def startLocalProxy(self, remote_url): |
549 | + proxy_factory = SnapProxyFactory( |
550 | + self.buildmanager, remote_url, timeout=60) |
551 | + proxy_listener = reactor.listenTCP(0, proxy_factory) |
552 | + self.addCleanup(proxy_listener.stopListening) |
553 | + return proxy_listener |
554 | + |
555 | + @defer.inlineCallbacks |
556 | + def assertCommandSuccess(self, command, extra_env=None): |
557 | + env = os.environ |
558 | + if extra_env is not None: |
559 | + env.update(extra_env) |
560 | + out, err, code = yield utils.getProcessOutputAndValue( |
561 | + command[0], command[1:], env=env, path=".") |
562 | + if code != 0: |
563 | + self.addDetail("stdout", text_content(out)) |
564 | + self.addDetail("stderr", text_content(err)) |
565 | + self.assertEqual(0, code) |
566 | + defer.returnValue(out) |
567 | + |
568 | + @defer.inlineCallbacks |
569 | + def test_fetch_via_proxy(self): |
570 | + remote_endpoint_listener = self.startFakeRemoteEndpoint() |
571 | + remote_endpoint_url = self.getListenerURL(remote_endpoint_listener) |
572 | + remote_proxy_listener = self.startFakeRemoteProxy() |
573 | + proxy_listener = self.startLocalProxy( |
574 | + self.getListenerURL(remote_proxy_listener)) |
575 | + out = yield self.assertCommandSuccess( |
576 | + [b"curl", remote_endpoint_url + b"a"], |
577 | + extra_env={b"http_proxy": self.getListenerURL(proxy_listener)}) |
578 | + self.assertEqual("a" * 1024, out) |
579 | + out = yield self.assertCommandSuccess( |
580 | + [b"curl", remote_endpoint_url + b"b"], |
581 | + extra_env={b"http_proxy": self.getListenerURL(proxy_listener)}) |
582 | + self.assertEqual("b" * 65536, out) |
583 | + |
584 | + # XXX cjwatson 2017-04-13: We should really test the HTTPS case as well, |
585 | + # but it's hard to see how to test that in a way that's independent of |
586 | + # the code under test since the stock twisted.web.proxy doesn't support |
587 | + # CONNECT. |
588 | |
589 | === modified file 'template-buildd-slave.conf' |
590 | --- template-buildd-slave.conf 2015-05-11 06:09:19 +0000 |
591 | +++ template-buildd-slave.conf 2018-06-10 12:47:59 +0000 |
592 | @@ -12,3 +12,6 @@ |
593 | |
594 | [translationtemplatesmanager] |
595 | resultarchive = translation-templates.tar.gz |
596 | + |
597 | +[snapmanager] |
598 | +proxyport = @SNAPPROXYPORT@ |
See also https:/ /bugs.launchpad .net/snapcraft/ +bug/1682534.