Merge lp:~alecu/ubuntuone-client/proxy-tunnel-server into lp:ubuntuone-client

Proposed by Alejandro J. Cura on 2012-02-28
Status: Merged
Approved by: Natalia Bidart on 2012-03-08
Approved revision: 1211
Merged at revision: 1202
Proposed branch: lp:~alecu/ubuntuone-client/proxy-tunnel-server
Merge into: lp:ubuntuone-client
Diff against target: 953 lines (+899/-1)
9 files modified
Makefile.am (+2/-1)
tests/proxy/__init__.py (+148/-0)
tests/proxy/ssl/dummy.cert (+19/-0)
tests/proxy/ssl/dummy.key (+16/-0)
tests/proxy/test_tunnel_server.py (+351/-0)
ubuntuone/proxy/__init__.py (+14/-0)
ubuntuone/proxy/common.py (+59/-0)
ubuntuone/proxy/logger.py (+37/-0)
ubuntuone/proxy/tunnel_server.py (+253/-0)
To merge this branch: bzr merge lp:~alecu/ubuntuone-client/proxy-tunnel-server
Reviewer Review Type Date Requested Status
Natalia Bidart 2012-02-28 Approve on 2012-03-08
Manuel de la Peña (community) 2012-02-28 Approve on 2012-03-07
Review via email: mp+95075@code.launchpad.net

Commit message

- The infrastructure for a QtNetwork based process to tunnel syncdaemon traffic thru proxies. (LP: #929207)

Description of the change

- The infrastructure for a QtNetwork based process to tunnel syncdaemon traffic thru proxies. (LP: #929207)

To post a comment you must log in.
Manuel de la Peña (mandel) wrote :

I dont think that the following code is correct:

787 + else:
788 + return QNetworkProxy(QNetworkProxy.HttpProxy,
789 + hostName=settings.get("host", ""),
790 + port=settings.get("port", 0),
791 + user=settings.get("username", ""),
792 + password=settings.get("password", ""))

The use of "" for the username and password will give problems when dealing with the auth proxies because the first attempt would be to use them and therefore you will get a 401 instead of a 407.

505 +
506 + def test_stop_but_never_connected(self):
507 + """Stop but it was never connected."""
508 + fake_protocol = FakeServerTunnelProtocol()
509 + client = tunnel_server.RemoteSocket(fake_protocol)
510 + yield client.stop()

There is a missing decorator in the test.

review: Needs Fixing
Alejandro J. Cura (alecu) wrote :

> I dont think that the following code is correct:
>
> 787 + else:
> 788 + return QNetworkProxy(QNetworkProxy.HttpProxy,
> 789 + hostName=settings.get("host", ""),
> 790 + port=settings.get("port", 0),
> 791 + user=settings.get("username", ""),
> 792 + password=settings.get("password", ""))
>
> The use of "" for the username and password will give problems when dealing
> with the auth proxies because the first attempt would be to use them and
> therefore you will get a 401 instead of a 407.

Those values are only used when the "settings" dictionary has no key with the given name.
And I'm using empty strings because that's what QNetworkProxy uses if no parameter is given:
 * https://qt-project.org/doc/qt-4.8/qnetworkproxy.html#QNetworkProxy-2

Since the current version of the tunnel is not getting the authentication, I think it's safe to use it like this. Perhaps we can add a comment or a TODO for it, and we can fix it in an upcoming branch to add authentication.

> 506 + def test_stop_but_never_connected(self):
> 507 + """Stop but it was never connected."""
> 508 + fake_protocol = FakeServerTunnelProtocol()
> 509 + client = tunnel_server.RemoteSocket(fake_protocol)
> 510 + yield client.stop()
>
> There is a missing decorator in the test.

Great catch! Thanks, I'm fixing it too.

1208. By Alejandro J. Cura on 2012-03-01

fixes requested on the review

1209. By Alejandro J. Cura on 2012-03-02

Use a QTimer to work around flaky signals

1210. By Alejandro J. Cura on 2012-03-06

merged with trunk

Manuel de la Peña (mandel) wrote :

I see no more issues.

review: Approve
Natalia Bidart (nataliabidart) wrote :

* Can't we re-use teh code from ubuntu-sso-client instead of defnining:

 +def build_proxy(settings):

?

* Most of our projects and source file use this syntax for file encoding:

# -*- coding: utf8 -*-

would you consider using that form?

* There should be spaces around the % in "HTTP/1.0 %d %s"%(code, description)

The branch looks good!

review: Needs Fixing
Alejandro J. Cura (alecu) wrote :

> * Can't we re-use teh code from ubuntu-sso-client instead of defnining:
>
> +def build_proxy(settings):
>
> ?

The bits of code that build the QNetworkProxy instance and configure the proxy settings in ussoc are currently in flux, because mandel is changing it to use a QNetworkProxyFactory and to query the proxy settings on each request. As soon as mandel's branch for ussoc lands we should be able to create a different branch to make u1-client use those bits from syncdaemon, so I think it makes sense to keep this branch as is, and refactor later.

> * Most of our projects and source file use this syntax for file encoding:
>
> # -*- coding: utf8 -*-
>
> would you consider using that form?

Sure, fixing that.

> * There should be spaces around the % in "HTTP/1.0 %d %s"%(code, description)

Fixing that too.

thanks for the review!

1211. By Alejandro J. Cura on 2012-03-07

fixes requested in review

Natalia Bidart (nataliabidart) wrote :

Looks great!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'Makefile.am'
2--- Makefile.am 2012-02-10 15:07:59 +0000
3+++ Makefile.am 2012-03-07 23:48:19 +0000
4@@ -53,7 +53,8 @@
5 test: logging.conf $(clientdefs_DATA) Makefile
6 echo "$(PYTHONPATH)"
7 if test "x$(builddir)" == "x$(srcdir)"; then \
8- PYTHONPATH="$(PYTHONPATH)" u1trial -r $(REACTOR) -p tests/platform/windows tests; \
9+ PYTHONPATH="$(PYTHONPATH)" u1trial -r $(REACTOR) -p tests/platform/windows,tests/proxy tests; \
10+ PYTHONPATH="$(PYTHONPATH)" u1trial -r qt4 -p tests/platform/windows tests/proxy; \
11 fi
12 rm -rf _trial_temp
13
14
15=== added directory 'tests/proxy'
16=== added file 'tests/proxy/__init__.py'
17--- tests/proxy/__init__.py 1970-01-01 00:00:00 +0000
18+++ tests/proxy/__init__.py 2012-03-07 23:48:19 +0000
19@@ -0,0 +1,148 @@
20+# Copyright 2012 Canonical Ltd.
21+#
22+# This program is free software: you can redistribute it and/or modify it
23+# under the terms of the GNU General Public License version 3, as published
24+# by the Free Software Foundation.
25+#
26+# This program is distributed in the hope that it will be useful, but
27+# WITHOUT ANY WARRANTY; without even the implied warranties of
28+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
29+# PURPOSE. See the GNU General Public License for more details.
30+#
31+# You should have received a copy of the GNU General Public License along
32+# with this program. If not, see <http://www.gnu.org/licenses/>.
33+"""Tests for the Ubuntu One proxy support."""
34+
35+from os import path
36+from StringIO import StringIO
37+
38+from twisted.application import internet, service
39+from twisted.internet import defer, ssl
40+from twisted.web import http, resource, server
41+
42+SAMPLE_CONTENT = "hello world!"
43+SIMPLERESOURCE = "simpleresource"
44+DUMMY_KEY_FILENAME = "dummy.key"
45+DUMMY_CERT_FILENAME = "dummy.cert"
46+
47+
48+class SaveHTTPChannel(http.HTTPChannel):
49+ """A save protocol to be used in tests."""
50+
51+ protocolInstance = None
52+
53+ # pylint: disable=C0103
54+ def connectionMade(self):
55+ """Keep track of the given protocol."""
56+ SaveHTTPChannel.protocolInstance = self
57+ http.HTTPChannel.connectionMade(self)
58+
59+
60+class SaveSite(server.Site):
61+ """A site that let us know when it's closed."""
62+
63+ protocol = SaveHTTPChannel
64+
65+ def __init__(self, *args, **kwargs):
66+ """Create a new instance."""
67+ server.Site.__init__(self, *args, **kwargs)
68+ # we disable the timeout in the tests, we will deal with it manually.
69+ # pylint: disable=C0103
70+ self.timeOut = None
71+
72+
73+class BaseMockWebServer(object):
74+ """A mock webserver for testing"""
75+
76+ def __init__(self):
77+ """Start up this instance."""
78+ self.root = self.get_root_resource()
79+ self.site = SaveSite(self.root)
80+ application = service.Application('web')
81+ self.service_collection = service.IServiceCollection(application)
82+ #pylint: disable=E1101
83+ self.tcpserver = internet.TCPServer(0, self.site)
84+ self.tcpserver.setServiceParent(self.service_collection)
85+ self.sslserver = internet.SSLServer(0, self.site, self.get_context())
86+ self.sslserver.setServiceParent(self.service_collection)
87+ self.service_collection.startService()
88+
89+ def get_dummy_path(self, filename):
90+ """Path pointing at the dummy certificate files."""
91+ base_path = path.dirname(__file__)
92+ return path.join(base_path, "ssl", filename)
93+
94+ def get_context(self):
95+ """Return an ssl context."""
96+ key_path = self.get_dummy_path(DUMMY_KEY_FILENAME)
97+ cert_path = self.get_dummy_path(DUMMY_CERT_FILENAME)
98+ return ssl.DefaultOpenSSLContextFactory(key_path, cert_path)
99+
100+ def get_root_resource(self):
101+ """Get the root resource with all the children."""
102+ raise NotImplementedError
103+
104+ def get_iri(self):
105+ """Build the iri for this mock server."""
106+ #pylint: disable=W0212
107+ port_num = self.tcpserver._port.getHost().port
108+ return u"http://0.0.0.0:%d/" % port_num
109+
110+ def get_ssl_iri(self):
111+ """Build the iri for the ssl mock server."""
112+ #pylint: disable=W0212
113+ port_num = self.sslserver._port.getHost().port
114+ return u"https://0.0.0.0:%d/" % port_num
115+
116+ def stop(self):
117+ """Shut it down."""
118+ #pylint: disable=E1101
119+ if self.site.protocol.protocolInstance:
120+ self.site.protocol.protocolInstance.timeoutConnection()
121+ return self.service_collection.stopService()
122+
123+
124+class SimpleResource(resource.Resource):
125+ """A simple web resource."""
126+
127+ def __init__(self):
128+ """Initialize this mock resource."""
129+ resource.Resource.__init__(self)
130+ self.rendered = defer.Deferred()
131+
132+ def render_GET(self, request):
133+ """Make a bit of html out of the resource's content."""
134+ if not self.rendered.called:
135+ self.rendered.callback(None)
136+ return SAMPLE_CONTENT
137+
138+
139+class MockWebServer(BaseMockWebServer):
140+ """A mock webserver."""
141+
142+ def __init__(self):
143+ """Initialize this mock server."""
144+ self.simple_resource = SimpleResource()
145+ super(MockWebServer, self).__init__()
146+
147+ def get_root_resource(self):
148+ """Get the root resource with all the children."""
149+ root = resource.Resource()
150+ root.putChild(SIMPLERESOURCE, self.simple_resource)
151+ return root
152+
153+
154+class FakeTransport(StringIO):
155+ """A fake transport that stores everything written to it."""
156+
157+ connected = True
158+ disconnecting = False
159+
160+ def loseConnection(self):
161+ """Mark the connection as lost."""
162+ self.connected = False
163+ self.disconnecting = True
164+
165+ def getPeer(self):
166+ """Return the peer IAddress."""
167+ return None
168
169=== added directory 'tests/proxy/ssl'
170=== added file 'tests/proxy/ssl/dummy.cert'
171--- tests/proxy/ssl/dummy.cert 1970-01-01 00:00:00 +0000
172+++ tests/proxy/ssl/dummy.cert 2012-03-07 23:48:19 +0000
173@@ -0,0 +1,19 @@
174+-----BEGIN CERTIFICATE-----
175+MIIDEDCCAnmgAwIBAgIJAM/bIJ77awBCMA0GCSqGSIb3DQEBBQUAMIGgMQswCQYD
176+VQQGEwJBUjETMBEGA1UECAwKRmFrZSBTdGF0ZTESMBAGA1UEBwwJRmFrZSBDaXR5
177+MRUwEwYDVQQKDAxGYWtlIENvbXBhbnkxFjAUBgNVBAsMDUZha2UgRGl2aXNpb24x
178+EjAQBgNVBAMMCUZha2UgTmFtZTElMCMGCSqGSIb3DQEJARYWZmFrZUBlbWFpbC5h
179+ZGRyZXNzLm5vdDAeFw0xMjAyMjIxOTI0MjBaFw0yMjAyMjMxOTI0MjBaMIGgMQsw
180+CQYDVQQGEwJBUjETMBEGA1UECAwKRmFrZSBTdGF0ZTESMBAGA1UEBwwJRmFrZSBD
181+aXR5MRUwEwYDVQQKDAxGYWtlIENvbXBhbnkxFjAUBgNVBAsMDUZha2UgRGl2aXNp
182+b24xEjAQBgNVBAMMCUZha2UgTmFtZTElMCMGCSqGSIb3DQEJARYWZmFrZUBlbWFp
183+bC5hZGRyZXNzLm5vdDCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA0hbliGty
184+HwfZixU609UHBQdbfO+oObrPIrIawWX5FxD6KhX4ei23idmpyYEcXLK4ivNlT4dW
185+27bvhtpf6/FBbu9e1YdwcdDNoXajr9Ia4NZJyANgo9b5UIsnyTc45NlnpZgRg5zc
186+Oz7Vwwr4qf6r1ljK/I2mAO7rlpH5Ak9J+RkCAwEAAaNQME4wHQYDVR0OBBYEFLwr
187+ps/JLNcfpSuuylMnkvImVvkgMB8GA1UdIwQYMBaAFLwrps/JLNcfpSuuylMnkvIm
188+VvkgMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADgYEAWYDBAr0MgpnBxIne
189+WRz8MX0/c7IqrEuZCYMSGnU7PoX3GdNk1Lkif1ufELKSoG8jY16CDgEl26GPxA1k
190+Tho7MWSLikLbuQYJs2saF9by0Y/Mrau0auxEnpHZ7pkybeKFnrIqiNKvTVMnjo5T
191+FMET5qEOKKvp9IOnezCYX1nYXyY=
192+-----END CERTIFICATE-----
193
194=== added file 'tests/proxy/ssl/dummy.key'
195--- tests/proxy/ssl/dummy.key 1970-01-01 00:00:00 +0000
196+++ tests/proxy/ssl/dummy.key 2012-03-07 23:48:19 +0000
197@@ -0,0 +1,16 @@
198+-----BEGIN PRIVATE KEY-----
199+MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBANIW5Yhrch8H2YsV
200+OtPVBwUHW3zvqDm6zyKyGsFl+RcQ+ioV+Hott4nZqcmBHFyyuIrzZU+HVtu274ba
201+X+vxQW7vXtWHcHHQzaF2o6/SGuDWScgDYKPW+VCLJ8k3OOTZZ6WYEYOc3Ds+1cMK
202++Kn+q9ZYyvyNpgDu65aR+QJPSfkZAgMBAAECgYBSxFh7TTExjmsjAyMg700LqyFc
203+8CHLVJBkL9ygkqb2cmbMC8nPgJFNSqY8T5Q35OUVQNyJ31zVxJVLAF9H2c0Xy48K
204+IkbS/hntyqlJYK1yfTbTHkDiweToE3Lm+55Do1TX04AyvBrwA1O/jNGi4xIlUEAy
205+1Bs8MrJ1E/j/XDn9/QJBAOuhPTgG3F7bKuBrQzv98CvC5o2Txf3vLY8nL8V24b3l
206+XgqzkDLhUxReBmmkGxZfKAju3+gXFvGGpbP7V8zShg8CQQDkQGs7kArFq/KR/GCh
207+CAmJaDWy4LJkSqzDHoJbTrS7YuqN6X6mW1xPRnWpYSxae38fJsCpG3Vq8Mv1Zl32
208+VPZXAkEAsAeE9JYri7GwFngLgoXzJr4z/xCmmU5VetyLk7l8a6Eu4E/FKj2rE0wq
209+/kDa+5ubDRFntLuLKGSu5gafUST1gQJABhdmBTfp4a6eEaFPntyNDJq4XCa8/Ao2
210+JBrrVa57Ckkwg0sI8z2a8A6sUzHhsiR7lwQ8vgaakpkMiGcL+Of5jwJAA/qX3PW+
211+9JXbjWxpgh7FHnZJNRZ8xSe47REGA7qS/nIlV9iRuf/9M+k3A5VqitfFxrjPwSyI
212+rvKTYkk13dL4hg==
213+-----END PRIVATE KEY-----
214
215=== added file 'tests/proxy/test_tunnel_server.py'
216--- tests/proxy/test_tunnel_server.py 1970-01-01 00:00:00 +0000
217+++ tests/proxy/test_tunnel_server.py 2012-03-07 23:48:19 +0000
218@@ -0,0 +1,351 @@
219+# Copyright 2012 Canonical Ltd.
220+#
221+# This program is free software: you can redistribute it and/or modify it
222+# under the terms of the GNU General Public License version 3, as published
223+# by the Free Software Foundation.
224+#
225+# This program is distributed in the hope that it will be useful, but
226+# WITHOUT ANY WARRANTY; without even the implied warranties of
227+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
228+# PURPOSE. See the GNU General Public License for more details.
229+#
230+# You should have received a copy of the GNU General Public License along
231+# with this program. If not, see <http://www.gnu.org/licenses/>.
232+"""Tests for the proxy tunnel."""
233+
234+from urlparse import urlparse
235+
236+from twisted.internet import defer, protocol, reactor
237+
238+from ubuntuone.devtools.testcases.squid import SquidTestCase
239+
240+from tests.proxy import (
241+ FakeTransport,
242+ MockWebServer,
243+ SAMPLE_CONTENT,
244+ SIMPLERESOURCE,
245+)
246+from ubuntuone.proxy import tunnel_server
247+from ubuntuone.proxy.tunnel_server import CRLF
248+
249+
250+FAKE_SESSION_TEMPLATE = (
251+ "CONNECT {0.netloc} HTTP/1.0" + CRLF +
252+ "Header1: value1" + CRLF +
253+ "Header2: value2" + CRLF +
254+ CRLF +
255+ "GET {0.path} HTTP/1.0" + CRLF + CRLF
256+)
257+
258+
259+class DisconnectingProtocol(protocol.Protocol):
260+ """A protocol that just disconnects."""
261+
262+ def connectionMade(self):
263+ """Upon connecting: just disconnect."""
264+ self.transport.loseConnection()
265+
266+
267+class DisconnectingClientFactory(protocol.ClientFactory):
268+ """A factory that fires a deferred on connection."""
269+
270+ def __init__(self):
271+ """Initialize this instance."""
272+ self.connected = defer.Deferred()
273+
274+ def buildProtocol(self, addr):
275+ """The connection was made."""
276+ proto = DisconnectingProtocol()
277+ if not self.connected.called:
278+ self.connected.callback(proto)
279+ return proto
280+
281+
282+class FakeProtocol(protocol.Protocol):
283+ """A protocol that forwards some data."""
284+
285+ def __init__(self, factory, data):
286+ """Initialize this fake."""
287+ self.factory = factory
288+ self.data = data
289+ self.received_data = []
290+
291+ def connectionMade(self):
292+ """Upon connection: send the stored data."""
293+ self.transport.write(self.data)
294+
295+ def dataReceived(self, data):
296+ """Some data was received."""
297+ self.received_data.append(data)
298+
299+ def connectionLost(self, reason):
300+ """The connection was lost, return the response."""
301+ response = "".join(self.received_data)
302+ if not self.factory.response.called:
303+ self.factory.response.callback(response)
304+
305+
306+class FakeClientFactory(protocol.ClientFactory):
307+ """A factory that forwards some data to the protocol."""
308+
309+ def __init__(self, data):
310+ """Initialize this fake."""
311+ self.data = data
312+ self.response = defer.Deferred()
313+
314+ def buildProtocol(self, addr):
315+ """The connection was made."""
316+ return FakeProtocol(self, self.data)
317+
318+
319+class TunnelIntegrationTestCase(SquidTestCase):
320+ """Basic tunnel integration tests."""
321+
322+ timeout = 3
323+
324+ @defer.inlineCallbacks
325+ def setUp(self):
326+ """Initialize this testcase."""
327+ yield super(TunnelIntegrationTestCase, self).setUp()
328+ self.ws = MockWebServer()
329+ self.addCleanup(self.ws.stop)
330+ self.dest_url = self.ws.get_iri().encode("utf-8") + SIMPLERESOURCE
331+ self.tunnel_server = tunnel_server.TunnelServer()
332+ self.addCleanup(self.tunnel_server.shutdown)
333+
334+ def test_init(self):
335+ """The tunnel is started."""
336+ self.assertNotEqual(self.tunnel_server.port, 0)
337+
338+ @defer.inlineCallbacks
339+ def test_accepts_connections(self):
340+ """The tunnel accepts incoming connections."""
341+ ncf = DisconnectingClientFactory()
342+ reactor.connectTCP("0.0.0.0", self.tunnel_server.port, ncf)
343+ yield ncf.connected
344+
345+ @defer.inlineCallbacks
346+ def test_complete_connection(self):
347+ """Test from the tunnel server down."""
348+ url = urlparse(self.dest_url)
349+ client = FakeClientFactory(FAKE_SESSION_TEMPLATE.format(url))
350+ reactor.connectTCP("0.0.0.0", self.tunnel_server.port, client)
351+ response = yield client.response
352+ self.assertIn(SAMPLE_CONTENT, response)
353+
354+
355+class FakeClient(object):
356+ """A fake destination client."""
357+
358+ protocol = None
359+ connection_result = defer.succeed(True)
360+
361+ def connect(self, hostport):
362+ """Establish a connection with the other end."""
363+ return self.connection_result
364+
365+ def write(self, data):
366+ """Write some data to the other end."""
367+ if data == 'GET /simpleresource HTTP/1.0\r\n\r\n':
368+ self.protocol.transport.write(SAMPLE_CONTENT)
369+
370+ def stop(self):
371+ """Stop this fake client."""
372+
373+
374+class ServerTunnelProtocolTestCase(SquidTestCase):
375+ """Tests for the ServerTunnelProtocol."""
376+
377+ @defer.inlineCallbacks
378+ def setUp(self):
379+ """Initialize this test instance."""
380+ yield super(ServerTunnelProtocolTestCase, self).setUp()
381+ self.ws = MockWebServer()
382+ self.addCleanup(self.ws.stop)
383+ self.dest_url = self.ws.get_iri().encode("utf-8") + SIMPLERESOURCE
384+ self.transport = FakeTransport()
385+ self.fake_client = FakeClient()
386+ self.proto = tunnel_server.ServerTunnelProtocol(
387+ lambda _: self.fake_client)
388+ self.fake_client.protocol = self.proto
389+ self.proto.transport = self.transport
390+
391+ def test_broken_request(self):
392+ """Broken request."""
393+ self.proto.dataReceived("Broken request." + CRLF)
394+ self.assertTrue(self.transport.getvalue().startswith("HTTP/1.0 400 "),
395+ "A broken request must fail.")
396+
397+ def test_wrong_method(self):
398+ """Wrong method."""
399+ self.proto.dataReceived("GET http://slashdot.org HTTP/1.0" + CRLF)
400+ self.assertTrue(self.transport.getvalue().startswith("HTTP/1.0 405 "),
401+ "Using a wrong method fails.")
402+
403+ def test_invalid_http_version(self):
404+ """Invalid HTTP version."""
405+ self.proto.dataReceived("CONNECT 127.0.0.1:9999 HTTP/1.1" + CRLF)
406+ self.assertTrue(self.transport.getvalue().startswith("HTTP/1.0 505 "),
407+ "Invalid http version is not allowed.")
408+
409+ def test_connection_is_established(self):
410+ """The response code is sent."""
411+ expected = "HTTP/1.0 200 Proxy connection established" + CRLF
412+ self.proto.dataReceived("CONNECT 127.0.0.1:9999 HTTP/1.0" + CRLF * 2)
413+ self.assertTrue(self.transport.getvalue().startswith(expected),
414+ "First line must be the response status")
415+
416+ def test_connection_fails(self):
417+ """The connection to the other end fails, and it's handled."""
418+ error = tunnel_server.ConnectionError()
419+ self.patch(self.fake_client, "connection_result", defer.fail(error))
420+ expected = "HTTP/1.0 500 Connection error" + CRLF
421+ self.proto.dataReceived("CONNECT 127.0.0.1:9999 HTTP/1.0" + CRLF * 2)
422+ self.assertTrue(self.transport.getvalue().startswith(expected),
423+ "The connection should fail at this point.")
424+
425+ def test_headers_stored(self):
426+ """The request headers are stored."""
427+ expected = [
428+ ("Header1", "value1"),
429+ ("Header2", "value2"),
430+ ]
431+ self.proto.dataReceived("CONNECT 127.0.0.1:9999 HTTP/1.0" + CRLF +
432+ "Header1: value1" + CRLF +
433+ "Header2: value2" + CRLF + CRLF)
434+ self.assertEqual(self.proto.received_headers, expected)
435+
436+ def test_successful_connect(self):
437+ """A successful connect thru the tunnel."""
438+ url = urlparse(self.dest_url)
439+ data = FAKE_SESSION_TEMPLATE.format(url)
440+ self.proto.dataReceived(data)
441+ lines = self.transport.getvalue().split(CRLF)
442+ self.assertEqual(lines[-1], SAMPLE_CONTENT)
443+
444+ def test_header_split(self):
445+ """Test a header with many colons."""
446+ self.proto.header_line("key: host:port")
447+ self.assertIn("key", dict(self.proto.received_headers))
448+
449+
450+class FakeServerTunnelProtocol(object):
451+ """A fake ServerTunnelProtocol."""
452+
453+ def __init__(self):
454+ """Initialize this fake tunnel."""
455+ self.response_received = defer.Deferred()
456+
457+ def response_data_received(self, data):
458+ """Fire the response deferred."""
459+ if not self.response_received.called:
460+ self.response_received.callback(data)
461+
462+ def remote_disconnected(self):
463+ """The remote server disconnected."""
464+
465+
466+class RemoteSocketTestCase(SquidTestCase):
467+ """Tests for the client that connects to the other side."""
468+
469+ timeout = 3
470+ get_proxy_settings = lambda _: {}
471+
472+ @defer.inlineCallbacks
473+ def setUp(self):
474+ """Initialize this testcase."""
475+ yield super(RemoteSocketTestCase, self).setUp()
476+ self.ws = MockWebServer()
477+ self.addCleanup(self.ws.stop)
478+ self.dest_url = self.ws.get_iri().encode("utf-8") + SIMPLERESOURCE
479+
480+ self.addCleanup(tunnel_server.QNetworkProxy.setApplicationProxy,
481+ tunnel_server.QNetworkProxy.applicationProxy())
482+ tunnel_server.QNetworkProxy.setApplicationProxy(
483+ tunnel_server.build_proxy(self.get_proxy_settings()))
484+
485+ def test_invalid_port(self):
486+ """A request with an invalid port fails with a 400."""
487+ protocol = tunnel_server.ServerTunnelProtocol(
488+ tunnel_server.RemoteSocket)
489+ protocol.transport = FakeTransport()
490+ protocol.dataReceived("CONNECT 127.0.0.1:wrong_port HTTP/1.0" +
491+ CRLF * 2)
492+
493+ status_line = protocol.transport.getvalue()
494+ self.assertTrue(status_line.startswith("HTTP/1.0 400 "),
495+ "The port must be an integer.")
496+
497+ @defer.inlineCallbacks
498+ def test_connection_is_finished_when_stopping(self):
499+ """The client disconnects when requested."""
500+ fake_protocol = FakeServerTunnelProtocol()
501+ client = tunnel_server.RemoteSocket(fake_protocol)
502+ url = urlparse(self.dest_url)
503+ yield client.connect(url.netloc)
504+ yield client.stop()
505+
506+ @defer.inlineCallbacks
507+ def test_stop_but_never_connected(self):
508+ """Stop but it was never connected."""
509+ fake_protocol = FakeServerTunnelProtocol()
510+ client = tunnel_server.RemoteSocket(fake_protocol)
511+ yield client.stop()
512+
513+ @defer.inlineCallbacks
514+ def test_client_write(self):
515+ """Data written to the client is sent to the other side."""
516+ fake_protocol = FakeServerTunnelProtocol()
517+ client = tunnel_server.RemoteSocket(fake_protocol)
518+ self.addCleanup(client.stop)
519+ url = urlparse(self.dest_url)
520+ yield client.connect(url.netloc)
521+ client.write("GET /simpleresource HTTP/1.0" + CRLF * 2)
522+ yield self.ws.simple_resource.rendered
523+
524+ @defer.inlineCallbacks
525+ def test_client_read(self):
526+ """Data received by the client is written into the transport."""
527+ fake_protocol = FakeServerTunnelProtocol()
528+ client = tunnel_server.RemoteSocket(fake_protocol)
529+ self.addCleanup(client.stop)
530+ url = urlparse(self.dest_url)
531+ yield client.connect(url.netloc)
532+ client.write("GET /simpleresource HTTP/1.0" + CRLF * 2)
533+ yield self.ws.simple_resource.rendered
534+ data = yield fake_protocol.response_received
535+ _headers, content = str(data).split(CRLF * 2, 1)
536+ self.assertEqual(content, SAMPLE_CONTENT)
537+
538+
539+class AnonProxyRemoteSocketTestCase(RemoteSocketTestCase):
540+ """Tests for the client going thru an anonymous proxy."""
541+
542+ get_proxy_settings = RemoteSocketTestCase.get_nonauth_proxy_settings
543+
544+ def parse_headers(self, raw_headers):
545+ """Parse the headers."""
546+ lines = raw_headers.split(CRLF)
547+ header_lines = lines[1:]
548+ headers_pairs = (l.split(":", 1) for l in header_lines)
549+ return dict((k.lower(), v.strip()) for k, v in headers_pairs)
550+
551+ @defer.inlineCallbacks
552+ def test_verify_client_uses_proxy(self):
553+ """Verify that the client uses the proxy."""
554+ fake_protocol = FakeServerTunnelProtocol()
555+ client = tunnel_server.RemoteSocket(fake_protocol)
556+ self.addCleanup(client.stop)
557+ url = urlparse(self.dest_url)
558+ yield client.connect(url.netloc)
559+ client.write("GET /simpleresource HTTP/1.0" + CRLF * 2)
560+ yield self.ws.simple_resource.rendered
561+ data = yield fake_protocol.response_received
562+ raw_headers, _content = str(data).split(CRLF * 2, 1)
563+ self.parse_headers(raw_headers)
564+
565+
566+class AuthenticatedProxyRemoteSocketTestCase(AnonProxyRemoteSocketTestCase):
567+ """Tests for the client going thru an authenticated proxy."""
568+
569+ get_proxy_settings = RemoteSocketTestCase.get_auth_proxy_settings
570
571=== added directory 'ubuntuone/proxy'
572=== added file 'ubuntuone/proxy/__init__.py'
573--- ubuntuone/proxy/__init__.py 1970-01-01 00:00:00 +0000
574+++ ubuntuone/proxy/__init__.py 2012-03-07 23:48:19 +0000
575@@ -0,0 +1,14 @@
576+# Copyright 2012 Canonical Ltd.
577+#
578+# This program is free software: you can redistribute it and/or modify it
579+# under the terms of the GNU General Public License version 3, as published
580+# by the Free Software Foundation.
581+#
582+# This program is distributed in the hope that it will be useful, but
583+# WITHOUT ANY WARRANTY; without even the implied warranties of
584+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
585+# PURPOSE. See the GNU General Public License for more details.
586+#
587+# You should have received a copy of the GNU General Public License along
588+# with this program. If not, see <http://www.gnu.org/licenses/>.
589+"""Ubuntu One proxy support."""
590
591=== added file 'ubuntuone/proxy/common.py'
592--- ubuntuone/proxy/common.py 1970-01-01 00:00:00 +0000
593+++ ubuntuone/proxy/common.py 2012-03-07 23:48:19 +0000
594@@ -0,0 +1,59 @@
595+# -*- coding: utf8 -*-
596+#
597+# Copyright 2012 Canonical Ltd.
598+#
599+# This program is free software: you can redistribute it and/or modify it
600+# under the terms of the GNU General Public License version 3, as published
601+# by the Free Software Foundation.
602+#
603+# This program is distributed in the hope that it will be useful, but
604+# WITHOUT ANY WARRANTY; without even the implied warranties of
605+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
606+# PURPOSE. See the GNU General Public License for more details.
607+#
608+# You should have received a copy of the GNU General Public License along
609+# with this program. If not, see <http://www.gnu.org/licenses/>.
610+"""Common classes to the tunnel client and server."""
611+
612+from twisted.protocols import basic
613+
614+CRLF = "\r\n"
615+
616+
617+class BaseTunnelProtocol(basic.LineReceiver):
618+ """CONNECT base protocol for tunnelling connections."""
619+
620+ delimiter = CRLF
621+
622+ def __init__(self):
623+ """Initialize this protocol."""
624+ self._first_line = True
625+ self.received_headers = []
626+
627+ def header_line(self, line):
628+ """Handle each header line received."""
629+ key, value = line.split(":", 1)
630+ value = value.strip()
631+ self.received_headers.append((key, value))
632+
633+ def lineReceived(self, line):
634+ """Process a line in the header."""
635+ if self._first_line:
636+ self._first_line = False
637+ self.handle_first_line(line)
638+ else:
639+ if line:
640+ self.header_line(line)
641+ else:
642+ self.setRawMode()
643+ self.headers_done()
644+
645+ def remote_disconnected(self):
646+ """The remote end closed the connection."""
647+ self.transport.loseConnection()
648+
649+ def format_headers(self, headers):
650+ """Format some headers as a few response lines."""
651+ return "".join("%s: %s" % item + CRLF for item in headers.items())
652+
653+
654
655=== added file 'ubuntuone/proxy/logger.py'
656--- ubuntuone/proxy/logger.py 1970-01-01 00:00:00 +0000
657+++ ubuntuone/proxy/logger.py 2012-03-07 23:48:19 +0000
658@@ -0,0 +1,37 @@
659+# ubuntuone.syncdaemon.logger - logging utilities
660+#
661+# Copyright 2009-2012 Canonical Ltd.
662+#
663+# This program is free software: you can redistribute it and/or modify it
664+# under the terms of the GNU General Public License version 3, as published
665+# by the Free Software Foundation.
666+#
667+# This program is distributed in the hope that it will be useful, but
668+# WITHOUT ANY WARRANTY; without even the implied warranties of
669+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
670+# PURPOSE. See the GNU General Public License for more details.
671+#
672+# You should have received a copy of the GNU General Public License along
673+# with this program. If not, see <http://www.gnu.org/licenses/>.
674+
675+"""SyncDaemon logging utilities and config."""
676+
677+import logging
678+import os
679+
680+from ubuntuone.logger import (
681+ _DEBUG_LOG_LEVEL,
682+ basic_formatter,
683+ CustomRotatingFileHandler,
684+)
685+
686+from ubuntuone.platform.xdg_base_directory import ubuntuone_log_dir
687+
688+
689+LOGFILENAME = os.path.join(ubuntuone_log_dir, 'proxy.log')
690+logger = logging.getLogger("ubuntuone.proxy")
691+logger.setLevel(_DEBUG_LOG_LEVEL)
692+handler = CustomRotatingFileHandler(filename=LOGFILENAME)
693+handler.setFormatter(basic_formatter)
694+handler.setLevel(_DEBUG_LOG_LEVEL)
695+logger.addHandler(handler)
696
697=== added file 'ubuntuone/proxy/tunnel_server.py'
698--- ubuntuone/proxy/tunnel_server.py 1970-01-01 00:00:00 +0000
699+++ ubuntuone/proxy/tunnel_server.py 2012-03-07 23:48:19 +0000
700@@ -0,0 +1,253 @@
701+# -*- coding: utf8 -*-
702+#
703+# Copyright 2012 Canonical Ltd.
704+#
705+# This program is free software: you can redistribute it and/or modify it
706+# under the terms of the GNU General Public License version 3, as published
707+# by the Free Software Foundation.
708+#
709+# This program is distributed in the hope that it will be useful, but
710+# WITHOUT ANY WARRANTY; without even the implied warranties of
711+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
712+# PURPOSE. See the GNU General Public License for more details.
713+#
714+# You should have received a copy of the GNU General Public License along
715+# with this program. If not, see <http://www.gnu.org/licenses/>.
716+"""A tunnel through proxies.
717+
718+The layers in a tunneled proxied connection:
719+
720+↓ tunnelclient - initiates tcp to tunnelserver, request outward connection
721+↕ client protocol - started after the tunneclient gets connected
722+---process boundary---
723+↕ tunnelserver - creates a tunnel instance per incoming connection
724+↕ tunnel - hold a qtcpsocket to tunnelclient, and srvtunnelproto to the remote
725+↕ servertunnelprotocol - gets CONNECT from tunnelclient, creates a remotesocket
726+↕ remotesocket - connects to the destination server via a proxy
727+↕ proxy server - goes thru firewalls
728+↑ server - dialogues with the client protocol
729+
730+"""
731+
732+
733+from PyQt4.QtCore import QCoreApplication, QTimer
734+from PyQt4.QtNetwork import (
735+ QAbstractSocket,
736+ QHostAddress,
737+ QNetworkProxy,
738+ QTcpServer,
739+ QTcpSocket,
740+)
741+from twisted.internet import defer, interfaces
742+from zope.interface import implements
743+
744+from ubuntuone.proxy.common import BaseTunnelProtocol, CRLF
745+from ubuntuone.proxy.logger import logger
746+
747+DEFAULT_CODE = 500
748+DEFAULT_DESCRIPTION = "Connection error"
749+
750+
751+class ConnectionError(Exception):
752+ """The client failed connecting to the destination."""
753+
754+ def __init__(self, code=DEFAULT_CODE, description=DEFAULT_DESCRIPTION):
755+ self.code = code
756+ self.description = description
757+
758+
759+class ProxyAuthenticationError(ConnectionError):
760+ """Credentials mismatch going thru a proxy."""
761+
762+
763+def build_proxy(settings):
764+ """Create a QNetworkProxy from these settings."""
765+ if "host" not in settings or "port" not in settings:
766+ return QNetworkProxy(QNetworkProxy.NoProxy)
767+ else:
768+ # TODO: add authentication here, to replace the empty user/pass
769+ return QNetworkProxy(QNetworkProxy.HttpProxy,
770+ hostName=settings.get("host", ""),
771+ port=settings.get("port", 0),
772+ user=settings.get("username", ""),
773+ password=settings.get("password", ""))
774+
775+
776+class RemoteSocket(QTcpSocket):
777+ """A dumb connection through a proxy to a destination hostport."""
778+
779+ def __init__(self, tunnel_protocol):
780+ """Initialize this object."""
781+ super(RemoteSocket, self).__init__()
782+ self.protocol = tunnel_protocol
783+ self.disconnected.connect(self.handle_disconnected)
784+ self.connected_d = defer.Deferred()
785+ self.connected.connect(self.handle_connected)
786+ self.buffered_data = []
787+
788+ def handle_connected(self):
789+ """When connected, send all pending data."""
790+ self.connected_d.callback(None)
791+ for d in self.buffered_data:
792+ super(RemoteSocket, self).write(d)
793+ self.buffered_data = []
794+
795+ def handle_disconnected(self):
796+ """Do something with disconnections."""
797+ logger.debug("Remote socket disconnected")
798+ self.protocol.remote_disconnected()
799+
800+ def write(self, data):
801+ """Write data to the remote end, buffering if not connected."""
802+ if self.state() == QAbstractSocket.ConnectedState:
803+ super(RemoteSocket, self).write(data)
804+ else:
805+ self.buffered_data.append(data)
806+
807+ def connect(self, hostport):
808+ """Try to establish the connection to the remote end."""
809+ host, port = hostport.split(":")
810+
811+ try:
812+ port = int(port)
813+ except ValueError:
814+ raise ConnectionError(400, "Destination port must be an integer.")
815+
816+ self.readyRead.connect(self.handle_ready_read)
817+ # TODO: handle the following signals in an upcoming branch
818+ #self.error.connect(...)
819+ #self.proxyAuthenticationRequired.connect(...)
820+ self.connectToHost(host, port)
821+
822+ return self.connected_d
823+
824+ def handle_ready_read(self):
825+ """Forward data from the remote end to the parent protocol."""
826+ data = self.readAll()
827+ self.protocol.response_data_received(data)
828+
829+ @defer.inlineCallbacks
830+ def stop(self):
831+ """Finish and cleanup."""
832+ self.disconnectFromHost()
833+ while self.state() != self.UnconnectedState:
834+ d = defer.Deferred()
835+ QTimer.singleShot(100, lambda: d.callback(None))
836+ yield d
837+
838+
839+class ServerTunnelProtocol(BaseTunnelProtocol):
840+ """CONNECT sever protocol for tunnelling connections."""
841+
842+ def __init__(self, client_class):
843+ """Initialize this protocol."""
844+ BaseTunnelProtocol.__init__(self)
845+ self.hostport = ""
846+ self.client = client_class(self)
847+
848+ def error_response(self, code, description):
849+ """Write a response with an error, and disconnect."""
850+ self.write_transport("HTTP/1.0 %d %s" % (code, description) + CRLF * 2)
851+ self.transport.loseConnection()
852+ self.client.stop()
853+ self.clearLineBuffer()
854+
855+ def write_transport(self, data):
856+ """Write a response in the transport."""
857+ self.transport.write(data)
858+
859+ def handle_first_line(self, line):
860+ """Special handling for the first line received."""
861+ try:
862+ method, hostport, proto_version = line.split(" ", 2)
863+ if proto_version != "HTTP/1.0":
864+ self.error_response(505, "HTTP Version Not Supported")
865+ return
866+ if method != "CONNECT":
867+ self.error_response(405, "Only the CONNECT method is allowed")
868+ return
869+ self.hostport = hostport
870+ except ValueError:
871+ self.error_response(400, "Bad request")
872+
873+ @defer.inlineCallbacks
874+ def headers_done(self):
875+ """An empty line was received, start connecting and switch mode."""
876+ try:
877+ yield self.client.connect(self.hostport)
878+ response_headers = {
879+ "Server": "Ubuntu One proxy tunnel",
880+ }
881+ self.write_transport("HTTP/1.0 200 Proxy connection established" +
882+ CRLF + self.format_headers(response_headers) +
883+ CRLF)
884+ except ConnectionError as e:
885+ self.error_response(e.code, e.description)
886+
887+ def rawDataReceived(self, data):
888+ """Tunnel all raw data straight to the other side."""
889+ self.client.write(data)
890+
891+ def response_data_received(self, data):
892+ """Return data coming from the other side."""
893+ logger.debug("writing data to the transport: %r", data)
894+ self.write_transport(data)
895+
896+
897+class Tunnel(object):
898+ """An instance of a running tunnel."""
899+
900+ implements(interfaces.ITransport)
901+
902+ def __init__(self, local_socket):
903+ """Initialize this Tunnel instance."""
904+ self.disconnecting = False
905+ self.local_socket = local_socket
906+ self.protocol = ServerTunnelProtocol(RemoteSocket)
907+ self.protocol.transport = self
908+ local_socket.readyRead.connect(self.server_ready_read)
909+ local_socket.disconnected.connect(self.local_disconnected)
910+
911+ def server_ready_read(self):
912+ """Data available on the local end. Move it forward."""
913+ data = bytes(self.local_socket.readAll())
914+ self.protocol.dataReceived(data)
915+
916+ def write(self, data):
917+ """Data available on the remote end. Bring it back."""
918+ self.local_socket.write(data)
919+
920+ def loseConnection(self):
921+ """The remote end disconnected."""
922+ logger.debug("disconnecting local end.")
923+ self.local_socket.close()
924+
925+ def local_disconnected(self):
926+ """The local end disconnected."""
927+ # TODO: handle this case in an upcoming branch
928+
929+
930+class TunnelServer(object):
931+ """A server for tunnel instances."""
932+
933+ def __init__(self):
934+ """Initialize this tunnel instance."""
935+ self.tunnels = []
936+ self.server = QTcpServer(QCoreApplication.instance())
937+ self.server.newConnection.connect(self.new_connection)
938+ self.server.listen(QHostAddress.LocalHost, 0)
939+
940+ def new_connection(self):
941+ """On a new connection create a new tunnel instance."""
942+ local_socket = self.server.nextPendingConnection()
943+ tunnel = Tunnel(local_socket)
944+ self.tunnels.append(tunnel)
945+
946+ def shutdown(self):
947+ """Terminate every connection."""
948+ # TODO: handle this gracefully in an upcoming branch
949+
950+ @property
951+ def port(self):
952+ """The port where this server listens."""
953+ return self.server.serverPort()

Subscribers

People subscribed via source and target branches