Merge lp:~alecu/ubuntuone-client/proxy-tunnel-server into lp:ubuntuone-client

Proposed by Alejandro J. Cura on 2012-02-28
Status: Merged
Approved by: Natalia Bidart on 2012-03-08
Approved revision: 1211
Merged at revision: 1202
Proposed branch: lp:~alecu/ubuntuone-client/proxy-tunnel-server
Merge into: lp:ubuntuone-client
Diff against target: 953 lines (+899/-1)
9 files modified
Makefile.am (+2/-1)
tests/proxy/__init__.py (+148/-0)
tests/proxy/ssl/dummy.cert (+19/-0)
tests/proxy/ssl/dummy.key (+16/-0)
tests/proxy/test_tunnel_server.py (+351/-0)
ubuntuone/proxy/__init__.py (+14/-0)
ubuntuone/proxy/common.py (+59/-0)
ubuntuone/proxy/logger.py (+37/-0)
ubuntuone/proxy/tunnel_server.py (+253/-0)
To merge this branch: bzr merge lp:~alecu/ubuntuone-client/proxy-tunnel-server
Reviewer Review Type Date Requested Status
Natalia Bidart 2012-02-28 Approve on 2012-03-08
Manuel de la Peña (community) 2012-02-28 Approve on 2012-03-07
Review via email: mp+95075@code.launchpad.net

Commit message

- The infrastructure for a QtNetwork based process to tunnel syncdaemon traffic thru proxies. (LP: #929207)

Description of the change

- The infrastructure for a QtNetwork based process to tunnel syncdaemon traffic thru proxies. (LP: #929207)

To post a comment you must log in.
Manuel de la Peña (mandel) wrote :

I dont think that the following code is correct:

787 + else:
788 + return QNetworkProxy(QNetworkProxy.HttpProxy,
789 + hostName=settings.get("host", ""),
790 + port=settings.get("port", 0),
791 + user=settings.get("username", ""),
792 + password=settings.get("password", ""))

The use of "" for the username and password will give problems when dealing with the auth proxies because the first attempt would be to use them and therefore you will get a 401 instead of a 407.

505 +
506 + def test_stop_but_never_connected(self):
507 + """Stop but it was never connected."""
508 + fake_protocol = FakeServerTunnelProtocol()
509 + client = tunnel_server.RemoteSocket(fake_protocol)
510 + yield client.stop()

There is a missing decorator in the test.

review: Needs Fixing
Alejandro J. Cura (alecu) wrote :

> I dont think that the following code is correct:
>
> 787 + else:
> 788 + return QNetworkProxy(QNetworkProxy.HttpProxy,
> 789 + hostName=settings.get("host", ""),
> 790 + port=settings.get("port", 0),
> 791 + user=settings.get("username", ""),
> 792 + password=settings.get("password", ""))
>
> The use of "" for the username and password will give problems when dealing
> with the auth proxies because the first attempt would be to use them and
> therefore you will get a 401 instead of a 407.

Those values are only used when the "settings" dictionary has no key with the given name.
And I'm using empty strings because that's what QNetworkProxy uses if no parameter is given:
 * https://qt-project.org/doc/qt-4.8/qnetworkproxy.html#QNetworkProxy-2

Since the current version of the tunnel is not getting the authentication, I think it's safe to use it like this. Perhaps we can add a comment or a TODO for it, and we can fix it in an upcoming branch to add authentication.

> 506 + def test_stop_but_never_connected(self):
> 507 + """Stop but it was never connected."""
> 508 + fake_protocol = FakeServerTunnelProtocol()
> 509 + client = tunnel_server.RemoteSocket(fake_protocol)
> 510 + yield client.stop()
>
> There is a missing decorator in the test.

Great catch! Thanks, I'm fixing it too.

1208. By Alejandro J. Cura on 2012-03-01

fixes requested on the review

1209. By Alejandro J. Cura on 2012-03-02

Use a QTimer to work around flaky signals

1210. By Alejandro J. Cura on 2012-03-06

merged with trunk

Manuel de la Peña (mandel) wrote :

I see no more issues.

review: Approve
Natalia Bidart (nataliabidart) wrote :

* Can't we re-use teh code from ubuntu-sso-client instead of defnining:

 +def build_proxy(settings):

?

* Most of our projects and source file use this syntax for file encoding:

# -*- coding: utf8 -*-

would you consider using that form?

* There should be spaces around the % in "HTTP/1.0 %d %s"%(code, description)

The branch looks good!

review: Needs Fixing
Alejandro J. Cura (alecu) wrote :

> * Can't we re-use teh code from ubuntu-sso-client instead of defnining:
>
> +def build_proxy(settings):
>
> ?

The bits of code that build the QNetworkProxy instance and configure the proxy settings in ussoc are currently in flux, because mandel is changing it to use a QNetworkProxyFactory and to query the proxy settings on each request. As soon as mandel's branch for ussoc lands we should be able to create a different branch to make u1-client use those bits from syncdaemon, so I think it makes sense to keep this branch as is, and refactor later.

> * Most of our projects and source file use this syntax for file encoding:
>
> # -*- coding: utf8 -*-
>
> would you consider using that form?

Sure, fixing that.

> * There should be spaces around the % in "HTTP/1.0 %d %s"%(code, description)

Fixing that too.

thanks for the review!

1211. By Alejandro J. Cura on 2012-03-07

fixes requested in review

Natalia Bidart (nataliabidart) wrote :

Looks great!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'Makefile.am'
--- Makefile.am 2012-02-10 15:07:59 +0000
+++ Makefile.am 2012-03-07 23:48:19 +0000
@@ -53,7 +53,8 @@
53test: logging.conf $(clientdefs_DATA) Makefile53test: logging.conf $(clientdefs_DATA) Makefile
54 echo "$(PYTHONPATH)"54 echo "$(PYTHONPATH)"
55 if test "x$(builddir)" == "x$(srcdir)"; then \55 if test "x$(builddir)" == "x$(srcdir)"; then \
56 PYTHONPATH="$(PYTHONPATH)" u1trial -r $(REACTOR) -p tests/platform/windows tests; \56 PYTHONPATH="$(PYTHONPATH)" u1trial -r $(REACTOR) -p tests/platform/windows,tests/proxy tests; \
57 PYTHONPATH="$(PYTHONPATH)" u1trial -r qt4 -p tests/platform/windows tests/proxy; \
57 fi58 fi
58 rm -rf _trial_temp59 rm -rf _trial_temp
5960
6061
=== added directory 'tests/proxy'
=== added file 'tests/proxy/__init__.py'
--- tests/proxy/__init__.py 1970-01-01 00:00:00 +0000
+++ tests/proxy/__init__.py 2012-03-07 23:48:19 +0000
@@ -0,0 +1,148 @@
1# Copyright 2012 Canonical Ltd.
2#
3# This program is free software: you can redistribute it and/or modify it
4# under the terms of the GNU General Public License version 3, as published
5# by the Free Software Foundation.
6#
7# This program is distributed in the hope that it will be useful, but
8# WITHOUT ANY WARRANTY; without even the implied warranties of
9# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
10# PURPOSE. See the GNU General Public License for more details.
11#
12# You should have received a copy of the GNU General Public License along
13# with this program. If not, see <http://www.gnu.org/licenses/>.
14"""Tests for the Ubuntu One proxy support."""
15
16from os import path
17from StringIO import StringIO
18
19from twisted.application import internet, service
20from twisted.internet import defer, ssl
21from twisted.web import http, resource, server
22
23SAMPLE_CONTENT = "hello world!"
24SIMPLERESOURCE = "simpleresource"
25DUMMY_KEY_FILENAME = "dummy.key"
26DUMMY_CERT_FILENAME = "dummy.cert"
27
28
29class SaveHTTPChannel(http.HTTPChannel):
30 """A save protocol to be used in tests."""
31
32 protocolInstance = None
33
34 # pylint: disable=C0103
35 def connectionMade(self):
36 """Keep track of the given protocol."""
37 SaveHTTPChannel.protocolInstance = self
38 http.HTTPChannel.connectionMade(self)
39
40
41class SaveSite(server.Site):
42 """A site that let us know when it's closed."""
43
44 protocol = SaveHTTPChannel
45
46 def __init__(self, *args, **kwargs):
47 """Create a new instance."""
48 server.Site.__init__(self, *args, **kwargs)
49 # we disable the timeout in the tests, we will deal with it manually.
50 # pylint: disable=C0103
51 self.timeOut = None
52
53
54class BaseMockWebServer(object):
55 """A mock webserver for testing"""
56
57 def __init__(self):
58 """Start up this instance."""
59 self.root = self.get_root_resource()
60 self.site = SaveSite(self.root)
61 application = service.Application('web')
62 self.service_collection = service.IServiceCollection(application)
63 #pylint: disable=E1101
64 self.tcpserver = internet.TCPServer(0, self.site)
65 self.tcpserver.setServiceParent(self.service_collection)
66 self.sslserver = internet.SSLServer(0, self.site, self.get_context())
67 self.sslserver.setServiceParent(self.service_collection)
68 self.service_collection.startService()
69
70 def get_dummy_path(self, filename):
71 """Path pointing at the dummy certificate files."""
72 base_path = path.dirname(__file__)
73 return path.join(base_path, "ssl", filename)
74
75 def get_context(self):
76 """Return an ssl context."""
77 key_path = self.get_dummy_path(DUMMY_KEY_FILENAME)
78 cert_path = self.get_dummy_path(DUMMY_CERT_FILENAME)
79 return ssl.DefaultOpenSSLContextFactory(key_path, cert_path)
80
81 def get_root_resource(self):
82 """Get the root resource with all the children."""
83 raise NotImplementedError
84
85 def get_iri(self):
86 """Build the iri for this mock server."""
87 #pylint: disable=W0212
88 port_num = self.tcpserver._port.getHost().port
89 return u"http://0.0.0.0:%d/" % port_num
90
91 def get_ssl_iri(self):
92 """Build the iri for the ssl mock server."""
93 #pylint: disable=W0212
94 port_num = self.sslserver._port.getHost().port
95 return u"https://0.0.0.0:%d/" % port_num
96
97 def stop(self):
98 """Shut it down."""
99 #pylint: disable=E1101
100 if self.site.protocol.protocolInstance:
101 self.site.protocol.protocolInstance.timeoutConnection()
102 return self.service_collection.stopService()
103
104
105class SimpleResource(resource.Resource):
106 """A simple web resource."""
107
108 def __init__(self):
109 """Initialize this mock resource."""
110 resource.Resource.__init__(self)
111 self.rendered = defer.Deferred()
112
113 def render_GET(self, request):
114 """Make a bit of html out of the resource's content."""
115 if not self.rendered.called:
116 self.rendered.callback(None)
117 return SAMPLE_CONTENT
118
119
120class MockWebServer(BaseMockWebServer):
121 """A mock webserver."""
122
123 def __init__(self):
124 """Initialize this mock server."""
125 self.simple_resource = SimpleResource()
126 super(MockWebServer, self).__init__()
127
128 def get_root_resource(self):
129 """Get the root resource with all the children."""
130 root = resource.Resource()
131 root.putChild(SIMPLERESOURCE, self.simple_resource)
132 return root
133
134
135class FakeTransport(StringIO):
136 """A fake transport that stores everything written to it."""
137
138 connected = True
139 disconnecting = False
140
141 def loseConnection(self):
142 """Mark the connection as lost."""
143 self.connected = False
144 self.disconnecting = True
145
146 def getPeer(self):
147 """Return the peer IAddress."""
148 return None
0149
=== added directory 'tests/proxy/ssl'
=== added file 'tests/proxy/ssl/dummy.cert'
--- tests/proxy/ssl/dummy.cert 1970-01-01 00:00:00 +0000
+++ tests/proxy/ssl/dummy.cert 2012-03-07 23:48:19 +0000
@@ -0,0 +1,19 @@
1-----BEGIN CERTIFICATE-----
2MIIDEDCCAnmgAwIBAgIJAM/bIJ77awBCMA0GCSqGSIb3DQEBBQUAMIGgMQswCQYD
3VQQGEwJBUjETMBEGA1UECAwKRmFrZSBTdGF0ZTESMBAGA1UEBwwJRmFrZSBDaXR5
4MRUwEwYDVQQKDAxGYWtlIENvbXBhbnkxFjAUBgNVBAsMDUZha2UgRGl2aXNpb24x
5EjAQBgNVBAMMCUZha2UgTmFtZTElMCMGCSqGSIb3DQEJARYWZmFrZUBlbWFpbC5h
6ZGRyZXNzLm5vdDAeFw0xMjAyMjIxOTI0MjBaFw0yMjAyMjMxOTI0MjBaMIGgMQsw
7CQYDVQQGEwJBUjETMBEGA1UECAwKRmFrZSBTdGF0ZTESMBAGA1UEBwwJRmFrZSBD
8aXR5MRUwEwYDVQQKDAxGYWtlIENvbXBhbnkxFjAUBgNVBAsMDUZha2UgRGl2aXNp
9b24xEjAQBgNVBAMMCUZha2UgTmFtZTElMCMGCSqGSIb3DQEJARYWZmFrZUBlbWFp
10bC5hZGRyZXNzLm5vdDCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEA0hbliGty
11HwfZixU609UHBQdbfO+oObrPIrIawWX5FxD6KhX4ei23idmpyYEcXLK4ivNlT4dW
1227bvhtpf6/FBbu9e1YdwcdDNoXajr9Ia4NZJyANgo9b5UIsnyTc45NlnpZgRg5zc
13Oz7Vwwr4qf6r1ljK/I2mAO7rlpH5Ak9J+RkCAwEAAaNQME4wHQYDVR0OBBYEFLwr
14ps/JLNcfpSuuylMnkvImVvkgMB8GA1UdIwQYMBaAFLwrps/JLNcfpSuuylMnkvIm
15VvkgMAwGA1UdEwQFMAMBAf8wDQYJKoZIhvcNAQEFBQADgYEAWYDBAr0MgpnBxIne
16WRz8MX0/c7IqrEuZCYMSGnU7PoX3GdNk1Lkif1ufELKSoG8jY16CDgEl26GPxA1k
17Tho7MWSLikLbuQYJs2saF9by0Y/Mrau0auxEnpHZ7pkybeKFnrIqiNKvTVMnjo5T
18FMET5qEOKKvp9IOnezCYX1nYXyY=
19-----END CERTIFICATE-----
020
=== added file 'tests/proxy/ssl/dummy.key'
--- tests/proxy/ssl/dummy.key 1970-01-01 00:00:00 +0000
+++ tests/proxy/ssl/dummy.key 2012-03-07 23:48:19 +0000
@@ -0,0 +1,16 @@
1-----BEGIN PRIVATE KEY-----
2MIICdgIBADANBgkqhkiG9w0BAQEFAASCAmAwggJcAgEAAoGBANIW5Yhrch8H2YsV
3OtPVBwUHW3zvqDm6zyKyGsFl+RcQ+ioV+Hott4nZqcmBHFyyuIrzZU+HVtu274ba
4X+vxQW7vXtWHcHHQzaF2o6/SGuDWScgDYKPW+VCLJ8k3OOTZZ6WYEYOc3Ds+1cMK
5+Kn+q9ZYyvyNpgDu65aR+QJPSfkZAgMBAAECgYBSxFh7TTExjmsjAyMg700LqyFc
68CHLVJBkL9ygkqb2cmbMC8nPgJFNSqY8T5Q35OUVQNyJ31zVxJVLAF9H2c0Xy48K
7IkbS/hntyqlJYK1yfTbTHkDiweToE3Lm+55Do1TX04AyvBrwA1O/jNGi4xIlUEAy
81Bs8MrJ1E/j/XDn9/QJBAOuhPTgG3F7bKuBrQzv98CvC5o2Txf3vLY8nL8V24b3l
9XgqzkDLhUxReBmmkGxZfKAju3+gXFvGGpbP7V8zShg8CQQDkQGs7kArFq/KR/GCh
10CAmJaDWy4LJkSqzDHoJbTrS7YuqN6X6mW1xPRnWpYSxae38fJsCpG3Vq8Mv1Zl32
11VPZXAkEAsAeE9JYri7GwFngLgoXzJr4z/xCmmU5VetyLk7l8a6Eu4E/FKj2rE0wq
12/kDa+5ubDRFntLuLKGSu5gafUST1gQJABhdmBTfp4a6eEaFPntyNDJq4XCa8/Ao2
13JBrrVa57Ckkwg0sI8z2a8A6sUzHhsiR7lwQ8vgaakpkMiGcL+Of5jwJAA/qX3PW+
149JXbjWxpgh7FHnZJNRZ8xSe47REGA7qS/nIlV9iRuf/9M+k3A5VqitfFxrjPwSyI
15rvKTYkk13dL4hg==
16-----END PRIVATE KEY-----
017
=== added file 'tests/proxy/test_tunnel_server.py'
--- tests/proxy/test_tunnel_server.py 1970-01-01 00:00:00 +0000
+++ tests/proxy/test_tunnel_server.py 2012-03-07 23:48:19 +0000
@@ -0,0 +1,351 @@
1# Copyright 2012 Canonical Ltd.
2#
3# This program is free software: you can redistribute it and/or modify it
4# under the terms of the GNU General Public License version 3, as published
5# by the Free Software Foundation.
6#
7# This program is distributed in the hope that it will be useful, but
8# WITHOUT ANY WARRANTY; without even the implied warranties of
9# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
10# PURPOSE. See the GNU General Public License for more details.
11#
12# You should have received a copy of the GNU General Public License along
13# with this program. If not, see <http://www.gnu.org/licenses/>.
14"""Tests for the proxy tunnel."""
15
16from urlparse import urlparse
17
18from twisted.internet import defer, protocol, reactor
19
20from ubuntuone.devtools.testcases.squid import SquidTestCase
21
22from tests.proxy import (
23 FakeTransport,
24 MockWebServer,
25 SAMPLE_CONTENT,
26 SIMPLERESOURCE,
27)
28from ubuntuone.proxy import tunnel_server
29from ubuntuone.proxy.tunnel_server import CRLF
30
31
32FAKE_SESSION_TEMPLATE = (
33 "CONNECT {0.netloc} HTTP/1.0" + CRLF +
34 "Header1: value1" + CRLF +
35 "Header2: value2" + CRLF +
36 CRLF +
37 "GET {0.path} HTTP/1.0" + CRLF + CRLF
38)
39
40
41class DisconnectingProtocol(protocol.Protocol):
42 """A protocol that just disconnects."""
43
44 def connectionMade(self):
45 """Upon connecting: just disconnect."""
46 self.transport.loseConnection()
47
48
49class DisconnectingClientFactory(protocol.ClientFactory):
50 """A factory that fires a deferred on connection."""
51
52 def __init__(self):
53 """Initialize this instance."""
54 self.connected = defer.Deferred()
55
56 def buildProtocol(self, addr):
57 """The connection was made."""
58 proto = DisconnectingProtocol()
59 if not self.connected.called:
60 self.connected.callback(proto)
61 return proto
62
63
64class FakeProtocol(protocol.Protocol):
65 """A protocol that forwards some data."""
66
67 def __init__(self, factory, data):
68 """Initialize this fake."""
69 self.factory = factory
70 self.data = data
71 self.received_data = []
72
73 def connectionMade(self):
74 """Upon connection: send the stored data."""
75 self.transport.write(self.data)
76
77 def dataReceived(self, data):
78 """Some data was received."""
79 self.received_data.append(data)
80
81 def connectionLost(self, reason):
82 """The connection was lost, return the response."""
83 response = "".join(self.received_data)
84 if not self.factory.response.called:
85 self.factory.response.callback(response)
86
87
88class FakeClientFactory(protocol.ClientFactory):
89 """A factory that forwards some data to the protocol."""
90
91 def __init__(self, data):
92 """Initialize this fake."""
93 self.data = data
94 self.response = defer.Deferred()
95
96 def buildProtocol(self, addr):
97 """The connection was made."""
98 return FakeProtocol(self, self.data)
99
100
101class TunnelIntegrationTestCase(SquidTestCase):
102 """Basic tunnel integration tests."""
103
104 timeout = 3
105
106 @defer.inlineCallbacks
107 def setUp(self):
108 """Initialize this testcase."""
109 yield super(TunnelIntegrationTestCase, self).setUp()
110 self.ws = MockWebServer()
111 self.addCleanup(self.ws.stop)
112 self.dest_url = self.ws.get_iri().encode("utf-8") + SIMPLERESOURCE
113 self.tunnel_server = tunnel_server.TunnelServer()
114 self.addCleanup(self.tunnel_server.shutdown)
115
116 def test_init(self):
117 """The tunnel is started."""
118 self.assertNotEqual(self.tunnel_server.port, 0)
119
120 @defer.inlineCallbacks
121 def test_accepts_connections(self):
122 """The tunnel accepts incoming connections."""
123 ncf = DisconnectingClientFactory()
124 reactor.connectTCP("0.0.0.0", self.tunnel_server.port, ncf)
125 yield ncf.connected
126
127 @defer.inlineCallbacks
128 def test_complete_connection(self):
129 """Test from the tunnel server down."""
130 url = urlparse(self.dest_url)
131 client = FakeClientFactory(FAKE_SESSION_TEMPLATE.format(url))
132 reactor.connectTCP("0.0.0.0", self.tunnel_server.port, client)
133 response = yield client.response
134 self.assertIn(SAMPLE_CONTENT, response)
135
136
137class FakeClient(object):
138 """A fake destination client."""
139
140 protocol = None
141 connection_result = defer.succeed(True)
142
143 def connect(self, hostport):
144 """Establish a connection with the other end."""
145 return self.connection_result
146
147 def write(self, data):
148 """Write some data to the other end."""
149 if data == 'GET /simpleresource HTTP/1.0\r\n\r\n':
150 self.protocol.transport.write(SAMPLE_CONTENT)
151
152 def stop(self):
153 """Stop this fake client."""
154
155
156class ServerTunnelProtocolTestCase(SquidTestCase):
157 """Tests for the ServerTunnelProtocol."""
158
159 @defer.inlineCallbacks
160 def setUp(self):
161 """Initialize this test instance."""
162 yield super(ServerTunnelProtocolTestCase, self).setUp()
163 self.ws = MockWebServer()
164 self.addCleanup(self.ws.stop)
165 self.dest_url = self.ws.get_iri().encode("utf-8") + SIMPLERESOURCE
166 self.transport = FakeTransport()
167 self.fake_client = FakeClient()
168 self.proto = tunnel_server.ServerTunnelProtocol(
169 lambda _: self.fake_client)
170 self.fake_client.protocol = self.proto
171 self.proto.transport = self.transport
172
173 def test_broken_request(self):
174 """Broken request."""
175 self.proto.dataReceived("Broken request." + CRLF)
176 self.assertTrue(self.transport.getvalue().startswith("HTTP/1.0 400 "),
177 "A broken request must fail.")
178
179 def test_wrong_method(self):
180 """Wrong method."""
181 self.proto.dataReceived("GET http://slashdot.org HTTP/1.0" + CRLF)
182 self.assertTrue(self.transport.getvalue().startswith("HTTP/1.0 405 "),
183 "Using a wrong method fails.")
184
185 def test_invalid_http_version(self):
186 """Invalid HTTP version."""
187 self.proto.dataReceived("CONNECT 127.0.0.1:9999 HTTP/1.1" + CRLF)
188 self.assertTrue(self.transport.getvalue().startswith("HTTP/1.0 505 "),
189 "Invalid http version is not allowed.")
190
191 def test_connection_is_established(self):
192 """The response code is sent."""
193 expected = "HTTP/1.0 200 Proxy connection established" + CRLF
194 self.proto.dataReceived("CONNECT 127.0.0.1:9999 HTTP/1.0" + CRLF * 2)
195 self.assertTrue(self.transport.getvalue().startswith(expected),
196 "First line must be the response status")
197
198 def test_connection_fails(self):
199 """The connection to the other end fails, and it's handled."""
200 error = tunnel_server.ConnectionError()
201 self.patch(self.fake_client, "connection_result", defer.fail(error))
202 expected = "HTTP/1.0 500 Connection error" + CRLF
203 self.proto.dataReceived("CONNECT 127.0.0.1:9999 HTTP/1.0" + CRLF * 2)
204 self.assertTrue(self.transport.getvalue().startswith(expected),
205 "The connection should fail at this point.")
206
207 def test_headers_stored(self):
208 """The request headers are stored."""
209 expected = [
210 ("Header1", "value1"),
211 ("Header2", "value2"),
212 ]
213 self.proto.dataReceived("CONNECT 127.0.0.1:9999 HTTP/1.0" + CRLF +
214 "Header1: value1" + CRLF +
215 "Header2: value2" + CRLF + CRLF)
216 self.assertEqual(self.proto.received_headers, expected)
217
218 def test_successful_connect(self):
219 """A successful connect thru the tunnel."""
220 url = urlparse(self.dest_url)
221 data = FAKE_SESSION_TEMPLATE.format(url)
222 self.proto.dataReceived(data)
223 lines = self.transport.getvalue().split(CRLF)
224 self.assertEqual(lines[-1], SAMPLE_CONTENT)
225
226 def test_header_split(self):
227 """Test a header with many colons."""
228 self.proto.header_line("key: host:port")
229 self.assertIn("key", dict(self.proto.received_headers))
230
231
232class FakeServerTunnelProtocol(object):
233 """A fake ServerTunnelProtocol."""
234
235 def __init__(self):
236 """Initialize this fake tunnel."""
237 self.response_received = defer.Deferred()
238
239 def response_data_received(self, data):
240 """Fire the response deferred."""
241 if not self.response_received.called:
242 self.response_received.callback(data)
243
244 def remote_disconnected(self):
245 """The remote server disconnected."""
246
247
248class RemoteSocketTestCase(SquidTestCase):
249 """Tests for the client that connects to the other side."""
250
251 timeout = 3
252 get_proxy_settings = lambda _: {}
253
254 @defer.inlineCallbacks
255 def setUp(self):
256 """Initialize this testcase."""
257 yield super(RemoteSocketTestCase, self).setUp()
258 self.ws = MockWebServer()
259 self.addCleanup(self.ws.stop)
260 self.dest_url = self.ws.get_iri().encode("utf-8") + SIMPLERESOURCE
261
262 self.addCleanup(tunnel_server.QNetworkProxy.setApplicationProxy,
263 tunnel_server.QNetworkProxy.applicationProxy())
264 tunnel_server.QNetworkProxy.setApplicationProxy(
265 tunnel_server.build_proxy(self.get_proxy_settings()))
266
267 def test_invalid_port(self):
268 """A request with an invalid port fails with a 400."""
269 protocol = tunnel_server.ServerTunnelProtocol(
270 tunnel_server.RemoteSocket)
271 protocol.transport = FakeTransport()
272 protocol.dataReceived("CONNECT 127.0.0.1:wrong_port HTTP/1.0" +
273 CRLF * 2)
274
275 status_line = protocol.transport.getvalue()
276 self.assertTrue(status_line.startswith("HTTP/1.0 400 "),
277 "The port must be an integer.")
278
279 @defer.inlineCallbacks
280 def test_connection_is_finished_when_stopping(self):
281 """The client disconnects when requested."""
282 fake_protocol = FakeServerTunnelProtocol()
283 client = tunnel_server.RemoteSocket(fake_protocol)
284 url = urlparse(self.dest_url)
285 yield client.connect(url.netloc)
286 yield client.stop()
287
288 @defer.inlineCallbacks
289 def test_stop_but_never_connected(self):
290 """Stop but it was never connected."""
291 fake_protocol = FakeServerTunnelProtocol()
292 client = tunnel_server.RemoteSocket(fake_protocol)
293 yield client.stop()
294
295 @defer.inlineCallbacks
296 def test_client_write(self):
297 """Data written to the client is sent to the other side."""
298 fake_protocol = FakeServerTunnelProtocol()
299 client = tunnel_server.RemoteSocket(fake_protocol)
300 self.addCleanup(client.stop)
301 url = urlparse(self.dest_url)
302 yield client.connect(url.netloc)
303 client.write("GET /simpleresource HTTP/1.0" + CRLF * 2)
304 yield self.ws.simple_resource.rendered
305
306 @defer.inlineCallbacks
307 def test_client_read(self):
308 """Data received by the client is written into the transport."""
309 fake_protocol = FakeServerTunnelProtocol()
310 client = tunnel_server.RemoteSocket(fake_protocol)
311 self.addCleanup(client.stop)
312 url = urlparse(self.dest_url)
313 yield client.connect(url.netloc)
314 client.write("GET /simpleresource HTTP/1.0" + CRLF * 2)
315 yield self.ws.simple_resource.rendered
316 data = yield fake_protocol.response_received
317 _headers, content = str(data).split(CRLF * 2, 1)
318 self.assertEqual(content, SAMPLE_CONTENT)
319
320
321class AnonProxyRemoteSocketTestCase(RemoteSocketTestCase):
322 """Tests for the client going thru an anonymous proxy."""
323
324 get_proxy_settings = RemoteSocketTestCase.get_nonauth_proxy_settings
325
326 def parse_headers(self, raw_headers):
327 """Parse the headers."""
328 lines = raw_headers.split(CRLF)
329 header_lines = lines[1:]
330 headers_pairs = (l.split(":", 1) for l in header_lines)
331 return dict((k.lower(), v.strip()) for k, v in headers_pairs)
332
333 @defer.inlineCallbacks
334 def test_verify_client_uses_proxy(self):
335 """Verify that the client uses the proxy."""
336 fake_protocol = FakeServerTunnelProtocol()
337 client = tunnel_server.RemoteSocket(fake_protocol)
338 self.addCleanup(client.stop)
339 url = urlparse(self.dest_url)
340 yield client.connect(url.netloc)
341 client.write("GET /simpleresource HTTP/1.0" + CRLF * 2)
342 yield self.ws.simple_resource.rendered
343 data = yield fake_protocol.response_received
344 raw_headers, _content = str(data).split(CRLF * 2, 1)
345 self.parse_headers(raw_headers)
346
347
348class AuthenticatedProxyRemoteSocketTestCase(AnonProxyRemoteSocketTestCase):
349 """Tests for the client going thru an authenticated proxy."""
350
351 get_proxy_settings = RemoteSocketTestCase.get_auth_proxy_settings
0352
=== added directory 'ubuntuone/proxy'
=== added file 'ubuntuone/proxy/__init__.py'
--- ubuntuone/proxy/__init__.py 1970-01-01 00:00:00 +0000
+++ ubuntuone/proxy/__init__.py 2012-03-07 23:48:19 +0000
@@ -0,0 +1,14 @@
1# Copyright 2012 Canonical Ltd.
2#
3# This program is free software: you can redistribute it and/or modify it
4# under the terms of the GNU General Public License version 3, as published
5# by the Free Software Foundation.
6#
7# This program is distributed in the hope that it will be useful, but
8# WITHOUT ANY WARRANTY; without even the implied warranties of
9# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
10# PURPOSE. See the GNU General Public License for more details.
11#
12# You should have received a copy of the GNU General Public License along
13# with this program. If not, see <http://www.gnu.org/licenses/>.
14"""Ubuntu One proxy support."""
015
=== added file 'ubuntuone/proxy/common.py'
--- ubuntuone/proxy/common.py 1970-01-01 00:00:00 +0000
+++ ubuntuone/proxy/common.py 2012-03-07 23:48:19 +0000
@@ -0,0 +1,59 @@
1# -*- coding: utf8 -*-
2#
3# Copyright 2012 Canonical Ltd.
4#
5# This program is free software: you can redistribute it and/or modify it
6# under the terms of the GNU General Public License version 3, as published
7# by the Free Software Foundation.
8#
9# This program is distributed in the hope that it will be useful, but
10# WITHOUT ANY WARRANTY; without even the implied warranties of
11# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
12# PURPOSE. See the GNU General Public License for more details.
13#
14# You should have received a copy of the GNU General Public License along
15# with this program. If not, see <http://www.gnu.org/licenses/>.
16"""Common classes to the tunnel client and server."""
17
18from twisted.protocols import basic
19
20CRLF = "\r\n"
21
22
23class BaseTunnelProtocol(basic.LineReceiver):
24 """CONNECT base protocol for tunnelling connections."""
25
26 delimiter = CRLF
27
28 def __init__(self):
29 """Initialize this protocol."""
30 self._first_line = True
31 self.received_headers = []
32
33 def header_line(self, line):
34 """Handle each header line received."""
35 key, value = line.split(":", 1)
36 value = value.strip()
37 self.received_headers.append((key, value))
38
39 def lineReceived(self, line):
40 """Process a line in the header."""
41 if self._first_line:
42 self._first_line = False
43 self.handle_first_line(line)
44 else:
45 if line:
46 self.header_line(line)
47 else:
48 self.setRawMode()
49 self.headers_done()
50
51 def remote_disconnected(self):
52 """The remote end closed the connection."""
53 self.transport.loseConnection()
54
55 def format_headers(self, headers):
56 """Format some headers as a few response lines."""
57 return "".join("%s: %s" % item + CRLF for item in headers.items())
58
59
060
=== added file 'ubuntuone/proxy/logger.py'
--- ubuntuone/proxy/logger.py 1970-01-01 00:00:00 +0000
+++ ubuntuone/proxy/logger.py 2012-03-07 23:48:19 +0000
@@ -0,0 +1,37 @@
1# ubuntuone.syncdaemon.logger - logging utilities
2#
3# Copyright 2009-2012 Canonical Ltd.
4#
5# This program is free software: you can redistribute it and/or modify it
6# under the terms of the GNU General Public License version 3, as published
7# by the Free Software Foundation.
8#
9# This program is distributed in the hope that it will be useful, but
10# WITHOUT ANY WARRANTY; without even the implied warranties of
11# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
12# PURPOSE. See the GNU General Public License for more details.
13#
14# You should have received a copy of the GNU General Public License along
15# with this program. If not, see <http://www.gnu.org/licenses/>.
16
17"""SyncDaemon logging utilities and config."""
18
19import logging
20import os
21
22from ubuntuone.logger import (
23 _DEBUG_LOG_LEVEL,
24 basic_formatter,
25 CustomRotatingFileHandler,
26)
27
28from ubuntuone.platform.xdg_base_directory import ubuntuone_log_dir
29
30
31LOGFILENAME = os.path.join(ubuntuone_log_dir, 'proxy.log')
32logger = logging.getLogger("ubuntuone.proxy")
33logger.setLevel(_DEBUG_LOG_LEVEL)
34handler = CustomRotatingFileHandler(filename=LOGFILENAME)
35handler.setFormatter(basic_formatter)
36handler.setLevel(_DEBUG_LOG_LEVEL)
37logger.addHandler(handler)
038
=== added file 'ubuntuone/proxy/tunnel_server.py'
--- ubuntuone/proxy/tunnel_server.py 1970-01-01 00:00:00 +0000
+++ ubuntuone/proxy/tunnel_server.py 2012-03-07 23:48:19 +0000
@@ -0,0 +1,253 @@
1# -*- coding: utf8 -*-
2#
3# Copyright 2012 Canonical Ltd.
4#
5# This program is free software: you can redistribute it and/or modify it
6# under the terms of the GNU General Public License version 3, as published
7# by the Free Software Foundation.
8#
9# This program is distributed in the hope that it will be useful, but
10# WITHOUT ANY WARRANTY; without even the implied warranties of
11# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
12# PURPOSE. See the GNU General Public License for more details.
13#
14# You should have received a copy of the GNU General Public License along
15# with this program. If not, see <http://www.gnu.org/licenses/>.
16"""A tunnel through proxies.
17
18The layers in a tunneled proxied connection:
19
20↓ tunnelclient - initiates tcp to tunnelserver, request outward connection
21↕ client protocol - started after the tunneclient gets connected
22---process boundary---
23↕ tunnelserver - creates a tunnel instance per incoming connection
24↕ tunnel - hold a qtcpsocket to tunnelclient, and srvtunnelproto to the remote
25↕ servertunnelprotocol - gets CONNECT from tunnelclient, creates a remotesocket
26↕ remotesocket - connects to the destination server via a proxy
27↕ proxy server - goes thru firewalls
28↑ server - dialogues with the client protocol
29
30"""
31
32
33from PyQt4.QtCore import QCoreApplication, QTimer
34from PyQt4.QtNetwork import (
35 QAbstractSocket,
36 QHostAddress,
37 QNetworkProxy,
38 QTcpServer,
39 QTcpSocket,
40)
41from twisted.internet import defer, interfaces
42from zope.interface import implements
43
44from ubuntuone.proxy.common import BaseTunnelProtocol, CRLF
45from ubuntuone.proxy.logger import logger
46
47DEFAULT_CODE = 500
48DEFAULT_DESCRIPTION = "Connection error"
49
50
51class ConnectionError(Exception):
52 """The client failed connecting to the destination."""
53
54 def __init__(self, code=DEFAULT_CODE, description=DEFAULT_DESCRIPTION):
55 self.code = code
56 self.description = description
57
58
59class ProxyAuthenticationError(ConnectionError):
60 """Credentials mismatch going thru a proxy."""
61
62
63def build_proxy(settings):
64 """Create a QNetworkProxy from these settings."""
65 if "host" not in settings or "port" not in settings:
66 return QNetworkProxy(QNetworkProxy.NoProxy)
67 else:
68 # TODO: add authentication here, to replace the empty user/pass
69 return QNetworkProxy(QNetworkProxy.HttpProxy,
70 hostName=settings.get("host", ""),
71 port=settings.get("port", 0),
72 user=settings.get("username", ""),
73 password=settings.get("password", ""))
74
75
76class RemoteSocket(QTcpSocket):
77 """A dumb connection through a proxy to a destination hostport."""
78
79 def __init__(self, tunnel_protocol):
80 """Initialize this object."""
81 super(RemoteSocket, self).__init__()
82 self.protocol = tunnel_protocol
83 self.disconnected.connect(self.handle_disconnected)
84 self.connected_d = defer.Deferred()
85 self.connected.connect(self.handle_connected)
86 self.buffered_data = []
87
88 def handle_connected(self):
89 """When connected, send all pending data."""
90 self.connected_d.callback(None)
91 for d in self.buffered_data:
92 super(RemoteSocket, self).write(d)
93 self.buffered_data = []
94
95 def handle_disconnected(self):
96 """Do something with disconnections."""
97 logger.debug("Remote socket disconnected")
98 self.protocol.remote_disconnected()
99
100 def write(self, data):
101 """Write data to the remote end, buffering if not connected."""
102 if self.state() == QAbstractSocket.ConnectedState:
103 super(RemoteSocket, self).write(data)
104 else:
105 self.buffered_data.append(data)
106
107 def connect(self, hostport):
108 """Try to establish the connection to the remote end."""
109 host, port = hostport.split(":")
110
111 try:
112 port = int(port)
113 except ValueError:
114 raise ConnectionError(400, "Destination port must be an integer.")
115
116 self.readyRead.connect(self.handle_ready_read)
117 # TODO: handle the following signals in an upcoming branch
118 #self.error.connect(...)
119 #self.proxyAuthenticationRequired.connect(...)
120 self.connectToHost(host, port)
121
122 return self.connected_d
123
124 def handle_ready_read(self):
125 """Forward data from the remote end to the parent protocol."""
126 data = self.readAll()
127 self.protocol.response_data_received(data)
128
129 @defer.inlineCallbacks
130 def stop(self):
131 """Finish and cleanup."""
132 self.disconnectFromHost()
133 while self.state() != self.UnconnectedState:
134 d = defer.Deferred()
135 QTimer.singleShot(100, lambda: d.callback(None))
136 yield d
137
138
139class ServerTunnelProtocol(BaseTunnelProtocol):
140 """CONNECT sever protocol for tunnelling connections."""
141
142 def __init__(self, client_class):
143 """Initialize this protocol."""
144 BaseTunnelProtocol.__init__(self)
145 self.hostport = ""
146 self.client = client_class(self)
147
148 def error_response(self, code, description):
149 """Write a response with an error, and disconnect."""
150 self.write_transport("HTTP/1.0 %d %s" % (code, description) + CRLF * 2)
151 self.transport.loseConnection()
152 self.client.stop()
153 self.clearLineBuffer()
154
155 def write_transport(self, data):
156 """Write a response in the transport."""
157 self.transport.write(data)
158
159 def handle_first_line(self, line):
160 """Special handling for the first line received."""
161 try:
162 method, hostport, proto_version = line.split(" ", 2)
163 if proto_version != "HTTP/1.0":
164 self.error_response(505, "HTTP Version Not Supported")
165 return
166 if method != "CONNECT":
167 self.error_response(405, "Only the CONNECT method is allowed")
168 return
169 self.hostport = hostport
170 except ValueError:
171 self.error_response(400, "Bad request")
172
173 @defer.inlineCallbacks
174 def headers_done(self):
175 """An empty line was received, start connecting and switch mode."""
176 try:
177 yield self.client.connect(self.hostport)
178 response_headers = {
179 "Server": "Ubuntu One proxy tunnel",
180 }
181 self.write_transport("HTTP/1.0 200 Proxy connection established" +
182 CRLF + self.format_headers(response_headers) +
183 CRLF)
184 except ConnectionError as e:
185 self.error_response(e.code, e.description)
186
187 def rawDataReceived(self, data):
188 """Tunnel all raw data straight to the other side."""
189 self.client.write(data)
190
191 def response_data_received(self, data):
192 """Return data coming from the other side."""
193 logger.debug("writing data to the transport: %r", data)
194 self.write_transport(data)
195
196
197class Tunnel(object):
198 """An instance of a running tunnel."""
199
200 implements(interfaces.ITransport)
201
202 def __init__(self, local_socket):
203 """Initialize this Tunnel instance."""
204 self.disconnecting = False
205 self.local_socket = local_socket
206 self.protocol = ServerTunnelProtocol(RemoteSocket)
207 self.protocol.transport = self
208 local_socket.readyRead.connect(self.server_ready_read)
209 local_socket.disconnected.connect(self.local_disconnected)
210
211 def server_ready_read(self):
212 """Data available on the local end. Move it forward."""
213 data = bytes(self.local_socket.readAll())
214 self.protocol.dataReceived(data)
215
216 def write(self, data):
217 """Data available on the remote end. Bring it back."""
218 self.local_socket.write(data)
219
220 def loseConnection(self):
221 """The remote end disconnected."""
222 logger.debug("disconnecting local end.")
223 self.local_socket.close()
224
225 def local_disconnected(self):
226 """The local end disconnected."""
227 # TODO: handle this case in an upcoming branch
228
229
230class TunnelServer(object):
231 """A server for tunnel instances."""
232
233 def __init__(self):
234 """Initialize this tunnel instance."""
235 self.tunnels = []
236 self.server = QTcpServer(QCoreApplication.instance())
237 self.server.newConnection.connect(self.new_connection)
238 self.server.listen(QHostAddress.LocalHost, 0)
239
240 def new_connection(self):
241 """On a new connection create a new tunnel instance."""
242 local_socket = self.server.nextPendingConnection()
243 tunnel = Tunnel(local_socket)
244 self.tunnels.append(tunnel)
245
246 def shutdown(self):
247 """Terminate every connection."""
248 # TODO: handle this gracefully in an upcoming branch
249
250 @property
251 def port(self):
252 """The port where this server listens."""
253 return self.server.serverPort()

Subscribers

People subscribed via source and target branches