Merge lp:~elachuni/piston-mini-client/timeouts into lp:piston-mini-client
- timeouts
- Merge into trunk
Status: | Merged | ||||||||
---|---|---|---|---|---|---|---|---|---|
Approved by: | Anthony Lenton | ||||||||
Approved revision: | 55 | ||||||||
Merged at revision: | 53 | ||||||||
Proposed branch: | lp:~elachuni/piston-mini-client/timeouts | ||||||||
Merge into: | lp:piston-mini-client | ||||||||
Diff against target: |
268 lines (+137/-8) 6 files modified
doc/envvars.rst (+3/-0) doc/tuning.rst (+17/-0) piston_mini_client/__init__.py (+31/-8) piston_mini_client/consts.py (+1/-0) piston_mini_client/failhandlers.py (+12/-0) piston_mini_client/tests/test_timeout.py (+73/-0) |
||||||||
To merge this branch: | bzr merge lp:~elachuni/piston-mini-client/timeouts | ||||||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
software-store-developers | Pending | ||
Review via email: mp+100417@code.launchpad.net |
Commit message
Description of the change
This branch adds code for configuring and handling socket timeouts, to fix bug #937865
While I was there, I also wrapped other socket exceptions so that all errors are raised as APIError (or subclasses of that).
Jonathan Lange (jml) wrote : | # |
Michael Vogt (mvo) wrote : | # |
On Mon, Apr 02, 2012 at 02:12:26PM -0000, Jonathan Lange wrote:
[..]
> * Have you had a chance to test this with a pathological endpoint as described in bug 937865? I don't know off the top of my head how to make such a thing, but would be happy to figure it out together with you. I don't think such a thing needs to be added to the automatic test suite, but it would be good to have at least once-off verification that the patch achieves its purpose.
[..]
We recently added some code to software-center to test slow network,
it maybe useful for this test as well (we borrowed it from bzr iirc):
Use: "tc qdisc del dev eth0 root" (or wlan0 or whatnot) to reset the
slowness again.
Cheers,
Michael
- 54. By Anthony Lenton
-
Changed default default_timeout from 0 to None.
- 55. By Anthony Lenton
-
Added a test to check default default_timeout.
Anthony Lenton (elachuni) wrote : | # |
> Thanks for looking into this Anthony.
>
> A few thoughts,
>
> * Does the timeout span the whole request & response, or just the time it
> takes to get a connected socket? If the former, then a timeout per request
> would be very helpful.
It affects both connect, read and write, but I afaik *each* of these operations gets the same timeout, so it's not a timeout for the whole API-level operation. Testing with a reasonably slow api and long request (staging ubuntu-
Python 2.7.3rc2 (default, Mar 21 2012, 21:13:11)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import timeit
>>> import sreclient
>>>
>>> api = sreclient.
... service_root='http://
... timeout=0.5, log_filename=
>>>
>>> def stmt():
... try:
... api._get(
... extra_headers=
... except:
... print "Operation timed out!"
...
>>>
>>> timeit.repeat(stmt, number=1, repeat=3)
Operation timed out!
Operation timed out!
Operation timed out!
[3.723790884017
re: Adding a timeout per request, I think this would make sense but httplib2 doesn't offer a per-request timeout, as it does its best to reuse connections between requests and only sets the timeout when it instantiates the connection (although I believe you can update the timeout on a socket no prob, so there should be a way to do this). I would be interested in looking into this option a bit more.
> * Have you had a chance to test this with a pathological endpoint as
> described in bug 937865? I don't know off the top of my head how to make such
> a thing, but would be happy to figure it out together with you. I don't think
> such a thing needs to be added to the automatic test suite, but it would be
> good to have at least once-off verification that the patch achieves its
> purpose.
The test above seems to say that it *doesn't* provide the kind of timeout you were asking for :/
> * It would be helpful if the timeout exception included the time taken and a
> pointer to some docs on how to change the timeout.
Where would you expect to find the pointer to the docs, when you print the exception? in the docstring? somewhere else?
> Thanks again,
> jml
Cheers,
achuni.
Preview Diff
1 | === modified file 'doc/envvars.rst' | |||
2 | --- doc/envvars.rst 2012-03-30 21:35:01 +0000 | |||
3 | +++ doc/envvars.rst 2012-04-02 14:26:18 +0000 | |||
4 | @@ -13,6 +13,9 @@ | |||
5 | 13 | ``piston_mini_client`` will log all requests and responses, including | 13 | ``piston_mini_client`` will log all requests and responses, including |
6 | 14 | headers, to this location. | 14 | headers, to this location. |
7 | 15 | 15 | ||
8 | 16 | * ``PISTON_MINI_CLIENT_DEFAULT_TIMEOUT``: Is used as a socket timeout for | ||
9 | 17 | instances that don't explicitly set a timeout. Should be in seconds. | ||
10 | 18 | |||
11 | 16 | * ``http_proxy`` / ``https_proxy``: ``piston_mini_client`` will check these | 19 | * ``http_proxy`` / ``https_proxy``: ``piston_mini_client`` will check these |
12 | 17 | variables to determine if a proxy should be used for each scheme. | 20 | variables to determine if a proxy should be used for each scheme. |
13 | 18 | The `SocksiPy <http://socksipy.sourceforge.net/>`_ module is needed for | 21 | The `SocksiPy <http://socksipy.sourceforge.net/>`_ module is needed for |
14 | 19 | 22 | ||
15 | === modified file 'doc/tuning.rst' | |||
16 | --- doc/tuning.rst 2012-03-30 21:35:01 +0000 | |||
17 | +++ doc/tuning.rst 2012-04-02 14:26:18 +0000 | |||
18 | @@ -176,3 +176,20 @@ | |||
19 | 176 | ``PISTON_MINI_CLIENT_LOG_FILENAME`` in the environment to point a file, | 176 | ``PISTON_MINI_CLIENT_LOG_FILENAME`` in the environment to point a file, |
20 | 177 | and all ``PistonAPI`` instances will use this location by default. That is, | 177 | and all ``PistonAPI`` instances will use this location by default. That is, |
21 | 178 | unless they're explicitly being instantiated to log elsewhere. | 178 | unless they're explicitly being instantiated to log elsewhere. |
22 | 179 | |||
23 | 180 | ================= | ||
24 | 181 | Handling timeouts | ||
25 | 182 | ================= | ||
26 | 183 | When you instantiate a ``PistonAPI`` you can provide an optional ``timeout`` | ||
27 | 184 | argument that will be used as a socket timeout for the requests that instance | ||
28 | 185 | makes. To explicitly set no timeout, pass in ``timeout=0``. If you leave | ||
29 | 186 | the default ``timeout=None``, the instance will first check for an environment | ||
30 | 187 | variable ``PISTON_MINI_CLIENT_DEFAULT_TIMEOUT``, and if that is undefined or | ||
31 | 188 | invalid, then the class's default timeout will be used; this can be defined | ||
32 | 189 | by setting a ``default_timeout`` class attribute when | ||
33 | 190 | writing the API class. Finally, if the class's default timeout is | ||
34 | 191 | also ``None``, Python's system-wide socket default timeout will be used. | ||
35 | 192 | |||
36 | 193 | You can't currently define timeouts on a per-request basis. If you need to | ||
37 | 194 | change the timeout used for certain requests, you'll need to use a new | ||
38 | 195 | ``PistonAPI`` instance. | ||
39 | 179 | 196 | ||
40 | === modified file 'piston_mini_client/__init__.py' | |||
41 | --- piston_mini_client/__init__.py 2012-03-30 21:35:01 +0000 | |||
42 | +++ piston_mini_client/__init__.py 2012-04-02 14:26:18 +0000 | |||
43 | @@ -16,6 +16,7 @@ | |||
44 | 16 | 16 | ||
45 | 17 | import json | 17 | import json |
46 | 18 | import os | 18 | import os |
47 | 19 | import socket | ||
48 | 19 | import urllib | 20 | import urllib |
49 | 20 | from datetime import datetime | 21 | from datetime import datetime |
50 | 21 | from functools import wraps | 22 | from functools import wraps |
51 | @@ -23,13 +24,16 @@ | |||
52 | 23 | 24 | ||
53 | 24 | from piston_mini_client.failhandlers import ( | 25 | from piston_mini_client.failhandlers import ( |
54 | 25 | APIError, | 26 | APIError, |
55 | 27 | ExceptionFailHandler, | ||
56 | 26 | format_request, | 28 | format_request, |
57 | 27 | format_response, | 29 | format_response, |
59 | 28 | ExceptionFailHandler, | 30 | SocketError, |
60 | 31 | TimeoutError, | ||
61 | 29 | ) | 32 | ) |
62 | 30 | from piston_mini_client.consts import ( | 33 | from piston_mini_client.consts import ( |
63 | 31 | DISABLE_SSL_VALIDATION_ENVVAR, | 34 | DISABLE_SSL_VALIDATION_ENVVAR, |
64 | 32 | LOG_FILENAME_ENVVAR, | 35 | LOG_FILENAME_ENVVAR, |
65 | 36 | TIMEOUT_ENVVAR, | ||
66 | 33 | ) | 37 | ) |
67 | 34 | 38 | ||
68 | 35 | 39 | ||
69 | @@ -195,6 +199,7 @@ | |||
70 | 195 | It's a small wrapper around ``httplib2`` to allow for a bit of state to | 199 | It's a small wrapper around ``httplib2`` to allow for a bit of state to |
71 | 196 | be stored (like the service root) so that you don't need to repeat | 200 | be stored (like the service root) so that you don't need to repeat |
72 | 197 | yourself as much. | 201 | yourself as much. |
73 | 202 | |||
74 | 198 | It's not intended to be used directly. Children classes should implement | 203 | It's not intended to be used directly. Children classes should implement |
75 | 199 | methods that actually call out to the api methods. | 204 | methods that actually call out to the api methods. |
76 | 200 | 205 | ||
77 | @@ -203,20 +208,16 @@ | |||
78 | 203 | methods provided by this class. | 208 | methods provided by this class. |
79 | 204 | """ | 209 | """ |
80 | 205 | SUPPORTED_SCHEMAS = ("http", "https") | 210 | SUPPORTED_SCHEMAS = ("http", "https") |
81 | 206 | |||
82 | 207 | default_service_root = '' | 211 | default_service_root = '' |
83 | 208 | |||
84 | 209 | default_content_type = 'application/json' | 212 | default_content_type = 'application/json' |
86 | 210 | 213 | default_timeout = None | |
87 | 211 | fail_handler = ExceptionFailHandler | 214 | fail_handler = ExceptionFailHandler |
88 | 212 | |||
89 | 213 | extra_headers = None | 215 | extra_headers = None |
90 | 214 | |||
91 | 215 | serializers = None | 216 | serializers = None |
92 | 216 | 217 | ||
93 | 217 | def __init__(self, service_root=None, cachedir=None, auth=None, | 218 | def __init__(self, service_root=None, cachedir=None, auth=None, |
94 | 218 | offline_mode=False, disable_ssl_validation=False, | 219 | offline_mode=False, disable_ssl_validation=False, |
96 | 219 | log_filename=None): | 220 | log_filename=None, timeout=None): |
97 | 220 | """Initialize a ``PistonAPI``. | 221 | """Initialize a ``PistonAPI``. |
98 | 221 | 222 | ||
99 | 222 | ``service_root`` is the url to the server's service root. | 223 | ``service_root`` is the url to the server's service root. |
100 | @@ -241,6 +242,15 @@ | |||
101 | 241 | 242 | ||
102 | 242 | If you pass in a ``log_filename``, all requests and responses | 243 | If you pass in a ``log_filename``, all requests and responses |
103 | 243 | including headers will be logged to this file. | 244 | including headers will be logged to this file. |
104 | 245 | |||
105 | 246 | ``timeout`` will be used as a socket timeout for all calls this | ||
106 | 247 | instance makes. To explicitly set no timeout, set timeout=0. The | ||
107 | 248 | default timeout=None will first check for an environment variable | ||
108 | 249 | ``PISTON_MINI_CLIENT_DEFAULT_TIMEOUT`` and try to use that. If this | ||
109 | 250 | environment variable is not found or it is an invalid float, the | ||
110 | 251 | class's ``default_timeout`` will be used. Finally, if the class's | ||
111 | 252 | default is also None, Python's default timeout for sockets will be | ||
112 | 253 | used. All these should be in seconds. | ||
113 | 244 | """ | 254 | """ |
114 | 245 | if service_root is None: | 255 | if service_root is None: |
115 | 246 | service_root = self.default_service_root | 256 | service_root = self.default_service_root |
116 | @@ -259,6 +269,12 @@ | |||
117 | 259 | self._auth = auth | 269 | self._auth = auth |
118 | 260 | self._offline_mode = offline_mode | 270 | self._offline_mode = offline_mode |
119 | 261 | self._disable_ssl_validation = disable_ssl_validation | 271 | self._disable_ssl_validation = disable_ssl_validation |
120 | 272 | if timeout is None: | ||
121 | 273 | try: | ||
122 | 274 | timeout = float(os.environ.get(TIMEOUT_ENVVAR)) | ||
123 | 275 | except (TypeError, ValueError): | ||
124 | 276 | timeout = self.default_timeout | ||
125 | 277 | self._timeout = timeout | ||
126 | 262 | # create one httplib2.Http object per scheme so that we can | 278 | # create one httplib2.Http object per scheme so that we can |
127 | 263 | # have per-scheme proxy settings (see also Issue 26 | 279 | # have per-scheme proxy settings (see also Issue 26 |
128 | 264 | # http://code.google.com/p/httplib2/issues/detail?id=26) | 280 | # http://code.google.com/p/httplib2/issues/detail?id=26) |
129 | @@ -279,6 +295,7 @@ | |||
130 | 279 | try: | 295 | try: |
131 | 280 | http = httplib2.Http( | 296 | http = httplib2.Http( |
132 | 281 | cache=self._httplib2_cache, | 297 | cache=self._httplib2_cache, |
133 | 298 | timeout=self._timeout, | ||
134 | 282 | disable_ssl_certificate_validation=True, | 299 | disable_ssl_certificate_validation=True, |
135 | 283 | proxy_info=proxy_info) | 300 | proxy_info=proxy_info) |
136 | 284 | except TypeError: | 301 | except TypeError: |
137 | @@ -286,7 +303,7 @@ | |||
138 | 286 | pass | 303 | pass |
139 | 287 | if http is None: | 304 | if http is None: |
140 | 288 | http = httplib2.Http(cache=self._httplib2_cache, | 305 | http = httplib2.Http(cache=self._httplib2_cache, |
142 | 289 | proxy_info=proxy_info) | 306 | timeout=self._timeout, proxy_info=proxy_info) |
143 | 290 | return http | 307 | return http |
144 | 291 | 308 | ||
145 | 292 | def _get_proxy_info(self, scheme): | 309 | def _get_proxy_info(self, scheme): |
146 | @@ -498,6 +515,12 @@ | |||
147 | 498 | raise APIError('Unable to connect to %s' % self._service_root) | 515 | raise APIError('Unable to connect to %s' % self._service_root) |
148 | 499 | else: | 516 | else: |
149 | 500 | raise | 517 | raise |
150 | 518 | except socket.timeout, e: | ||
151 | 519 | raise TimeoutError('Timed out attempting to connect to %s' % | ||
152 | 520 | self._service_root) | ||
153 | 521 | except (socket.gaierror, socket.error), e: | ||
154 | 522 | raise SocketError('connecting to %s: %s' % (self._service_root, | ||
155 | 523 | e.message)) | ||
156 | 501 | if self.log_filename: | 524 | if self.log_filename: |
157 | 502 | self._dump_response(response, response_body) | 525 | self._dump_response(response, response_body) |
158 | 503 | handler = self.fail_handler(url, method, body, headers) | 526 | handler = self.fail_handler(url, method, body, headers) |
159 | 504 | 527 | ||
160 | === modified file 'piston_mini_client/consts.py' | |||
161 | --- piston_mini_client/consts.py 2012-03-30 21:35:01 +0000 | |||
162 | +++ piston_mini_client/consts.py 2012-04-02 14:26:18 +0000 | |||
163 | @@ -5,3 +5,4 @@ | |||
164 | 5 | DISABLE_SSL_VALIDATION_ENVVAR = 'PISTON_MINI_CLIENT_DISABLE_SSL_VALIDATION' | 5 | DISABLE_SSL_VALIDATION_ENVVAR = 'PISTON_MINI_CLIENT_DISABLE_SSL_VALIDATION' |
165 | 6 | DEBUG_ENVVAR = 'PISTON_MINI_CLIENT_DEBUG' | 6 | DEBUG_ENVVAR = 'PISTON_MINI_CLIENT_DEBUG' |
166 | 7 | LOG_FILENAME_ENVVAR = 'PISTON_MINI_CLIENT_LOG_FILENAME' | 7 | LOG_FILENAME_ENVVAR = 'PISTON_MINI_CLIENT_LOG_FILENAME' |
167 | 8 | TIMEOUT_ENVVAR = 'PISTON_MINI_CLIENT_DEFAULT_TIMEOUT' | ||
168 | 8 | 9 | ||
169 | === modified file 'piston_mini_client/failhandlers.py' | |||
170 | --- piston_mini_client/failhandlers.py 2012-03-30 21:35:01 +0000 | |||
171 | +++ piston_mini_client/failhandlers.py 2012-04-02 14:26:18 +0000 | |||
172 | @@ -66,6 +66,18 @@ | |||
173 | 66 | return self.msg | 66 | return self.msg |
174 | 67 | 67 | ||
175 | 68 | 68 | ||
176 | 69 | class SocketError(APIError): | ||
177 | 70 | def __init__(self, msg): | ||
178 | 71 | self.msg = msg | ||
179 | 72 | |||
180 | 73 | def __str__(self): | ||
181 | 74 | return self.msg | ||
182 | 75 | |||
183 | 76 | |||
184 | 77 | class TimeoutError(SocketError): | ||
185 | 78 | pass | ||
186 | 79 | |||
187 | 80 | |||
188 | 69 | class BaseFailHandler(object): | 81 | class BaseFailHandler(object): |
189 | 70 | """A base class for fail handlers. | 82 | """A base class for fail handlers. |
190 | 71 | 83 | ||
191 | 72 | 84 | ||
192 | === added file 'piston_mini_client/tests/test_timeout.py' | |||
193 | --- piston_mini_client/tests/test_timeout.py 1970-01-01 00:00:00 +0000 | |||
194 | +++ piston_mini_client/tests/test_timeout.py 2012-04-02 14:26:18 +0000 | |||
195 | @@ -0,0 +1,73 @@ | |||
196 | 1 | # -*- coding: utf-8 -*- | ||
197 | 2 | # Copyright 2010-2012 Canonical Ltd. This software is licensed under the | ||
198 | 3 | # GNU Lesser General Public License version 3 (see the file LICENSE). | ||
199 | 4 | |||
200 | 5 | import socket | ||
201 | 6 | from unittest import TestCase | ||
202 | 7 | from mock import patch | ||
203 | 8 | from piston_mini_client import PistonAPI | ||
204 | 9 | from piston_mini_client.failhandlers import APIError | ||
205 | 10 | |||
206 | 11 | |||
207 | 12 | class LazyAPI(PistonAPI): | ||
208 | 13 | default_service_root = 'http://test.info/api/1.0/' | ||
209 | 14 | default_timeout = 42 | ||
210 | 15 | |||
211 | 16 | def sleep(self, amount): | ||
212 | 17 | return self._get('/snooze/%s/' % amount) | ||
213 | 18 | |||
214 | 19 | |||
215 | 20 | class TimeoutTestCase(TestCase): | ||
216 | 21 | @patch('os.environ.get') | ||
217 | 22 | @patch('httplib2.Http') | ||
218 | 23 | def test_timeout_in_constructor_wins(self, mock_http, mock_get): | ||
219 | 24 | mock_get.return_value = '3.14' | ||
220 | 25 | api = LazyAPI(timeout=1) | ||
221 | 26 | self.assertEqual(1, api._timeout) | ||
222 | 27 | mock_http.assert_called_with(cache=None, proxy_info=None, timeout=1, | ||
223 | 28 | disable_ssl_certificate_validation=True) | ||
224 | 29 | |||
225 | 30 | @patch('os.environ.get') | ||
226 | 31 | @patch('httplib2.Http') | ||
227 | 32 | def test_timeout_in_env_beats_class_default(self, mock_http, mock_get): | ||
228 | 33 | mock_get.return_value = '3.14' | ||
229 | 34 | api = LazyAPI() | ||
230 | 35 | self.assertEqual(3.14, api._timeout) | ||
231 | 36 | mock_http.assert_called_with(cache=None, proxy_info=None, timeout=3.14, | ||
232 | 37 | disable_ssl_certificate_validation=True) | ||
233 | 38 | |||
234 | 39 | @patch('os.environ.get') | ||
235 | 40 | @patch('httplib2.Http') | ||
236 | 41 | def test_no_envvar_falls_back_to_class_default(self, mock_http, mock_get): | ||
237 | 42 | mock_get.return_value = None | ||
238 | 43 | api = LazyAPI() | ||
239 | 44 | self.assertEqual(42, api._timeout) | ||
240 | 45 | mock_http.assert_called_with(cache=None, proxy_info=None, timeout=42) | ||
241 | 46 | |||
242 | 47 | @patch('os.environ.get') | ||
243 | 48 | @patch('httplib2.Http') | ||
244 | 49 | def test_no_nothing_falls_back_to_system_default(self, mock_http, | ||
245 | 50 | mock_get): | ||
246 | 51 | class DefaultAPI(PistonAPI): | ||
247 | 52 | default_service_root = 'http://test.info/api/1.0/' | ||
248 | 53 | |||
249 | 54 | mock_get.return_value = None | ||
250 | 55 | api = DefaultAPI() | ||
251 | 56 | self.assertEqual(None, api._timeout) | ||
252 | 57 | mock_http.assert_called_with(cache=None, proxy_info=None, timeout=None) | ||
253 | 58 | |||
254 | 59 | @patch('os.environ.get') | ||
255 | 60 | @patch('httplib2.Http') | ||
256 | 61 | def test_invalid_envvar_uses_class_default(self, mock_http, mock_get): | ||
257 | 62 | mock_get.return_value = 'invalid' | ||
258 | 63 | api = LazyAPI() | ||
259 | 64 | self.assertEqual(42, api._timeout) | ||
260 | 65 | mock_http.assert_called_with(cache=None, proxy_info=None, timeout=42, | ||
261 | 66 | disable_ssl_certificate_validation=True) | ||
262 | 67 | |||
263 | 68 | @patch('httplib2.HTTPConnectionWithTimeout.connect') | ||
264 | 69 | def test_timeout_is_handled_by_failhandler(self, mock_connect): | ||
265 | 70 | mock_connect.side_effect = socket.timeout | ||
266 | 71 | api = LazyAPI() | ||
267 | 72 | |||
268 | 73 | self.assertRaises(APIError, api.sleep, 2) |
Thanks for looking into this Anthony.
A few thoughts,
* Does the timeout span the whole request & response, or just the time it takes to get a connected socket? If the former, then a timeout per request would be very helpful.
* Have you had a chance to test this with a pathological endpoint as described in bug 937865? I don't know off the top of my head how to make such a thing, but would be happy to figure it out together with you. I don't think such a thing needs to be added to the automatic test suite, but it would be good to have at least once-off verification that the patch achieves its purpose.
* It would be helpful if the timeout exception included the time taken and a pointer to some docs on how to change the timeout.
Thanks again,
jml