Merge lp:~leonardr/launchpad/toggle-representation-cache into lp:launchpad
- toggle-representation-cache
- Merge into devel
Status: | Merged |
---|---|
Approved by: | Graham Binns |
Approved revision: | no longer in the source branch. |
Merged at revision: | 11011 |
Proposed branch: | lp:~leonardr/launchpad/toggle-representation-cache |
Merge into: | lp:launchpad |
Diff against target: |
513 lines (+331/-9) 14 files modified
configs/development/launchpad-lazr.conf (+3/-0) lib/canonical/config/schema-lazr.conf (+3/-0) lib/canonical/database/sqlbase.py (+7/-0) lib/canonical/launchpad/pagetests/webservice/cache.txt (+77/-0) lib/canonical/launchpad/rest/configuration.py (+4/-0) lib/canonical/launchpad/testing/pages.py (+12/-0) lib/canonical/launchpad/zcml/webservice.zcml (+5/-0) lib/lp/registry/stories/webservice/xx-project-registry.txt (+2/-0) lib/lp/services/memcache/doc/restful-cache.txt (+143/-0) lib/lp/services/memcache/restful.py (+63/-0) lib/lp/services/memcache/tests/test_doc.py (+8/-8) lib/lp/soyuz/stories/webservice/xx-builds.txt (+1/-0) lib/lp/soyuz/stories/webservice/xx-source-package-publishing.txt (+2/-0) versions.cfg (+1/-1) |
To merge this branch: | bzr merge lp:~leonardr/launchpad/toggle-representation-cache |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Aaron Bentley (community) | Approve | ||
Graham Binns (community) | code | Approve | |
Paul Hummer (community) | code | Approve | |
Jelmer Vernooij (community) | code | Approve | |
Review via email: mp+26725@code.launchpad.net |
Commit message
Description of the change
This branch implements the enable_
Leonard Richardson (leonardr) wrote : | # |
I need another review of the last couple revisions. While going through stub's review of my earlier branch, I remembered that we planned to set a default expiration time for items in the cache, so that when we tested on staging the cache wouldn't contain bad data indefinitely. I've implemented that code and a test, as well as responding to Jelmer's feedback and some more of Stuart's.
(Stuart's review is here: https:/
I couldn't think of any way to test the expiration except from actually sleeping for 2 seconds, since I don't think we have a mocked-up memcached for use in testing. Suggestions are welcome.
Paul Hummer (rockstar) : | # |
Leonard Richardson (leonardr) wrote : | # |
I need one more tiny review. I got a lots of test failures because memcached thinks space is a control character. So after creating the key, I replace any spaces with periods.
Graham Binns (gmb) : | # |
Leonard Richardson (leonardr) wrote : | # |
I was still getting test failures, so I made a number of changes (which required a new version of lazr.restful: see https:/
Aaron Bentley (abentley) wrote : | # |
As discussed in IRC
- Please catch storm.exception
- If possible, narrow the scope of the related try/except block.
- Please be more specific in applying defaults, so that cached values that evaluate to False are handled correctly.
Preview Diff
1 | === modified file 'configs/development/launchpad-lazr.conf' | |||
2 | --- configs/development/launchpad-lazr.conf 2010-05-27 07:05:20 +0000 | |||
3 | +++ configs/development/launchpad-lazr.conf 2010-06-15 13:39:43 +0000 | |||
4 | @@ -275,6 +275,9 @@ | |||
5 | 275 | [vhost.api] | 275 | [vhost.api] |
6 | 276 | hostname: api.launchpad.dev | 276 | hostname: api.launchpad.dev |
7 | 277 | rooturl: https://api.launchpad.dev/ | 277 | rooturl: https://api.launchpad.dev/ |
8 | 278 | # Turn this on once we've solved cache invalidation problems and are | ||
9 | 279 | # ready to test. | ||
10 | 280 | # enable_server_side_representation_cache: True | ||
11 | 278 | 281 | ||
12 | 279 | [vhost.blueprints] | 282 | [vhost.blueprints] |
13 | 280 | hostname: blueprints.launchpad.dev | 283 | hostname: blueprints.launchpad.dev |
14 | 281 | 284 | ||
15 | === modified file 'lib/canonical/config/schema-lazr.conf' | |||
16 | --- lib/canonical/config/schema-lazr.conf 2010-06-08 15:13:20 +0000 | |||
17 | +++ lib/canonical/config/schema-lazr.conf 2010-06-15 13:39:43 +0000 | |||
18 | @@ -1950,6 +1950,9 @@ | |||
19 | 1950 | [vhost.api] | 1950 | [vhost.api] |
20 | 1951 | # This key should be removed once the production configs have been updated. | 1951 | # This key should be removed once the production configs have been updated. |
21 | 1952 | beta_test_team: disabled | 1952 | beta_test_team: disabled |
22 | 1953 | enable_server_side_representation_cache: False | ||
23 | 1954 | # By default, cache representations for 4 hours. | ||
24 | 1955 | representation_cache_expiration_time: 14400 | ||
25 | 1953 | 1956 | ||
26 | 1954 | [vhost.blueprints] | 1957 | [vhost.blueprints] |
27 | 1955 | 1958 | ||
28 | 1956 | 1959 | ||
29 | === modified file 'lib/canonical/database/sqlbase.py' | |||
30 | --- lib/canonical/database/sqlbase.py 2010-03-24 17:37:26 +0000 | |||
31 | +++ lib/canonical/database/sqlbase.py 2010-06-15 13:39:43 +0000 | |||
32 | @@ -28,6 +28,8 @@ | |||
33 | 28 | from zope.interface import implements | 28 | from zope.interface import implements |
34 | 29 | from zope.security.proxy import removeSecurityProxy | 29 | from zope.security.proxy import removeSecurityProxy |
35 | 30 | 30 | ||
36 | 31 | from lazr.restful.interfaces import IRepresentationCache | ||
37 | 32 | |||
38 | 31 | from canonical.config import config, dbconfig | 33 | from canonical.config import config, dbconfig |
39 | 32 | from canonical.database.interfaces import ISQLBase | 34 | from canonical.database.interfaces import ISQLBase |
40 | 33 | 35 | ||
41 | @@ -246,6 +248,11 @@ | |||
42 | 246 | """Inverse of __eq__.""" | 248 | """Inverse of __eq__.""" |
43 | 247 | return not (self == other) | 249 | return not (self == other) |
44 | 248 | 250 | ||
45 | 251 | def __storm_flushed__(self): | ||
46 | 252 | """Invalidate the web service cache.""" | ||
47 | 253 | cache = getUtility(IRepresentationCache) | ||
48 | 254 | cache.delete(self) | ||
49 | 255 | |||
50 | 249 | alreadyInstalledMsg = ("A ZopelessTransactionManager with these settings is " | 256 | alreadyInstalledMsg = ("A ZopelessTransactionManager with these settings is " |
51 | 250 | "already installed. This is probably caused by calling initZopeless twice.") | 257 | "already installed. This is probably caused by calling initZopeless twice.") |
52 | 251 | 258 | ||
53 | 252 | 259 | ||
54 | === added file 'lib/canonical/launchpad/pagetests/webservice/cache.txt' | |||
55 | --- lib/canonical/launchpad/pagetests/webservice/cache.txt 1970-01-01 00:00:00 +0000 | |||
56 | +++ lib/canonical/launchpad/pagetests/webservice/cache.txt 2010-06-15 13:39:43 +0000 | |||
57 | @@ -0,0 +1,77 @@ | |||
58 | 1 | ************************ | ||
59 | 2 | The representation cache | ||
60 | 3 | ************************ | ||
61 | 4 | |||
62 | 5 | Launchpad stores JSON representations of objects in a memcached | ||
63 | 6 | cache. The full cache functionality is tested in lazr.restful and in | ||
64 | 7 | lib/lp/services/memcache/doc/restful-cache.txt. This is just a simple | ||
65 | 8 | integration test. | ||
66 | 9 | |||
67 | 10 | By default, the cache is disabled, even in the testrunner | ||
68 | 11 | environment. (This will change once we improve the Launchpad cache's | ||
69 | 12 | cache invalidation.) Let's enable the cache just for this test. | ||
70 | 13 | |||
71 | 14 | >>> from canonical.config import config | ||
72 | 15 | >>> config.vhost.api.enable_server_side_representation_cache | ||
73 | 16 | False | ||
74 | 17 | >>> config.push('enable cache', """\ | ||
75 | 18 | ... [vhost.api] | ||
76 | 19 | ... enable_server_side_representation_cache: True | ||
77 | 20 | ... """) | ||
78 | 21 | |||
79 | 22 | Now we need to get a reference to the cache object, so we can look | ||
80 | 23 | inside. | ||
81 | 24 | |||
82 | 25 | >>> from zope.component import getUtility | ||
83 | 26 | >>> from lazr.restful.interfaces import IRepresentationCache | ||
84 | 27 | >>> cache = getUtility(IRepresentationCache) | ||
85 | 28 | |||
86 | 29 | Since the cache is keyed by the underlying database object, we also | ||
87 | 30 | need one of those objects. | ||
88 | 31 | |||
89 | 32 | >>> from lp.registry.interfaces.person import IPersonSet | ||
90 | 33 | >>> login(ANONYMOUS) | ||
91 | 34 | >>> person = getUtility(IPersonSet).getByName('salgado') | ||
92 | 35 | >>> key = cache.key_for(person, 'application/json', 'devel') | ||
93 | 36 | >>> logout() | ||
94 | 37 | |||
95 | 38 | The cache starts out empty. | ||
96 | 39 | |||
97 | 40 | >>> print cache.get_by_key(key) | ||
98 | 41 | None | ||
99 | 42 | |||
100 | 43 | Retrieving a representation of an object populates the cache. | ||
101 | 44 | |||
102 | 45 | >>> ignore = webservice.get("/~salgado", api_version="devel").jsonBody() | ||
103 | 46 | |||
104 | 47 | >>> cache.get_by_key(key) | ||
105 | 48 | '{...}' | ||
106 | 49 | |||
107 | 50 | Once the cache is populated with a representation, the cached | ||
108 | 51 | representation is used in preference to generating a new | ||
109 | 52 | representation of that object. We can verify this by putting a fake | ||
110 | 53 | value into the cache and retrieving a representation of the | ||
111 | 54 | corresponding object. | ||
112 | 55 | |||
113 | 56 | >>> import simplejson | ||
114 | 57 | >>> cache.set_by_key(key, simplejson.dumps("Fake representation")) | ||
115 | 58 | |||
116 | 59 | >>> print webservice.get("/~salgado", api_version="devel").jsonBody() | ||
117 | 60 | Fake representation | ||
118 | 61 | |||
119 | 62 | If there's a problem with the cache or the invalidation code, we can | ||
120 | 63 | disable the cache by setting a configuration variable. | ||
121 | 64 | |||
122 | 65 | Cleanup: re-disable the cache. | ||
123 | 66 | |||
124 | 67 | >>> ignore = config.pop('enable cache') | ||
125 | 68 | |||
126 | 69 | Note that documents are never served from a disabled cache, even if the | ||
127 | 70 | cache is populated. | ||
128 | 71 | |||
129 | 72 | >>> print webservice.get("/~salgado", api_version="devel").jsonBody() | ||
130 | 73 | {...} | ||
131 | 74 | |||
132 | 75 | Cleanup: clear the cache. | ||
133 | 76 | |||
134 | 77 | >>> cache.delete(person) | ||
135 | 0 | 78 | ||
136 | === modified file 'lib/canonical/launchpad/rest/configuration.py' | |||
137 | --- lib/canonical/launchpad/rest/configuration.py 2010-05-17 20:03:02 +0000 | |||
138 | +++ lib/canonical/launchpad/rest/configuration.py 2010-06-15 13:39:43 +0000 | |||
139 | @@ -63,6 +63,10 @@ | |||
140 | 63 | return request | 63 | return request |
141 | 64 | 64 | ||
142 | 65 | @property | 65 | @property |
143 | 66 | def enable_server_side_representation_cache(self): | ||
144 | 67 | return config.vhost.api.enable_server_side_representation_cache | ||
145 | 68 | |||
146 | 69 | @property | ||
147 | 66 | def default_batch_size(self): | 70 | def default_batch_size(self): |
148 | 67 | return config.launchpad.default_batch_size | 71 | return config.launchpad.default_batch_size |
149 | 68 | 72 | ||
150 | 69 | 73 | ||
151 | === modified file 'lib/canonical/launchpad/testing/pages.py' | |||
152 | --- lib/canonical/launchpad/testing/pages.py 2010-03-30 09:40:48 +0000 | |||
153 | +++ lib/canonical/launchpad/testing/pages.py 2010-06-15 13:39:43 +0000 | |||
154 | @@ -40,6 +40,7 @@ | |||
155 | 40 | from canonical.launchpad.webapp.interfaces import OAuthPermission | 40 | from canonical.launchpad.webapp.interfaces import OAuthPermission |
156 | 41 | from canonical.launchpad.webapp.url import urlsplit | 41 | from canonical.launchpad.webapp.url import urlsplit |
157 | 42 | from canonical.testing import PageTestLayer | 42 | from canonical.testing import PageTestLayer |
158 | 43 | from lazr.restful.interfaces import IRepresentationCache | ||
159 | 43 | from lazr.restful.testing.webservice import WebServiceCaller | 44 | from lazr.restful.testing.webservice import WebServiceCaller |
160 | 44 | from lp.testing import ( | 45 | from lp.testing import ( |
161 | 45 | ANONYMOUS, launchpadlib_for, login, login_person, logout) | 46 | ANONYMOUS, launchpadlib_for, login, login_person, logout) |
162 | @@ -672,6 +673,16 @@ | |||
163 | 672 | return LaunchpadWebServiceCaller(consumer_key, access_token.key) | 673 | return LaunchpadWebServiceCaller(consumer_key, access_token.key) |
164 | 673 | 674 | ||
165 | 674 | 675 | ||
166 | 676 | def ws_uncache(obj): | ||
167 | 677 | """Manually remove an object from the web service representation cache. | ||
168 | 678 | |||
169 | 679 | Directly modifying a data model object during a test may leave | ||
170 | 680 | invalid data in the representation cache. | ||
171 | 681 | """ | ||
172 | 682 | cache = getUtility(IRepresentationCache) | ||
173 | 683 | cache.delete(obj) | ||
174 | 684 | |||
175 | 685 | |||
176 | 675 | def setupDTCBrowser(): | 686 | def setupDTCBrowser(): |
177 | 676 | """Testbrowser configured for Distribution Translations Coordinators. | 687 | """Testbrowser configured for Distribution Translations Coordinators. |
178 | 677 | 688 | ||
179 | @@ -776,6 +787,7 @@ | |||
180 | 776 | test.globs['print_tag_with_id'] = print_tag_with_id | 787 | test.globs['print_tag_with_id'] = print_tag_with_id |
181 | 777 | test.globs['PageTestLayer'] = PageTestLayer | 788 | test.globs['PageTestLayer'] = PageTestLayer |
182 | 778 | test.globs['stop'] = stop | 789 | test.globs['stop'] = stop |
183 | 790 | test.globs['ws_uncache'] = ws_uncache | ||
184 | 779 | 791 | ||
185 | 780 | 792 | ||
186 | 781 | class PageStoryTestCase(unittest.TestCase): | 793 | class PageStoryTestCase(unittest.TestCase): |
187 | 782 | 794 | ||
188 | === modified file 'lib/canonical/launchpad/zcml/webservice.zcml' | |||
189 | --- lib/canonical/launchpad/zcml/webservice.zcml 2010-03-26 19:11:50 +0000 | |||
190 | +++ lib/canonical/launchpad/zcml/webservice.zcml 2010-06-15 13:39:43 +0000 | |||
191 | @@ -16,6 +16,11 @@ | |||
192 | 16 | provides="lazr.restful.interfaces.IWebServiceConfiguration"> | 16 | provides="lazr.restful.interfaces.IWebServiceConfiguration"> |
193 | 17 | </utility> | 17 | </utility> |
194 | 18 | 18 | ||
195 | 19 | <utility | ||
196 | 20 | factory="lp.services.memcache.restful.MemcachedStormRepresentationCache" | ||
197 | 21 | provides="lazr.restful.interfaces.IRepresentationCache"> | ||
198 | 22 | </utility> | ||
199 | 23 | |||
200 | 19 | <securedutility | 24 | <securedutility |
201 | 20 | class="canonical.launchpad.systemhomes.WebServiceApplication" | 25 | class="canonical.launchpad.systemhomes.WebServiceApplication" |
202 | 21 | provides="canonical.launchpad.interfaces.IWebServiceApplication"> | 26 | provides="canonical.launchpad.interfaces.IWebServiceApplication"> |
203 | 22 | 27 | ||
204 | === modified file 'lib/lp/registry/stories/webservice/xx-project-registry.txt' | |||
205 | --- lib/lp/registry/stories/webservice/xx-project-registry.txt 2010-06-11 18:05:59 +0000 | |||
206 | +++ lib/lp/registry/stories/webservice/xx-project-registry.txt 2010-06-15 13:39:43 +0000 | |||
207 | @@ -1250,6 +1250,8 @@ | |||
208 | 1250 | 1250 | ||
209 | 1251 | >>> logout() | 1251 | >>> logout() |
210 | 1252 | 1252 | ||
211 | 1253 | >>> from lazr.restful.interfaces import IRepresentationCache | ||
212 | 1254 | >>> ws_uncache(mmm) | ||
213 | 1253 | >>> mmm = webservice.get("/mega-money-maker").jsonBody() | 1255 | >>> mmm = webservice.get("/mega-money-maker").jsonBody() |
214 | 1254 | >>> print mmm['display_name'] | 1256 | >>> print mmm['display_name'] |
215 | 1255 | Mega Money Maker | 1257 | Mega Money Maker |
216 | 1256 | 1258 | ||
217 | === added file 'lib/lp/services/memcache/doc/restful-cache.txt' | |||
218 | --- lib/lp/services/memcache/doc/restful-cache.txt 1970-01-01 00:00:00 +0000 | |||
219 | +++ lib/lp/services/memcache/doc/restful-cache.txt 2010-06-15 13:39:43 +0000 | |||
220 | @@ -0,0 +1,143 @@ | |||
221 | 1 | **************************************** | ||
222 | 2 | The Storm/memcached representation cache | ||
223 | 3 | **************************************** | ||
224 | 4 | |||
225 | 5 | The web service library lazr.restful will store the representations it | ||
226 | 6 | generates in a cache, if a suitable cache implementation is | ||
227 | 7 | provided. We implement a cache that stores representations of Storm | ||
228 | 8 | objects in memcached. | ||
229 | 9 | |||
230 | 10 | >>> login('foo.bar@canonical.com') | ||
231 | 11 | |||
232 | 12 | >>> from lp.services.memcache.restful import ( | ||
233 | 13 | ... MemcachedStormRepresentationCache) | ||
234 | 14 | >>> cache = MemcachedStormRepresentationCache() | ||
235 | 15 | |||
236 | 16 | An object's cache key is derived from its Storm metadata: its database | ||
237 | 17 | table name and its primary key. | ||
238 | 18 | |||
239 | 19 | >>> from zope.component import getUtility | ||
240 | 20 | >>> from lp.registry.interfaces.person import IPersonSet | ||
241 | 21 | >>> person = getUtility(IPersonSet).getByName('salgado') | ||
242 | 22 | |||
243 | 23 | >>> cache_key = cache.key_for( | ||
244 | 24 | ... person, 'media/type', 'web-service-version') | ||
245 | 25 | >>> print person.id, cache_key | ||
246 | 26 | 29 Person(29,),testrunner,media/type,web-service-version | ||
247 | 27 | |||
248 | 28 | >>> from operator import attrgetter | ||
249 | 29 | >>> languages = sorted(person.languages, key=attrgetter('englishname')) | ||
250 | 30 | >>> for language in languages: | ||
251 | 31 | ... cache_key = cache.key_for( | ||
252 | 32 | ... language, 'media/type', 'web-service-version') | ||
253 | 33 | ... print language.id, cache_key | ||
254 | 34 | 119 Language(119,),testrunner,media/type,web-service-version | ||
255 | 35 | 521 Language(521,),testrunner,media/type,web-service-version | ||
256 | 36 | |||
257 | 37 | The cache starts out empty. | ||
258 | 38 | |||
259 | 39 | >>> json_type = 'application/json' | ||
260 | 40 | |||
261 | 41 | >>> print cache.get(person, json_type, "v1", default="missing") | ||
262 | 42 | missing | ||
263 | 43 | |||
264 | 44 | Add a representation to the cache, and you can retrieve it later. | ||
265 | 45 | |||
266 | 46 | >>> cache.set(person, json_type, "beta", | ||
267 | 47 | ... "This is a representation for version beta.") | ||
268 | 48 | |||
269 | 49 | >>> print cache.get(person, json_type, "beta") | ||
270 | 50 | This is a representation for version beta. | ||
271 | 51 | |||
272 | 52 | If an object has no Storm metadata, it is currently not cached at all. | ||
273 | 53 | |||
274 | 54 | >>> from lp.hardwaredb.interfaces.hwdb import IHWDBApplication | ||
275 | 55 | >>> hwdb_app = getUtility(IHWDBApplication) | ||
276 | 56 | >>> cache.set(hwdb_app, 'media/type', 'web-service-version', 'data') | ||
277 | 57 | >>> print cache.get(hwdb_app, 'media/type', 'web-service-version') | ||
278 | 58 | None | ||
279 | 59 | |||
280 | 60 | A single object can cache different representations for different | ||
281 | 61 | web service versions. | ||
282 | 62 | |||
283 | 63 | >>> cache.set(person, json_type, '1.0', | ||
284 | 64 | ... 'This is a different representation for version 1.0.') | ||
285 | 65 | |||
286 | 66 | >>> print cache.get(person, json_type, "1.0") | ||
287 | 67 | This is a different representation for version 1.0. | ||
288 | 68 | |||
289 | 69 | The web service version doesn't have to actually be defined in the | ||
290 | 70 | configuration. (But you shouldn't use this--see below!) | ||
291 | 71 | |||
292 | 72 | >>> cache.set(person, json_type, 'no-such-version', | ||
293 | 73 | ... 'This is a representation for a nonexistent version.') | ||
294 | 74 | |||
295 | 75 | >>> print cache.get(person, json_type, "no-such-version") | ||
296 | 76 | This is a representation for a nonexistent version. | ||
297 | 77 | |||
298 | 78 | A single object can also cache different representations for different | ||
299 | 79 | media types, not just application/json. (But you shouldn't use | ||
300 | 80 | this--see below!) | ||
301 | 81 | |||
302 | 82 | >>> cache.set(person, 'media/type', '1.0', | ||
303 | 83 | ... 'This is a representation for a strange media type.') | ||
304 | 84 | |||
305 | 85 | >>> print cache.get(person, "media/type", "1.0") | ||
306 | 86 | This is a representation for a strange media type. | ||
307 | 87 | |||
308 | 88 | When a Launchpad object is modified, its JSON representations for | ||
309 | 89 | recognized web service versions are automatically removed from the | ||
310 | 90 | cache. | ||
311 | 91 | |||
312 | 92 | >>> person.addressline1 = "New address" | ||
313 | 93 | >>> from canonical.launchpad.ftests import syncUpdate | ||
314 | 94 | >>> syncUpdate(person) | ||
315 | 95 | |||
316 | 96 | >>> print cache.get(person, json_type, "beta", default="missing") | ||
317 | 97 | missing | ||
318 | 98 | |||
319 | 99 | >>> print cache.get(person, json_type, "1.0", default="missing") | ||
320 | 100 | missing | ||
321 | 101 | |||
322 | 102 | But non-JSON representations, and representations for unrecognized web | ||
323 | 103 | service versions, are _not_ removed from the cache. (This is why you | ||
324 | 104 | shouldn't put such representations into the cache.) | ||
325 | 105 | |||
326 | 106 | >>> print cache.get(person, json_type, "no-such-version") | ||
327 | 107 | This is a representation for a nonexistent version. | ||
328 | 108 | |||
329 | 109 | >>> print cache.get(person, "media/type", "1.0") | ||
330 | 110 | This is a representation for a strange media type. | ||
331 | 111 | |||
332 | 112 | Expiration time | ||
333 | 113 | =============== | ||
334 | 114 | |||
335 | 115 | Objects will automatically be purged from the cache after a | ||
336 | 116 | configurable time. The default is 4 hours (14400 seconds) | ||
337 | 117 | |||
338 | 118 | >>> from canonical.config import config | ||
339 | 119 | >>> print config.vhost.api.representation_cache_expiration_time | ||
340 | 120 | 14400 | ||
341 | 121 | |||
342 | 122 | Let's change that. | ||
343 | 123 | |||
344 | 124 | >>> from canonical.config import config | ||
345 | 125 | >>> config.push('short expiration time', """\ | ||
346 | 126 | ... [vhost.api] | ||
347 | 127 | ... representation_cache_expiration_time: 1 | ||
348 | 128 | ... """) | ||
349 | 129 | |||
350 | 130 | >>> cache.set(person, "some/type", "version", "Some representation") | ||
351 | 131 | >>> print cache.get(person, "some/type", "version") | ||
352 | 132 | Some representation | ||
353 | 133 | |||
354 | 134 | Now objects are purged from the cache after one second. | ||
355 | 135 | |||
356 | 136 | >>> import time | ||
357 | 137 | >>> time.sleep(2) | ||
358 | 138 | >>> print cache.get(person, "some/type", "version") | ||
359 | 139 | None | ||
360 | 140 | |||
361 | 141 | Cleanup. | ||
362 | 142 | |||
363 | 143 | >>> ignore = config.pop("short expiration time") | ||
364 | 0 | 144 | ||
365 | === added file 'lib/lp/services/memcache/restful.py' | |||
366 | --- lib/lp/services/memcache/restful.py 1970-01-01 00:00:00 +0000 | |||
367 | +++ lib/lp/services/memcache/restful.py 2010-06-15 13:39:43 +0000 | |||
368 | @@ -0,0 +1,63 @@ | |||
369 | 1 | # Copyright 2010 Canonical Ltd. This software is licensed under the | ||
370 | 2 | # GNU Affero General Public License version 3 (see the file LICENSE). | ||
371 | 3 | |||
372 | 4 | """Storm/memcached implementation of lazr.restful's representation cache.""" | ||
373 | 5 | |||
374 | 6 | import storm | ||
375 | 7 | |||
376 | 8 | from zope.component import getUtility | ||
377 | 9 | from zope.security.proxy import removeSecurityProxy | ||
378 | 10 | from zope.traversing.browser import absoluteURL | ||
379 | 11 | |||
380 | 12 | from canonical.config import config | ||
381 | 13 | from lp.services.memcache.interfaces import IMemcacheClient | ||
382 | 14 | from lazr.restful.simple import BaseRepresentationCache | ||
383 | 15 | from lazr.restful.utils import get_current_web_service_request | ||
384 | 16 | |||
385 | 17 | __metaclass__ = type | ||
386 | 18 | __all__ = [ | ||
387 | 19 | 'MemcachedStormRepresentationCache', | ||
388 | 20 | ] | ||
389 | 21 | |||
390 | 22 | |||
391 | 23 | class MemcachedStormRepresentationCache(BaseRepresentationCache): | ||
392 | 24 | """Caches lazr.restful representations of Storm objects in memcached.""" | ||
393 | 25 | |||
394 | 26 | def __init__(self): | ||
395 | 27 | """Initialize with the memcached client.""" | ||
396 | 28 | self.client = getUtility(IMemcacheClient) | ||
397 | 29 | |||
398 | 30 | def key_for(self, obj, media_type, version): | ||
399 | 31 | """See `BaseRepresentationCache`.""" | ||
400 | 32 | obj = removeSecurityProxy(obj) | ||
401 | 33 | try: | ||
402 | 34 | storm_info = storm.info.get_obj_info(obj) | ||
403 | 35 | except storm.exceptions.ClassInfoError, e: | ||
404 | 36 | # There's no Storm data for this object. Don't cache it, | ||
405 | 37 | # since we don't know how to invalidate the cache. | ||
406 | 38 | return self.DO_NOT_CACHE | ||
407 | 39 | table_name = storm_info.cls_info.table | ||
408 | 40 | primary_key = tuple(var.get() for var in storm_info.primary_vars) | ||
409 | 41 | identifier = table_name + repr(primary_key) | ||
410 | 42 | |||
411 | 43 | key = (identifier | ||
412 | 44 | + ',' + config._instance_name | ||
413 | 45 | + ',' + media_type + ',' + str(version)).replace(' ', '.') | ||
414 | 46 | return key | ||
415 | 47 | |||
416 | 48 | def get_by_key(self, key, default=None): | ||
417 | 49 | """See `BaseRepresentationCache`.""" | ||
418 | 50 | value = self.client.get(key) | ||
419 | 51 | if value is None: | ||
420 | 52 | value = default | ||
421 | 53 | return value | ||
422 | 54 | |||
423 | 55 | def set_by_key(self, key, value): | ||
424 | 56 | """See `BaseRepresentationCache`.""" | ||
425 | 57 | self.client.set( | ||
426 | 58 | key, value, | ||
427 | 59 | time=config.vhost.api.representation_cache_expiration_time) | ||
428 | 60 | |||
429 | 61 | def delete_by_key(self, key): | ||
430 | 62 | """See `BaseRepresentationCache`.""" | ||
431 | 63 | self.client.delete(key) | ||
432 | 0 | 64 | ||
433 | === modified file 'lib/lp/services/memcache/tests/test_doc.py' | |||
434 | --- lib/lp/services/memcache/tests/test_doc.py 2010-03-03 11:00:42 +0000 | |||
435 | +++ lib/lp/services/memcache/tests/test_doc.py 2010-06-15 13:39:43 +0000 | |||
436 | @@ -7,9 +7,7 @@ | |||
437 | 7 | 7 | ||
438 | 8 | import os.path | 8 | import os.path |
439 | 9 | from textwrap import dedent | 9 | from textwrap import dedent |
440 | 10 | import unittest | ||
441 | 11 | 10 | ||
442 | 12 | from zope.component import getUtility | ||
443 | 13 | import zope.pagetemplate.engine | 11 | import zope.pagetemplate.engine |
444 | 14 | from zope.pagetemplate.pagetemplate import PageTemplate | 12 | from zope.pagetemplate.pagetemplate import PageTemplate |
445 | 15 | from zope.publisher.browser import TestRequest | 13 | from zope.publisher.browser import TestRequest |
446 | @@ -17,9 +15,7 @@ | |||
447 | 17 | from canonical.launchpad.testing.systemdocs import ( | 15 | from canonical.launchpad.testing.systemdocs import ( |
448 | 18 | LayeredDocFileSuite, setUp, tearDown) | 16 | LayeredDocFileSuite, setUp, tearDown) |
449 | 19 | from canonical.testing.layers import LaunchpadFunctionalLayer, MemcachedLayer | 17 | from canonical.testing.layers import LaunchpadFunctionalLayer, MemcachedLayer |
450 | 20 | from lp.services.memcache.interfaces import IMemcacheClient | ||
451 | 21 | from lp.services.testing import build_test_suite | 18 | from lp.services.testing import build_test_suite |
452 | 22 | from lp.testing import TestCase | ||
453 | 23 | 19 | ||
454 | 24 | 20 | ||
455 | 25 | here = os.path.dirname(os.path.realpath(__file__)) | 21 | here = os.path.dirname(os.path.realpath(__file__)) |
456 | @@ -59,11 +55,15 @@ | |||
457 | 59 | test.globs['MemcachedLayer'] = MemcachedLayer | 55 | test.globs['MemcachedLayer'] = MemcachedLayer |
458 | 60 | 56 | ||
459 | 61 | 57 | ||
460 | 58 | def suite_for_doctest(filename): | ||
461 | 59 | return LayeredDocFileSuite( | ||
462 | 60 | '../doc/%s' % filename, | ||
463 | 61 | setUp=memcacheSetUp, tearDown=tearDown, | ||
464 | 62 | layer=LaunchpadFunctionalLayer) | ||
465 | 63 | |||
466 | 62 | special = { | 64 | special = { |
471 | 63 | 'tales-cache.txt': LayeredDocFileSuite( | 65 | 'tales-cache.txt': suite_for_doctest('tales-cache.txt'), |
472 | 64 | '../doc/tales-cache.txt', | 66 | 'restful-cache.txt': suite_for_doctest('restful-cache.txt'), |
469 | 65 | setUp=memcacheSetUp, tearDown=tearDown, | ||
470 | 66 | layer=LaunchpadFunctionalLayer), | ||
473 | 67 | } | 67 | } |
474 | 68 | 68 | ||
475 | 69 | 69 | ||
476 | 70 | 70 | ||
477 | === modified file 'lib/lp/soyuz/stories/webservice/xx-builds.txt' | |||
478 | --- lib/lp/soyuz/stories/webservice/xx-builds.txt 2010-05-17 11:30:31 +0000 | |||
479 | +++ lib/lp/soyuz/stories/webservice/xx-builds.txt 2010-06-15 13:39:43 +0000 | |||
480 | @@ -93,6 +93,7 @@ | |||
481 | 93 | 93 | ||
482 | 94 | >>> build = getUtility(IBinaryPackageBuildSet).getByBuildID(26) | 94 | >>> build = getUtility(IBinaryPackageBuildSet).getByBuildID(26) |
483 | 95 | >>> build.upload_log = build.log | 95 | >>> build.upload_log = build.log |
484 | 96 | >>> ws_uncache(build) | ||
485 | 96 | 97 | ||
486 | 97 | >>> logout() | 98 | >>> logout() |
487 | 98 | 99 | ||
488 | 99 | 100 | ||
489 | === modified file 'lib/lp/soyuz/stories/webservice/xx-source-package-publishing.txt' | |||
490 | --- lib/lp/soyuz/stories/webservice/xx-source-package-publishing.txt 2010-02-26 14:49:00 +0000 | |||
491 | +++ lib/lp/soyuz/stories/webservice/xx-source-package-publishing.txt 2010-06-15 13:39:43 +0000 | |||
492 | @@ -152,6 +152,8 @@ | |||
493 | 152 | ... pub.sourcepackagerelease.dscsigningkey = None | 152 | ... pub.sourcepackagerelease.dscsigningkey = None |
494 | 153 | >>> logout() | 153 | >>> logout() |
495 | 154 | 154 | ||
496 | 155 | >>> ws_uncache(pub) | ||
497 | 156 | |||
498 | 155 | Query the source again: | 157 | Query the source again: |
499 | 156 | 158 | ||
500 | 157 | >>> pubs = webservice.named_get( | 159 | >>> pubs = webservice.named_get( |
501 | 158 | 160 | ||
502 | === modified file 'versions.cfg' | |||
503 | --- versions.cfg 2010-06-09 19:45:18 +0000 | |||
504 | +++ versions.cfg 2010-06-15 13:39:43 +0000 | |||
505 | @@ -28,7 +28,7 @@ | |||
506 | 28 | lazr.delegates = 1.1.0 | 28 | lazr.delegates = 1.1.0 |
507 | 29 | lazr.enum = 1.1.2 | 29 | lazr.enum = 1.1.2 |
508 | 30 | lazr.lifecycle = 1.1 | 30 | lazr.lifecycle = 1.1 |
510 | 31 | lazr.restful = 0.9.26 | 31 | lazr.restful = 0.9.29 |
511 | 32 | lazr.restfulclient = 0.9.14 | 32 | lazr.restfulclient = 0.9.14 |
512 | 33 | lazr.smtptest = 1.1 | 33 | lazr.smtptest = 1.1 |
513 | 34 | lazr.testing = 0.1.1 | 34 | lazr.testing = 0.1.1 |
In lib/lp/ services/ memcache/ doc/restful- cache.txt you're missing an asterisk on the first line.
It'd be nice to have docstrings for the methods in MemcachedStormR epresentationCa che, even if they just say "See `BaseRepository Cache`. "