Merge lp:~ntt-pf-lab/nova/monkey_patch_notification into lp:~hudson-openstack/nova/trunk

Proposed by Nachi Ueno
Status: Merged
Approved by: Ed Leafe
Approved revision: 1463
Merged at revision: 1482
Proposed branch: lp:~ntt-pf-lab/nova/monkey_patch_notification
Merge into: lp:~hudson-openstack/nova/trunk
Diff against target: 426 lines (+245/-0)
16 files modified
bin/nova-api (+1/-0)
bin/nova-api-ec2 (+1/-0)
bin/nova-api-os (+1/-0)
bin/nova-compute (+1/-0)
bin/nova-network (+1/-0)
bin/nova-objectstore (+1/-0)
bin/nova-scheduler (+3/-0)
bin/nova-volume (+1/-0)
nova/flags.py (+9/-0)
nova/notifier/api.py (+27/-0)
nova/tests/monkey_patch_example/__init__.py (+33/-0)
nova/tests/monkey_patch_example/example_a.py (+29/-0)
nova/tests/monkey_patch_example/example_b.py (+30/-0)
nova/tests/test_notifier.py (+21/-0)
nova/tests/test_utils.py (+45/-0)
nova/utils.py (+41/-0)
To merge this branch: bzr merge lp:~ntt-pf-lab/nova/monkey_patch_notification
Reviewer Review Type Date Requested Status
Ed Leafe (community) Approve
Alex Meade (community) Approve
Joshua McKenty (community) Approve
Matt Dietz Pending
Jesse Andrews Pending
Review via email: mp+72262@code.launchpad.net

Commit message

I added notifications decorator for each API call using monkey_patching.
By this merge, users can get API call notification from any modules.

Description of the change

I added notifications decorator for each API call using monkey_patching.
By this merge, users can get API call notification from any modules.

I added some following Flags.

New Flags
- monkey_patch: Whether monkey patched or not
- monkey_patch_modules:
 Module list representing monkey patched module and decorator
 This value is colon separated
   module:decorator

In addition, I wrote notify decorator which notify functions calls.

To post a comment you must log in.
Revision history for this message
Joshua McKenty (joshua-mckenty) wrote :

lgtm - this will make it much easier to debug things.

review: Approve
Revision history for this message
Alex Meade (alex-meade) wrote :

122: Should this be "whether to log monkey patching"?

Is there any way tests can be written to test this functionality?

136, 174: Can some doc-strings be added here?

Revision history for this message
Alex Meade (alex-meade) wrote :

I also get this when building a virtualenv:

  Could not find any downloads that satisfy the requirement pyclbr (from -r /home/alexmeade/nova/monkey_patch_notification/tools/pip-requires (line 37))
No distributions at all found for pyclbr

Revision history for this message
William Wolf (throughnothing) wrote :

I think we should have tests that test the False, and True state of the monkey_patch FLAG, if possible.

Also, agree with Alex that doc strings (and even additional comments) would be helpful on these functions as monkey_patch() is fairly confusing/complex.

I think pyclbr may be standard, which means it doesn't need to be in pip-requires at all?

Revision history for this message
Nachi Ueno (nati-ueno) wrote :

Hi Alex, William

Thank you for your review!

I am going to work
1. add test case
2. check pyclbr problem
3. Add doc strings

Thanks
2011/8/22 William Wolf <email address hidden>:
> I think we should have tests that test the False, and True state of the monkey_patch FLAG, if possible.
>
> Also, agree with Alex that doc strings (and even additional comments) would be helpful on these functions as monkey_patch() is fairly confusing/complex.
>
> I think pyclbr may be standard, which means it doesn't need to be in pip-requires at all?
> --
> https://code.launchpad.net/~ntt-pf-lab/nova/monkey_patch_notification/+merge/72262
> You proposed lp:~ntt-pf-lab/nova/monkey_patch_notification for merging.
>

--
Nachi Ueno
email:<email address hidden>
twitter:http://twitter.com/nati

Revision history for this message
Nachi Ueno (nati-ueno) wrote :

Hi Alex, William

I added docstring, tests and fixed pyclbr problem.
Would you please check it again?

Revision history for this message
Alex Meade (alex-meade) wrote :

maybe rename the 'example' directory so we know they are for monkey patch notification?

405, 411: typos

137: docstrings need to be changed a tad, see the HACKING file

review: Needs Fixing
Revision history for this message
Nachi Ueno (nati-ueno) wrote :

HI Alex

Thanks again
I fixed your point, so would you please check it

Revision history for this message
Alex Meade (alex-meade) wrote :

Good work, hate to be a nitpicker but:

184, 227, 264: Docstring should be all on one line with a period

211, 248: Should this be a NASA copyright? And should be 2011 not 2010

154-156: Should be lined up with FLAGS

293: Should be on one line
 example_api = nova.notifier.api.notify_decorator(

Revision history for this message
Nachi Ueno (nati-ueno) wrote :

Thanks Alex again
You are very helpful

184, 227, 264: Docstring should be all on one line with a period <- Fixed

211, 248: Should this be a NASA copyright? And should be 2011 not 2010 <- Fixed

154-156: Should be lined up with FLAGS <- Fixed (I addred default_publisher_id flag)

293: Should be on one line <- Fixed
 example_api = nova.notifier.api.notify_decorator(

Thanks a lot

Revision history for this message
Alex Meade (alex-meade) wrote :

Nice I like it

review: Approve
Revision history for this message
Nachi Ueno (nati-ueno) wrote :

Thanks Alex!

Revision history for this message
Ed Leafe (ed-leafe) wrote :

Minor doc issue: The flag description on lines 127-8 uses an incorrect line continuation. The string as written on one line would be:
'Module list representing monkey patched module and decorator'

The correct way of writing that flag declaration would be:
DEFINE_list('monkey_patch_modules',
        ['nova.api.ec2.cloud:nova.notifier.api.notify_decorator',
        'nova.compute.api:nova.notifier.api.notify_decorator'],
        'Module list representing monkey '
        'patched module and decorator')

Revision history for this message
Nachi Ueno (nati-ueno) wrote :

Thanks Ed!

Revision history for this message
Nachi Ueno (nati-ueno) wrote :

Hi Ed
I fixed your point, would you please check it again?

Revision history for this message
Ed Leafe (ed-leafe) wrote :

ok, looks good now.

review: Approve
Revision history for this message
Nachi Ueno (nati-ueno) wrote :

Thanks!

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'bin/nova-api'
2--- bin/nova-api 2011-08-18 17:55:39 +0000
3+++ bin/nova-api 2011-08-23 19:08:38 +0000
4@@ -45,6 +45,7 @@
5 utils.default_flagfile()
6 flags.FLAGS(sys.argv)
7 logging.setup()
8+ utils.monkey_patch()
9 servers = []
10 for api in flags.FLAGS.enabled_apis:
11 servers.append(service.WSGIService(api))
12
13=== modified file 'bin/nova-api-ec2'
14--- bin/nova-api-ec2 2011-08-18 18:31:28 +0000
15+++ bin/nova-api-ec2 2011-08-23 19:08:38 +0000
16@@ -41,6 +41,7 @@
17 utils.default_flagfile()
18 flags.FLAGS(sys.argv)
19 logging.setup()
20+ utils.monkey_patch()
21 server = service.WSGIService('ec2')
22 service.serve(server)
23 service.wait()
24
25=== modified file 'bin/nova-api-os'
26--- bin/nova-api-os 2011-08-18 18:31:28 +0000
27+++ bin/nova-api-os 2011-08-23 19:08:38 +0000
28@@ -41,6 +41,7 @@
29 utils.default_flagfile()
30 flags.FLAGS(sys.argv)
31 logging.setup()
32+ utils.monkey_patch()
33 server = service.WSGIService('osapi')
34 service.serve(server)
35 service.wait()
36
37=== modified file 'bin/nova-compute'
38--- bin/nova-compute 2011-08-18 18:28:02 +0000
39+++ bin/nova-compute 2011-08-23 19:08:38 +0000
40@@ -43,6 +43,7 @@
41 utils.default_flagfile()
42 flags.FLAGS(sys.argv)
43 logging.setup()
44+ utils.monkey_patch()
45 server = service.Service.create(binary='nova-compute')
46 service.serve(server)
47 service.wait()
48
49=== modified file 'bin/nova-network'
50--- bin/nova-network 2011-08-18 18:28:02 +0000
51+++ bin/nova-network 2011-08-23 19:08:38 +0000
52@@ -43,6 +43,7 @@
53 utils.default_flagfile()
54 flags.FLAGS(sys.argv)
55 logging.setup()
56+ utils.monkey_patch()
57 server = service.Service.create(binary='nova-network')
58 service.serve(server)
59 service.wait()
60
61=== modified file 'bin/nova-objectstore'
62--- bin/nova-objectstore 2011-08-18 17:55:39 +0000
63+++ bin/nova-objectstore 2011-08-23 19:08:38 +0000
64@@ -49,6 +49,7 @@
65 utils.default_flagfile()
66 FLAGS(sys.argv)
67 logging.setup()
68+ utils.monkey_patch()
69 router = s3server.S3Application(FLAGS.buckets_path)
70 server = wsgi.Server("S3 Objectstore",
71 router,
72
73=== modified file 'bin/nova-scheduler'
74--- bin/nova-scheduler 2011-08-18 18:28:43 +0000
75+++ bin/nova-scheduler 2011-08-23 19:08:38 +0000
76@@ -22,6 +22,7 @@
77 import eventlet
78 eventlet.monkey_patch()
79
80+import gettext
81 import os
82 import sys
83
84@@ -33,6 +34,7 @@
85 if os.path.exists(os.path.join(possible_topdir, 'nova', '__init__.py')):
86 sys.path.insert(0, possible_topdir)
87
88+gettext.install('nova', unicode=1)
89
90 from nova import flags
91 from nova import log as logging
92@@ -43,6 +45,7 @@
93 utils.default_flagfile()
94 flags.FLAGS(sys.argv)
95 logging.setup()
96+ utils.monkey_patch()
97 server = service.Service.create(binary='nova-scheduler')
98 service.serve(server)
99 service.wait()
100
101=== modified file 'bin/nova-volume'
102--- bin/nova-volume 2011-08-18 18:28:02 +0000
103+++ bin/nova-volume 2011-08-23 19:08:38 +0000
104@@ -43,6 +43,7 @@
105 utils.default_flagfile()
106 flags.FLAGS(sys.argv)
107 logging.setup()
108+ utils.monkey_patch()
109 server = service.Service.create(binary='nova-volume')
110 service.serve(server)
111 service.wait()
112
113=== modified file 'nova/flags.py'
114--- nova/flags.py 2011-08-23 14:31:34 +0000
115+++ nova/flags.py 2011-08-23 19:08:38 +0000
116@@ -404,3 +404,12 @@
117 'Command prefix to use for running commands as root')
118
119 DEFINE_bool('use_ipv6', False, 'use ipv6')
120+
121+DEFINE_bool('monkey_patch', False,
122+ 'Whether to log monkey patching')
123+
124+DEFINE_list('monkey_patch_modules',
125+ ['nova.api.ec2.cloud:nova.notifier.api.notify_decorator',
126+ 'nova.compute.api:nova.notifier.api.notify_decorator'],
127+ 'Module list representing monkey '
128+ 'patched module and decorator')
129
130=== modified file 'nova/notifier/api.py'
131--- nova/notifier/api.py 2011-07-29 19:09:17 +0000
132+++ nova/notifier/api.py 2011-08-23 19:08:38 +0000
133@@ -25,6 +25,9 @@
134
135 flags.DEFINE_string('default_notification_level', 'INFO',
136 'Default notification level for outgoing notifications')
137+flags.DEFINE_string('default_publisher_id', FLAGS.host,
138+ 'Default publisher_id for outgoing notifications')
139+
140
141 WARN = 'WARN'
142 INFO = 'INFO'
143@@ -39,6 +42,30 @@
144 pass
145
146
147+def notify_decorator(name, fn):
148+ """ decorator for notify which is used from utils.monkey_patch()
149+
150+ :param name: name of the function
151+ :param function: - object of the function
152+ :returns: function -- decorated function
153+
154+ """
155+ def wrapped_func(*args, **kwarg):
156+ body = {}
157+ body['args'] = []
158+ body['kwarg'] = {}
159+ for arg in args:
160+ body['args'].append(arg)
161+ for key in kwarg:
162+ body['kwarg'][key] = kwarg[key]
163+ notify(FLAGS.default_publisher_id,
164+ name,
165+ FLAGS.default_notification_level,
166+ body)
167+ return fn(*args, **kwarg)
168+ return wrapped_func
169+
170+
171 def publisher_id(service, host=None):
172 if not host:
173 host = FLAGS.host
174
175=== added directory 'nova/tests/monkey_patch_example'
176=== added file 'nova/tests/monkey_patch_example/__init__.py'
177--- nova/tests/monkey_patch_example/__init__.py 1970-01-01 00:00:00 +0000
178+++ nova/tests/monkey_patch_example/__init__.py 2011-08-23 19:08:38 +0000
179@@ -0,0 +1,33 @@
180+# vim: tabstop=4 shiftwidth=4 softtabstop=4
181+
182+# Copyright 2011 OpenStack LLC.
183+# All Rights Reserved.
184+#
185+# Licensed under the Apache License, Version 2.0 (the "License"); you may
186+# not use this file except in compliance with the License. You may obtain
187+# a copy of the License at
188+#
189+# http://www.apache.org/licenses/LICENSE-2.0
190+#
191+# Unless required by applicable law or agreed to in writing, software
192+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
193+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
194+# License for the specific language governing permissions and limitations
195+# under the License.
196+"""Example Module for testing utils.monkey_patch()."""
197+
198+
199+CALLED_FUNCTION = []
200+
201+
202+def example_decorator(name, function):
203+ """ decorator for notify which is used from utils.monkey_patch()
204+
205+ :param name: name of the function
206+ :param function: - object of the function
207+ :returns: function -- decorated function
208+ """
209+ def wrapped_func(*args, **kwarg):
210+ CALLED_FUNCTION.append(name)
211+ return function(*args, **kwarg)
212+ return wrapped_func
213
214=== added file 'nova/tests/monkey_patch_example/example_a.py'
215--- nova/tests/monkey_patch_example/example_a.py 1970-01-01 00:00:00 +0000
216+++ nova/tests/monkey_patch_example/example_a.py 2011-08-23 19:08:38 +0000
217@@ -0,0 +1,29 @@
218+# vim: tabstop=4 shiftwidth=4 softtabstop=4
219+
220+# Copyright 2011 OpenStack LLC.
221+# All Rights Reserved.
222+#
223+# Licensed under the Apache License, Version 2.0 (the "License"); you may
224+# not use this file except in compliance with the License. You may obtain
225+# a copy of the License at
226+#
227+# http://www.apache.org/licenses/LICENSE-2.0
228+#
229+# Unless required by applicable law or agreed to in writing, software
230+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
231+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
232+# License for the specific language governing permissions and limitations
233+# under the License.
234+"""Example Module A for testing utils.monkey_patch()."""
235+
236+
237+def example_function_a():
238+ return 'Example function'
239+
240+
241+class ExampleClassA():
242+ def example_method(self):
243+ return 'Example method'
244+
245+ def example_method_add(self, arg1, arg2):
246+ return arg1 + arg2
247
248=== added file 'nova/tests/monkey_patch_example/example_b.py'
249--- nova/tests/monkey_patch_example/example_b.py 1970-01-01 00:00:00 +0000
250+++ nova/tests/monkey_patch_example/example_b.py 2011-08-23 19:08:38 +0000
251@@ -0,0 +1,30 @@
252+# vim: tabstop=4 shiftwidth=4 softtabstop=4
253+
254+# Copyright 2011 OpenStack LLC.
255+# All Rights Reserved.
256+#
257+# Licensed under the Apache License, Version 2.0 (the "License"); you may
258+# not use this file except in compliance with the License. You may obtain
259+# a copy of the License at
260+#
261+# http://www.apache.org/licenses/LICENSE-2.0
262+#
263+# Unless required by applicable law or agreed to in writing, software
264+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
265+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
266+# License for the specific language governing permissions and limitations
267+# under the License.
268+
269+"""Example Module B for testing utils.monkey_patch()."""
270+
271+
272+def example_function_b():
273+ return 'Example function'
274+
275+
276+class ExampleClassB():
277+ def example_method(self):
278+ return 'Example method'
279+
280+ def example_method_add(self, arg1, arg2):
281+ return arg1 + arg2
282
283=== modified file 'nova/tests/test_notifier.py'
284--- nova/tests/test_notifier.py 2011-06-01 14:17:00 +0000
285+++ nova/tests/test_notifier.py 2011-08-23 19:08:38 +0000
286@@ -134,3 +134,24 @@
287 self.assertEqual(msg['event_type'], 'error_notification')
288 self.assertEqual(msg['priority'], 'ERROR')
289 self.assertEqual(msg['payload']['error'], 'foo')
290+
291+ def test_send_notification_by_decorator(self):
292+ self.notify_called = False
293+
294+ def example_api(arg1, arg2):
295+ return arg1 + arg2
296+
297+ example_api = nova.notifier.api.notify_decorator(
298+ 'example_api',
299+ example_api)
300+
301+ def mock_notify(cls, *args):
302+ self.notify_called = True
303+
304+ self.stubs.Set(nova.notifier.no_op_notifier, 'notify',
305+ mock_notify)
306+
307+ class Mock(object):
308+ pass
309+ self.assertEqual(3, example_api(1, 2))
310+ self.assertEqual(self.notify_called, True)
311
312=== modified file 'nova/tests/test_utils.py'
313--- nova/tests/test_utils.py 2011-08-19 17:01:25 +0000
314+++ nova/tests/test_utils.py 2011-08-23 19:08:38 +0000
315@@ -18,6 +18,7 @@
316 import os
317 import tempfile
318
319+import nova
320 from nova import exception
321 from nova import test
322 from nova import utils
323@@ -394,3 +395,47 @@
324 self.assertTrue(ret[0].startswith(u"<module 'datetime' from "))
325 self.assertTrue(ret[1].startswith(u'<function foo at 0x'))
326 self.assertEquals(ret[2], u'<built-in function dir>')
327+
328+
329+class MonkeyPatchTestCase(test.TestCase):
330+ """Unit test for utils.monkey_patch()."""
331+ def setUp(self):
332+ super(MonkeyPatchTestCase, self).setUp()
333+ self.example_package = 'nova.tests.monkey_patch_example.'
334+ self.flags(
335+ monkey_patch=True,
336+ monkey_patch_modules=[self.example_package + 'example_a' + ':'
337+ + self.example_package + 'example_decorator'])
338+
339+ def test_monkey_patch(self):
340+ utils.monkey_patch()
341+ nova.tests.monkey_patch_example.CALLED_FUNCTION = []
342+ from nova.tests.monkey_patch_example import example_a, example_b
343+
344+ self.assertEqual('Example function', example_a.example_function_a())
345+ exampleA = example_a.ExampleClassA()
346+ exampleA.example_method()
347+ ret_a = exampleA.example_method_add(3, 5)
348+ self.assertEqual(ret_a, 8)
349+
350+ self.assertEqual('Example function', example_b.example_function_b())
351+ exampleB = example_b.ExampleClassB()
352+ exampleB.example_method()
353+ ret_b = exampleB.example_method_add(3, 5)
354+
355+ self.assertEqual(ret_b, 8)
356+ package_a = self.example_package + 'example_a.'
357+ self.assertTrue(package_a + 'example_function_a'
358+ in nova.tests.monkey_patch_example.CALLED_FUNCTION)
359+
360+ self.assertTrue(package_a + 'ExampleClassA.example_method'
361+ in nova.tests.monkey_patch_example.CALLED_FUNCTION)
362+ self.assertTrue(package_a + 'ExampleClassA.example_method_add'
363+ in nova.tests.monkey_patch_example.CALLED_FUNCTION)
364+ package_b = self.example_package + 'example_b.'
365+ self.assertFalse(package_b + 'example_function_b'
366+ in nova.tests.monkey_patch_example.CALLED_FUNCTION)
367+ self.assertFalse(package_b + 'ExampleClassB.example_method'
368+ in nova.tests.monkey_patch_example.CALLED_FUNCTION)
369+ self.assertFalse(package_b + 'ExampleClassB.example_method_add'
370+ in nova.tests.monkey_patch_example.CALLED_FUNCTION)
371
372=== modified file 'nova/utils.py'
373--- nova/utils.py 2011-08-22 23:35:09 +0000
374+++ nova/utils.py 2011-08-23 19:08:38 +0000
375@@ -35,6 +35,7 @@
376 import time
377 import types
378 import uuid
379+import pyclbr
380 from xml.sax import saxutils
381
382 from eventlet import event
383@@ -860,3 +861,43 @@
384 except ValueError:
385 return False
386 return True
387+
388+
389+def monkey_patch():
390+ """ If the Flags.monkey_patch set as True,
391+ this functuion patches a decorator
392+ for all functions in specified modules.
393+ You can set decorators for each modules
394+ using FLAGS.monkey_patch_modules.
395+ The format is "Module path:Decorator function".
396+ Example: 'nova.api.ec2.cloud:nova.notifier.api.notify_decorator'
397+
398+ Parameters of the decorator is as follows.
399+ (See nova.notifier.api.notify_decorator)
400+
401+ name - name of the function
402+ function - object of the function
403+ """
404+ # If FLAGS.monkey_patch is not True, this function do nothing.
405+ if not FLAGS.monkey_patch:
406+ return
407+ # Get list of modules and decorators
408+ for module_and_decorator in FLAGS.monkey_patch_modules:
409+ module, decorator_name = module_and_decorator.split(':')
410+ # import decorator function
411+ decorator = import_class(decorator_name)
412+ __import__(module)
413+ # Retrieve module information using pyclbr
414+ module_data = pyclbr.readmodule_ex(module)
415+ for key in module_data.keys():
416+ # set the decorator for the class methods
417+ if isinstance(module_data[key], pyclbr.Class):
418+ clz = import_class("%s.%s" % (module, key))
419+ for method, func in inspect.getmembers(clz, inspect.ismethod):
420+ setattr(clz, method,\
421+ decorator("%s.%s.%s" % (module, key, method), func))
422+ # set the decorator for the function
423+ if isinstance(module_data[key], pyclbr.Function):
424+ func = import_class("%s.%s" % (module, key))
425+ setattr(sys.modules[module], key,\
426+ decorator("%s.%s" % (module, key), func))