Merge lp:~lifeless/python-oops-tools/amqp into lp:python-oops-tools

Proposed by Robert Collins
Status: Merged
Approved by: Robert Collins
Approved revision: no longer in the source branch.
Merged at revision: 2
Proposed branch: lp:~lifeless/python-oops-tools/amqp
Merge into: lp:python-oops-tools
Diff against target: 408 lines (+242/-29)
8 files modified
buildout.cfg (+1/-1)
setup.py (+2/-0)
src/oopstools/NEWS.txt (+7/-2)
src/oopstools/README.txt (+15/-4)
src/oopstools/oops/models.py (+37/-20)
src/oopstools/oops/test/test_amqp2disk.py (+40/-0)
src/oopstools/scripts/amqp2disk.py (+137/-0)
versions.cfg (+3/-2)
To merge this branch: bzr merge lp:~lifeless/python-oops-tools/amqp
Reviewer Review Type Date Requested Status
Steve Kowalik (community) code Approve
Review via email: mp+79505@code.launchpad.net

Commit message

Add AMQP support and improve docs.

Description of the change

Build on the recent improvements in oops-datedir-repo and oops-amqp to add an amqp queue worker that will take oopses from amqp and load them straight into OOPS-tools. Also fix docs (0.6 is the current release) and stop telling folk to edit product.cfg in-place.

To post a comment you must log in.
Steve Kowalik (stevenk) wrote :

This looks fine to me.

review: Approve (code)
lp:~lifeless/python-oops-tools/amqp updated
2. By Robert Collins on 2011-10-17

Add AMQP support and improve docs.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'buildout.cfg'
2--- buildout.cfg 2011-10-13 20:18:51 +0000
3+++ buildout.cfg 2011-10-16 23:09:17 +0000
4@@ -54,7 +54,7 @@
5 recipe = djangorecipe
6 version = 1.3
7 project = oopstools
8-projectegg = oops-tools
9+projectegg = oopstools
10 settings = settings
11 test = oopstools
12 eggs = oops-tools
13
14=== modified file 'setup.py'
15--- setup.py 2011-10-13 20:18:51 +0000
16+++ setup.py 2011-10-16 23:09:17 +0000
17@@ -57,6 +57,7 @@
18 'launchpadlib',
19 'lazr.config',
20 'oops',
21+ 'oops-amqp',
22 'oops-datedir-repo',
23 'pytz',
24 'setuptools',
25@@ -79,6 +80,7 @@
26 ),
27 entry_points=dict(
28 console_scripts=[ # `console_scripts` is a magic name to setuptools
29+ 'amqp2disk = oopstools.scripts.amqp2disk:main',
30 'analyse_error_reports = oopstools.scripts.analyse_error_reports:main',
31 'load_sample_data = oopstools.scripts.load_sample_data:main',
32 'update_db = oopstools.scripts.update_db:main',
33
34=== modified file 'src/oopstools/NEWS.txt'
35--- src/oopstools/NEWS.txt 2011-10-13 20:18:51 +0000
36+++ src/oopstools/NEWS.txt 2011-10-16 23:09:17 +0000
37@@ -2,8 +2,13 @@
38 NEWS for oopstools
39 ===================
40
41-0.5 (UNRELEASED)
42-================
43+NEXT
44+====
45+
46+* Added AMQP support via the bin/amqp2disk script. (Robert Collins)
47+
48+0.6
49+===
50
51 * Initial release
52
53
54=== modified file 'src/oopstools/README.txt'
55--- src/oopstools/README.txt 2011-10-13 20:18:51 +0000
56+++ src/oopstools/README.txt 2011-10-16 23:09:17 +0000
57@@ -51,19 +51,21 @@
58 Deployment using mod_wsgi
59 =========================
60
61-Update the production.cfg file with your database configuration and the paths
62-to your OOPS directories:
63+Create a custom cfg file - start with production.cfg and take a copy. Update
64+your copy with your database configuration and the paths to your OOPS
65+directories:
66
67 [configuration]
68 db-name = /path/to/your/oops.db
69 index-template = 'index.html'
70-oopsdir = /path/to/oops/reports
71+oopsdir = /path/where/rsynced/oopses/are/found
72+ /another/such/path
73
74 Update settings.py setting a custom SECRET_KEY
75
76 To deploy oops tools make sure all the dependecies are installed.
77
78- * bin/buildout -c production.cfg
79+ * bin/buildout -c yourfilename.cfg
80
81 * Run bin/django syncdb
82
83@@ -71,6 +73,15 @@
84
85 * Copy apache/oops-tools.dev.mod_wsgi to /etc/apache2/sites-available/
86
87+AMQP Integration
88+================
89+
90+The script bin/amqp2disk is an AMQP handler that will receive OOPS reports over
91+AMQP and publish them locally to disk as well as loading the metadata directly
92+into the oops-tools database. To use this you will need to config your OOPS
93+creation to publish over AMQP. If you are using Python then the oops-amqp
94+module will help you do this.
95+
96 Running locally
97 ===============
98
99
100=== modified file 'src/oopstools/oops/models.py'
101--- src/oopstools/oops/models.py 2011-10-13 20:18:51 +0000
102+++ src/oopstools/oops/models.py 2011-10-16 23:09:17 +0000
103@@ -21,6 +21,8 @@
104 import datetime
105 import os.path
106
107+from pytz import utc
108+
109 from django.db import models
110 from django.db.models.signals import pre_save
111 import oops_datedir_repo.serializer
112@@ -278,7 +280,11 @@
113 prefix = oops.get('reporter')
114 if not prefix:
115 # Legacy support for pre-reporter using OOPSes.
116- prefix = oops_re.match(oopsid).group('oopsprefix')
117+ prefix_match = oops_re.match(oopsid)
118+ if prefix_match is not None:
119+ prefix = prefix_match.group('oopsprefix')
120+ else:
121+ prefix = 'UNKNOWN'
122 prefix = prefix.upper()
123 try:
124 prefix = Prefix.objects.get(value__exact=prefix)
125@@ -329,9 +335,10 @@
126 if total_time < 0:
127 total_time = 0
128 # Get the oops infestation
129- exception_type = oops.get('type')
130+ exception_type = oops.get('type') or ''
131+ exception_value = oops.get('value') or ''
132 exception_value = _normalize_exception_value(
133- exception_type, oops.get('value'), prefix)
134+ exception_type, exception_value, prefix)
135 try:
136 infestation = Infestation.objects.get(
137 exception_type__exact=exception_type,
138@@ -361,10 +368,13 @@
139 most_expensive_statement = conform(most_expensive_statement, 200)
140 url = conform(oops.get('url') or '', MAX_URL_LEN)
141 informational = oops.get('informational', 'False').lower() == 'true'
142+ oops_date = oops.get('time')
143+ if oops_date is None:
144+ oops_date = datetime.datetime.now(utc)
145 data.update(
146 oopsid = oopsid,
147 prefix = prefix,
148- date = oops.get('time').replace(microsecond=0),
149+ date = oops_date.replace(microsecond=0),
150 # Missing pageids are urls because that suits our queries.
151 pageid = oops.get('topic') or url,
152 url = url,
153@@ -377,7 +387,28 @@
154 is_bot = _robot_pat.search(data['user_agent']) is not None,
155 statements_count = len(statements),
156 )
157- return data, req_vars, statements, oops.get('tb_text')
158+ return data, req_vars, statements, oops.get('tb_text') or ''
159+
160+
161+def parsed_oops_to_model_oops(parsed_oops, pathname):
162+ """Convert an oops report dict to an Oops object."""
163+ data, req_vars, statements, traceback = _get_oops_tuple(parsed_oops)
164+ data['pathname'] = pathname
165+ res = Oops(**data)
166+ res.appinstance = res.get_appinstance()
167+ res.save()
168+ # Get it again. Otherwise we have discrepancies between old and
169+ # new oops objects: old ones have unicode attributes, and new
170+ # ones have string attributes, for instance. Ideally the message
171+ # conversion would have converted everything to unicode, but it
172+ # doesn't easily.
173+ res = Oops.objects.get(oopsid__exact=parsed_oops['id'])
174+ res.parsed_oops = parsed_oops
175+ res.req_vars = req_vars
176+ res.statements = statements
177+ res.traceback = traceback
178+ res.save()
179+ return res
180
181
182 class Oops(models.Model):
183@@ -432,21 +463,7 @@
184 try:
185 res = cls.objects.get(oopsid__exact=oopsid)
186 except cls.DoesNotExist:
187- data, req_vars, statements, traceback = _get_oops_tuple(parsed_oops)
188- data['pathname'] = pathname
189- res = cls(**data)
190- res.appinstance = res.get_appinstance()
191- res.save()
192- # Get it again. Otherwise we have discrepancies between old and
193- # new oops objects: old ones have unicode attributes, and new
194- # ones have string attributes, for instance. Ideally the message
195- # conversion would have converted everything to unicode, but it
196- # doesn't easily.
197- res = cls.objects.get(oopsid__exact=oopsid)
198- res.parsed_oops = parsed_oops
199- res.req_vars = req_vars
200- res.statements = statements
201- res.traceback = traceback
202+ res = parsed_oops_to_model_oops(parsed_oops, pathname)
203 return res
204
205 @readproperty
206
207=== added file 'src/oopstools/oops/test/test_amqp2disk.py'
208--- src/oopstools/oops/test/test_amqp2disk.py 1970-01-01 00:00:00 +0000
209+++ src/oopstools/oops/test/test_amqp2disk.py 2011-10-16 23:09:17 +0000
210@@ -0,0 +1,40 @@
211+# Copyright 2005-2011 Canonical Ltd. All rights reserved.
212+#
213+# This program is free software: you can redistribute it and/or modify
214+# it under the terms of the GNU Affero General Public License as published by
215+# the Free Software Foundation, either version 3 of the License, or
216+# (at your option) any later version.
217+#
218+# This program is distributed in the hope that it will be useful,
219+# but WITHOUT ANY WARRANTY; without even the implied warranty of
220+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
221+# GNU Affero General Public License for more details.
222+#
223+# You should have received a copy of the GNU Affero General Public License
224+# along with this program. If not, see <http://www.gnu.org/licenses/>.
225+
226+
227+import os.path
228+
229+import bson
230+from fixtures import TempDir
231+from testtools import TestCase
232+
233+from oopstools.oops.models import Oops
234+from oopstools.scripts import amqp2disk
235+
236+
237+class TestOOPSConfig(TestCase):
238+
239+ def test_publishes_disk_and_DB(self):
240+ self.root_dir = self.useFixture(TempDir()).path
241+ config = amqp2disk.make_amqp_config(self.root_dir)
242+ orig_report = {'id': '12345'}
243+ report = dict(orig_report)
244+ ids = config.publish(report)
245+ self.assertEqual(['12345', '12345'], ids)
246+ with open(report['datedir_repo_filepath'], 'rb') as fp:
247+ disk_report = bson.loads(fp.read())
248+ self.assertEqual(disk_report, orig_report)
249+ model_report = Oops.objects.get(oopsid='12345')
250+ self.assertNotEqual(None, model_report)
251
252=== added file 'src/oopstools/scripts/amqp2disk.py'
253--- src/oopstools/scripts/amqp2disk.py 1970-01-01 00:00:00 +0000
254+++ src/oopstools/scripts/amqp2disk.py 2011-10-16 23:09:17 +0000
255@@ -0,0 +1,137 @@
256+# Copyright 2011 Canonical Ltd. All rights reserved.
257+#
258+# This program is free software: you can redistribute it and/or modify
259+# it under the terms of the GNU Affero General Public License as published by
260+# the Free Software Foundation, either version 3 of the License, or
261+# (at your option) any later version.
262+#
263+# This program is distributed in the hope that it will be useful,
264+# but WITHOUT ANY WARRANTY; without even the implied warranty of
265+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
266+# GNU Affero General Public License for more details.
267+#
268+# You should have received a copy of the GNU Affero General Public License
269+# along with this program. If not, see <http://www.gnu.org/licenses/>.
270+
271+# Receive OOPS reports from AMQP and publish them into the oops-tools
272+# repository.
273+
274+__metaclass__ = type
275+
276+import sys
277+import optparse
278+import StringIO
279+from textwrap import dedent
280+
281+import amqplib.client_0_8 as amqp
282+from oops import Config
283+import oops_amqp
284+import oops_datedir_repo
285+
286+from oopstools.oops.helpers import parsedate, load_prefixes
287+from oopstools.oops.models import (
288+ Oops,
289+ parsed_oops_to_model_oops,
290+ Prefix,
291+ Report,
292+ )
293+from oopstools.oops.oopsstore import OopsStore
294+from oopstools.oops import dbsummaries
295+from oopstools.oops.summaries import (
296+ WebAppErrorSummary,
297+ CheckwatchesErrorSummary,
298+ CodeHostingWithRemoteSectionSummary,
299+ GenericErrorSummary,
300+)
301+
302+
303+def main(argv=None):
304+ if argv is None:
305+ argv=sys.argv
306+ usage = dedent("""\
307+ %prog [options]
308+
309+ The following options must be supplied:
310+ --output
311+ --host
312+ --username
313+ --password
314+ --vhost
315+ --queue
316+
317+ e.g.
318+ amqp2disk --output /srv/oops-tools/amqpoopses --host "localhost:3472" \\
319+ --username "guest" --password "guest" --vhost "/" --queue "oops"
320+
321+ The AMQP environment should be setup in advance with a persistent queue
322+ bound to your exchange : using transient queues would allow OOPSes to
323+ be lost if the amqp2disk process were to be shutdown for a non-trivial
324+ duration. The --bind-to option will cause the queue to be created and
325+ bound to the given exchange. This is only needed the first time as it
326+ is created persistently.
327+ """)
328+ description = "Load OOPS reports into oops-tools from AMQP."
329+ parser = optparse.OptionParser(
330+ description=description, usage=usage)
331+ parser.add_option('--output', help="Root directory to store OOPSes in.")
332+ parser.add_option('--host', help="AQMP host / host:port.")
333+ parser.add_option('--username', help="AQMP username.")
334+ parser.add_option('--password', help="AQMP password.")
335+ parser.add_option('--vhost', help="AMQP vhost.")
336+ parser.add_option('--queue', help="AMQP queue name.")
337+ parser.add_option(
338+ '--bind-to', help="AMQP exchange to bind to (only needed once).")
339+ options, args = parser.parse_args(argv[1:])
340+ def needed(optname):
341+ if getattr(options, optname, None) is None:
342+ raise ValueError('option "%s" must be supplied' % optname)
343+ needed('host')
344+ needed('output')
345+ needed('username')
346+ needed('password')
347+ needed('vhost')
348+ needed('queue')
349+ connection = amqp.Connection(host=options.host, userid=options.username,
350+ password=options.password, virtual_host=options.vhost)
351+ channel = connection.channel()
352+ if options.bind_to:
353+ channel.queue_declare(options.queue, durable=True, auto_delete=False)
354+ channel.queue_bind(options.queue, options.bind_to)
355+ config = make_amqp_config(options.output)
356+ receiver = oops_amqp.Receiver(config, channel, options.queue)
357+ try:
358+ receiver.run_forever()
359+ except KeyboardInterrupt:
360+ pass
361+
362+
363+def db_publisher(report):
364+ """Publish OOPS reports to the oops-tools django store."""
365+ # the first publisher will either inherit or assign, so this should be
366+ # impossible.
367+ assert report['id'] is not None
368+ # Some fallback methods could lead to duplicate paths into the DB: exit
369+ # early if the OOPS is already loaded.
370+ try:
371+ res = Oops.objects.get(oopsid__exact=report['id'])
372+ except Oops.DoesNotExist:
373+ res = parsed_oops_to_model_oops(
374+ report, report['datedir_repo_filepath'])
375+ return res.oopsid
376+ return None
377+
378+
379+def make_amqp_config(output_dir):
380+ """Create an OOPS Config for republishing amqp OOPSes.
381+
382+ An OOPS published to this config will be written to disk and then loaded
383+ into the database.
384+
385+ :param output_dir: The directory to write OOPSes too.
386+ """
387+ config = Config()
388+ disk_publisher = oops_datedir_repo.DateDirRepo(
389+ output_dir, inherit_id=True, stash_path=True)
390+ config.publishers.append(disk_publisher.publish)
391+ config.publishers.append(db_publisher)
392+ return config
393
394=== modified file 'versions.cfg'
395--- versions.cfg 2011-10-13 20:18:51 +0000
396+++ versions.cfg 2011-10-16 23:09:17 +0000
397@@ -19,8 +19,9 @@
398 launchpadlib = 1.6.0
399 lazr.config = 1.1.3
400 mechanize = 0.1.11
401-oops = 0.0.7
402-oops-datedir-repo = 0.0.7
403+oops = 0.0.9
404+oops-amqp = 0.0.1
405+oops-datedir-repo = 0.0.9
406 setuptools = 0.6c11
407 z3c.recipe.filetemplate = 2.0.3
408 z3c.recipe.sphinxdoc = 0.0.8

Subscribers

People subscribed via source and target branches

to all changes: