Merge lp:~ahasenack/txstatsd/txstatsd-packaging into lp:txstatsd
- txstatsd-packaging
- Merge into trunk
Status: | Superseded |
---|---|
Proposed branch: | lp:~ahasenack/txstatsd/txstatsd-packaging |
Merge into: | lp:txstatsd |
Diff against target: |
1035 lines (+792/-22) 20 files modified
.bzrignore (+1/-0) debian/changelog (+5/-0) debian/compat (+1/-0) debian/control (+15/-0) debian/copyright (+43/-0) debian/docs (+4/-0) debian/postinst (+39/-0) debian/postrm (+37/-0) debian/preinst (+35/-0) debian/prerm (+38/-0) debian/rules (+13/-0) example-stats-client.tac (+10/-3) statsd.tac (+1/-2) txstatsd/metrics.py (+24/-2) txstatsd/process.py (+171/-0) txstatsd/protocol.py (+9/-8) txstatsd/report.py (+46/-0) txstatsd/service.py (+32/-7) txstatsd/tests/test_process.py (+266/-0) txstatsd/tests/test_service.py (+2/-0) |
To merge this branch: | bzr merge lp:~ahasenack/txstatsd/txstatsd-packaging |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Landscape | Pending | ||
Landscape | Pending | ||
Review via email:
|
Commit message
Description of the change
This branch adds a debian/ structure to the txstatsd source tree. It builds and installs, and python sees the module after installation, as does twistd.
The {post,pre}inst and {post,pre}rm scripts are straight from dh_make, I didn't change those, and seem to do the right thing. After installation, I checked and postinst had a section added by dh_pysupport.
These are the lint warnings I get:
W: python-txstatsd source: out-of-
W: python-txstatsd: maintainer-
W: python-txstatsd: maintainer-
![](/+icing/build/overlay/assets/skins/sam/images/close.gif)
Sidnei da Silva (sidnei) wrote : | # |
- 15. By Andreas Hasenack
-
Merged with trunk (the ~landscape one, not upstream).
- 16. By Andreas Hasenack
-
Added dependency for python-psutil.
- 17. By Andreas Hasenack
-
Include example-
stats-client. tac in the documentation. - 18. By Andreas Hasenack
-
One more example file for the docs.
- 19. By Andreas Hasenack
-
- Merged with trunk again.
- Added statsd.tac to the documentation directory. - 20. By Andreas Hasenack
-
Change source package name to just txstatsd.
- 21. By Andreas Hasenack
-
- Changed source package name to just txstatsd
- Changed build dependency from python-twisted to python-twisted-core - 22. By Andreas Hasenack
-
Merged with trunk to grab missing report.py.
- 23. By Andreas Hasenack
-
Adjusted dependencies.
- 24. By Andreas Hasenack
-
Actually, no need for python-dev as this is noarch.
- 25. By Andreas Hasenack
-
Fixed version/release numbers.
Unmerged revisions
- 25. By Andreas Hasenack
-
Fixed version/release numbers.
- 24. By Andreas Hasenack
-
Actually, no need for python-dev as this is noarch.
- 23. By Andreas Hasenack
-
Adjusted dependencies.
- 22. By Andreas Hasenack
-
Merged with trunk to grab missing report.py.
- 21. By Andreas Hasenack
-
- Changed source package name to just txstatsd
- Changed build dependency from python-twisted to python-twisted-core - 20. By Andreas Hasenack
-
Change source package name to just txstatsd.
- 19. By Andreas Hasenack
-
- Merged with trunk again.
- Added statsd.tac to the documentation directory. - 18. By Andreas Hasenack
-
One more example file for the docs.
- 17. By Andreas Hasenack
-
Include example-
stats-client. tac in the documentation. - 16. By Andreas Hasenack
-
Added dependency for python-psutil.
Preview Diff
1 | === modified file '.bzrignore' |
2 | --- .bzrignore 2011-06-20 19:11:22 +0000 |
3 | +++ .bzrignore 2011-07-12 14:22:56 +0000 |
4 | @@ -1,1 +1,2 @@ |
5 | _trial_temp* |
6 | +twisted/plugins/dropin.cache |
7 | |
8 | === added directory 'debian' |
9 | === added file 'debian/changelog' |
10 | --- debian/changelog 1970-01-01 00:00:00 +0000 |
11 | +++ debian/changelog 2011-07-12 14:22:56 +0000 |
12 | @@ -0,0 +1,5 @@ |
13 | +txstatsd (0.1.0-0ubuntu1) unstable; urgency=low |
14 | + |
15 | + * Initial Release. |
16 | + |
17 | + -- Andreas Hasenack <andreas@canonical.com> Mon, 11 Jul 2011 19:29:34 -0300 |
18 | |
19 | === added file 'debian/compat' |
20 | --- debian/compat 1970-01-01 00:00:00 +0000 |
21 | +++ debian/compat 2011-07-12 14:22:56 +0000 |
22 | @@ -0,0 +1,1 @@ |
23 | +7 |
24 | |
25 | === added file 'debian/control' |
26 | --- debian/control 1970-01-01 00:00:00 +0000 |
27 | +++ debian/control 2011-07-12 14:22:56 +0000 |
28 | @@ -0,0 +1,15 @@ |
29 | +Source: txstatsd |
30 | +Section: python |
31 | +Priority: extra |
32 | +Maintainer: Andreas Hasenack <andreas@canonical.com> |
33 | +Build-Depends: debhelper (>= 7), python-twisted-core, python-support |
34 | +Standards-Version: 3.8.3 |
35 | +Homepage: https://launchpad.net/txstatsd |
36 | + |
37 | +Package: python-txstatsd |
38 | +Architecture: all |
39 | +Depends: ${python:Depends}, ${misc:Depends}, python-psutil, python-twisted-core |
40 | +Description: A network daemon for aggregating statistics |
41 | + A network daemon for aggregating statistics (counters and timers), rolling |
42 | + them up, then sending them to graphite. Really a port of |
43 | + https://github.com/etsy/statsd to twisted. |
44 | |
45 | === added file 'debian/copyright' |
46 | --- debian/copyright 1970-01-01 00:00:00 +0000 |
47 | +++ debian/copyright 2011-07-12 14:22:56 +0000 |
48 | @@ -0,0 +1,43 @@ |
49 | +This work was packaged for Debian by: |
50 | + |
51 | + Andreas Hasenack <andreas@canonical.com> on Mon, 11 Jul 2011 19:29:34 -0300 |
52 | + |
53 | +It was downloaded from: |
54 | + |
55 | + https://launchpad.net/txstatsd |
56 | + |
57 | +Upstream Author(s): |
58 | + |
59 | + Sidnei da Silva <sidnei@canonical.com> |
60 | + |
61 | +Copyright: |
62 | + |
63 | + <Copyright (C) 2011 Canonical Services Ltd> |
64 | + |
65 | +License: |
66 | + |
67 | + Permission is hereby granted, free of charge, to any person obtaining |
68 | + a copy of this software and associated documentation files (the |
69 | + "Software"), to deal in the Software without restriction, including |
70 | + without limitation the rights to use, copy, modify, merge, publish, |
71 | + distribute, sublicense, and/or sell copies of the Software, and to |
72 | + permit persons to whom the Software is furnished to do so, subject to |
73 | + the following conditions: |
74 | + |
75 | + The above copyright notice and this permission notice shall be |
76 | + included in all copies or substantial portions of the Software. |
77 | + |
78 | + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, |
79 | + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF |
80 | + MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. |
81 | + IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY |
82 | + CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, |
83 | + TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE |
84 | + SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
85 | + |
86 | +The Debian packaging is: |
87 | + |
88 | + Copyright (C) 2011 Andreas Hasenack <andreas@canonical.com> |
89 | + |
90 | +and is licensed under MIT. |
91 | + |
92 | |
93 | === added file 'debian/docs' |
94 | --- debian/docs 1970-01-01 00:00:00 +0000 |
95 | +++ debian/docs 2011-07-12 14:22:56 +0000 |
96 | @@ -0,0 +1,4 @@ |
97 | +README |
98 | +example-stats-client.tac |
99 | +txstatsd.conf-example |
100 | +statsd.tac |
101 | |
102 | === added file 'debian/postinst' |
103 | --- debian/postinst 1970-01-01 00:00:00 +0000 |
104 | +++ debian/postinst 2011-07-12 14:22:56 +0000 |
105 | @@ -0,0 +1,39 @@ |
106 | +#!/bin/sh |
107 | +# postinst script for python-txstatsd |
108 | +# |
109 | +# see: dh_installdeb(1) |
110 | + |
111 | +set -e |
112 | + |
113 | +# summary of how this script can be called: |
114 | +# * <postinst> `configure' <most-recently-configured-version> |
115 | +# * <old-postinst> `abort-upgrade' <new version> |
116 | +# * <conflictor's-postinst> `abort-remove' `in-favour' <package> |
117 | +# <new-version> |
118 | +# * <postinst> `abort-remove' |
119 | +# * <deconfigured's-postinst> `abort-deconfigure' `in-favour' |
120 | +# <failed-install-package> <version> `removing' |
121 | +# <conflicting-package> <version> |
122 | +# for details, see http://www.debian.org/doc/debian-policy/ or |
123 | +# the debian-policy package |
124 | + |
125 | + |
126 | +case "$1" in |
127 | + configure) |
128 | + ;; |
129 | + |
130 | + abort-upgrade|abort-remove|abort-deconfigure) |
131 | + ;; |
132 | + |
133 | + *) |
134 | + echo "postinst called with unknown argument \`$1'" >&2 |
135 | + exit 1 |
136 | + ;; |
137 | +esac |
138 | + |
139 | +# dh_installdeb will replace this with shell code automatically |
140 | +# generated by other debhelper scripts. |
141 | + |
142 | +#DEBHELPER# |
143 | + |
144 | +exit 0 |
145 | |
146 | === added file 'debian/postrm' |
147 | --- debian/postrm 1970-01-01 00:00:00 +0000 |
148 | +++ debian/postrm 2011-07-12 14:22:56 +0000 |
149 | @@ -0,0 +1,37 @@ |
150 | +#!/bin/sh |
151 | +# postrm script for python-txstatsd |
152 | +# |
153 | +# see: dh_installdeb(1) |
154 | + |
155 | +set -e |
156 | + |
157 | +# summary of how this script can be called: |
158 | +# * <postrm> `remove' |
159 | +# * <postrm> `purge' |
160 | +# * <old-postrm> `upgrade' <new-version> |
161 | +# * <new-postrm> `failed-upgrade' <old-version> |
162 | +# * <new-postrm> `abort-install' |
163 | +# * <new-postrm> `abort-install' <old-version> |
164 | +# * <new-postrm> `abort-upgrade' <old-version> |
165 | +# * <disappearer's-postrm> `disappear' <overwriter> |
166 | +# <overwriter-version> |
167 | +# for details, see http://www.debian.org/doc/debian-policy/ or |
168 | +# the debian-policy package |
169 | + |
170 | + |
171 | +case "$1" in |
172 | + purge|remove|upgrade|failed-upgrade|abort-install|abort-upgrade|disappear) |
173 | + ;; |
174 | + |
175 | + *) |
176 | + echo "postrm called with unknown argument \`$1'" >&2 |
177 | + exit 1 |
178 | + ;; |
179 | +esac |
180 | + |
181 | +# dh_installdeb will replace this with shell code automatically |
182 | +# generated by other debhelper scripts. |
183 | + |
184 | +#DEBHELPER# |
185 | + |
186 | +exit 0 |
187 | |
188 | === added file 'debian/preinst' |
189 | --- debian/preinst 1970-01-01 00:00:00 +0000 |
190 | +++ debian/preinst 2011-07-12 14:22:56 +0000 |
191 | @@ -0,0 +1,35 @@ |
192 | +#!/bin/sh |
193 | +# preinst script for python-txstatsd |
194 | +# |
195 | +# see: dh_installdeb(1) |
196 | + |
197 | +set -e |
198 | + |
199 | +# summary of how this script can be called: |
200 | +# * <new-preinst> `install' |
201 | +# * <new-preinst> `install' <old-version> |
202 | +# * <new-preinst> `upgrade' <old-version> |
203 | +# * <old-preinst> `abort-upgrade' <new-version> |
204 | +# for details, see http://www.debian.org/doc/debian-policy/ or |
205 | +# the debian-policy package |
206 | + |
207 | + |
208 | +case "$1" in |
209 | + install|upgrade) |
210 | + ;; |
211 | + |
212 | + abort-upgrade) |
213 | + ;; |
214 | + |
215 | + *) |
216 | + echo "preinst called with unknown argument \`$1'" >&2 |
217 | + exit 1 |
218 | + ;; |
219 | +esac |
220 | + |
221 | +# dh_installdeb will replace this with shell code automatically |
222 | +# generated by other debhelper scripts. |
223 | + |
224 | +#DEBHELPER# |
225 | + |
226 | +exit 0 |
227 | |
228 | === added file 'debian/prerm' |
229 | --- debian/prerm 1970-01-01 00:00:00 +0000 |
230 | +++ debian/prerm 2011-07-12 14:22:56 +0000 |
231 | @@ -0,0 +1,38 @@ |
232 | +#!/bin/sh |
233 | +# prerm script for python-txstatsd |
234 | +# |
235 | +# see: dh_installdeb(1) |
236 | + |
237 | +set -e |
238 | + |
239 | +# summary of how this script can be called: |
240 | +# * <prerm> `remove' |
241 | +# * <old-prerm> `upgrade' <new-version> |
242 | +# * <new-prerm> `failed-upgrade' <old-version> |
243 | +# * <conflictor's-prerm> `remove' `in-favour' <package> <new-version> |
244 | +# * <deconfigured's-prerm> `deconfigure' `in-favour' |
245 | +# <package-being-installed> <version> `removing' |
246 | +# <conflicting-package> <version> |
247 | +# for details, see http://www.debian.org/doc/debian-policy/ or |
248 | +# the debian-policy package |
249 | + |
250 | + |
251 | +case "$1" in |
252 | + remove|upgrade|deconfigure) |
253 | + ;; |
254 | + |
255 | + failed-upgrade) |
256 | + ;; |
257 | + |
258 | + *) |
259 | + echo "prerm called with unknown argument \`$1'" >&2 |
260 | + exit 1 |
261 | + ;; |
262 | +esac |
263 | + |
264 | +# dh_installdeb will replace this with shell code automatically |
265 | +# generated by other debhelper scripts. |
266 | + |
267 | +#DEBHELPER# |
268 | + |
269 | +exit 0 |
270 | |
271 | === added file 'debian/rules' |
272 | --- debian/rules 1970-01-01 00:00:00 +0000 |
273 | +++ debian/rules 2011-07-12 14:22:56 +0000 |
274 | @@ -0,0 +1,13 @@ |
275 | +#!/usr/bin/make -f |
276 | +# -*- makefile -*- |
277 | +# Sample debian/rules that uses debhelper. |
278 | +# This file was originally written by Joey Hess and Craig Small. |
279 | +# As a special exception, when this file is copied by dh-make into a |
280 | +# dh-make output file, you may use that output file without restriction. |
281 | +# This special exception was added by Craig Small in version 0.37 of dh-make. |
282 | + |
283 | +# Uncomment this to turn on verbose mode. |
284 | +#export DH_VERBOSE=1 |
285 | + |
286 | +%: |
287 | + dh $@ |
288 | |
289 | === modified file 'example-stats-client.tac' |
290 | --- example-stats-client.tac 2011-06-23 14:19:21 +0000 |
291 | +++ example-stats-client.tac 2011-07-12 14:22:56 +0000 |
292 | @@ -8,11 +8,18 @@ |
293 | |
294 | from txstatsd.protocol import StatsDClientProtocol |
295 | from txstatsd.metrics import TransportMeter |
296 | +from txstatsd.process import PROCESS_STATS |
297 | +from txstatsd.report import ReportingService |
298 | |
299 | |
300 | application = Application("example-stats-client") |
301 | -meter = TransportMeter(prefix=socket.gethostname()) |
302 | - |
303 | +meter = TransportMeter(prefix=socket.gethostname() + ".example-client") |
304 | + |
305 | +reporting = ReportingService() |
306 | +reporting.setServiceParent(application) |
307 | + |
308 | +for report in PROCESS_STATS: |
309 | + reporting.schedule(report, 10, meter.increment) |
310 | |
311 | def random_walker(name): |
312 | """Meters a random walk.""" |
313 | @@ -35,5 +42,5 @@ |
314 | t.start(0.5, now=False) |
315 | |
316 | |
317 | -protocol = StatsDClientProtocol("127.0.0.1", 8125, meter) |
318 | +protocol = StatsDClientProtocol("127.0.0.1", 8125, meter, 6000) |
319 | reactor.listenUDP(0, protocol) |
320 | |
321 | === modified file 'statsd.tac' |
322 | --- statsd.tac 2011-07-04 21:43:01 +0000 |
323 | +++ statsd.tac 2011-07-12 14:22:56 +0000 |
324 | @@ -4,6 +4,5 @@ |
325 | |
326 | application = Application("statsd") |
327 | |
328 | -statsd_service = service.createService(service.StatsdOptions()) |
329 | +statsd_service = service.createService(service.StatsDOptions()) |
330 | statsd_service.setServiceParent(application) |
331 | - |
332 | |
333 | === modified file 'txstatsd/metrics.py' |
334 | --- txstatsd/metrics.py 2011-07-05 05:27:22 +0000 |
335 | +++ txstatsd/metrics.py 2011-07-12 14:22:56 +0000 |
336 | @@ -55,6 +55,18 @@ |
337 | raise NotImplementedError() |
338 | |
339 | |
340 | +class InProcessMeter(BaseMeter): |
341 | + """A meter that can be used inside the C{StatsD} daemon itself.""" |
342 | + |
343 | + def __init__(self, processor, prefix="", sample_rate=1): |
344 | + self.processor = processor |
345 | + BaseMeter.__init__(self, prefix=prefix, sample_rate=sample_rate) |
346 | + |
347 | + def write(self, data): |
348 | + """Pass the data along directly to the C{Processor}.""" |
349 | + self.processor.process(data) |
350 | + |
351 | + |
352 | class Meter(BaseMeter): |
353 | """A trivial, non-Twisted-dependent meter.""" |
354 | |
355 | @@ -118,12 +130,22 @@ |
356 | host = None |
357 | port = None |
358 | |
359 | - def connected(self, transport, host, port): |
360 | + def __init__(self, prefix="", sample_rate=1, |
361 | + connect_callback=None, disconnect_callback=None): |
362 | + self.connect_callback = connect_callback |
363 | + self.disconnect_callback = disconnect_callback |
364 | + BaseMeter.__init__(self, prefix=prefix, sample_rate=sample_rate) |
365 | + |
366 | + def connect(self, transport, host, port): |
367 | self.transport = transport |
368 | self.host = host |
369 | self.port = port |
370 | + if self.connect_callback is not None: |
371 | + self.connect_callback() |
372 | |
373 | - def disconnected(self): |
374 | + def disconnect(self): |
375 | + if self.disconnect_callback is not None: |
376 | + self.disconnect_callback() |
377 | self.transport = self.host = self.port = None |
378 | |
379 | def write(self, data): |
380 | |
381 | === added file 'txstatsd/process.py' |
382 | --- txstatsd/process.py 1970-01-01 00:00:00 +0000 |
383 | +++ txstatsd/process.py 2011-07-12 14:22:56 +0000 |
384 | @@ -0,0 +1,171 @@ |
385 | +import os |
386 | +import socket |
387 | +import psutil |
388 | + |
389 | +from functools import update_wrapper |
390 | + |
391 | +from twisted.internet import defer, fdesc, error |
392 | + |
393 | + |
394 | +MEMINFO_KEYS = ("MemTotal:", "MemFree:", "Buffers:", |
395 | + "Cached:", "SwapCached:", "SwapTotal:", |
396 | + "SwapFree:") |
397 | + |
398 | +MULTIPLIERS = {"kB": 1024, "mB": 1024 * 1024} |
399 | + |
400 | + |
401 | +def load_file(filename): |
402 | + """Load a file into memory with non blocking reads.""" |
403 | + |
404 | + fd = os.open(filename, os.O_RDONLY) |
405 | + fdesc.setNonBlocking(fd) |
406 | + |
407 | + chunks = [] |
408 | + d = defer.Deferred() |
409 | + |
410 | + def read_loop(data=None): |
411 | + """Inner loop.""" |
412 | + if data is not None: |
413 | + chunks.append(data) |
414 | + r = fdesc.readFromFD(fd, read_loop) |
415 | + if isinstance(r, error.ConnectionDone): |
416 | + os.close(fd) |
417 | + d.callback("".join(chunks)) |
418 | + elif r is not None: |
419 | + os.close(fd) |
420 | + d.errback(r) |
421 | + |
422 | + read_loop("") |
423 | + return d |
424 | + |
425 | + |
426 | +def parse_meminfo(data, prefix="sys.mem"): |
427 | + """Parse data from /proc/meminfo.""" |
428 | + result = {} |
429 | + |
430 | + for line in data.split("\n"): |
431 | + if not line: |
432 | + continue |
433 | + parts = [x for x in line.split(" ") if x] |
434 | + if not parts[0] in MEMINFO_KEYS: |
435 | + continue |
436 | + |
437 | + multiple = 1 |
438 | + |
439 | + if len(parts) == 3: |
440 | + multiple = MULTIPLIERS[parts[2]] |
441 | + |
442 | + # remove ':' |
443 | + label = parts[0][:-1] |
444 | + amount = int(parts[1]) * multiple |
445 | + result[prefix + "." + label] = amount |
446 | + |
447 | + return result |
448 | + |
449 | + |
450 | +def parse_loadavg(data, prefix="sys.loadavg"): |
451 | + """Parse data from /proc/loadavg.""" |
452 | + return dict(zip( |
453 | + (prefix + ".oneminute", |
454 | + prefix + ".fiveminutes", |
455 | + prefix + ".fifthteenminutes"), |
456 | + [float(x) for x in data.split()[:3]])) |
457 | + |
458 | + |
459 | +def report_process_memory_and_cpu(process=psutil.Process(os.getpid()), |
460 | + prefix="proc"): |
461 | + """Report memory and CPU stats for C{process}.""" |
462 | + vsize, rss = process.get_memory_info() |
463 | + utime, stime = process.get_cpu_times() |
464 | + result = {prefix + ".cpu.percent": process.get_cpu_percent(), |
465 | + prefix + ".cpu.user": utime, |
466 | + prefix + ".cpu.system": stime, |
467 | + prefix + ".memory.percent": process.get_memory_percent(), |
468 | + prefix + ".memory.vsize": vsize, |
469 | + prefix + ".memory.rss": rss} |
470 | + if getattr(process, "get_num_threads", None) is not None: |
471 | + result[prefix + ".threads"] = process.get_num_threads() |
472 | + return result |
473 | + |
474 | + |
475 | +def report_process_io_counters(process=psutil.Process(os.getpid()), |
476 | + prefix="proc.io"): |
477 | + """Report IO statistics for C{process}.""" |
478 | + result = {} |
479 | + if getattr(process, "get_io_counters", None) is not None: |
480 | + (read_count, write_count, |
481 | + read_bytes, write_bytes) = process.get_io_counters() |
482 | + result.update({ |
483 | + prefix + ".read.count": read_count, |
484 | + prefix + ".write.count": write_count, |
485 | + prefix + ".read.bytes": read_bytes, |
486 | + prefix + ".write.bytes": write_bytes}) |
487 | + return result |
488 | + |
489 | + |
490 | +def report_process_net_stats(process=psutil.Process(os.getpid()), |
491 | + prefix="proc.net"): |
492 | + """Report active connection statistics for C{process}.""" |
493 | + result = {} |
494 | + if getattr(process, "get_connections", None) is not None: |
495 | + for connection in process.get_connections(): |
496 | + fd, family, _type, laddr, raddr, status = connection |
497 | + if _type == socket.SOCK_STREAM: |
498 | + key = prefix + ".status.%s" % status.lower() |
499 | + if not key in result: |
500 | + result[key] = 1 |
501 | + else: |
502 | + result[key] += 1 |
503 | + return result |
504 | + |
505 | + |
506 | +def report_system_stats(prefix="sys"): |
507 | + cpu_times = psutil.cpu_times() |
508 | + return {prefix + ".cpu.idle": cpu_times.idle, |
509 | + prefix + ".cpu.iowait": cpu_times.iowait, |
510 | + prefix + ".cpu.irq": cpu_times.irq, |
511 | + prefix + ".cpu.nice": cpu_times.nice, |
512 | + prefix + ".cpu.system": cpu_times.system, |
513 | + prefix + ".cpu.user": cpu_times.user} |
514 | + |
515 | + |
516 | +def report_threadpool_stats(threadpool, prefix="threadpool"): |
517 | + """Report stats about a given threadpool.""" |
518 | + def report(): |
519 | + return {prefix + ".working": len(threadpool.working), |
520 | + prefix + ".queue": threadpool.q.qsize(), |
521 | + prefix + ".waiters": len(threadpool.waiters), |
522 | + prefix + ".threads": len(threadpool.threads)} |
523 | + update_wrapper(report, report_threadpool_stats) |
524 | + return report |
525 | + |
526 | + |
527 | +def report_reactor_stats(reactor, prefix="reactor"): |
528 | + """Report statistics about a twisted reactor.""" |
529 | + def report(): |
530 | + return {prefix + ".readers": len(reactor.getReaders()), |
531 | + prefix + ".writers": len(reactor.getWriters())} |
532 | + |
533 | + update_wrapper(report, report_reactor_stats) |
534 | + return report |
535 | + |
536 | + |
537 | +def report_file_stats(filename, parser): |
538 | + """Read statistics from a file and report them.""" |
539 | + def report(): |
540 | + deferred = load_file(filename) |
541 | + deferred.addCallback(parser) |
542 | + return deferred |
543 | + update_wrapper(report, report_file_stats) |
544 | + return report |
545 | + |
546 | + |
547 | +PROCESS_STATS = (report_process_memory_and_cpu,) |
548 | + |
549 | +IO_STATS = (report_process_io_counters,) |
550 | + |
551 | +NET_STATS = (report_process_net_stats,) |
552 | + |
553 | +SYSTEM_STATS = (report_file_stats("/proc/meminfo", parse_meminfo), |
554 | + report_file_stats("/proc/loadavg", parse_loadavg), |
555 | + report_system_stats) |
556 | |
557 | === modified file 'txstatsd/protocol.py' |
558 | --- txstatsd/protocol.py 2011-06-22 00:18:58 +0000 |
559 | +++ txstatsd/protocol.py 2011-07-12 14:22:56 +0000 |
560 | @@ -1,7 +1,7 @@ |
561 | import logging |
562 | |
563 | from twisted.python import log |
564 | -from twisted.internet import task |
565 | +from twisted.internet import task, defer |
566 | from twisted.protocols.basic import LineOnlyReceiver |
567 | from twisted.internet.protocol import ( |
568 | DatagramProtocol, ReconnectingClientFactory) |
569 | @@ -27,21 +27,22 @@ |
570 | class StatsDClientProtocol(DatagramProtocol): |
571 | """A Twisted-based implementation of the StatsD client protocol. |
572 | |
573 | - Data is sent via ConnectedUDP to a StatsD server for aggregation. |
574 | + Data is sent via UDP to a StatsD server for aggregation. |
575 | """ |
576 | |
577 | - def __init__(self, host, port, meter): |
578 | + def __init__(self, host, port, meter, interval=None): |
579 | self.host = host |
580 | self.port = port |
581 | self.meter = meter |
582 | + self.interval = interval |
583 | |
584 | def startProtocol(self): |
585 | """Connect to destination host.""" |
586 | - self.meter.connected(self.transport, self.host, self.port) |
587 | + self.meter.connect(self.transport, self.host, self.port) |
588 | |
589 | def stopProtocol(self): |
590 | """Connection was lost.""" |
591 | - self.meter.disconnected() |
592 | + self.meter.disconnect() |
593 | |
594 | |
595 | class GraphiteProtocol(LineOnlyReceiver): |
596 | @@ -65,7 +66,7 @@ |
597 | log.msg("Connected. Scheduling flush to now + %ds." % |
598 | (self.interval / 1000), logLevel=logging.DEBUG) |
599 | self.flush_task = task.LoopingCall(self.flushProcessor) |
600 | - self.flush_task.start(self.interval / 1000) |
601 | + self.flush_task.start(self.interval / 1000, False) |
602 | |
603 | def connectionLost(self, reason): |
604 | """ |
605 | @@ -77,12 +78,12 @@ |
606 | log.msg("Canceling scheduled flush.", logLevel=logging.DEBUG) |
607 | self.flush_task.stop() |
608 | |
609 | + @defer.inlineCallbacks |
610 | def flushProcessor(self): |
611 | """Flush messages queued in the processor to Graphite.""" |
612 | - log.msg("Flushing messages.", logLevel=logging.DEBUG) |
613 | for message in self.processor.flush(interval=self.interval): |
614 | for line in message.splitlines(): |
615 | - self.sendLine(line) |
616 | + yield self.sendLine(line) |
617 | |
618 | |
619 | class GraphiteClientFactory(ReconnectingClientFactory): |
620 | |
621 | === added file 'txstatsd/report.py' |
622 | --- txstatsd/report.py 1970-01-01 00:00:00 +0000 |
623 | +++ txstatsd/report.py 2011-07-12 14:22:56 +0000 |
624 | @@ -0,0 +1,46 @@ |
625 | +from twisted.internet.defer import maybeDeferred |
626 | +from twisted.internet.task import LoopingCall |
627 | +from twisted.python import log |
628 | + |
629 | +from functools import wraps |
630 | + |
631 | +from twisted.application.service import Service |
632 | + |
633 | + |
634 | +class ReportingService(Service): |
635 | + |
636 | + def __init__(self): |
637 | + self.tasks = [] |
638 | + |
639 | + def schedule(self, function, interval, report_function): |
640 | + """ |
641 | + Schedule C{function} to be called every C{interval} seconds and then |
642 | + report gathered metrics to C{Graphite} using C{report_function}. |
643 | + """ |
644 | + task = LoopingCall(self.wrapped(function, report_function)) |
645 | + self.tasks.append((task, interval)) |
646 | + |
647 | + def wrapped(self, function, report_function): |
648 | + def report_metrics(metrics): |
649 | + """For each metric returned, call C{report_function} with it.""" |
650 | + for name, value in metrics.items(): |
651 | + report_function(name, value) |
652 | + return metrics |
653 | + |
654 | + @wraps(function) |
655 | + def wrapper(): |
656 | + """Wrap C{function} to report metrics or log a failure.""" |
657 | + deferred = maybeDeferred(function) |
658 | + deferred.addCallback(report_metrics) |
659 | + deferred.addErrback(lambda failure: log.err( |
660 | + failure, "Error while processing %s" % function.func_name)) |
661 | + return deferred |
662 | + return wrapper |
663 | + |
664 | + def startService(self): |
665 | + for task, interval in self.tasks: |
666 | + task.start(interval, now=False) |
667 | + |
668 | + def stopService(self): |
669 | + for task, interval in self.tasks: |
670 | + task.stop() |
671 | |
672 | === modified file 'txstatsd/service.py' |
673 | --- txstatsd/service.py 2011-07-05 05:27:22 +0000 |
674 | +++ txstatsd/service.py 2011-07-12 14:22:56 +0000 |
675 | @@ -1,11 +1,16 @@ |
676 | +import socket |
677 | import ConfigParser |
678 | |
679 | from twisted.application.internet import TCPClient, UDPServer |
680 | from twisted.application.service import MultiService |
681 | +from twisted.internet import reactor |
682 | from twisted.python import usage, util |
683 | |
684 | +from txstatsd import process |
685 | +from txstatsd.metrics import InProcessMeter |
686 | from txstatsd.processor import MessageProcessor |
687 | from txstatsd.protocol import GraphiteClientFactory, StatsDServerProtocol |
688 | +from txstatsd.report import ReportingService |
689 | |
690 | _unset = object() |
691 | |
692 | @@ -73,13 +78,17 @@ |
693 | """ |
694 | glue_parameters = [ |
695 | ["carbon-cache-host", "h", "localhost", |
696 | - "The host where carbon cache is listening."], |
697 | + "The host where carbon cache is listening."], |
698 | ["carbon-cache-port", "p", 2003, |
699 | - "The port where carbon cache is listening.", int], |
700 | + "The port where carbon cache is listening.", int], |
701 | ["listen-port", "l", 8125, |
702 | - "The UDP port where we will listen.", int], |
703 | + "The UDP port where we will listen.", int], |
704 | ["flush-interval", "i", 10000, |
705 | - "The number of milliseconds between each flush.", int], |
706 | + "The number of milliseconds between each flush.", int], |
707 | + ["prefix", "p", None, |
708 | + "Prefix to use when reporting stats.", str], |
709 | + ["report", "r", None, |
710 | + "Which additional stats to report {process|net|io|system}.", str], |
711 | ] |
712 | |
713 | |
714 | @@ -89,11 +98,27 @@ |
715 | service = MultiService() |
716 | service.setName("statsd") |
717 | processor = MessageProcessor() |
718 | + prefix = options["prefix"] |
719 | + if prefix is None: |
720 | + prefix = socket.gethostname() + ".statsd" |
721 | + |
722 | + meter = InProcessMeter(processor, prefix=prefix) |
723 | + |
724 | + if options["report"] is not None: |
725 | + reporting = ReportingService() |
726 | + reporting.setServiceParent(service) |
727 | + reporting.schedule( |
728 | + process.report_reactor_stats(reactor), 10, meter.increment) |
729 | + reports = [name.strip() for name in options["report"].split(",")] |
730 | + for report_name in reports: |
731 | + for reporter in getattr(process, "%s_STATS" % |
732 | + report_name.upper(), ()): |
733 | + reporting.schedule(reporter, 10, meter.increment) |
734 | |
735 | factory = GraphiteClientFactory(processor, options["flush-interval"]) |
736 | - client = TCPClient( |
737 | - options["carbon-cache-host"], options["carbon-cache-port"], |
738 | - factory) |
739 | + client = TCPClient(options["carbon-cache-host"], |
740 | + options["carbon-cache-port"], |
741 | + factory) |
742 | client.setServiceParent(service) |
743 | |
744 | listener = UDPServer(options["listen-port"], |
745 | |
746 | === added file 'txstatsd/tests/test_process.py' |
747 | --- txstatsd/tests/test_process.py 1970-01-01 00:00:00 +0000 |
748 | +++ txstatsd/tests/test_process.py 2011-07-12 14:22:56 +0000 |
749 | @@ -0,0 +1,266 @@ |
750 | +import os |
751 | +import psutil |
752 | + |
753 | +from mocker import MockerTestCase |
754 | +from twisted.internet import defer |
755 | +from twisted.trial.unittest import TestCase |
756 | + |
757 | +from txstatsd.process import ( |
758 | + load_file, parse_meminfo, parse_loadavg, |
759 | + report_process_memory_and_cpu, report_process_io_counters, |
760 | + report_process_net_stats, report_system_stats, |
761 | + report_reactor_stats, report_threadpool_stats) |
762 | + |
763 | + |
764 | +meminfo = """\ |
765 | +MemTotal: 8190436 kB |
766 | +MemFree: 995724 kB |
767 | +Buffers: 8052 kB |
768 | +Cached: 344824 kB |
769 | +SwapCached: 170828 kB |
770 | +Active: 4342436 kB |
771 | +Inactive: 907076 kB |
772 | +Active(anon): 4210168 kB |
773 | +Inactive(anon): 756096 kB |
774 | +Active(file): 132268 kB |
775 | +Inactive(file): 150980 kB |
776 | +Unevictable: 641692 kB |
777 | +Mlocked: 641676 kB |
778 | +SwapTotal: 23993336 kB |
779 | +SwapFree: 22750588 kB |
780 | +Dirty: 740 kB |
781 | +Writeback: 0 kB |
782 | +AnonPages: 5453396 kB |
783 | +Mapped: 259524 kB |
784 | +Shmem: 69952 kB |
785 | +Slab: 142444 kB |
786 | +SReclaimable: 83188 kB |
787 | +SUnreclaim: 59256 kB |
788 | +KernelStack: 16144 kB |
789 | +PageTables: 88384 kB |
790 | +NFS_Unstable: 0 kB |
791 | +Bounce: 0 kB |
792 | +WritebackTmp: 0 kB |
793 | +CommitLimit: 28088552 kB |
794 | +Committed_AS: 11178564 kB |
795 | +VmallocTotal: 34359738367 kB |
796 | +VmallocUsed: 156208 kB |
797 | +VmallocChunk: 34359579400 kB |
798 | +HardwareCorrupted: 0 kB |
799 | +HugePages_Total: 0 |
800 | +HugePages_Free: 0 |
801 | +HugePages_Rsvd: 0 |
802 | +HugePages_Surp: 0 |
803 | +Hugepagesize: 2048 kB |
804 | +DirectMap4k: 7968640 kB |
805 | +DirectMap2M: 415744 kB""" |
806 | + |
807 | + |
808 | +class TestSystemPerformance(TestCase, MockerTestCase): |
809 | + """Test system performance monitoring.""" |
810 | + |
811 | + def assertSuccess(self, deferred, result=None): |
812 | + """ |
813 | + Assert that the given C{deferred} results in the given C{result}. |
814 | + """ |
815 | + self.assertTrue(isinstance(deferred, defer.Deferred)) |
816 | + return deferred.addCallback(self.assertEqual, result) |
817 | + |
818 | + def test_read(self): |
819 | + """We can read files non blocking.""" |
820 | + d = load_file(__file__) |
821 | + return self.assertSuccess(d, open(__file__).read()) |
822 | + |
823 | + def test_loadinfo(self): |
824 | + """We understand loadinfo.""" |
825 | + loadinfo = "1.02 1.08 1.14 2/2015 19420" |
826 | + self.assertEqual(parse_loadavg(loadinfo), { |
827 | + "sys.loadavg.oneminute": 1.02, |
828 | + "sys.loadavg.fiveminutes": 1.08, |
829 | + "sys.loadavg.fifthteenminutes": 1.14}) |
830 | + |
831 | + def test_meminfo(self): |
832 | + """We understand meminfo.""" |
833 | + r = parse_meminfo(meminfo) |
834 | + self.assertEqual(r['sys.mem.Buffers'], 8052 * 1024) |
835 | + self.assert_('sys.mem.HugePages_Rsvd' not in r) |
836 | + |
837 | + def test_statinfo(self): |
838 | + """System stat info is collected through psutil.""" |
839 | + cpu_times = psutil.cpu_times() |
840 | + mock = self.mocker.replace("psutil.cpu_times") |
841 | + self.expect(mock()).result(cpu_times) |
842 | + self.mocker.replay() |
843 | + |
844 | + result = report_system_stats() |
845 | + self.assertEqual(cpu_times.idle, result["sys.cpu.idle"]) |
846 | + self.assertEqual(cpu_times.iowait, result["sys.cpu.iowait"]) |
847 | + self.assertEqual(cpu_times.irq, result["sys.cpu.irq"]) |
848 | + self.assertEqual(cpu_times.nice, result["sys.cpu.nice"]) |
849 | + self.assertEqual(cpu_times.system, result["sys.cpu.system"]) |
850 | + self.assertEqual(cpu_times.user, result["sys.cpu.user"]) |
851 | + |
852 | + def test_self_statinfo(self): |
853 | + """ |
854 | + Process stat info is collected through psutil. |
855 | + |
856 | + If the L{Process} implementation does not have C{get_num_threads} then |
857 | + the number of threads will not be included in the output. |
858 | + """ |
859 | + process = psutil.Process(os.getpid()) |
860 | + vsize, rss = process.get_memory_info() |
861 | + utime, stime = process.get_cpu_times() |
862 | + cpu_percent = process.get_cpu_percent() |
863 | + memory_percent = process.get_memory_percent() |
864 | + |
865 | + mock = self.mocker.mock() |
866 | + self.expect(mock.get_memory_info()).result((vsize, rss)) |
867 | + self.expect(mock.get_cpu_times()).result((utime, stime)) |
868 | + self.expect(mock.get_cpu_percent()).result(cpu_percent) |
869 | + self.expect(mock.get_memory_percent()).result(memory_percent) |
870 | + self.expect(mock.get_num_threads).result(None) |
871 | + self.mocker.replay() |
872 | + |
873 | + result = report_process_memory_and_cpu(process=mock) |
874 | + self.assertEqual(utime, result["proc.cpu.user"]) |
875 | + self.assertEqual(stime, result["proc.cpu.system"]) |
876 | + self.assertEqual(cpu_percent, result["proc.cpu.percent"]) |
877 | + self.assertEqual(vsize, result["proc.memory.vsize"]) |
878 | + self.assertEqual(rss, result["proc.memory.rss"]) |
879 | + self.assertEqual(memory_percent, result["proc.memory.percent"]) |
880 | + self.failIf("proc.threads" in result) |
881 | + |
882 | + def test_self_statinfo_with_num_threads(self): |
883 | + """ |
884 | + Process stat info is collected through psutil. |
885 | + |
886 | + If the L{Process} implementation contains C{get_num_threads} then the |
887 | + number of threads will be included in the output. |
888 | + |
889 | + """ |
890 | + process = psutil.Process(os.getpid()) |
891 | + vsize, rss = process.get_memory_info() |
892 | + utime, stime = process.get_cpu_times() |
893 | + cpu_percent = process.get_cpu_percent() |
894 | + memory_percent = process.get_memory_percent() |
895 | + |
896 | + mock = self.mocker.mock() |
897 | + self.expect(mock.get_memory_info()).result((vsize, rss)) |
898 | + self.expect(mock.get_cpu_times()).result((utime, stime)) |
899 | + self.expect(mock.get_cpu_percent()).result(cpu_percent) |
900 | + self.expect(mock.get_memory_percent()).result(memory_percent) |
901 | + self.expect(mock.get_num_threads()).result(1) |
902 | + self.mocker.replay() |
903 | + |
904 | + result = report_process_memory_and_cpu(process=mock) |
905 | + self.assertEqual(utime, result["proc.cpu.user"]) |
906 | + self.assertEqual(stime, result["proc.cpu.system"]) |
907 | + self.assertEqual(cpu_percent, result["proc.cpu.percent"]) |
908 | + self.assertEqual(vsize, result["proc.memory.vsize"]) |
909 | + self.assertEqual(rss, result["proc.memory.rss"]) |
910 | + self.assertEqual(memory_percent, result["proc.memory.percent"]) |
911 | + self.assertEqual(1, result["proc.threads"]) |
912 | + |
913 | + def test_ioinfo(self): |
914 | + """Process IO info is collected through psutil.""" |
915 | + mock = self.mocker.mock() |
916 | + self.expect(mock.get_io_counters).result(None) |
917 | + self.mocker.replay() |
918 | + |
919 | + # If the version of psutil doesn't have the C{get_io_counters}, |
920 | + # then io stats are not included in the output. |
921 | + result = report_process_io_counters(process=mock) |
922 | + self.failIf("proc.io.read.count" in result) |
923 | + self.failIf("proc.io.write.count" in result) |
924 | + self.failIf("proc.io.read.bytes" in result) |
925 | + self.failIf("proc.io.write.bytes" in result) |
926 | + |
927 | + def test_ioinfo_with_get_io_counters(self): |
928 | + """ |
929 | + Process IO info is collected through psutil. |
930 | + |
931 | + If C{get_io_counters} is implemented by the L{Process} object, |
932 | + then io information will be returned with the process information. |
933 | + """ |
934 | + io_counters = (10, 42, 125, 16) |
935 | + |
936 | + mock = self.mocker.mock() |
937 | + self.expect(mock.get_io_counters).result(mock) |
938 | + self.expect(mock.get_io_counters()).result(io_counters) |
939 | + self.mocker.replay() |
940 | + |
941 | + result = report_process_io_counters(process=mock) |
942 | + self.assertEqual(10, result["proc.io.read.count"]) |
943 | + self.assertEqual(42, result["proc.io.write.count"]) |
944 | + self.assertEqual(125, result["proc.io.read.bytes"]) |
945 | + self.assertEqual(16, result["proc.io.write.bytes"]) |
946 | + |
947 | + def test_netinfo_no_get_connections(self): |
948 | + """ |
949 | + Process connection info is collected through psutil. |
950 | + |
951 | + If the version of psutil doesn't implement C{get_connections} for |
952 | + L{Process}, then no information is returned. |
953 | + """ |
954 | + mock = self.mocker.mock() |
955 | + self.expect(mock.get_connections).result(None) |
956 | + self.mocker.replay() |
957 | + |
958 | + # If the version of psutil doesn't have the C{get_io_counters}, |
959 | + # then io stats are not included in the output. |
960 | + result = report_process_net_stats(process=mock) |
961 | + self.failIf("proc.net.status.established" in result) |
962 | + |
963 | + def test_netinfo_with_get_connections(self): |
964 | + """ |
965 | + Process connection info is collected through psutil. |
966 | + |
967 | + If the version of psutil implements C{get_connections} for L{Process}, |
968 | + then a count of connections in each state is returned. |
969 | + """ |
970 | + connections = [ |
971 | + (115, 2, 1, ("10.0.0.1", 48776), |
972 | + ("93.186.135.91", 80), "ESTABLISHED"), |
973 | + (117, 2, 1, ("10.0.0.1", 43761), |
974 | + ("72.14.234.100", 80), "CLOSING"), |
975 | + (119, 2, 1, ("10.0.0.1", 60759), |
976 | + ("72.14.234.104", 80), "ESTABLISHED"), |
977 | + (123, 2, 1, ("10.0.0.1", 51314), |
978 | + ("72.14.234.83", 443), "SYN_SENT") |
979 | + ] |
980 | + |
981 | + mock = self.mocker.mock() |
982 | + self.expect(mock.get_connections).result(mock) |
983 | + self.expect(mock.get_connections()).result(connections) |
984 | + self.mocker.replay() |
985 | + |
986 | + result = report_process_net_stats(process=mock) |
987 | + self.assertEqual(2, result["proc.net.status.established"]) |
988 | + self.assertEqual(1, result["proc.net.status.closing"]) |
989 | + self.assertEqual(1, result["proc.net.status.syn_sent"]) |
990 | + |
991 | + def test_reactor_stats(self): |
992 | + """Given a twisted reactor, pull out some stats from it.""" |
993 | + mock = self.mocker.mock() |
994 | + self.expect(mock.getReaders()).result([None, None, None]) |
995 | + self.expect(mock.getWriters()).result([None, None]) |
996 | + self.mocker.replay() |
997 | + |
998 | + result = report_reactor_stats(mock)() |
999 | + self.assertEqual(3, result["reactor.readers"]) |
1000 | + self.assertEqual(2, result["reactor.writers"]) |
1001 | + |
1002 | + def test_threadpool_stats(self): |
1003 | + """Given a twisted threadpool, pull out some stats from it.""" |
1004 | + mock = self.mocker.mock() |
1005 | + self.expect(mock.q.qsize()).result(42) |
1006 | + self.expect(mock.threads).result(6 * [None]) |
1007 | + self.expect(mock.waiters).result(2 * [None]) |
1008 | + self.expect(mock.working).result(4 * [None]) |
1009 | + self.mocker.replay() |
1010 | + |
1011 | + result = report_threadpool_stats(mock)() |
1012 | + self.assertEqual(42, result["threadpool.queue"]) |
1013 | + self.assertEqual(6, result["threadpool.threads"]) |
1014 | + self.assertEqual(2, result["threadpool.waiters"]) |
1015 | + self.assertEqual(4, result["threadpool.working"]) |
1016 | |
1017 | === modified file 'txstatsd/tests/test_service.py' |
1018 | --- txstatsd/tests/test_service.py 2011-07-05 05:27:22 +0000 |
1019 | +++ txstatsd/tests/test_service.py 2011-07-12 14:22:56 +0000 |
1020 | @@ -6,6 +6,7 @@ |
1021 | |
1022 | |
1023 | class GlueOptionsTestCase(TestCase): |
1024 | + |
1025 | def test_defaults(self): |
1026 | """ |
1027 | Defaults get passed over to the instance. |
1028 | @@ -94,6 +95,7 @@ |
1029 | |
1030 | |
1031 | class ServiceTests(TestCase): |
1032 | + |
1033 | def test_service(self): |
1034 | """ |
1035 | The StatsD service can be created. |
python-psutil needs to be added as a dependency.