Merge lp:~paulgear/charms/trusty/ntp/fix-divide-by-zero into lp:charms/trusty/ntp

[Paul Gear]

I've fixed the upstream version of check_ntpmon.py; this commit merges the current head version from https://github.com/paulgear/check_ntpmon

[Whit Morriss (whitmo)]

Thanks Paul!

LGTM +1. Merges cleanly, tests pass (demonstrating successful deployment).

My only gripe is that there are no unittests to cover this bug fix (or the python code in general). While this fix *looks* fine, having some test to prove it would be really nice!

-w

review: Approve

[lp:~paulgear/charms/trusty/ntp/fix-divide-by-zero]

19. By Charles Butler on 2015-04-16

[r=lazypower] Charles Butler 2015-04-16 [lazypower] Ammended unit test to specify private-address
Brad Marshall 2015-04-01 [merge] [bradm] Fixed unit name on the test
Whit Morriss 2015-03-31 specify unit
Brad Marshall 2015-03-31 [bradm] Merge in nrpe fixes
Brad Marshall 2015-03-27 [bradm] Remove the total_source line
Brad Marshall 2015-03-27 [bradm] Tidied up extra unneeded lines.
Brad Marshall 2015-03-27 [bradm] Deploy ntp subordinate before adding relation to ntpmaster, c...
Brad Marshall 2015-03-27 [bradm] Apply the timeout to the sentry
Brad Marshall 2015-03-27 [bradm] Add missing type to nagios_servicegroups
Brad Marshall 2015-03-27 [bradm] Fixed nagios_servicegroup setting
Brad Marshall 2015-03-27 [bradm] Added default empty string to nagios_servicegroup, unit tests...
Brad Marshall 2015-03-25 [merge] [bradm] Fixed conflicts with upstream
Brad Marshall 2015-03-17 [bradm] Add docs on auto peers and peers config options.
Brad Marshall 2015-03-16 [bradm] Fixed use_iburst variable to work properly
Brad Marshall 2015-03-16 [bradm] Added use_iburst config variable, tidy up warning message
Brad Marshall 2015-03-16 [bradm] Fixed formatting on warning
Brad Marshall 2015-03-16 [bradm] Not using brackets inside strings
Brad Marshall 2015-03-16 [bradm] Add backslash for continued line
Brad Marshall 2015-03-16 [bradm] Added restrict line for new ntp sources to ntp.conf. Add peer...
Brad Marshall 2015-03-16 [bradm] Gave source a default value, made auto_peers a boolean
Brad Marshall 2015-03-16 [bradm] Add auto_peers config option
Brad Marshall 2015-03-16 [bradm] First cut of ntp peers relationship

[Paul Gear (paulgear)]

Hi Whit,

I have a reasonably comprehensive test suite for it upstream: https://github.com/paulgear/ntpmon/blob/master/test_check_ntpmon.py; where would be the most appropriate point to run it?

[Whit Morriss (whitmo)]

I think in this case what would be best is unit tests that exercise the
hook code (which seem to be generally missing, not your fault). The ntpmon
tests do look handy for exercising a deployment in general though.

-w

On Thu, Apr 16, 2015 at 6:14 PM, Paul Gear <email address hidden> wrote:

> Hi Whit,
>
> I have a reasonably comprehensive test suite for it upstream:
> https://github.com/paulgear/ntpmon/blob/master/test_check_ntpmon.py;
> where would be the most appropriate point to run it?
> --
>
> https://code.launchpad.net/~paulgear/charms/trusty/ntp/fix-divide-by-zero/+merge/255660
> You are reviewing the proposed merge of
> lp:~paulgear/charms/trusty/ntp/fix-divide-by-zero into lp:charms/trusty/ntp.
>

--
---------------
D. Whit Morriss
Developer, Juju Ecosystem
Canonical USA

[Paul Gear (paulgear)]

I've done a quick rebase of this code to use the latest version of trunk. Is there anything else needed before this can be merged? Should I be asking for it to be merged another branch instead of trunk?

[lp:~paulgear/charms/trusty/ntp/fix-divide-by-zero]

20. By Paul Gear on 2015-04-20

Merge merge upstream check_ntpmon.py

21. By Paul Gear on 2015-04-27

Update to upstream check_ntpmon

Notable changes:
- detects the case when ntpd is not running correctly
- always returns OK until ntpd has been running for long enough to get 8 successful polls

22. By Paul Gear on 2015-04-27

Install new prerequisite for upstream check_ntpmon

[Adam Israel (aisrael)]

Hi Paul,

I had a chance to review your merge proposal today. The current tests pass cleanly. As far as your new test upstream, that could be integrated into the charm's tests/ directory; something like 20-test_check_ntpmon. Still, this looks good to me, especially as it fixes a bug. You have my +1.

[Adam Israel (aisrael)]

review: Approve

[Paul Gear (paulgear)]

> Hi Paul,
>
> I had a chance to review your merge proposal today. The current tests pass
> cleanly. As far as your new test upstream, that could be integrated into the
> charm's tests/ directory; something like 20-test_check_ntpmon. Still, this
> looks good to me, especially as it fixes a bug. You have my +1.

Thanks Adam. My impression of the charm tests/ directory was that it was intended for deployment tests rather than unit tests. Is that not the case? If not, I can easily add the test there.

Regards,
Paul

[lp:~paulgear/charms/trusty/ntp/fix-divide-by-zero]

23. By Paul Gear on 2015-05-14

Add check_ntpmon unit tests from https://github.com/paulgear/check_ntpmon/blob/master/test_check_ntpmon.py

24. By Paul Gear on 2015-05-14

Move unit tests to where they belong, add path to tested module; install software dependency

25. By Paul Gear on 2015-05-14

Fix comment

[Paul Gear (paulgear)]

I've added the unit tests for check_ntpmon to the unit_tests directory now.

[lp:~paulgear/charms/trusty/ntp/fix-divide-by-zero]

26. By Paul Gear on 2015-05-14

Fix setup script

[Tim Van Steenburgh (tvansteenburgh)]

LGTM, thanks Paul!

review: Approve

=== modified file 'files/nagios/check_ntpmon.py'
--- files/nagios/check_ntpmon.py	2015-03-23 00:39:09 +0000
+++ files/nagios/check_ntpmon.py	2015-05-14 10:36:53 +0000
@@ -20,28 +20,30 @@
 #
 
 import argparse
+import psutil
 import re
 import subprocess
 import sys
+import time
 import traceback
 import warnings
 
 
 def ishostnamey(name):
     """Return true if the passed name is roughly hostnamey.  NTP is rather casual about how it
-    reports hostnames and IP addresses, so we can't be too strict.  This function simply tests
+    reports hostnames and IP addresses, so we can't be too strict.  This method simply tests
     that all of the characters in the string are letters, digits, dash, or period."""
     return re.search(r'^[\w.-]*$', name) is not None and name.find('_') == -1
 
 
 def isipaddressy(name):
     """Return true if the passed name is roughly IP addressy.  NTP is rather casual about how it
-    reports hostnames and IP addresses, so we can't be too strict.  This function simply tests
+    reports hostnames and IP addresses, so we can't be too strict.  This method simply tests
     that all of the characters in the string are hexadecimal digits, period, or colon."""
     return re.search(r'^[0-9a-f.:]*$', name) is not None
 
 
-class CheckNTPMon(object):
+class CheckNTPMonSilent(object):
 
     def __init__(self,
                  warnpeers=2,
@@ -50,7 +52,6 @@
                  critoffset=50,
                  warnreach=75,
                  critreach=50):
-
         self.warnpeers = warnpeers
         self.okpeers = okpeers
         self.warnoffset = warnoffset
@@ -59,65 +60,130 @@
         self.critreach = critreach
 
     def peers(self, n):
+        if n >= self.okpeers:
+            return (0, "OK: %d usable peers" % (n))
+        elif n < self.warnpeers:
+            return (2, "CRITICAL: Too few peers (%d) - must be at least %d" %
+                    (n, self.warnpeers))
+        else:
+            return (1, "WARNING: Too few peers (%d) - should be at least %d" %
+                    (n, self.okpeers))
+
+    def offset(self, offset):
+        if abs(offset) > self.critoffset:
+            return (2,
+                    "CRITICAL: Offset too high (%g) - must be less than %g" %
+                    (offset, self.critoffset))
+        if abs(offset) > self.warnoffset:
+            return (1,
+                    "WARNING: Offset too high (%g) - should be less than %g" %
+                    (offset, self.warnoffset))
+        else:
+            return (0, "OK: Offset normal (%g)" % (offset))
+
+    def reachability(self, percent):
+        if percent < 0 or percent > 100:
+            raise ValueError('Value must be a percentage')
+        if percent <= self.critreach:
+            return (
+                2,
+                "CRITICAL: Reachability too low (%g%%) - must be more than %g%%" %
+                (percent, self.critreach))
+        elif percent <= self.warnreach:
+            return (
+                1,
+                "WARNING: Reachability too low (%g%%) - should be more than %g%%" %
+                (percent, self.warnreach))
+        else:
+            return (0, "OK: Reachability normal (%g%%)" % (percent))
+
+    def sync(self, synchost):
+        synced = len(synchost) > 0 and (ishostnamey(synchost) or
+                                        isipaddressy(synchost))
+        if synced:
+            return (0, "OK: time is in sync with %s" % (synchost))
+        else:
+            return (2, "CRITICAL: no sync host selected")
+
+    def is_silent(self):
+        return True
+
+    def dump(self):
+        print "warnpeers  = %d" % (self.warnpeers)
+        print "okpeers    = %d" % (self.okpeers)
+        print "warnoffset = %g" % (self.warnoffset)
+        print "critoffset = %g" % (self.critoffset)
+        print "warnreach  = %g" % (self.warnreach)
+        print "critreach  = %g" % (self.critreach)
+
+    @classmethod
+    def clone(cls, obj):
+        # must actually be a CheckNTPMonSilent object
+        assert obj.warnpeers is not None
+        if obj.is_silent():
+            return obj
+        else:
+            return CheckNTPMonSilent(warnpeers=obj.warnpeers,
+                                     okpeers=obj.okpeers,
+                                     warnoffset=obj.warnoffset,
+                                     critoffset=obj.critoffset,
+                                     warnreach=obj.warnreach,
+                                     critreach=obj.critreach)
+
+
+class CheckNTPMon(CheckNTPMonSilent):
+
+    """Version of CheckNTPMonSilent which prints out the diagnostic message and returns
+    integer return code instead of a list"""
+
+    def __init__(self,
+                 warnpeers=2,
+                 okpeers=4,
+                 warnoffset=10,
+                 critoffset=50,
+                 warnreach=75,
+                 critreach=50):
+        CheckNTPMonSilent.__init__(self, warnpeers, okpeers, warnoffset,
+                                   critoffset, warnreach, critreach)
+
+    def peers(self, n):
         """Return 0 if the number of peers is OK
         Return 1 if the number of peers is WARNING
         Return 2 if the number of peers is CRITICAL"""
-        if n >= self.okpeers:
-            print "OK: %d usable peers" % n
-            return 0
-        elif n < self.warnpeers:
-            print "CRITICAL: Too few peers (%d) - must be at least %d" % (n, self.warnpeers)
-            return 2
-        else:
-            print "WARNING: Too few peers (%d) - should be at least %d" % (n, self.okpeers)
-            return 1
+        code, msg = CheckNTPMonSilent.peers(self, n)
+        print msg
+        return code
 
     def offset(self, offset):
         """Return 0 if the offset is OK
         Return 1 if the offset is WARNING
         Return 2 if the offset is CRITICAL"""
-        if abs(offset) > self.critoffset:
-            print "CRITICAL: Offset too high (%g) - must be less than %g" % \
-                    (offset, self.critoffset)
-            return 2
-        if abs(offset) > self.warnoffset:
-            print "WARNING: Offset too high (%g) - should be less than %g" % \
-                    (offset, self.warnoffset)
-            return 1
-        else:
-            print "OK: Offset normal (%g)" % (offset)
-            return 0
+        code, msg = CheckNTPMonSilent.offset(self, offset)
+        print msg
+        return code
 
     def reachability(self, percent):
         """Return 0 if the reachability percentage is OK
         Return 1 if the reachability percentage is warning
         Return 2 if the reachability percentage is critical
         Raise a ValueError if reachability is not a percentage"""
-        if percent < 0 or percent > 100:
-            raise ValueError('Value must be a percentage')
-        if percent <= self.critreach:
-            print "CRITICAL: Reachability too low (%g%%) - must be more than %g%%" % \
-                    (percent, self.critreach)
-            return 2
-        elif percent <= self.warnreach:
-            print "WARNING: Reachability too low (%g%%) - should be more than %g%%" % \
-                    (percent, self.warnreach)
-            return 1
-        else:
-            print "OK: Reachability normal (%g%%)" % (percent)
-            return 0
+        code, msg = CheckNTPMonSilent.reachability(self, percent)
+        print msg
+        return code
 
     def sync(self, synchost):
-        """Return 0 if the synchost is non-zero in length and is a roughly valid host identifier, return 2 otherwise."""
-        synced = len(synchost) > 0 and (ishostnamey(synchost) or isipaddressy(synchost))
-        if synced:
-            print "OK: time is in sync with %s" % (synchost)
-        else:
-            print "CRITICAL: no sync host selected"
-        return 0 if synced else 2
+        """Return 0 if the synchost is non-zero in length and is a roughly valid host identifier
+        Return 2 otherwise"""
+        code, msg = CheckNTPMonSilent.sync(self, synchost)
+        print msg
+        return code
+
+    def is_silent(self):
+        return False
 
 
 class NTPPeers(object):
+
     """Turn the peer lines returned by 'ntpq -pn' into a data structure usable for checks."""
 
     noiselines = [
@@ -135,7 +201,8 @@
 
     def shouldignore(self, fields, l):
         if len(fields) != 10:
-            warnings.warn('Invalid ntpq peer line - there are %d fields: %s' % (len(fields), l))
+            warnings.warn('Invalid ntpq peer line - there are %d fields: %s' %
+                          (len(fields), l))
             return True
         if fields[1] in self.ignorepeers:
             return True
@@ -186,20 +253,23 @@
             if self.isnoiseline(l):
                 continue
 
-            # first column is the tally field, the rest are whitespace-separated fields
+            # first column is the tally field, the rest are
+            # whitespace-separated fields
             tally = l[0]
             fields = l[1:-1].split()
 
             if self.shouldignore(fields, l):
                 continue
 
-            fieldnames = ['peer', 'refid', 'stratum', 'type', 'lastpoll', 'interval', 'reach',
-                          'delay', 'offset', 'jitter']
+            fieldnames = ['peer', 'refid', 'stratum', 'type', 'lastpoll',
+                          'interval', 'reach', 'delay', 'offset', 'jitter']
             peerdata = dict(zip(fieldnames, fields))
 
             offset = abs(float(peerdata['offset']))
             if not self.parsetally(tally, peerdata, offset):
-                warnings.warn('Unknown tally code detected - please report a bug: %s' % (l))
+                warnings.warn(
+                    'Unknown tally code detected - please report a bug: %s' %
+                    (l))
                 continue
 
             self.ntpdata['peers'] += 1
@@ -208,35 +278,43 @@
             # reachability - this counts the number of bits set in the reachability field
             # (which is displayed in octal in the ntpq output)
             # http://stackoverflow.com/questions/9829578/fast-way-of-counting-bits-in-python
-            self.ntpdata['totalreach'] += bin(int(peerdata['reach'], 8)).count("1")
-
-        # reachability as a percentage of the last 8 polls, across all peers
-        self.ntpdata['reachability'] = float(self.ntpdata['totalreach']) * 100 / self.ntpdata['peers'] / 8
+            self.ntpdata['totalreach'] += bin(int(peerdata['reach'],
+                                                  8)).count("1")
 
         # average offsets
         if self.ntpdata['survivors'] > 0:
-            self.ntpdata['averageoffsetsurvivors'] = \
-                    self.ntpdata['offsetsurvivors'] / self.ntpdata['survivors']
+            self.ntpdata['averageoffsetsurvivors'] = self.ntpdata['offsetsurvivors'] / self.ntpdata['survivors']
         if self.ntpdata['discards'] > 0:
-            self.ntpdata['averageoffsetdiscards'] = \
-                    self.ntpdata['offsetdiscards'] / self.ntpdata['discards']
-        self.ntpdata['averageoffset'] = self.ntpdata['offsetall'] / self.ntpdata['peers']
+            self.ntpdata['averageoffsetdiscards'] = self.ntpdata['offsetdiscards'] / self.ntpdata['discards']
+
+        if self.ntpdata['peers'] > 0:
+            # precent average reachability of all peers over the last 8 polls
+            reach = float(self.ntpdata['totalreach']) * 100 / self.ntpdata['peers']
+            self.ntpdata['reachability'] = reach / 8
+
+            # average offset of all peers
+            self.ntpdata['averageoffset'] = self.ntpdata['offsetall'] / self.ntpdata['peers']
+        else:
+            # if there are no peers, reachability is zero and average offset is invalid
+            self.ntpdata['reachability'] = 0.0
+            self.ntpdata['averageoffset'] = float('nan')
 
     def dump(self):
         if self.ntpdata.get('syncpeer'):
-            print "Synced to: %s, offset %g ms" % \
-                    (self.ntpdata['syncpeer'], self.ntpdata['offsetsyncpeer'])
+            print "Synced to: %s, offset %g ms" % (
+                self.ntpdata['syncpeer'], self.ntpdata['offsetsyncpeer'])
         else:
             print "No remote sync peer"
-        print "%d total peers, average offset %g ms" % \
-                (self.ntpdata['peers'], self.ntpdata['averageoffset'])
+        print "%d total peers, average offset %g ms" % (
+            self.ntpdata['peers'], self.ntpdata['averageoffset'])
         if self.ntpdata['survivors'] > 0:
-            print "%d good peers, average offset %g ms" % \
-                    (self.ntpdata['survivors'], self.ntpdata['averageoffsetsurvivors'])
+            print "%d good peers, average offset %g ms" % (
+                self.ntpdata['survivors'], self.ntpdata['averageoffsetsurvivors'])
         if self.ntpdata['discards'] > 0:
-            print "%d discarded peers, average offset %g ms" % \
-                    (self.ntpdata['discards'], self.ntpdata['averageoffsetdiscards'])
-        print "Average reachability of all peers: %d%%" % (self.ntpdata['reachability'])
+            print "%d discarded peers, average offset %g ms" % (
+                self.ntpdata['discards'], self.ntpdata['averageoffsetdiscards'])
+        print "Average reachability of all peers: %d%%" % (
+            self.ntpdata['reachability'])
 
     def check_peers(self, check=None):
         """Check the number of usable peers"""
@@ -246,11 +324,13 @@
 
     def check_offset(self, check=None):
         """Check the offset from the sync peer, returning critical, warning,
-        or OK based on the CheckNTPMon results.
+            or OK based on the CheckNTPMon results.
         If there is no sync peer, use the average offset of survivors instead.
-        If there are no survivors, use the average offset of discards instead, and return warning as a minimum.
+        If there are no survivors, use the average offset of discards instead,
+            and return warning as a minimum.
         If there are no discards, return critical.
         """
+        result = 0
         if check is None:
             check = self.check if self.check else CheckNTPMon()
         if 'offsetsyncpeer' in self.ntpdata:
@@ -259,10 +339,20 @@
             return check.offset(self.ntpdata['averageoffsetsurvivors'])
         if 'averageoffsetdiscards' in self.ntpdata:
             result = check.offset(self.ntpdata['averageoffsetdiscards'])
-            return 1 if result < 1 else result
+            msg = "WARNING: No sync peer or survivors - used discard offsets"
+            if check.is_silent():
+                return [1 if result[0] < 1 else result[0],
+                          msg + " (" + result[1] + ")"]
+            else:
+                print msg
+                return 1 if result < 1 else result
         else:
-            print "CRITICAL: No peers for which to check offset"
-            return 2
+            ret = [2, "CRITICAL: No peers for which to check offset"]
+            if check.is_silent():
+                return ret
+            else:
+                print ret[1]
+                return ret[0]
 
     def check_reachability(self, check=None):
         """Check reachability of all peers"""
@@ -275,74 +365,166 @@
         if check is None:
             check = self.check if self.check else CheckNTPMon()
         if self.ntpdata.get('syncpeer') is None:
-            print "CRITICAL: No sync peer"
-            return 2
+            ret = [2, "CRITICAL: No sync peer"]
+            if check.is_silent():
+                return ret
+            else:
+                print ret[1]
+                return ret[0]
         return check.sync(self.ntpdata['syncpeer'])
 
     def checks(self, methods=None, check=None):
-        ret = 0
+        """Run the specified list of checks (or all of them if none is supplied)
+        and return the worst result.  Output only the diagnostic message for that
+        result."""
+
+        # ensure check exists and is silent
+        if check is None:
+            check = self.check if self.check else CheckNTPMonSilent()
+        if not check.is_silent():
+            check = CheckNTPMonSilent.clone(check)
+        assert check.is_silent()
+
         if not methods:
-            methods = [self.check_offset, self.check_peers, self.check_reachability, self.check_sync]
+            methods = [self.check_offset, self.check_peers,
+                       self.check_reachability, self.check_sync]
+
+        ret = -1
+        msg = None
         for method in methods:
-            check = method()
-            if ret < check:
-                ret = check
+            result = method(check=check)
+            if ret < result[0]:
+                ret = result[0]
+                msg = result[1]
+
+        if msg is None:
+            print "%s returned no results - please report a bug" % (method)
+            return 3
+        print msg
         return ret
 
     @staticmethod
     def query():
         lines = None
         try:
-            output = subprocess.check_output(["ntpq", "-pn"])
-            lines = output.split("\n")
+            null = open("/dev/null", "a")
+            output = subprocess.check_output(["ntpq", "-pn"], stderr=null)
+            if len(output) > 0:
+                lines = output.split("\n")
         except:
             traceback.print_exc(file=sys.stdout)
         return lines
 
 
+class NTPProcess(object):
+
+    def __init__(self, names=None):
+        """Look for ntpd or xntpd in the process table and save its process object."""
+        if names is None:
+            names = ["ntpd", "xntpd"]
+        self.proc = None
+        for proc in psutil.process_iter():
+            try:
+                if proc.name() in names:
+                    self.proc = proc
+                    break
+            except psutil.Error:
+                pass
+
+    def runtime(self):
+        """Return the length of time in seconds that the process has been running.
+        If ntpd is not running or any error occurs, return -1."""
+        if self.proc is None:
+            return -1
+        try:
+            now = time.time()
+            start = int(self.proc.create_time())
+            return now - start
+        except psutil.Error:
+            return -1
+
+
 def main():
     methodnames = ['offset', 'peers', 'reachability', 'sync']
     options = {
-        'warnpeers':  [  2, int,   'Minimum number of peers to be considered non-critical'],
-        'okpeers':    [  4, int,   'Minimum number of peers to be considered OK'],
-        'warnoffset': [ 10, float, 'Minimum offset to be considered warning'],
-        'critoffset': [ 50, float, 'Minimum offset to be considered critical'],
-        'warnreach':  [ 75, float, 'Minimum peer reachability percentage to be considered OK'],
-        'critreach':  [ 50, float, 'Minimum peer reachability percentage to be considered non-crtical'],
+        'warnpeers': [
+            2, int, 'Minimum number of peers to be considered non-critical',
+        ],
+        'okpeers': [
+            4, int, 'Minimum number of peers to be considered OK',
+        ],
+        'warnoffset': [
+            10, float, 'Minimum offset to be considered warning',
+        ],
+        'critoffset': [
+            50, float, 'Minimum offset to be considered critical',
+        ],
+        'warnreach': [
+            75, float, 'Minimum peer reachability percentage to be considered OK',
+        ],
+        'critreach': [
+            50, float, 'Minimum peer reachability percentage to be considered non-crtical',
+        ],
     }
 
     # Create check ranges; will be used by parse_args to store options
     checkntpmon = CheckNTPMon()
 
     # parse command line
-    parser = argparse.ArgumentParser(description='Nagios NTP check incorporating the logic of NTPmon')
-    parser.add_argument('--check', choices=methodnames,
-            help='Select check to run; if omitted, run all checks and return the worst result.')
-    parser.add_argument('--debug', action='store_true',
-            help='Include "ntpq -pn" output and internal state dump along with check results.')
+    parser = argparse.ArgumentParser(
+        description='Nagios NTP check incorporating the logic of NTPmon')
+    parser.add_argument(
+        '--check',
+        choices=methodnames,
+        help='Select check to run; if omitted, run all checks and return the worst result.')
+    parser.add_argument(
+        '--debug',
+        action='store_true',
+        help='Include "ntpq -pn" output and internal state dump along with check results.')
+    parser.add_argument(
+        '--run-time',
+        default=512,
+        type=int,
+        help='Time in seconds (default: 512) for which to always return OK after ntpd startup.')
+    parser.add_argument(
+        '--test',
+        action='store_true',
+        help='Accept "ntpq -pn" output on standard input instead of running it.')
     for o in options.keys():
         helptext = options[o][2] + ' (default: %d)' % (options[o][0])
-        parser.add_argument('--' + o, default=options[o][0], help=helptext, type=options[o][1])
+        parser.add_argument('--' + o,
+                            default=options[o][0],
+                            help=helptext,
+                            type=options[o][1])
     args = parser.parse_args(namespace=checkntpmon)
 
     # run ntpq
-    lines = NTPPeers.query()
+    lines = NTPPeers.query() if not args.test else [x.rstrip() for x in sys.stdin.readlines()]
     if lines is None:
         # Unknown result
-        print "Cannot get peers from ntpq."
-        print "Please check that an NTP server is installed and functional."
+        print "UNKNOWN: Cannot get peers from ntpq.  Please check that an NTP server is installed and running."
         sys.exit(3)
 
-    # initialise our object with the results of ntpq and our preferred check thresholds
+    # Don't report anything other than OK until ntpd has been running for at
+    # least enough time for 8 polling intervals of 64 seconds each.
+    age = NTPProcess().runtime()
+    if age > 0 and age <= args.run_time:
+        print "OK: ntpd still starting up (running %d seconds)" % age
+        sys.exit(0)
+
+    # initialise our object with the results of ntpq and our preferred check
+    # thresholds
     ntp = NTPPeers(lines, checkntpmon)
 
     if args.debug:
         print "\n".join(lines)
+        checkntpmon.dump()
         ntp.dump()
 
     # work out which method to run
     # (methods must be in the same order as methodnames above)
-    methods = [ntp.check_offset, ntp.check_peers, ntp.check_reachability, ntp.check_sync]
+    methods = [ntp.check_offset, ntp.check_peers, ntp.check_reachability,
+               ntp.check_sync]
     checkmethods = dict(zip(methodnames, methods))
 
     # if check argument is specified, run just that check
@@ -352,10 +534,11 @@
         ret = method()
     # else check all the methods
     else:
-        ret = ntp.checks(methods)
+        ret = ntp.checks()
 
     sys.exit(ret)
 
+
 if __name__ == "__main__":
     main()
 

=== modified file 'hooks/ntp_hooks.py'
--- hooks/ntp_hooks.py	2015-03-31 00:35:56 +0000
+++ hooks/ntp_hooks.py	2015-05-14 10:36:53 +0000
@@ -88,7 +88,8 @@
             'nrpe-external-master-relation-changed')
 def update_nrpe_config():
     # python-dbus is used by check_upstart_job
-    fetch.apt_install('python-dbus')
+    # python-psutil is used by check_ntpmon
+    fetch.apt_install(['python-dbus', 'python-psutil'])
     nagios_ntpmon_checks = hookenv.config('nagios_ntpmon_checks')
     if os.path.isdir(NAGIOS_PLUGINS):
         host.rsync(os.path.join(os.getenv('CHARM_DIR'), 'files', 'nagios',

=== modified file 'tests/00-setup'
--- tests/00-setup	2014-06-18 18:11:00 +0000
+++ tests/00-setup	2015-05-14 10:36:53 +0000
@@ -3,11 +3,11 @@
 set -ex 
 
 # Check if amulet is installed before adding repository and updating apt-get.
-dpkg -s amulet
-if [ $? -ne 0 ]; then
+if ! dpkg -s amulet; then
     sudo add-apt-repository -y ppa:juju/stable
     sudo apt-get update
     sudo apt-get install -y amulet
 fi
 
 # Install any additional python packages or software here.
+sudo apt-get install -y python-psutil

=== added directory 'unit_tests'
=== added file 'unit_tests/test_check_ntpmon.py'
--- unit_tests/test_check_ntpmon.py	1970-01-01 00:00:00 +0000
+++ unit_tests/test_check_ntpmon.py	2015-05-14 10:36:53 +0000
@@ -0,0 +1,492 @@
+#!/usr/bin/python
+#
+# Author:       Paul Gear
+# Copyright:    (c) 2015 Gear Consulting Pty Ltd <http://libertysys.com.au/>
+# License:      GPLv3 <http://www.gnu.org/licenses/gpl.html>
+# Description:  Test CheckNTPMon class
+#
+# This program is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the Free Software
+# Foundation, either version 3 of the License, or (at your option) any later
+# version.
+#
+# This program is distributed in the hope that it will be useful, but WITHOUT
+# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+# FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more
+# details.
+#
+# You should have received a copy of the GNU General Public License along with
+# this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import argparse
+import unittest
+import sys
+
+from pkg_resources import resource_filename
+
+# allow importing check_ntpmon from the correct directory
+sys.path.append(resource_filename(__name__, '../files/nagios'))
+
+from check_ntpmon import CheckNTPMon, CheckNTPMonSilent, NTPPeers
+
+testdata = [
+"""
+     remote           refid      st t when poll reach   delay   offset  jitter
+==============================================================================
+ 137.189.4.10    .STEP.          16 u    - 1024    0    0.000    0.000   0.000
+ 128.199.253.156 .STEP.          16 u    - 1024    0    0.000    0.000   0.000
+ 103.233.241.1   .STEP.          16 u    - 1024    0    0.000    0.000   0.000
+ 128.199.169.185 .STEP.          16 u    - 1024    0    0.000    0.000   0.000
+*91.189.89.199   131.188.3.220    2 u 1019 1024  377  346.845   -0.598   1.105
+ 192.189.54.33   .STEP.          16 u    - 1024    0    0.000    0.000   0.000
+ 129.250.35.250  .STEP.          16 u    - 1024    0    0.000    0.000   0.000
+ 27.54.95.11     .STEP.          16 u    - 1024    0    0.000    0.000   0.000
+ 54.252.129.186  .STEP.          16 u    - 1024    0    0.000    0.000   0.000
+""",
+"""
+     remote           refid      st t when poll reach   delay   offset  jitter
+==============================================================================
+-203.19.252.1    210.9.192.50     2 u  456 1024  377   47.629   -1.526  35.084
+*202.60.94.11    223.252.32.9     2 u  618 1024  377   18.785    0.259   0.371
++202.60.94.15    223.252.32.9     2 u  635 1024  377   19.333    0.038   0.851
++54.252.129.186  223.252.32.9     2 u  927 1024  377   36.072    0.337   1.043
+-192.168.1.2     203.23.237.200   3 u  788 1024  377    0.627   -0.605   0.162
+-192.168.1.1     203.23.237.200   3 u  123 1024  377    0.299   -0.941   0.329
+ 192.168.1.21    .INIT.          16 u    - 1024    0    0.000    0.000   0.000
+""",
+"""
+     remote           refid      st t when poll reach   delay   offset  jitter
+==============================================================================
+*91.189.94.4     131.188.3.220    2 u  338 1024  377    1.600  -194.54 171.548
+""",
+"""
+     remote           refid      st t when poll reach   delay   offset  jitter
+==============================================================================
+ 218.189.210.3   .INIT.          16 u    -   64    0    0.000    0.000   0.000
+ 103.224.117.98  .INIT.          16 u    -   64    0    0.000    0.000   0.000
+ 118.143.17.82   .INIT.          16 u    -   64    0    0.000    0.000   0.000
+ 91.189.89.199   192.93.2.20      2 u   21   64    7  336.631    0.913   0.223
+ 175.45.85.97    .INIT.          16 u    -   64    0    0.000    0.000   0.000
+ 192.189.54.33   .INIT.          16 u    -   64    0    0.000    0.000   0.000
+ 129.250.35.250  .INIT.          16 u    -   64    0    0.000    0.000   0.000
+ 54.252.165.245  .INIT.          16 u    -   64    0    0.000    0.000   0.000
+""",
+]
+
+demodata = [
+"""
+     remote  refid   st t when poll reach   delay   offset    disp
+=========================================================================
+*WWVB_SPEC(1)  .WWVB.  0 l  114   64  377     0.00   37.623   12.77
+ relay.hp.com    listo 2 u  225  512  377     6.93   34.052   10.79
+ cosl4.cup.hp.co listo 2 u  226  512  377     4.18   29.385   13.21
+ paloalto.cns.hp listo 2 u  235  512  377     9.80   33.487   11.61
+ chelmsford.cns. listo 2 u  233  512  377    88.79   30.462    9.66
+ atlanta.cns.hp. listo 2 u  231  512  377    67.44   32.909   12.86
+ colorado.cns.hp listo 2 u  233  512  377    43.70   30.077   18.63
+ boise.cns.hp.co listo 2 u  224  512  377    33.42   31.682    8.54
+""",
+"""
+     remote           refid      st t when poll reach   delay   offset  jitter
+==============================================================================
+ GENERIC(1)      .GPS.            0 l    4   64    1    0.000   -0.719   0.001
+ PPS(1)          .PPS.           16 l    -   64    0    0.000    0.000 4000.00
+ ltgpsdemo       .INIT.          16 u    3   64    0    0.000    0.000 4000.00
+""",
+"""
+     remote           refid      st t when poll reach   delay   offset  jitter
+==============================================================================
+ GENERIC(1)      .GPS.            0 l   38   64    7    0.000   -1.193   0.528
+ PPS(1)          .PPS.           16 l    -   64    0    0.000    0.000 4000.00
+ ltgpsdemo       .GPS.            1 u   33   64    1    0.624   -0.417   0.001
+""",
+"""
+     remote           refid      st t when poll reach   delay   offset  jitter
+==============================================================================
+*GENERIC(1)      .GPS.            0 l   45   64  377    0.000   -0.437   0.203
+ PPS(1)          .PPS.           16 l    -   64    0    0.000    0.000 4000.00
++ltgpsdemo       .PPS.            1 u  116  512  377    0.500    0.349   0.106
+""",
+# mangled from Cisco router demo output:
+"""
+ 27.54.95.11     .STEP.          16 u    -     64     0  0.000   0.000 15937.
++130.102.128.23  216.218.254.20   2 u   25     64    77 51.267  32.537 189.39
+ 128.184.34.53   169.254.0.1      3 u   64     64   122 49.115  29.474 1939.5
+*129.250.35.250  133.243.238.24   2 u   14     64   177 261.47   7.906 65.514
++129.250.35.251  133.243.238.24   2 u   55     64    77 255.70  13.942 190.86
+""",
+"""
+     remote           refid      st t when poll reach   delay   offset  jitter
+==============================================================================
+ ff05::101       .MCST.          16 u    -   64    0    0.000    0.000 4000.00
+*example.site.co .PPS.            1 u  320 1024  377    1.955   -1.234   1.368
+""",
+"""
+remote refid st t when poll reach delay offset jitter
+==========================================================
+-navobs1.oar.net .USNO. 1 u 958 1024 377 89.425 -6.073 0.695
+*navobs1.gatech. .GPS. 1 u 183 1024 375 82.102 1.639 0.281
+-NAVOBS1.MIT.EDU .PSC. 1 u 895 1024 377 90.912 -0.207 0.368
++navobs1.wustl.e .GPS. 1 u 48 1024 377 76.890 1.093 0.525
+-bigben.cac.wash .USNO. 1 u 924 1024 377 113.327 0.028 0.326
++tick.ucla.edu .GPS. 1 u 107 1024 377 102.470 2.032 0.482
+-ntp.alaska.edu .GPS. 1 u 881 1024 377 168.741 5.180 5.157
+-tock.mhpcc.hpc. .GPS. 1 u 933 1024 377 174.518 -1.094 0.054
+""",
+"""
+remote refid st t when poll reach delay offset disp
+==========================================================
++128.252.19.1 .GPS. 1 u 495 1024 377 30.90 -6.366 8.26
+*139.78.133.139 .USNO. 1 u 936 1024 377 48.43 -2.906 5.20
+""",
+"""
+remote refid st t when poll reach delay offset jitter
+======================================================
++navobs1.wustl.e .GPS. 1 u 241 256 377 77.626 1.744 0.195
+-tick.ucla.edu .GPS. 1 u 136 256 377 102.069 2.019 2.281
++ntp.alaska.edu .GPS. 1 u 207 256 377 168.971 0.528 6.612
+*GPS_NMEA(1) .GPS. 0 l 62 64 377 0.000 0.000 0.001
+LOCAL(0) .LOCL. 10 l 62 64 377 0.000 0.000 0.000
+""",
+"""
+     remote           refid      st t when poll reach   delay   offset  jitter
+==============================================================================
+*dione.cbane.org 204.123.2.5      2 u  509 1024  377   51.661   -3.343   0.279
++ns1.your-site.c 132.236.56.252   3 u  899 1024  377   48.395    2.047   1.006
++ntp.yoinks.net  129.7.1.66       2 u  930 1024  377    0.693    1.035   0.241
+ LOCAL(0)        .LOCL.          10 l   45   64  377    0.000    0.000   0.001
+""",
+"""
+remote refid st t when poll reach delay offset jitter
+======================================================================
+ 6s-ntp .ACST. 16 u - 64 0 0.000 0.000 0.002
+*ntp0.kostecke.n 192.168.19.2 3 u 225 1024 377 0.723 -3.463 1.889
+""",
+"""
+     remote refid st t when poll reach delay offset jitter
+==============================================================================
+ 10.35.60.40 .INIT. 16 u 505 1024 0 0.000 0.000 0.000
+ 10.35.60.41 .INIT. 16 u 471 1024 0 0.000 0.000 0.000
+""",
+]
+
+
+class TestCheckNTPMon(unittest.TestCase):
+
+    def test_offset(self):
+        check = CheckNTPMon()
+
+        for i in [50.01, 50.1, 51, 99, 100, 999]:
+            self.assertEqual(check.offset(i), 2, 'High offset non-critical')
+            self.assertEqual(check.offset(-i), 2, 'High offset non-critical')
+
+        for i in [10.01, 10.1, 11, 49, 49.99, 50]:
+            self.assertEqual(check.offset(i), 1, 'Moderate offset non-warning')
+            self.assertEqual(check.offset(-i), 1, 'Moderate offset non-warning')
+
+        for i in [0, 0.01, 1, 1.01, 9, 9.99, 10]:
+            self.assertEqual(check.offset(i), 0, 'Low offset non-OK')
+            self.assertEqual(check.offset(-i), 0, 'Low offset non-OK')
+
+    def test_peers(self):
+        check = CheckNTPMon()
+
+        for i in [-100, -10, -1, 0, 1]:
+            self.assertEqual(check.peers(i), 2, 'Low peers non-critical')
+
+        for i in [2, 3]:
+            self.assertEqual(check.peers(i), 1, 'Few peers non-warning')
+
+        for i in [4, 5, 6, 10, 100]:
+            self.assertEqual(check.peers(i), 0, 'High peers non-OK')
+
+    def test_reach(self):
+        check = CheckNTPMon()
+
+        for i in [0, 0.01, 1, 25, 49, 49.99, 50]:
+            self.assertEqual(check.reachability(i), 2, 'Low reachability non-critical')
+        for i in [50.01, 50.1, 74.99, 75]:
+            self.assertEqual(check.reachability(i), 1, 'Moderate reachability non-warning')
+        for i in [75.01, 76, 99, 100]:
+            self.assertEqual(check.reachability(i), 0, 'High reachability non-OK')
+        # check that invalid percentage causes exception
+        for i in [-100, -1, 100.01, 101, 1000]:
+            self.assertRaises(ValueError, check.reachability, (i))
+
+    def test_sync(self):
+        check = CheckNTPMon()
+
+        self.assertEqual(check.sync(''), 2, 'Invalid sync peer not detected')
+        self.assertEqual(check.sync('    '), 2, 'Invalid sync peer not detected')
+        self.assertEqual(check.sync('!@#$%^&*()'), 2, 'Invalid sync peer not detected')
+        self.assertEqual(check.sync('blah.example.com'), 0, 'Sync peer not detected')
+        self.assertEqual(check.sync('192.168.2.1'), 0, 'Sync peer not detected')
+        self.assertEqual(check.sync('fe80::1'), 0, 'Sync peer not detected')
+        self.assertEqual(check.sync('ds002.dedicated'), 0, 'Sync peer not detected')
+        self.assertEqual(check.sync('node01.au.serve'), 0, 'Sync peer not detected')
+
+    def test_NTPPeer0(self):
+        # check the parsing done by NTPPeers
+        ntp = NTPPeers(testdata[0].split("\n"))
+        self.assertEqual(ntp.ntpdata['syncpeer'], '91.189.89.199')
+        self.assertEqual(ntp.ntpdata['offsetsyncpeer'], 0.598)
+        self.assertEqual(ntp.ntpdata['survivors'], 1)
+        self.assertEqual(ntp.ntpdata['averageoffsetsurvivors'], 0.598)
+        self.assertEqual(ntp.ntpdata['discards'], 0)
+        self.assertEqual(ntp.ntpdata.get('averageoffsetdiscards'), None)
+        self.assertEqual(ntp.ntpdata['peers'], 1)
+        self.assertEqual(ntp.ntpdata['averageoffset'], 0.598)
+        self.assertEqual(ntp.ntpdata['reachability'], 100)
+
+        # run checks on the data
+        check = CheckNTPMon()
+        self.assertEqual(check.sync(ntp.ntpdata['syncpeer']), 0, 'Sync peer not detected')
+        self.assertEqual(check.offset(ntp.ntpdata['offsetsyncpeer']), 0, 'Low offset non-OK')
+        self.assertEqual(check.offset(ntp.ntpdata['averageoffsetsurvivors']), 0, 'Low offset non-OK')
+        self.assertEqual(check.offset(ntp.ntpdata['averageoffset']), 0, 'Low offset non-OK')
+        self.assertEqual(check.peers(ntp.ntpdata['peers']), 2, 'Low peers non-critical')
+        self.assertEqual(check.reachability(ntp.ntpdata['reachability']), 0,
+                'High reachability non-OK')
+
+        # run overall health checks
+        self.assertEqual(ntp.check_sync(), 0, 'Sync peer not detected')
+        self.assertEqual(ntp.check_offset(), 0, 'Low offset non-OK')
+        self.assertEqual(ntp.check_peers(), 2, 'Low peers non-critical')
+        self.assertEqual(ntp.check_reachability(), 0, 'High reachability non-OK')
+
+    def test_NTPPeer1(self):
+        # check the parsing done by NTPPeers
+        ntp = NTPPeers(testdata[1].split("\n"))
+        self.assertEqual(ntp.ntpdata['syncpeer'], '202.60.94.11')
+        self.assertEqual(ntp.ntpdata['offsetsyncpeer'], 0.259)
+        self.assertEqual(ntp.ntpdata['survivors'], 3)
+        self.assertEqual(ntp.ntpdata['averageoffsetsurvivors'], 0.21133333333333335)
+        self.assertEqual(ntp.ntpdata['discards'], 3)
+        self.assertEqual(ntp.ntpdata['averageoffsetdiscards'], 1.024)
+        self.assertEqual(ntp.ntpdata['peers'], 6)
+        self.assertEqual(ntp.ntpdata['averageoffset'], 0.6176666666666667)
+        self.assertEqual(ntp.ntpdata['reachability'], 100)
+
+        # run checks on the data
+        check = CheckNTPMon()
+        self.assertEqual(check.sync(ntp.ntpdata['syncpeer']), 0, 'Sync peer not detected')
+        self.assertEqual(check.offset(ntp.ntpdata['offsetsyncpeer']), 0, 'Low offset non-OK')
+        self.assertEqual(check.offset(ntp.ntpdata['averageoffsetsurvivors']), 0, 'Low offset non-OK')
+        self.assertEqual(check.offset(ntp.ntpdata['averageoffsetdiscards']), 0, 'Low offset non-OK')
+        self.assertEqual(check.offset(ntp.ntpdata['averageoffset']), 0, 'Low offset non-OK')
+        self.assertEqual(check.peers(ntp.ntpdata['peers']), 0, 'High peers non-OK')
+        self.assertEqual(check.reachability(ntp.ntpdata['reachability']), 0,
+                         'High reachability non-OK')
+
+        # run overall health checks
+        self.assertEqual(ntp.check_sync(), 0, 'Sync peer not detected')
+        self.assertEqual(ntp.check_offset(), 0, 'Low offset non-OK')
+        self.assertEqual(ntp.check_peers(), 0, 'High peers non-OK')
+        self.assertEqual(ntp.check_reachability(), 0, 'High reachability non-OK')
+
+    def test_NTPPeer2(self):
+        # check the parsing done by NTPPeers
+        ntp = NTPPeers(testdata[2].split("\n"))
+        self.assertEqual(ntp.ntpdata['syncpeer'], '91.189.94.4')
+        self.assertEqual(ntp.ntpdata['offsetsyncpeer'], 194.54)
+        self.assertEqual(ntp.ntpdata['survivors'], 1)
+        self.assertEqual(ntp.ntpdata['averageoffsetsurvivors'], 194.54)
+        self.assertEqual(ntp.ntpdata['discards'], 0)
+        self.assertEqual(ntp.ntpdata.get('averageoffsetdiscards'), None)
+        self.assertEqual(ntp.ntpdata['peers'], 1)
+        self.assertEqual(ntp.ntpdata['averageoffset'], 194.54)
+        self.assertEqual(ntp.ntpdata['reachability'], 100)
+
+        # run checks on the data
+        check = CheckNTPMon()
+        self.assertEqual(check.sync(ntp.ntpdata['syncpeer']), 0, 'Sync peer not detected')
+        self.assertEqual(check.offset(ntp.ntpdata['offsetsyncpeer']), 2, 'High offset non-critical')
+        self.assertEqual(check.offset(ntp.ntpdata['averageoffsetsurvivors']), 2, 'High offset non-critical')
+        self.assertEqual(ntp.ntpdata.get('averageoffsetdiscards'), None)
+        self.assertEqual(check.offset(ntp.ntpdata['averageoffset']), 2, 'High offset non-critical')
+        self.assertEqual(check.peers(ntp.ntpdata['peers']), 2, 'Low peers non-critical')
+        self.assertEqual(check.reachability(ntp.ntpdata['reachability']), 0,
+                         'High reachability non-OK')
+
+        # run overall health checks
+        self.assertEqual(ntp.check_sync(), 0, 'Sync peer not detected')
+        self.assertEqual(ntp.check_offset(), 2, 'High offset non-critical')
+        self.assertEqual(ntp.check_peers(), 2, 'Low peers non-critical')
+        self.assertEqual(ntp.check_reachability(), 0, 'High reachability non-OK')
+
+    def test_NTPPeer3(self):
+        # check the parsing done by NTPPeers
+        ntp = NTPPeers(testdata[3].split("\n"))
+        self.assertEqual(ntp.ntpdata.get('syncpeer'), None)
+        self.assertEqual(ntp.ntpdata.get('offsetsyncpeer'), None)
+        self.assertEqual(ntp.ntpdata['survivors'], 0)
+        self.assertEqual(ntp.ntpdata.get('averageoffsetsurvivors'), None)
+        self.assertEqual(ntp.ntpdata['discards'], 1)
+        self.assertEqual(ntp.ntpdata['averageoffsetdiscards'], 0.913)
+        self.assertEqual(ntp.ntpdata['peers'], 1)
+        self.assertEqual(ntp.ntpdata['averageoffset'], 0.913)
+        self.assertEqual(ntp.ntpdata['reachability'], 37.5)
+
+        # run checks on the data
+        check = CheckNTPMon()
+        self.assertEqual(check.offset(ntp.ntpdata['averageoffsetdiscards']), 0, 'Low offset non-OK')
+        self.assertEqual(check.offset(ntp.ntpdata['averageoffset']), 0, 'Low offset non-OK')
+        self.assertEqual(check.peers(ntp.ntpdata['peers']), 2, 'Low peers non-critical')
+        self.assertEqual(check.reachability(ntp.ntpdata['reachability']), 2,
+                         'Low reachability non-critical')
+
+        # run overall health checks
+        self.assertEqual(ntp.check_sync(), 2, 'Missing sync peer not detected')
+        self.assertEqual(ntp.check_offset(), 1, 'Missing sync peer/survivor offset non-warning')
+        self.assertEqual(ntp.check_peers(), 2, 'Low peers non-critical')
+        self.assertEqual(ntp.check_reachability(), 2, 'Low reachability non-critical')
+
+    def test_defaults(self):
+        c = CheckNTPMon()
+        self.assertEqual(c.warnpeers, 2)
+        self.assertEqual(c.okpeers, 4)
+        self.assertEqual(c.warnoffset, 10)
+        self.assertEqual(c.critoffset, 50)
+        self.assertEqual(c.warnreach, 75)
+        self.assertEqual(c.critreach, 50)
+
+    def test_non_default(self):
+        c = CheckNTPMon(1, 2, 9, 49, 80, 60)
+        self.assertEqual(c.warnpeers, 1)
+        self.assertEqual(c.okpeers, 2)
+        self.assertEqual(c.warnoffset, 9)
+        self.assertEqual(c.critoffset, 49)
+        self.assertEqual(c.warnreach, 80)
+        self.assertEqual(c.critreach, 60)
+
+    def test_clone(self):
+        ch = CheckNTPMon()
+        self.assertFalse(ch.is_silent())
+        c = CheckNTPMonSilent.clone(ch)
+        self.assertTrue(c.is_silent())
+        self.assertEqual(c.warnpeers, 2)
+        self.assertEqual(c.okpeers, 4)
+        self.assertEqual(c.warnoffset, 10)
+        self.assertEqual(c.critoffset, 50)
+        self.assertEqual(c.warnreach, 75)
+        self.assertEqual(c.critreach, 50)
+
+    def test_clone_non_default(self):
+        ch = CheckNTPMon(1, 2, 9, 49, 80, 60)
+        self.assertFalse(ch.is_silent())
+        c = CheckNTPMonSilent.clone(ch)
+        self.assertTrue(c.is_silent())
+        self.assertEqual(c.warnpeers, 1)
+        self.assertEqual(c.okpeers, 2)
+        self.assertEqual(c.warnoffset, 9)
+        self.assertEqual(c.critoffset, 49)
+        self.assertEqual(c.warnreach, 80)
+        self.assertEqual(c.critreach, 60)
+
+    def test_clone_silent(self):
+        """Cloning CheckNTPMonSilent should return itself"""
+        cs = CheckNTPMonSilent()
+        c = CheckNTPMonSilent.clone(cs)
+        self.assertEqual(c, cs)
+
+    def test_clone_inheritance(self):
+        """Cloning a child class should work too"""
+        class testobject(CheckNTPMonSilent):
+            def dump(self):
+                pass
+        obj = testobject()
+        self.assertEqual(obj.warnpeers, 2)
+        self.assertEqual(obj.okpeers, 4)
+        self.assertEqual(obj.warnoffset, 10)
+        self.assertEqual(obj.critoffset, 50)
+        self.assertEqual(obj.warnreach, 75)
+        self.assertEqual(obj.critreach, 50)
+        c = CheckNTPMonSilent.clone(obj)
+        self.assertEqual(c.warnpeers, 2)
+        self.assertEqual(c.okpeers, 4)
+        self.assertEqual(c.warnoffset, 10)
+        self.assertEqual(c.critoffset, 50)
+        self.assertEqual(c.warnreach, 75)
+        self.assertEqual(c.critreach, 50)
+
+    def test_clone_inheritance_non_silent(self):
+        """Cloning a non-silent child class should work too"""
+        class testobject(CheckNTPMon):
+            def dump(self):
+                pass
+        obj = testobject()
+        self.assertEqual(obj.warnpeers, 2)
+        self.assertEqual(obj.okpeers, 4)
+        self.assertEqual(obj.warnoffset, 10)
+        self.assertEqual(obj.critoffset, 50)
+        self.assertEqual(obj.warnreach, 75)
+        self.assertEqual(obj.critreach, 50)
+        c = CheckNTPMonSilent.clone(obj)
+        self.assertEqual(c.warnpeers, 2)
+        self.assertEqual(c.okpeers, 4)
+        self.assertEqual(c.warnoffset, 10)
+        self.assertEqual(c.critoffset, 50)
+        self.assertEqual(c.warnreach, 75)
+        self.assertEqual(c.critreach, 50)
+
+    def test_clone_inheritance_non_silent_made_silent(self):
+        """Don't try this at home, kids"""
+        class testobject(CheckNTPMon):
+            def is_silent(self):
+                return True
+        obj = testobject()
+        c = CheckNTPMonSilent.clone(obj)
+        self.assertEqual(c, obj)
+
+    def test_bad_clone(self):
+        """Cloning something that's not a CheckNTPMonSilent or CheckNTPMon should raise an AttributeError"""
+        class testobject(object):
+            pass
+        obj = testobject()
+        self.assertRaises(AttributeError, CheckNTPMonSilent.clone, obj)
+
+    def test_demos(self):
+        """Ensure that demo data is parsed successfully and doesn't produce exceptions or unknown results"""
+        for d in demodata:
+            ntp = NTPPeers(d.split("\n"))
+            ntp.dump()
+            methods = [ntp.check_offset, ntp.check_peers, ntp.check_reachability,
+                       ntp.check_sync, ntp.checks]
+            for method in methods:
+                ret = method()
+                self.assertIn(ret, [0, 1, 2],
+                    "Method %s returned invalid result parsing demo data:\n%s\nTry running with --show-demos." % (method, d))
+
+
+def demo():
+    """Duplicate of test_demos which shows full output"""
+    i = 0
+    for d in demodata:
+        print "Parsing demo data %d: %s" % (i, d)
+        ntp = NTPPeers(d.split("\n"))
+        i += 1
+        ntp.dump()
+        methods = [ntp.check_offset, ntp.check_peers, ntp.check_reachability,
+                   ntp.check_sync, ntp.checks]
+        for method in methods:
+            ret = method()
+            if ret not in [0, 1, 2]:
+                print "Method %s returned invalid result parsing demo data:\n%s" % (method, d)
+                sys.exit(3)
+
+
+if __name__ == "__main__":
+    # object to store parsed arguments
+    test_checkntpmon = CheckNTPMon()
+
+    # parse command line
+    parser = argparse.ArgumentParser(description='NTPmon test class')
+    parser.add_argument('--show-demos', action='store_true',
+        help='Show demo output.')
+    args = parser.parse_args(namespace=test_checkntpmon)
+    if test_checkntpmon.show_demos:
+        demo()
+    else:
+        unittest.main()
+