Merge lp:~michael.iatrou/maas/fix-1673378 into lp:~maas-committers/maas/trunk

[Michael Iatrou]

Add NTP connectivity test.

[Gavin Panella (allenap)]

I have a suggestion that the script should tidy up after itself, but otherwise this looks good.

review: Needs Fixing

[Michael Iatrou (michael.iatrou)]

Thank you Gavin! I have amended the patch per your feedback.

[Gavin Panella (allenap)]

Thanks.

I haven't tested this script. Have you had a chance?

review: Approve

[Ante Karamatić (ivoks)]

I'll test it right away

[Ante Karamatić (ivoks)]

I guess it needs a bit more intelligence. Test reports passed even if sync is not done:

...
16 Mar 14:44:42 ntpd[3744]: ntpd 4.2.8p4@1.3265-o Wed Oct 5 12:34:45 UTC 2016 (1): Starting
16 Mar 14:44:42 ntpd[3744]: Command line: ntpd -gq
16 Mar 14:44:42 ntpd[3744]: proto: precision = 0.046 usec (-24)
16 Mar 14:44:42 ntpd[3744]: Listen and drop on 0 v6wildcard [::]:123
16 Mar 14:44:42 ntpd[3744]: Listen and drop on 1 v4wildcard 0.0.0.0:123
16 Mar 14:44:42 ntpd[3744]: Listen normally on 2 lo 127.0.0.1:123
16 Mar 14:44:42 ntpd[3744]: Listen normally on 3 ens255f0 172.16.7.29:123
16 Mar 14:44:42 ntpd[3744]: Listen normally on 4 lo [::1]:123
16 Mar 14:44:42 ntpd[3744]: Listen normally on 5 ens255f0 [fe80::2e60:cff:fef9:6b6a%2]:123
16 Mar 14:44:42 ntpd[3744]: Listening on routing socket on fd #22 for interface updates
16 Mar 14:44:53 ntpd[3744]: ntpd: no servers found
...

When NTP is accessible, it works fine:

...
16 Mar 14:24:52 ntpd[3553]: ntpd 4.2.8p4@1.3265-o Wed Oct 5 12:34:45 UTC 2016 (1): Starting
16 Mar 14:24:52 ntpd[3553]: Command line: ntpd -gq
16 Mar 14:24:52 ntpd[3553]: proto: precision = 0.045 usec (-24)
16 Mar 14:24:52 ntpd[3553]: Listen and drop on 0 v6wildcard [::]:123
16 Mar 14:24:52 ntpd[3553]: Listen and drop on 1 v4wildcard 0.0.0.0:123
16 Mar 14:24:52 ntpd[3553]: Listen normally on 2 lo 127.0.0.1:123
16 Mar 14:24:52 ntpd[3553]: Listen normally on 3 ens255f0 172.16.7.29:123
16 Mar 14:24:52 ntpd[3553]: Listen normally on 4 lo [::1]:123
16 Mar 14:24:52 ntpd[3553]: Listen normally on 5 ens255f0 [fe80::2e60:cff:fef9:6b6a%2]:123
16 Mar 14:24:52 ntpd[3553]: Listening on routing socket on fd #22 for interface updates
16 Mar 14:24:53 ntpd[3553]: ntpd: time slew +0.001707 s
...

[Gavin Panella (allenap)]

Mmm, not sure ntpd is going to work at all, at least without a lot more work; see my comment.

review: Needs Fixing

[Michael Iatrou (michael.iatrou)]

> Mmm, not sure ntpd is going to work at all, at least without a lot more work;
> see my comment.

It seems that /var/lib/systemd/clock is not necessarily up to date:

$ timedatectl status
      Local time: Thu 2017-03-16 15:32:40 UTC
  Universal time: Thu 2017-03-16 15:32:40 UTC
        RTC time: n/a
       Time zone: Etc/UTC (UTC, +0000)
 Network time on: yes
NTP synchronized: yes
 RTC in local TZ: no
$ ll /var/lib/systemd/clock
-rw-r--r-- 1 systemd-timesync systemd-timesync 0 Mar 16 14:19 /var/lib/systemd/clock

[Ante Karamatić (ivoks)]

Gavin I see different results in my tests when I change NTP server in MAAS' settings (ntp.ubuntu.com works, nttypo.ubuntu.com doesn't). So, I guess NTP/ntp.conf does get updated?

[Ante Karamatić (ivoks)]

Failure:
sudo: unable to resolve host helped-swift
Reading package lists...
Building dependency tree...
Reading state information...
ntp is already the newest version (1:4.2.8p4+dfsg-3ubuntu5.3).
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
sudo: unable to resolve host helped-swift
sudo: unable to resolve host helped-swift
16 Mar 15:52:32 ntpd[3682]: ntpd 4.2.8p4@1.3265-o Wed Oct 5 12:34:45 UTC 2016 (1): Starting
16 Mar 15:52:32 ntpd[3682]: Command line: ntpd -gq
16 Mar 15:52:32 ntpd[3682]: proto: precision = 0.045 usec (-24)
16 Mar 15:52:32 ntpd[3682]: Listen and drop on 0 v6wildcard [::]:123
16 Mar 15:52:32 ntpd[3682]: Listen and drop on 1 v4wildcard 0.0.0.0:123
16 Mar 15:52:32 ntpd[3682]: Listen normally on 2 lo 127.0.0.1:123
16 Mar 15:52:32 ntpd[3682]: Listen normally on 3 ens255f0 172.16.7.29:123
16 Mar 15:52:32 ntpd[3682]: Listen normally on 4 lo [::1]:123
16 Mar 15:52:32 ntpd[3682]: Listen normally on 5 ens255f0 [fe80::2e60:cff:fef9:6b6a%2]:123
16 Mar 15:52:32 ntpd[3682]: Listening on routing socket on fd #22 for interface updates
16 Mar 15:52:42 ntpd[3682]: ntpd exiting on signal 15 (Terminated)
16 Mar 15:52:42 ntpd[3682]: 172.16.7.2 local addr 172.16.7.29 -> <null>

Success:
sudo: unable to resolve host helped-swift
Reading package lists...
Building dependency tree...
Reading state information...
ntp is already the newest version (1:4.2.8p4+dfsg-3ubuntu5.3).
0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
sudo: unable to resolve host helped-swift
sudo: unable to resolve host helped-swift
16 Mar 16:01:19 ntpd[3735]: ntpd 4.2.8p4@1.3265-o Wed Oct 5 12:34:45 UTC 2016 (1): Starting
16 Mar 16:01:19 ntpd[3735]: Command line: ntpd -gq
16 Mar 16:01:19 ntpd[3735]: proto: precision = 0.045 usec (-24)
16 Mar 16:01:19 ntpd[3735]: Listen and drop on 0 v6wildcard [::]:123
16 Mar 16:01:19 ntpd[3735]: Listen and drop on 1 v4wildcard 0.0.0.0:123
16 Mar 16:01:19 ntpd[3735]: Listen normally on 2 lo 127.0.0.1:123
16 Mar 16:01:19 ntpd[3735]: Listen normally on 3 ens255f0 172.16.7.29:123
16 Mar 16:01:19 ntpd[3735]: Listen normally on 4 lo [::1]:123
16 Mar 16:01:19 ntpd[3735]: Listen normally on 5 ens255f0 [fe80::2e60:cff:fef9:6b6a%2]:123
16 Mar 16:01:19 ntpd[3735]: Listening on routing socket on fd #22 for interface updates
16 Mar 16:01:20 ntpd[3735]: ntpd: time slew +0.009739 s
ntpd: time slew +0.009739s
sudo: unable to resolve host helped-swift
Reading package lists...
Building dependency tree...
Reading state information...
The following package was automatically installed and is no longer required:
libopts25
Use 'sudo apt autoremove' to remove it.
The following packages will be REMOVED:
ntp*
0 upgraded, 0 newly installed, 1 to remove and 3 not upgraded.
After this operation, 1,620 kB disk space will be freed.
(Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Read...

[Ante Karamatić (ivoks)]

Looks like ntp is already installed?

[Andres Rodriguez (andreserl)]

fwiw, in the commissioning environment we send cloud-init metadata for NTP configuration. This gets configured before anywhere else runs. As such, there is NTP configured and running in the ephemeral environment

[Gavin Panella (allenap)]

Just had a conversation with the team: ntp is already installed, as you've discovered Ante. MAAS instructs cloud-init to install and configure ntp. So, sorry, I've been leading you on a long path round the problem.

So, given that ntp is already installed and running, it might be possible to reduce the script to a single command:

  ntpq -np

which prints the active peers. For my local machine this prints:

       remote refid st t when poll reach delay offset jitter
  ==============================================================================
  *192.168.178.1 141.82.25.202 3 u 290 1024 377 2.379 0.483 0.793

[Ante Karamatić (ivoks)]

While that's good info to have, with broken NTP servers it also returns
results:

root@helium:~# ntpq -np
     remote refid st t when poll reach delay offset
 jitter
==============================================================================
 typo.ubuntu.com .POOL. 16 p - 64 0 0.000 0.000
0.000
root@helium:~# echo $?
0

On Thu, Mar 16, 2017 at 5:18 PM Gavin Panella <email address hidden>
wrote:

> Just had a conversation with the team: ntp is already installed, as you've
> discovered Ante. MAAS instructs cloud-init to install and configure ntp.
> So, sorry, I've been leading you on a long path round the problem.
>
> So, given that ntp is already installed and running, it might be possible
> to reduce the script to a single command:
>
> ntpq -np
>
> which prints the active peers. For my local machine this prints:
>
> remote refid st t when poll reach delay offset
> jitter
>
> ==============================================================================
> *192.168.178.1 141.82.25.202 3 u 290 1024 377 2.379 0.483
> 0.793
>
> --
> https://code.launchpad.net/~michael.iatrou/maas/fix-1673378/+merge/320037
> You are requested to review the proposed merge of
> lp:~michael.iatrou/maas/fix-1673378 into lp:maas.
>
--
Ante Karamatić
<email address hidden>
Canonical

[Gavin Panella (allenap)]

The ntpstat command (from the ntpstat) package might do what you need.
From ntpstat(1):

  DESCRIPTION
       ntpstat will report the synchronisation state of the NTP daemon
       running on the local machine. If the local system is found to be
       synchronised to a reference time source, ntpstat will report the
       approximate time accuracy.

  RETURNS
       ntpstat returns 0 if clock is synchronised. ntpstat returns 1 if
       clock is not synchronised. ntpstat returns 2 if clock state is
       indeterminant, for example if ntpd is not contactable.

The only downsides are that it's not installed by default and it's in
universe.

Running locally:

  $ ntpstat
  synchronised to NTP server (192.168.178.1) at stratum 4
     time correct to within 78 ms
     polling server every 1024 s

[Ante Karamatić (ivoks)]

I approve with latest changes which I tested with correct and incorrect NTPs.

review: Approve

[Gavin Panella (allenap)]

I think I've spotted a bug in the script. I've made too many mistakes today to trust myself right now, but please check it out.

review: Abstain

[Lee Trager (ltrager)]

LGTM, thanks for adding the NTP test.

review: Approve

[Paul Gear (paulgear)]

Or you could use the Nagios check (or a variant on it) from the NTP charm: https://git.launchpad.net/ntpmon/tree/src/check_ntpmon.py

=== modified file 'src/metadataserver/builtin_scripts/__init__.py'
--- src/metadataserver/builtin_scripts/__init__.py	2017-03-10 16:50:30 +0000
+++ src/metadataserver/builtin_scripts/__init__.py	2017-03-16 23:39:45 +0000
@@ -124,6 +124,14 @@
         timeout=60 * 60 * 12 + 60 * 5,
         filename='stress_ng_memory_long.sh',
         ),
+    BuiltinScript(
+        name='ntp',
+        title='NTP validation',
+        description='Run ntp clock set to verify NTP connectivity.',
+        tags=['network', 'internet'],
+        timeout=60,
+        filename='ntp.sh',
+        ),
 ]
 
 

=== added file 'src/metadataserver/builtin_scripts/ntp.sh'
--- src/metadataserver/builtin_scripts/ntp.sh	1970-01-01 00:00:00 +0000
+++ src/metadataserver/builtin_scripts/ntp.sh	2017-03-16 23:39:45 +0000
@@ -0,0 +1,27 @@
+#!/bin/sh -e
+#
+# ntp - Run ntp clock set to verify NTP connectivity.
+#
+# Author: Michael Iatrou <michael.iatrou (at) canonical.com>
+#
+# Copyright (C) 2017 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as
+# published by the Free Software Foundation, either version 3 of the
+# License, or (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+e=0
+ntpq -np
+sudo systemctl stop ntp.service
+sudo timeout 10 ntpd -gq || e=$?
+sudo systemctl start ntp.service
+exit $e