ntp brought up before network is ready; fails not resolve any ip or host names; ntp does not recover

I edited the bug report to add the daemon.log.txt as an attachment instead of being in the bug report to make it more readable.

Same issue here.. I'm using Ubuntu 7.10

See attached daemon.log snippet

Yes, I can replicate this exactly - same issue - please refer:

https://answers.launchpad.net/ubuntu/+question/13064

It seems DHCP is the issue. Here is more info from my system logs:

Sep 9 13:00:58 palantir avahi-daemon[5314]: No service found in /etc/avahi/services.
Sep 9 13:00:58 palantir avahi-daemon[5314]: Joining mDNS multicast group on interface eth0.IPv4 with address 192.168.1.30.
Sep 9 13:00:58 palantir avahi-daemon[5314]: New relevant interface eth0.IPv4 for mDNS.
Sep 9 13:00:58 palantir avahi-daemon[5314]: Network interface enumeration completed.
Sep 9 13:00:58 palantir avahi-daemon[5314]: Registering new address record for fe80::21c:23ff:fe92:b82e on eth0.*.
Sep 9 13:00:58 palantir avahi-daemon[5314]: Registering new address record for 192.168.1.30 on eth0.IPv4.
Sep 9 13:00:58 palantir avahi-daemon[5314]: Registering HINFO record with values 'I686'/'LINUX'.
Sep 9 13:00:58 palantir NetworkManager: <information>^Inm_device_init(): device's worker thread started, continuing.
Sep 9 13:00:58 palantir NetworkManager: <information>^INow managing wired Ethernet (802.3) device 'eth0'.
Sep 9 13:00:58 palantir NetworkManager: <information>^IDeactivating device eth0.
Sep 9 13:00:58 palantir avahi-daemon[5314]: Withdrawing address record for 192.168.1.30 on eth0.
Sep 9 13:00:58 palantir avahi-daemon[5314]: Leaving mDNS multicast group on interface eth0.IPv4 with address 192.168.1.30.
Sep 9 13:00:58 palantir avahi-daemon[5314]: Interface eth0.IPv4 no longer relevant for mDNS.
Sep 9 13:00:58 palantir avahi-daemon[5314]: Withdrawing address record for fe80::21c:23ff:fe92:b82e on eth0.
Sep 9 13:00:58 palantir NetworkManager: <information>^IWill activate wired connection 'eth0' because it now has a link.

...
...

and then later:

Sep 9 13:01:00 palantir ntpd[5676]: ntpd 4.2.2p4@1.1585-o Wed Mar 7 20:43:30 UTC 2007 (1)
Sep 9 13:01:00 palantir ntpd[5677]: precision = 1.000 usec
Sep 9 13:01:00 palantir ntpd[5677]: Listening on interface wildcard, 0.0.0.0#123 Disabled
Sep 9 13:01:00 palantir ntpd[5677]: Listening on interface wildcard, ::#123 Disabled
Sep 9 13:01:00 palantir ntpd[5677]: Listening on interface lo, ::1#123 Enabled
Sep 9 13:01:00 palantir ntpd[5677]: Listening on interface lo, 127.0.0.1#123 Enabled
Sep 9 13:01:00 palantir ntpd[5677]: kernel time sync status 0040
Sep 9 13:01:00 palantir ntpd[5677]: frequency initialized 5.764 PPM from /var/lib/ntp/ntp.drift
Sep 9 13:01:00 palantir NetworkManager: <information>^IDHCP daemon state is now 1 (starting) for interface eth0
Sep 9 13:01:01 palantir ntpd[5677]: sendto(192.168.1.20) (fd=16): Network is unreachable
Sep 9 13:01:02 palantir dhclient: DHCPREQUEST on eth0 to 255.255.255.255 port 67
Sep 9 13:01:02 palantir dhclient: DHCPACK from 192.168.1.23
Sep 9 13:01:02 palantir avahi-daemon[5314]: Joining mDNS multicast group on interface eth0.IPv4 with address 192.168.1.30.
Sep 9 13:01:02 palantir avahi-daemon[5314]: New relevant interface eth0.IPv4 for mDNS.
Sep 9 13:01:02 palantir avahi-daemon[5314]: Registering new address record for 192.168.1.30 on eth0.IPv4.
Sep 9 13:01:02 palantir NetworkManager: <information>^IDHCP daemon state is now 4 (reboot) for interf...

I 'fixed' this by moving "/etc/dhcp3/dhclient-enter-hooks.d/ntp" to "/etc/dhcp3/dhclient-exit-hooks.d/ntp".

Before the move a test PING of the new NTP servers failed with "ping: sendmsg: Network is unreachable".
This makes sends because the -enter- hooks are run before the DHCP stage (in this case, $reason=REBOOT) is complete, i.e. before the interface has finished configuring itself.

After the move, the test ping succeeded and ntp was happy.
I assume this is because the -exit- hooks are run after the DHCP stage has completed, i.e. after the interface is up and configured.

Assuming this is the Right Thing to do, the ntpdate hook script should probably be moved too.

Thomas:

Confirmed, yep, that works perfectly - and without moving ntpdate, this works fine too.

Nick

I 'fixed' this by moving "/etc/dhcp3/dhclient-enter-hooks.d/ntp" to "/etc/dhcp3/dhclient-exit-hooks.d/ntp".

This did not work for me in Gutsy.

@feisty_hot_lover

I modified the ntp hook script to add some debugging (see attached).

With the hook script in "-enter-" it was clear from the logs that the interface wasn't ready when NTP needed it.

Could you try this version of the hook script and post the log files (/tmp/dhclient-script.*.debug) ?

I tried your debug script in both enter & exit and the result was the same.

ntp(REBOOT): No servers
ntp(REBOOT): Nothing to unconfigure

I also got a couple of log entries I have not seen priviously with 7.04.

Oct 3 18:45:45 ubuntu ntpd[5639]: bind() fd 19, family 10, port 123, scope 2, addr fe80::205:5dff:fe53:2ba5, in6_is_addr_multicast=0 flags=17 fails: Cannot assign requested address

Oct 3 18:45:45 ubuntu ntpd[5639]: unable to create socket on eth1 (3) for fe80::205:5dff:fe53:2ba5#123

"No Servers" means that the DHCP server didn't pass on any NTP servers and/or DHCP client doesn't care about them.

Check your lease (/var/lib/dhcp3/dhclient.eth0.leases), does it mention "option ntp-servers ....." ?

Check your dhclient configuration file, does it ask for NTP servers (by default it doesn't, see bug #74164)?

sprinkmeier@serenity:~$ diff --unified /etc/dhcp3/dhclient.conf__ORIG /etc/dhcp3/dhclient.conf
--- /etc/dhcp3/dhclient.conf__ORIG 2007-09-07 09:23:22.000000000 +0930
+++ /etc/dhcp3/dhclient.conf 2007-09-11 11:30:17.000000000 +0930
@@ -16,9 +16,17 @@
 #send dhcp-lease-time 3600;
 #supersede domain-name "fugue.com home.vix.com";
 #prepend domain-name-servers 127.0.0.1;
+# ask for NTP servers and the DNS search path
 request subnet-mask, broadcast-address, time-offset, routers,
        domain-name, domain-name-servers, host-name,
- netbios-name-servers, netbios-scope;
+ netbios-name-servers, netbios-scope,
+ ntp-servers,;
+
 #require subnet-mask, domain-name-servers;
 timeout 30;
 #retry 60;

The bind error seems to be IPv6 related. Are you using IPv6 or plain old IPv4?
What does "ifconfig" say?

The lease file and the dhclient file did not contain any entries for ntp. I use IPv4 and ifconfig says:

eth1 Link encap:Ethernet HWaddr 00:05:5D:53:2B:A5
          inet addr:192.168.1.100 Bcast:192.168.1.255 Mask:255.255.255.0
          inet6 addr: fe80::205:5dff:fe53:2ba5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:19498 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12212 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:28280531 (26.9 MB) TX bytes:971217 (948.4 KB)
          Interrupt:19 Base address:0x6800

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:16436 Metric:1
          RX packets:46 errors:0 dropped:0 overruns:0 frame:0
          TX packets:46 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2300 (2.2 KB) TX bytes:2300 (2.2 KB)

"inet addr:192.168.1.100"
As expected. You're using an ADSL gateway as DHCP server?

"HWaddr 00:05:5D:53:2B:A5" and "inet6 addr: fe80::205:5dff:fe53:2ba5"
No IPv6 configuration (the "fe80::<MAC>" address is the address IPv6 assigns when it has no address, sort of like 169.254/24 in IPv4 (see RFC3330))

No errors or anything else amiss....

Add "ntp-servers" to the dhclinet configuration file and re-acquire the lease ("/etc/init.d/network restart").
If the lease file still doesn't mention NTP then double check the DHCP server configuration.
If it does check the -exit-hook logs and ntp status ("ntpq -p") again.

"As expected. You're using an ADSL gateway as DHCP server?"

Actually I use a cable modem and a router. Anyway I tried to add ntp-servers to dhclient but doing research I discovered this only works if the dhcp server is configured to give out ntp server addresses. Since I have no dhcp servers that give out ntp addresses it will not do any good to add ntp to dhclient.

The problems that remains is that ntp is being brought up and attempting to do it's thing before networking is brought up. We need someone to check if this problem exist with static networking and if it does then the fault lays with ntp simply being too early and the package maintainer will need to set ntp to come up after networking. However if ntp works just fine with static networking then the fault may lay with the dhcp system. The dhcp I believe is slower to bring up then static because all that stuff with acquiring ip addresses and such but I could be wrong.

I would like to add that I tested ntp with a static network setup. NTP works perfectly with a static network setup. Since I need ntp I will be using a static network config for now but I would prefer that ntp work with dhcp.

Does it work if you comment out the line "exit 0" in /etc/network/if-up.d/ntp?

To the maintainer(s): Why is this line in there by default? Would it hurt to restart ntp whenever a network interface comes up, and let users disable this if they want, instead of doing it the other way around?

@feisty,

If your DHCP server isn't providing NTP servers then DHCP should have nothing to do with NTP.
"/etc/rcS.d/S40networking" should have the NIC's up and configured long before "/etc/rc[2345].d/S23ntp" tries to start NTP.

Does you modem do 'lazy connects'?
I'm wondering if the DNS queries from NTP are the first thing the modem hears, causing it to connect.
If the modem takes too long to connect then the DNS queries will fail and NTP won't start.
Could you add a few IP addresses (rather than FQDN's) to your ntp.conf file?
What does "ntpq -p" say?

@Phillipp
"Would it hurt to restart ntp whenever a network interface comes up?"
Yes, but not much.
NTP builds up information on remote servers and local clock drift. This is lost on restart, meaning NTP won't work as well as it could (i.e. it'll take longer to sync, and even longer to tweak the local clock frequency).
Arguably this is not a big problem for most situations where all you want is accuracy within a few seconds (in which case "ntpdate" may be a better choice anyway).

The polling interval NTP uses tends to increase as time goes on (it starts at 2^6 seconds (roughly once a minute), I've seen it as high as 2^16 (roughly every 18 hours)).
Re-starting NTP will mean a shorter polling interval and more load on the servers (especially if you use "iburst").

I think the thinking behind the "exit 0" in "/etc/network/if-up.d/ntp" is that, if your network address changes, it was probably DHCP that changed it and DHCP should be responsible for re-starting NTP.

Thomas Sprinkmeier wrote:
> @feisty,
>
> If your DHCP server isn't providing NTP servers then DHCP should have nothing to do with NTP.
> "/etc/rcS.d/S40networking" should have the NIC's up and configured long before "/etc/rc[2345].d/S23ntp" tries to start NTP.
I see the bug on feisty with a fast local network and local dhcp server.

>
> Does you modem do 'lazy connects'?
> I'm wondering if the DNS queries from NTP are the first thing the modem hears, causing it to connect.
> If the modem takes too long to connect then the DNS queries will fail and NTP won't start.
> Could you add a few IP addresses (rather than FQDN's) to your ntp.conf file?
> What does "ntpq -p" say?
>
> @Phillipp
> "Would it hurt to restart ntp whenever a network interface comes up?"
> Yes, but not much.
> NTP builds up information on remote servers and local clock drift. This is lost on restart, meaning NTP won't work as well as it could (i.e. it'll take longer to sync, and even longer to tweak the local clock frequency).
> Arguably this is not a big problem for most situations where all you want is accuracy within a few seconds (in which case "ntpdate" may be a better choice anyway).

I don't think this is a problem: the local clock drift information is
stored in /var/lib/ntp/ntp.drift, so it is not lost upon restart. The
information about servers builds up fast, and in the meantime, ntp's
application of the stored drift info to the local clock keeps it
accurate to a fraction of a second.

As of 2005, the advice was that ntp does need to be restarted:
http://lists.ntp.isc.org/pipermail/questions/2005-August/006333.html

As of a week ago, it seems that the underlying ntp problem may have been
fixed: https://support.ntp.org/bugs/show_bug.cgi?id=51
(see the bottom of that bug report).

Eric

>
> The polling interval NTP uses tends to increase as time goes on (it starts at 2^6 seconds (roughly once a minute), I've seen it as high as 2^16 (roughly every 18 hours)).
> Re-starting NTP will mean a shorter polling interval and more load on the servers (especially if you use "iburst").
>
> I think the thinking behind the "exit 0" in "/etc/network/if-up.d/ntp"
> is that, if your network address changes, it was probably DHCP that
> changed it and DHCP should be responsible for re-starting NTP.
>

@Eric,

"I see the bug on feisty with a fast local network and local dhcp server."

Did you move "/etc/dhcp3/dhclient-enter-hooks.d/ntp" to "/etc/dhcp3/dhclient-exit-hooks.d/ntp" ?

My cable modem does not disconnect and I should not really call it a modem I suppose. My internet service is always up except for network outages which are pretty rare. The other possibility is that certain brands of routers are somehow broken or slow with regards to dhcp. My router is a Linksys BefSR41 version 3.0.

I tested again with dhcp using ip numbers and no go. I attached my daemon log output.

ntpq -p
No association ID's returned

Thomas Sprinkmeier wrote:
> @Eric,
>
> "I see the bug on feisty with a fast local network and local dhcp
> server."
>
> Did you move "/etc/dhcp3/dhclient-enter-hooks.d/ntp" to "/etc/dhcp3
> /dhclient-exit-hooks.d/ntp" ?
>

I hadn't when I wrote the message, so I did it right now, rebooted, and
ntp came up OK--I think this is the first time it has done so on this
installation, so it looks like it solves the problem, at least based on
this first trial.

Eric

I may be having a different problem than other posters on this bug report. My internet connection is a WPA-secured wireless connection. The interface is brought up at boot time, and NTP is started. However, the wireless card does not connect to the router before I log in and enter my keyring password (which unlocks the WPA password). At that time, ntpd has already figured out that it cannot connect to any of the time servers:

 Oct 7 14:17:43 localhost ntpd_initres[5377]: couldn't resolve `ntp.ubuntu.com', giving up on it

Now, when the wireless interface connects to the router and gets configured via DHCP, NTP does not do anything anymore.

I guess that most people using laptops with WPA-secured wireless networks suffer from the same problem. I believe the default configuration should work with this kind of setup (as it appears to be increasingly common).

NTP needs to have some kind of system implemented where if there is no connection to the internet then it lays in wait until a connection appears. One problem to consider is that ntp accuracy for intermittent connections such as on a laptop/notebook is way lower than it would be on a long term stable connection. When you have intermittent connections the ntpdate program is what you need.

I just now realized that in Gutsy 7.10 the networking is setup in roaming mode by default so I did some further testing.

Here is what works and what does not work in network configuration.

1. SUCCESS = /etc/dhcp3/dhclient-enter-hooks.d/ntp, roaming mode off, DHCP selected manually from drop down box, ip and or FQDN.

2. FAILURE = Roaming mode selected.

3. SUCCESS = Static networking.

It appears that selecting roaming mode is the root cause of the problem in Gutsy 7.10. Go into networking and untick the roaming mode and select DHCP from the drop down box and see if that works for you. I can't test Feisty as I have moved onto Gutsy 7.10. If someone knows how to turn off roaming in Feisty 7.04 please post that method here so that others can test the roaming theory.

Guys and Gals,

I updated to Gutsy today, and got the very same issue with ntpd.

WHY does ntpd need to rely on dhcp broadcasts or whatever??? It DOESN'T.

All that it needs is a network connection via the systems eth0 (or whatever).

The issue is somehow ntpd gets started before any network interface is UP and that fubars it. ntpd will find the route once it is up.

Nick

I'm just finishing a brand new Gutsy install.
I added NTP, moved the hook files:

root@laptop:/etc/dhcp3# mv dhclient-enter-hooks.d/*ntp dhclient-exit-hooks.d

I added "ntp-servers" to the dhclient.conf file (see bug #74164)

Now NTP starts as expected when the system boots. I haven't tried any post-boot network reconfigurations yet, but at least it works in the simple case.

ARGH!!!

Just rebooting to test again, and now DHCP isn't starting properly.

I switched the wireless from WEP to WPA, not sure if that's related or not.

Anyway, "sudo /etc/init.d/networking restart" after booting brings up DHCP and NTP just fine.

I edited the wlan0 section of "/etc/network/interfaces" to add a "pre-up" line:

iface wlan0 inet dhcp
wpa-psk ...
wpa-driver wext
wpa-key-mgmt WPA-PSK
wpa-proto WPA
wpa-ssid ...
pre-up /sbin/ifdown wlan0

That fixed the problem with the wireless not coming up on boot. I think it's WPA related, but I can't find anything matching in launchpad.

That, moving the hook script and requesting NTP servers, and everything seems to be working.

I'm having this problem in gutsy.
But...I've noticed an option to pass to the server entries in ntp.conf that might work.....

From the ntp documentation it's describe as follows.....

dynamic
    Allows a server/peer to be configured even if it is not reachable at configuration time. It is assumed that at some point in the future the network environment changes so that this server/peer can be reached. This option is useful to configure servers/peers on mobile systems with intermittent network access (e.g. wlan clients).

Will try enabling, and report back.

The 'dynamic' option didn't work for me.

Appears not to manage lack of DNS.
It is the lack of a valid DNS that causes NTP to fail....
And NTP isn't yet designed to cope with DNS changes/missing DNS.

Apparently there has been some discussion of making DNS queries dynamic on one of the mailing lists....
But I'm not clear of the conclusions.

I suspect that NTP in dynamic mode, together with Proxy DNS (http://packages.ubuntu.com/gutsy/net/pdnsd) should work.

Earlier I said that if you turn off roaming mode then ntp works. Well thats partially true. In plain vanilla DHCP with roaming mode off the ntp interface works but if I do a reboot ntp comes up as not working, but do another reboot and ntp works for me. So it goes like this, if you reboot then no ntp but reboot again ntp works.

1. If NTP is working when you reboot then it will not work.

2. If NTP working when you reboot then it will not work but if you reboot again it will work.

3. NTP=working=2 reboots in a row=working

4.NTP=working=1 reboot=not working

So basically when my ntp is working and I reboot it will not work so I reboot again to make it work cause I probably just too lazy to do /etc/init.d/ntp restart

As others, I confirm for Gutsy 7.10...

I do not see a consensus in the preceding comments regarding a "for-certain" solution. It is absolutely a situation that needs to be resolved, and, boy, would I like to see a documented change here that does.

heindsight, in http://ubuntuforums.org/showthread.php?t=647559&page=2 said:
"The problem is that the boot sequence goes something like this:

    * S12dbus: Starts dbus
          o NetworkManager is started and starts trying to bring up network interfaces
            Initially, it fails for interfaces using dhcp, since dhcdbd is not yet running.
    * S23ntp:Starts ntpd
      ntpd tries to associate with servers, but fails because network interfaces are not yet up.
    * S24dhcdbd: Starts dhcdbd
      NetworkManager can finally bring up interfaces that use dhcp
    * S40networking: Runs scripts in /etc/network/if-up.d
          o ntpdate tries to set the time, but fails because ntpd is already running."

I do not know if that is accurate, but it sure seems to be.

The bottom line is that the sequencing is messed up. It needs to be this (in my opinion, what the final fix should do):

 a. ethernet connection is up FOR GOOD (not to be stopped, or changed, or re-dhcp'ed, or restarted subsequently during the boot)

 b. ntpdate (if installed) runs, sets the time, completely exits, and releases all ports or other resources. Also, it would be EXTREMELY nice to have the option of running 'hwclock --systohc' after ntpdate, to write the time into the hw clock (If ntpd later tries to start, and hw clock was more than 1000sec off actual time, ntpd will fail to start--with notice appearing only in the syslog).

 c. ntpd (if installed) is started.

The ntpdate (ntpdate-debian) and ntp scripts should test for the necessary states to exist for them to properly run, and if they do not exist, schedule a retry when the conditions are met.

greetings,

As J Carerra mentioned, I was also having some problems with ntpd on Gutsy. I managed to solve my problems though, so I'm posting the solution
here in the hope that it might help someone else too.

My original analysis of the problem I was having with ntpd (which J Carerra has quoted here) was not quite correct (after hours of digging through init scripts I got slightly confused about which scripts run from rcS.d and which from rc2.d). The DHCP server in the building where I live doesn't provide NTP
servers, so I can't comment on what happens when your DHCP server does provide NTP servers.

I basically had two separate problems that were preventing ntpd and ntpdate from working properly:
* With NetworkManager managing my ethernet connection, ntpd was being started before eth0 was up, so ntpd couldn't associate with any other
   ntp servers. Then, when eth0 did come up, ntpdate couldn't run, because ntpd was already running.
* When I stopped NetworkManager from managing my ethernet connection by adding an entry for eth0 in /etc/network/interfaces, so that eth0 would be
   brought up by rcS.d/S40networking before ntpd was started by rc2.d/S23ntp everything worked most of the time. However /etc/network/if-up.d/ntpdate
   was running ntpdate in the background, which meant that every once in a while ntpdate would still be running by the time rc2.d/S23ntp tried to start
   ntpd, thus preventing ntpd from starting.

I managed to solve my ntp problems by writing a new script (which I'm attaching here) to replace /etc/network/if-up.d/ntpdate. This script works for me regardless of whether NetworkManager is managing my internet connection. If you want to try this solution, do the following:
* Leave the symlinks to /etc/init.d/ntp intact (the script won't start ntpd if it is disabled in the current runlevel)
* Deactivate (remove executable permissions from) /etc/network/if-up.d/ntpdate
* Put the new ntpd script at /etc/network/if-up.d/ntpd (and give it proper ownership and executable permissions)
* Enjoy (hopefully)

For every network interface that is brought up (except the loopback interface), this script:
* Stops ntpd (if it was running already)
* Runs /usr/bin/ntpdate-debian and
   - if ntpdate successfully sets the clock, also runs hwclock --systohc
* Starts ntpd again (but only if it is configured to start in the current runlevel)

Be warned though, that this script may cause misleading log entries (particularly if NetworkManager is not managing your internet connection) claiming that ntpd failed to start when it actually did start. Unfortunately I don't really see a way of preventing this from happening.

BUG CONFIRMED
A WORKAROUND FOUND

heindsight's method worked for me, with one additional change that he also suggested at
http://ubuntuforums.org/showthread.php?t=647559&page=5 ...

specifically, changing S24dhcdbd to S13dhcdbd in /etc/rc[2,3,4,5].d (I used "sysv-rc-conf -p ' by the way) so NetworkManager gets eth0 up faster .

On my computer, that eliminated eth0 trying and failing to start about seven times before succeeding.

With that change to dhcdbd startup priority and the previously suggested change, I end up with:

- eth0 starts one time
- there is an extraneous ntpd start that fails to get associations
- heindsight's script stops ntp, runs ntpdate, then restarts ntp, successfully

Other than the extraneous ntpd "too early" start, this is exactly the sequence desired, so I consider my system fixed.

This, or an equivalent change accomplishing the same thing, needs to be implemented by the devs ASAP.

BTW, what does "Declined for Gutsy by Henrik Nilsen Omma " mean?

Does that mean the developers have ruled out fixing it?

Ubuntu hardy ntpd doesn't work fine.
It can't synchronize the server.
It is look like work fine, but doesn't use ntp server.

     remote refid st t when poll reach delay offset jitter
==============================================================================
 europium.canoni 192.36.133.17 2 u 2 64 1 35.144 15.201 0.001
 rubicon.ki.iif. 32.47.27.45 2 u 1 64 1 1.957 25.593 0.001
 metronoom.dmz.c PPS(0) 2 u - 64 1 33.797 39.092 0.001

Jan 18 06:03:45 kub-kal ntpd[5341]: synchronized to 91.189.94.4, stratum 2
Jan 18 06:04:50 kub-kal ntpd[5341]: time reset +456.480996 s
Jan 18 06:23:16 kub-kal ntpd[5341]: synchronized to 91.189.94.4, stratum 2
Jan 18 06:32:55 kub-kal ntpd[5341]: no servers reachable

Jan 16 10:04:10 kub-kal ntpd[5341]: sendto(91.189.94.4) (fd=24): Invalid argument
Jan 16 10:05:13 kub-kal ntpd[5341]: sendto(91.189.94.4) (fd=24): Invalid argument
Jan 16 10:06:18 kub-kal ntpd[5341]: sendto(91.189.94.4) (fd=24): Invalid argument
Jan 16 10:06:35 kub-kal ntpd[5341]: Deleting interface #8 ppp0, 61.161.241.160#123, interface stats: received=660, sent=663, dropped=3, active_time=44701 secs
Jan 16 10:08:14 kub-kal ntpdate[15475]: can't find host ntp.ubuntu.com
Jan 16 10:08:14 kub-kal ntpdate[15475]: no servers can be used, exiting
Jan 16 10:08:14 kub-kal ntpdate[15534]: can't find host ntp.ubuntu.com
Jan 16 10:08:14 kub-kal ntpdate[15534]: no servers can be used, exiting
Jan 16 10:08:14 kub-kal ntpdate[15595]: can't find host ntp.ubuntu.com
Jan 16 10:08:14 kub-kal ntpdate[15595]: no servers can be used, exiting
Jan 16 10:10:34 kub-kal ntpd[5341]: no servers reachable

A note to all:

updates released sometime shortly before 1-27-08, when I installed them in Gutsy, REVERTED the change of S24dhcdbd to S13dhcdbd start priority in rc5.d back to S24 !!
(I used "sysv-rc-conf -p ' tool to make the previous changes by the way)

This reversion caused the previous behavior of multiple attempts to start eth0 to return. If you installed these updates, you should check your system and ensure none of the changes you previously made to correct this bug were not undone like mine were.

I do not know about effects in any systems other than Gutsy. Wish these updates had fixed the Gutsy bug, but I see at the top of this bug report that it is "declined" for Gutsy.

This patch stops ntp before running ntpdate, then starts it again afterwards.

This bug was fixed in the package ntp - 1:4.2.4p4+dfsg-3ubuntu2

---------------
ntp (1:4.2.4p4+dfsg-3ubuntu2) hardy; urgency=low

  * Stop ntp before running ntpdate when an interface
    comes up, then start again afterwards (LP: #114505)

 -- Onno Benschop <email address hidden> Thu, 6 Mar 2008 14:00:42 +0900

This IS NOT fixed. Using 4.2.4p4+dfsg-3ubuntu2 on 8.04.1 I have 3 machines doing this and ALL failing.

The NTP server comes up & down like a yoyo during system boot - which is laughable.
The when ntpd is executed from init.d to bring the services up as expected it fails becuase the socket isnt available.
ntpq -p after login states 'no association IDs'.

ifdown --all
ifup --all

fixes - as does /etc/init.d/ntp restart

The machines that you are having issues with, were they fresh hardy installs, upgrades, or a mixture?

How are their network interfaces managed?

Are they running Ubuntu, or Ubuntu-server?

In re-reading this bug-report, I can see there are many issues that appear to affect this bug and the NTP server restarting several times during boot appears to be related to the several ways in which it's started.

While not ideal by any stretch of the imagination, the aim was to get to a point after boot at which time ntpdate had succesfully run and NTP - if installed - was running.

From your report, this does not appear to be the case.

Hi,

All fresh installs. ntp installed POST installation via apt-get.
Ubuntu Server 8.04.1 & Kubuntu Desktop 8.04.1 KDE4 Remix.
Mixture of DHCP and static IP.
 - As an example the desktop I am email from had 4 network interfaces. 2
static IP and 2 DHCP all specified under /etc/network/interfaces. 2
static both cabled, gateway specified on one of them. 2 marked as DHCP
not connected. Only eth0 and eth1 marked as auto. This typical of 5
server installs and 3 desktop installs which are all have this issue.

The machine it *does* work is a laptop with eth0 (DHCP not connected)
and ath0 (WPA wireless managed by knetworkmanager). This seems to always
come up OK. But that is probably because the wireless network comes up
under ifup post-login and the ifup scripts are probably doing their job.
I cannot remember if it works properly under a cabled eth0 with DHCP as
I always use the wireless on the laptop.

Hope this helps,
Barry

Onno Benschop wrote:
> The machines that you are having issues with, were they fresh hardy
> installs, upgrades, or a mixture?
>
> How are their network interfaces managed?
>
> Are they running Ubuntu, or Ubuntu-server?
>
> In re-reading this bug-report, I can see there are many issues that
> appear to affect this bug and the NTP server restarting several times
> during boot appears to be related to the several ways in which it's
> started.
>
> While not ideal by any stretch of the imagination, the aim was to get to
> a point after boot at which time ntpdate had succesfully run and NTP -
> if installed - was running.
>
> >From your report, this does not appear to be the case.
>
> ** Changed in: ntp (Ubuntu Hardy)
> Status: New => Incomplete
>
>

--
Sincerely Yours

Copyright Witness Net Support
<email address hidden>

www.copyrightwitness.com
Registration centre for copyright works.

This e-mail and any attachments are confidential and intended for the addressee only.
The information in this mail does not amount to legal advice or opinion. Any views or legal references are those of the author and are based on personal opinion or understanding only.

I just finished a fresh U8.10-rc install.
Plain vanilla, all I added was the NTP package.

    sprinkmeier@u810:~$ ls -l /etc/rc5.d/S2[38]*
    lrwxrwxrwx 1 root root 13 Oct 26 14:18 /etc/rc5.d/S23ntp -> ../init.d/ntp
    lrwxrwxrwx 1 root root 24 Oct 26 10:10 /etc/rc5.d/S28NetworkManager -> ../init.d/NetworkManager

NTP starts before Network Manager, meaning that NTP won't have DNS available at startup.
It tends to work, but moving NTP's startup after Network Managers should make it more reliable. It will also allow NTPDATE to run (it's called from the ip-up scripts, but will fail as NTPD will already own the socket).

    sprinkmeier@u810:~$ grep ntp-server /etc/dhcp3/dhclient.conf
    sprinkmeier@u810:~$

NTP servers still aren't requested by default. If whoever set up your DHCP server knows enought to advertise NTP server(s) then chances are you should use them (or at the very least PREPEND them).

Hi,

I disagree with this diagnosis..

S28NetworkManager is a red-herring. For a start Hardy doesnt have
this...Looking at the startup for Hardy:

S15bind9
S16openvpn
S16ssh
S17mysql-ndb-mgm
S17portmap
S18avahi-daemon
S18mysql-ndb
S19mysql
S20apmd
S20apport
S20cupsys
S20cyrus2.2
S20exim4
S20hotkey-setup
S20jboss
S20nfs-common
S20nfs-kernel-server
S20nvidia-kernel
S20powernowd
S20rsync
S20samba
S20saslauthd
S20ser2net
S20winbind
S20xl2tpd
S23ntp

As you can see S23ntp is up WAY after bind9. DNS clients resolving using
gethostbyname() only need the network up - and on worst case (i.e. my
machine with a caching DNS server) it needs to be up after S15bind9. It
seems to me the problem is that NTP is run before the *network* cards
are brought up properly (certainly before eth0). By the time it hits
S23ntp the damage was already done a long time before.

Resolution is based around /etc/nsswitch.conf, /etc/hosts and
/etc/resolv.conf. As it happens I am running caching DNS servers (bind9)
on all these machines as well - to fail to obtain the host that means
NTP is being executed PRIOR to S15bind9.

Now I have to say...the situation with the NTP daemon is a farce and
there have been so many people 'tampering' with NTPD startup that quite
frankly the use of /etc/rc2.d scripts is being completely undermined.
What is needed is a proper solution here rather than yet more tampering.
The NTP daemon is being started already from around 3 different places.
It is the ordering and interaction of these which is causing the
problems - coupled with the fact that the NTP daemon itself does not
perform any kind of retry-recovery internally.

Thomas wrote:
> I just finished a fresh U8.10-rc install.
> Plain vanilla, all I added was the NTP package.
>
> sprinkmeier@u810:~$ ls -l /etc/rc5.d/S2[38]*
> lrwxrwxrwx 1 root root 13 Oct 26 14:18 /etc/rc5.d/S23ntp -> ../init.d/ntp
> lrwxrwxrwx 1 root root 24 Oct 26 10:10 /etc/rc5.d/S28NetworkManager -> ../init.d/NetworkManager
>
> NTP starts before Network Manager, meaning that NTP won't have DNS available at startup.
> It tends to work, but moving NTP's startup after Network Managers should make it more reliable. It will also allow NTPDATE to run (it's called from the ip-up scripts, but will fail as NTPD will already own the socket).
>
> sprinkmeier@u810:~$ grep ntp-server /etc/dhcp3/dhclient.conf
> sprinkmeier@u810:~$
>
> NTP servers still aren't requested by default. If whoever set up your
> DHCP server knows enought to advertise NTP server(s) then chances are
> you should use them (or at the very least PREPEND them).
>
>

--
Sincerely Yours

Copyright Witness Net Support
<email address hidden>

www.copyrightwitness.com
Registration centre for copyright works.

This e-mail and any attachments are confidential and intended for the addressee only.
The information in this mail does not amount to legal advice or opinion. Any views or legal references are those of the author and are based on personal opinion or understanding only.

Moderators may delete this post if you wish..

I just have to say it is truly incredible this issue is not fixed in Hardy. It has been known for a long time and the fundamental cause is known--the necessary sequence of getting the internet connection to be fully functioning (eth0, dhcp, etc) before running ntpdate to completion, then running ntp.

ntp is so widely used, it is amazing this wasn't satisfactorily addressed.

As for the tampering, we HAVE TO DO THAT TO GET THE DANG THING TO WORK.

re tampering :) Know how you feel.

Because of this I feel it is better to fix the problem at source (pun intended!) and have raised #288914 [ https://bugs.launchpad.net/ubuntu/+source/ntp/+bug/288914 ] to try to prompt discussions. When I get a chance I will d/l the source and fix it properly....but im a couple of months away of being able to fit that into my workload.

The question I have yet to resolve is *what* is pulling NTPD up prior to S15bind9 being executed (and I reckon it happens more than once!).

If NTP worked as I suggested in 288914 then all this messing about in dhcp scripts, ifup scripts etc would completely disappear and we can stick with just executing ntpd once from S23ntp quite happily - or even earlier if desired.

On 25/10/08 04:42, cwsupport wrote:
> The NTP server comes up & down like a yoyo during system boot
How did you determine this?

--
Onno Benschop

Connected via Optus B3 at S31°54'06" - E115°50'39" (Yokine, WA)
--
()/)/)() ..ASCII for Onno..
|>>? ..EBCDIC for Onno..
--- -. -. --- ..Morse for Onno..

ITmaze - ABN: 56 178 057 063 - ph: 04 1219 8888 - <email address hidden>

I don't feel that opening another bug for the same issue is helpful and
have marked Bug #288914 as a duplicate.

On 27/10/08 02:13, cwsupport wrote:
> The question I have yet to resolve is *what* is pulling NTPD up prior to
> S15bind9 being executed (and I reckon it happens more than once!)
This is of much more interest than any other issue.

The patch I submitted to this bug brings NTP down and restarts it when
an interface comes up. It was based on the fix that was supplied by
heindsight. It was applied after testing and confirming, like J Carrera
did, that it worked.

I don't have a machine with multiple interfaces available, but I suspect
now that the code runs for *each* interface, causing NTP to be restarted
multiple times. While annoying, this should not actually cause it to fail.

I'm much more interested in attempting to determine what starts NTP and
how to fix that.

--
Onno Benschop

Connected via Optus B3 at S31°54'06" - E115°50'39" (Yokine, WA)
--
()/)/)() ..ASCII for Onno..
|>>? ..EBCDIC for Onno..
--- -. -. --- ..Morse for Onno..

ITmaze - ABN: 56 178 057 063 - ph: 04 1219 8888 - <email address hidden>

Onno, would you please elaborate on the "patch" you posted...

-What files does it modify, delete, or add?
-What, in detail, does it do, exactly, step by step?
-How does one employ/deploy it?
-To what versions can it be applied?

Thanks.

See above, date: 2008-03-07

Just to be clear, this patch has already been accepted and applied into NTP - it was what closed this bug report on 2008-03-08 and now, seven months later, there has been a further report indicating that there is still a problem.

Hi,

Onno Benschop wrote:
> On 25/10/08 04:42, cwsupport wrote:
>
>> The NTP server comes up & down like a yoyo during system boot
>>
> How did you determine this?
>
>
I see lots of ntp started messages during startup.

It is illuminating to look at syslog in an editor/viewer just after a boot and search for 'ntp.' That will show you the ntpd and ntpdate calls--easily showing if multiple attempts are happening. You can also see if ntpd is making associations, look to see if interfaces are up, if dhcp addresses have been assigned before ntp is trying to start, and if ntpdate ran before ntpd.

Also, I have thought that the fix for this should also add the 'write to hardware switch' to ntpdate IF AND ONLY IF ntpdate is successful. This will avoid the rare but possible case of the machine clock being more than 1000 secs wrong from actual, causing ntpd to go silent without notification (other than logging).

J Carrera, I do not know what you mean by "'write to hardware switch' to ntpdate".

>J Carrera, I do not know what you mean by "'write to hardware switch' to ntpdate".

Sorry, I am on long travel and don't have my linux machine. As I recall it is something involving "--hwclock" or something like that. It may even be a separate command rather than an ntpdate switch. Maybe "date --hwclock"?

At any rate it is a command that writes the system time into the hardware clock--thus updating the hwclock to the correct current time. This is useful because system time is set during initial boot to the hardware clock time. If the hardware clock is more than 1000 sec off, ntp will fail when it tries to start, and it does not recover until manual intervention.

I did some web research. The command is "hwclock --systohc"

This should be part of the fix (or at least made an option somehow) to run after ntpdate,
for the reason previously cited.

I have track this down to the ifup script causing the issue.

S23ntp starts a process with pid 7083.

After start ps -ef shows ntpd with pid of 3628.
Immediately before this is 2939 udev and after this is 5502 portmap.

This shows just how early this ntp process is started.

The machines I run have

nsswitch.conf: hosts: files mdns4_minimal [NOTFOUND=return] dns mdns4
resolv.conf: <EMPTY!>

All these machines are hard-wire static IP run a minimum of a caching DNS server. Therefore to resolve ANY external hosts NTPD *must* be started after bind9.

However the ifup ntpdate script completely undermines the rc2.d functionality.

Removing the /etc/network/if-up.d/ntpdate script completely removed the problem and everything operates as normal.

if-up scripts are also fundamentally flawed - the bringing up of the interface of a hard-wired network card will result in the interface coming UP *even* if the cable is unplugged. Subsequent plugging in of the cable will do nothing except the network will now work - which wont cause an event that restarts NTPD - again this is due to the flaw in NTPD that if it cannot resolve or cannot connect to a host it ignores the specified server.

As an interim fix these scripts should check to see if NTPD is running - if NOT then ntpd SHOULD NOT be started. This then ensures correct functionality on both hard-wired and wireless systems.
------------------
Case 1: hardwired

udev brings up
if-up.d/ntpdate runs - no ntpd server is currently running so it does nothing.
S15bind9 comes up
S23ntp comes up - ntpdate-debian ive put in this script corrects the clock and NTPD runs.

Case 2: wireless

udev brings up eth0
if-up.d/ntpdate runs - no ntpd server is currently running so it does nothing.
S15bind9 comes up
S23ntp comes up ntpdate-debian fails but the server runs.

networkmanager subsequently brings up ath0 and connects.
(i believe) if-up.d/ntpdate now reruns - pulls down NTPD, runs ntpdate (no longer required if you have ntpdate-debian in the init.d script which is more sensible) and then brings up NTPD again because *it was running* prior to the i/face coming up.
-----------------

I believe the following fixes are required - 2 solutions offered:

1. Best Solution
a) calls to ntpdate-debian should set the hwclock as discussed above.
b) ntpd if corrected to behave appropriately (panic threshold and server connect retries)

2. Interim Solution
a) calls to ntpdate-debian should set the hwclock as discussed above.
b) only pull up the ntpd server (or cal ntpdate even) in if-up scripts if it was previously running. After all - ntpdate should only be used automatically if ntp as a service has been started. If the service is never started the admin is stating that it does not want the system to automatically modify the date - merely having the s/w installed should not cause it to modify the clock automatically - its needs to be installed and the service started.

cwsupport, this should be a great help to getting this fixed (has been known a LONG time--should have been fixed in Gutsy and Hardy). However, there is one part of your logic I do not understand:

>After all - ntpdate should only be used automatically if ntp as a service has been started.

I think I disagree. ntpdate is not dependent on ntpd running...it is a stand alone function. So, I don't understand why you are saying to run ntpdate only if ntpd is running?????

Indeed, ntpdate should be run before ntp...the sequence should be...

internet connectivity fully up--interface with IP address assigned =and= DNS available
ntpdate, then test for success (probably should retry on failures)
if ntpdate successful,
hwclock --systohc (to ensure if you are starting on a machine with a very wrong hw clock, it is made 'nearly right')
ntpd started

The ntpdate/hwclock part is to prevent a case evolving where ntp is attempted to start during boot on a machine where the hw clock is more than 1000 sec wrong and sysclock is set to the hw clock. If the boot sequence allows that to happen, ntp will fail and there is no direct indication that has happened--only syslog entries. The system clock and hw clock remain wrong because system has been set to the hw clock. This situation will exist if
hw clock is very wrong
ntpdate fails or is not run.

Here is a patch to /etc/init.d/ntp that will ensure that the NTP service comes up and the hwclock is re-sync'd even if the panic-threshold would normally stop ntpd from syncing.

This may not be applicable to everyone but I believe it should be the default behaviour - maybe a switch could be applied in /etc/default to handle this?

Here is a patch that works in conjunction with the previous patch.

If the ntp daemon is running already it is simply restarted (previous patch includes syncing ntpdate and setting the hwclock in the init.d script).

If the ntp daemon is not running already ntpdate-debian is called and the hwclock is set BUT the ntp daemon is NOT started.

oops. re: the last patch.

I still believe that the option to set the date via ntpdate automatically should be based on options in /etc/default and that merely having the ntp package installed should not invoke functionality in a mandatory manner without being able to easily switch off that functionality.

RE: the previous three...

hmmmm...I am sure I have seen cases where ntpd is running, but no associations are active; therefore, I am not sure that just testing if ntpd is running fully covers all possible cases..

Hi,

Yes thats the whole problem! The no-association IDs is caused by ntpd
coming up too early - a restart fixes this.
What the fix does is PRIOR to S23ntp running ntpd will not be started
due to the if-up script - this means that subsequently when S23ntp runs
S15bind9 has already come up and therefore addresses on a local DNS
based server will resolve.

Testing shows this works a treat and doesnt interfere with my laptop's
wireless card connection either.

The test about ntpd running is merely to determine whether or not to use
the init.d script and restart. If its not running you wont get 'no
association IDs' - you will get 'connection refused' when using ntpq.

Hope this helps,
Barry

J Carerra wrote:
> RE: the previous three...
>
> hmmmm...I am sure I have seen cases where ntpd is running, but no
> associations are active; therefore, I am not sure that just testing if
> ntpd is running fully covers all possible cases..
>
>

--
Sincerely Yours

Copyright Witness Net Support
<email address hidden>

www.copyrightwitness.com
Registration centre for copyright works.

This e-mail and any attachments are confidential and intended for the addressee only.
The information in this mail does not amount to legal advice or opinion. Any views or legal references are those of the author and are based on personal opinion or understanding only.

You say "addresses on a local DNS server will resolve." What about the vast majority of people who do not have a local DNS server and rely on internet DNS?

It will not affect them in any way at all. The ntpd will come up at the
right time as specified in rc2.d.

Resolution without local DNS will result in gethostbyname() going out to
an external DNS and this functionality will not be affected.

The fix is therefore transparent to most users (hence why it probably
hasnt been picked up so far).

The fix will be a god-send to anyone running a server that also hosts
DNS (bind9 typically). The fix is therefore more important to
corporates. Most end-users not running DNS servers probably wonder what
all the current fuss is about.

TTFN
Barry

J Carerra wrote:
> You say "addresses on a local DNS server will resolve." What about the
> vast majority of people who do not have a local DNS server and rely on
> internet DNS?
>
>

--
Sincerely Yours

Copyright Witness Net Support
<email address hidden>

www.copyrightwitness.com
Registration centre for copyright works.

This e-mail and any attachments are confidential and intended for the addressee only.
The information in this mail does not amount to legal advice or opinion. Any views or legal references are those of the author and are based on personal opinion or understanding only.

I probably won't add anything new to this, but here it goes:

I've just installed Hardy for a mail/dns/webserver with a NTP server also and I ran into this little 'bug'.

- machine boots, NTP is stopped and started when eth0 (statically configured) comes up
- down the line, NTP is again started with rc2.d/S23ntp (obviously nothing happens because it's already running)
- I log in, check ntpq -p, i get nothing ("No association ID's returned").

I have my own DNS server running on localhost, and I have /etc/resolv.conf with

nameserver 127.0.0.1

a.k.a. I'm using my own nameserver, which isn't available at network start, becase it starts later (but before S23).

So this is a server install with little to none network ups/downs, nothing dynamic, nothing fancy, just the good old static network config, and the fact that the nameserver used is 127.0.0.1.

I can see how one would try to do a good thing by (re)syncing the clock whenever the network is (re)started, but doing that for _every single network interface_ the machine has ?

If I happen to have 3-4 interfaces, NTP gets restarted that many times, that's just silly. And what if I have eth1 for internal LAN and do an ifdown/ifup eth1 (because say I changed the IP address, or whatever), which has nothing to do with NTP ? It still gets restarted. It doesn't seem the right thing to do IMHO.

This is *STILL* an issue in Karmic Koala 9.10 BETA.

NOTE: The patch works for Karmic Koala as well.

Hardy has seen the end of its life and is no longer receiving any updates. Marking the Hardy task for this ticket as "Won't Fix".