squid3 gets killed at startup with dnsmasq

Status changed to 'Confirmed' because the bug affects multiple users.

I experience the same issue, but have NetworkManager installed - although it does not manage the eth0 interface (I am using guessnet for that).

So I think what is happening is that the network interface is coming up before squid has installed a handler for SIGHUP.

*tons* of stuff happens between squid starting, and squid installing its signal handlers. I haven't gone through all of the code but I'm betting at least one or two stall on not having a network yet. It would seem that squid should probably ignore SIGHUP almost immediately on startup to avoid this issue.

I've pushed up a branch that makes that change, and I'll put up packages in a PPA soon that affected users can test.

Where does the HUP signal come from?
Could "squid3 -k reconfigure" get used there instead? (if it's specific to squid3)

(I assume you've meant that squid's handler is being installed to late to map it to "reload config" instead)

squid3 includes a resolvconf script to run whenever DNS servers change:

#!/bin/sh

PATH="/usr/sbin:/usr/bin:/sbin:/bin"

# Make squid aware of changes to resolv.conf
if status squid3 | grep "start/running" > /dev/null; then
 reload squid3
fi

reconfigure also ends a HUP:

            if (!strncmp(optarg, "reconfigure", strlen(optarg)))
                /** \li On reconfigure send SIGHUP. */
                opt_send_signal = SIGHUP;

And yes, thats what I mean, there's a lag between when squid might receive the HUP, and when it will handle it properly. I'd rather ignore that HUP than die, since most of what it is doing at that exact time is reading its configuration and then attempting DNS lookups. I haven't looked deep enough into the code yet to see if it could handle the full reconfiguration earlier.

Do you think adding a workaround like "sleep 5" above the "reload" into the resolvconf script makes sense and is good enough for a workaround in Ubuntu Precise?
Would it be possible to add a loop to detect if the SIGHUP handler is properly setup already (and send HUP only after that)?
Another idea might be to "restart" instead of "reload".

Sorry, I've just now recognized that you have a patch already, and so any workarounds might be void.
Have you uploaded it to a PPA already?

Do you want to propose this patch upstream in Squid's Bugzilla?
This would allow to gain feedback on it before including it in Ubuntu.

I haven't had a chance to test it locally yet, other than building the package. I was hoping to simulate the problem. Hopefully will test in a few days.

I'm seeing a similar crash on a 12.04 32-bit server in similar circumstances, however the error is different. Before I open a separate bug report I thought I'd check with Marco if he sees the following reports in "/var/log/squid3/access.log.1" ?

11:12:24 | Reconfiguring Squid Cache (version 3.1.19)...
11:12:24 | FD 14 Closing HTTP Connection
11:12:24 | assertion failed: disk.cc377: "fd >= 0"

I see this in dmesg:

init: squid3 main process (3562) killed by ABRT signal

Oops, correction! the error message is in /var/log/squid3/cache.log.1.

I've found an upstream bug that matches the symptoms I see exactly so I've created a new report bug #988802 "squid3 killed by ABRT signal. assertion failed: disk.cc377: "fd >= 0" "

Have the same behaviour on a x86_64 server freshly-upgraded from Oneiric to Precise.
On system reboot, squid3 does not start properly - gets killed by HUP signal as recorded in syslog.
Logging into the system by ssh and manual starting service squid3 helps to resolve the problem.
But having squid3 starting normally on bootup would be very beneficial.

I strongly suspect this misbehaviour happens due to incomplete integration of upstart, ifupdown and resolvconf packages functionality. I use system without NetworkManager (it's a server after all), my interfaces are controlled by ifupdown and by ppp scripts. Probably upstart starts ifupdown too early, again as it usually happens when you have resolvsconf in universe instead of main.

And btw, my DNS server on this host is bind9, not dnsmasq.

And yes, I reproduced this on i386 as well as on x86_64

For what it's worth, this does not happen with the squid-deb-proxy package (which uses squid with an optimized config) - at least after several boot processes.

The init/start script is different, for example, it uses
    "start on (local-filesystems and net-device-up IFACE!=lo)"
instead of
    "start on runlevel [2345]".

Excerpts from Daniel Hahler's message of Tue May 15 08:53:06 UTC 2012:
> For what it's worth, this does not happen with the squid-deb-proxy
> package (which uses squid with an optimized config) - at least after
> several boot processes.
>
> The init/start script is different, for example, it uses
> "start on (local-filesystems and net-device-up IFACE!=lo)"

This is basically wrong. A system may have more than one interface and
there is no guarantee that one of them is "the right one".

That said, this makes perfect sense, because with NM, this will delay the
start until *after* the first interface comes up, whereas with runlevel 2,
that event will actually come before the first interface is up, so that
starts the window where squid is starting up, polling for DNS entries
and failing, and then it gets HUP just as the interface comes up.

My patch to ignore HUP early on should solve this issue, but I haven't
pushed it back upstream yet.

I'm running 12.04 precise 64bit (upgraded from oneric) squid3 (ver 3.1.19) and bind9 and dansguardian. Was working fine until an hour ago when I apt-get update and apt-get upgrade. Now squid3 is killed by HUP (so says syslog) and I have to restart squid3 to get everything working.

Here is the patch I have come up with, very simple, but I'm not sure of the full ramifications:

This bug was fixed in the package squid3 - 3.1.19-1ubuntu5

---------------
squid3 (3.1.19-1ubuntu5) quantal; urgency=low

  * d/squid3.upstart: Work around squid not handling SIGHUP by
    adding respawn to upstart job. (LP: #978356)
 -- Clint Byrum <email address hidden> Tue, 19 Jun 2012 15:35:19 -0700

the squid patch looks reasonable to me; please forward to the squid-dev list with a [PATCH] subject.

Hello Marco, or anyone else affected,

Accepted squid3 into precise-proposed. The package will build now and be available in a few hours. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Fixed for me:

Jun 23 00:04:44 server kernel: [ 14.922376] init: squid3 main process (1148) killed by ABRT signal
Jun 23 00:04:44 server kernel: [ 14.922405] init: squid3 main process ended, respawning

This bug was fixed in the package squid3 - 3.1.19-1ubuntu3.12.04.1

---------------
squid3 (3.1.19-1ubuntu3.12.04.1) precise-proposed; urgency=low

  * d/squid3.upstart: Work around squid not handling SIGHUP by
    adding respawn to upstart job. (LP: #978356)
 -- Clint Byrum <email address hidden> Tue, 19 Jun 2012 16:32:10 -0700

Sorry, I've been too busy / other problems since today.
I've upgraded the installation that was affected (3.1.19-1ubuntu3.12.04.1 has been installed), removed the workaround in rc.conf, rebooted and works fine now!
Thanks a lot for the fix