[FFe] squid: Update to latest upstream release (3.5)

In case we are unable to merge Squid 3.5+, I have prepared debdiffs to upgrade the current squid3-3.3.8-1ubuntu15 in proposed to 3.3.8-1ubuntu16 and from there to 3.3.14-0ubuntu1.

See LP: #1496223 debdiffs for:
1. gcc5 transition of libecap2 (the current libecap3 on proposed will be unused and should be safely removed)
2. update from 3.3.8-1ubuntu15 (currently FTBFS on proposed) to 3.3.8-1ubuntu16

Then LP: #1502178 debdiff to update from 3.3.8-1ubuntu16 to 3.3.14-0ubuntu1.

Year after year I am astonished that squid has been kept back in Ubuntu, when all other distros update this important and hugely popular software with every release.
Squid 3.3 became DEPRECATED in 2013 !!!

http://wiki.squid-cache.org/Squid-3.3

Please read the note from the squid page:

"Squid-3.3 is CONSIDERED DANGEROUS as the security people say. Due to unfixed vulnerabilities CVE-2014-7141, CVE-2014-7142 and CVE-2014-6270 and any other recently discovered issues."

PLEASE update it to 3.5.x (as 3.4 is also marked DEPRECATED - bold letters from squid website, not mine) in 16.04 LTS.

Note that the Ubuntu packages have had CVE-2014-7141 and CVE-2014-7142 fixed; CVE-2014-6270 is still open. We've rated CVE-2014-6270 as a low priority issue and will update it when a higher priority issue is found.

http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-7141.html
http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-7142.html
http://people.canonical.com/~ubuntu-security/cve/2014/CVE-2014-6270.html

Thanks

I would not consider a buffer overflow and code execution as low priority, especially when this program is likely to run on a firewall or network gateway.

Is there a better timeline than when "we feel like there's a real issue" we'll update? We are now 2 generations depreciated...

e-Vent, we rated this issue "low" because:

- snmp is not enabled by default
- squid's snmp listener can listen on specific interfaces
- local iptables / ufw rules probably already allow only specific services on the hosts that run squid
- network firewalls / routers probably already allow only specific services on the networks that run squid

In general allowing untrusted access to SNMP is not a good idea regardless if this is fixed.

We have limited resources and we have to prioritize the work we do accordingly. If you have the time and inclination to prepare and test a patch for this issue, we'd be happy to sponsor updates. See https://wiki.ubuntu.com/SecurityTeam/UpdatePreparation for more details.

Thanks

I will only add that even in the best of circumstances with perfect firewalling, a low privilege sysadmin or helpdesk member/troubleshooter could easily use this overflow as a hop to privilege escalation and/or willful damage.

I'm hoping to get squid updated in Xenial within the next two weeks.

Requesting a brief (1-3 week) FFe for this please. I have a merge ready, but there is a transitional bug I want to fix. conffile renaming is currently not working in some scenarios, which I believe also affects Debian. It is complicated because of a migration in Debian of squid3 -> squid which overlaps Ubuntu taking squid3 as default ahead of Debian in the past, so there are overlapping package renames going on (crossing the Ubuntu delta) as well as the conffile rename, which isn't working.

I'm still working on fixing this, and I'd prefer to land it when it's ready rather than right now, because then I don't have to worry about unsticking users caught by the current bug in another set of transitional code.

I was delayed by what turned out to be a race condition affecting the init.d script and system, where systemd cannot find a pid file before the daemon has written it. A workaround for this is trivial (sleep 4 or something), but I'd prefer not to land this without fixing it properly either. But if the release team prefers an earlier landing to grant the FFe, I can fix the first and leave the second worked around and fix it properly afterwards. That would allow users to test the actual binaries and functionality better.

I feel it's quite important to get squid to 3.5 in 16.04, as the current version is quite out of date.

@Release Team, may I have a review of the FFe request please?

Proposed merge ready in ppa:racb/experimental, version ~dev5. git tree with detailed Ubuntu delta at https://git.launchpad.net/~racb/ubuntu/+source/squid3/log/?h=merge

I will test the upgrade paths again but I believe they're all working correctly now.

On Wed, Feb 24, 2016 at 04:48:38PM -0000, Robie Basak wrote:
> @Release Team, may I have a review of the FFe request please?
>
> Proposed merge ready in ppa:racb/experimental, version ~dev5. git tree
> with detailed Ubuntu delta at
> https://git.launchpad.net/~racb/ubuntu/+source/squid3/log/?h=merge
>
> I will test the upgrade paths again but I believe they're all working
> correctly now.

Are you ready to upload now?

It's still quite early, so go ahead. I think I would be optimally happy
if this were added to manual milestone testing so that QA people (or
whoever) check that squid works/upgrades/etc - but I'll leave that point
to you.

Cheers,

--
Iain Lane [ <email address hidden> ]
Debian Developer [ <email address hidden> ]
Ubuntu Developer [ <email address hidden> ]

11:26 <rbasak> Laney: thank you for responding to bug 1473691. I have notes to test against squidguard, squid-deb-proxy and to check upgrade paths more throroughly. Would you prefer that I do this before upload?
11:26 <ubot5`> bug 1473691 in squid3 (Ubuntu) "[FFe] squid: Update to latest upstream release (3.5)" [Wishlist,In progress] https://launchpad.net/bugs/1473691
11:26 <rbasak> Apart from that it's ready - no known issues.
11:27 <rbasak> The disadvantage presumably is that nobody else can also test until I've uploaded.
11:27 <rbasak> (easily)
11:27 <Laney> rbasak: You could upload and then block in proposed until you've done
11:27 <Laney> Up to you
11:27 <rbasak> That sounds like a good plan. I'll do that, thanks.

Uploaded to xenial-proposed. This will dep-wait on libecap3 NEW though, and libecap will also FTBFS on 32-bit architectures currently, pending a symbols file fixup.

In the meantime, if others could please test from my PPA above, that would be helpful to inform us about removing block-proposed and help get this migrated to the release pocket.

Now built in xenial-proposed, so you can test from there.

On a xenial system, when enabling xenial-proposed and running "apt-get install squid", I get the below error. Note that I have 'DAEMON=nope' in /etc/default/squid3 to stop squid3 from starting (because it's only installed for squid-deb-proxy).

Setting up squid (3.5.12-1ubuntu1) ...
Removing obsolete conffile /etc/squid3/msntauth.conf ...
Removing obsolete conffile /etc/init/squid3.conf ...
Failed to stop squid3.service: Unit squid3.service not loaded.
invoke-rc.d: initscript squid3, action "stop" failed.
dpkg: error processing package squid (--configure):
 subprocess installed post-installation script returned error exit status 5
Processing triggers for systemd (229-2ubuntu1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for ufw (0.35-0ubuntu1) ...
Errors were encountered while processing:
 squid
E: Sub-process /usr/bin/dpkg returned an error code (1)

The above error doesn't appear to have anything to do with your DAEMON being set to gibberish but, rather, squid3 being removed (along with its init script) before squid.postinst tries to stop it. This is probably due to the Conflicts that should be a Breaks. Experimenting locally.

I did some upgrade testing on squid3 and there are some troubles apparently, I suspect and will try to modify debian/control to make it work (at least for upgrade from xenial to xenial-proposed).

-----
From precise to trusty with squid3 + squid-deb-proxy +squidguard (to check squid/squid3 renaming with ubuntu deltas)

Last week it was ok (3.3.8-1ubuntu6.4), now upgrading from squid(3) + squid-deb-proxy + squidguard is broken 3.3.8-1ubuntu6.6.
So it appears that I won't be able to test the full upgrade path from precise to xenial

Squidguard would have created some troubles as it was still using /etc/squid or /var/log/squid in precise version. It would have been interesting to test because of this, but last upstream broke release upgrade (last week it failed at starting service but continued upgrade, now it doesn't allow the upgrade to finish).

----

The following are my upgrade tests from trusty to xenial:

long story short: it fails.

I'm in a two stages update due to the squid package being in proposed, first stage from trusty to xenial current, and then to xenial-proposed.
I did the test two times one with installing with 'squid', and then with 'squid3'.

Both trusty setup were working squid3 + squidguard +squid-deb-proxy, and basic conf was apply to check.

First step upgrade caused squidguard to stop working (certainly some API related stuff), and squid-deb-proxy service seems broken, as I can run it by hand. squid3 was still working.

squid3 service is renamed to squid, we need to warn or take appropriate disposition as done in docker.io upgrade path.

Second step upgrade broke squid3. Logs/reference to follow.

----
squidguard and squid-deb-proxy are universe, but the upgrade path must be checked with them also to get some real-world upgrade path.

----

http://paste.ubuntu.com/15480862/

Kick,

Thanks for the thorough upgrade test. I was able to recreate and debug the proposed package install script. It appears that the squid.postinst expects to find a /etc/init.d/squid3 service script if we also have an existing /etc/squid3 directory. However, in my testing, we do have the /etc/squid3, but the service script itself is called /etc/init.d/squid.

Manually fixing up this in the /var/lib/dpkg/info/squid.postinst and then running sudo apt-get -f install will allow squid upgrade to proposed levels complete successfully.

After the upgrade, I can confirm that squid-deb-proxy and squidGuard work correctly. I'm attaching a debdiff with a fix for this issue (we can check which init script is present and use that to stop/start the squid service after install).

I build squid from the merge branch from rabc's git repo with the attached debdiff, created a local ppa (including a copy of the libecap3 package) and performed an apt-get update && apt-get install squid which upgrades and installs successfully.

After this, squid, squid-deb-proxy and squidGuard all continue to work.

Hi All,

Ubuntu is to release in just a few weeks from now (3.5?) and as I understand it, the new version of squid is changing the service/init script name from squid3 to squid. Provided that we are very close to release, shouldn't we, Ubuntu, maintain the squid3 init script at least to provide backwards compatibility?

This will definitely break people that rely on squid3 being a init script.

Hi Andres,

It will break people, yes. But isn't a new release the right time to
make such a change?

We're taking care of squid-deb-proxy and squidguard in the archive. Is
there any other consumer that is in the archive that you're bothered
about?

Old Ubuntu to New Debian debdiff

Old Ubuntu to New Ubuntu debdiff

I've attached new debdiff integrating the change to the squid.postinst fix. We would like to get this build uploaded into proposed (and locked as the current one is) to continue wider testing of the new squid.

Ryan, for ease of review and sponsorship please provide an incremental debdiff against the package that's already in xenial-proposed.

Nevermind, it seems someone already sponsored this upload. So, for future reference... :)

if test -d /etc/squid3 && dpkg --compare-versions "$2" lt '3.5'; then
       #
       # handle the case where we have /etc/squid3 but the init script is
       # named squid.
       #
       service=squid3
       if ! test -e /etc/init.d/$service ; then
               service=squid
       fi
       invoke-rc.d $service stop
       invoke-rc.d $service start
else
       invoke-rc.d squid restart
fi

I had to look very carefully at the packages in trusty and wily to try to figure out what this code is supposed to do, and I'm still not sure. At first I believed it was buggy, then I came to believe it was just pointless, now I'm convinced again that it's buggy.

The trusty version of the squid3 package ships an /etc/init/squid3.conf upstart job and no init script.
The wily version of the squid3 package ships both an /etc/init/squid3.conf upstart job, and an /etc/init.d/squid3 init script.

You are first looking for a script named /etc/init.d/squid3, and if it's absent, you are stopping and starting the 'squid' service. But if it's *present*, you are stopping and starting the squid3 service, which is certainly wrong.

So on upgrade from trusty, you will have removed /etc/init/squid3.conf in the preinst of squid, and the old version of the squid3 package will have stopped the squid3 service in its prerm because squid Conflicts: old squid3 package and is being removed. Then squid postinst stops squid (unnecessary because not previously started), then starts squid. So the code to get there is convoluted but this will work.

But on upgrade from wily, the old version of the squid3 package will have stopped the squid3 service in its prerm because squid Conflicts: old squid3 package and is being removed; and the new squid preinst will have removed /etc/init/squid3.conf; but /etc/init.d/squid3, which is a conffile, will still be on disk, so your code now tries to stop and then start the old squid3 service, instead of stopping squid3 (which, again, is redundant) and starting squid.

As far as I can see, the entire versioned code block here is wrong, and you should *only* be restarting squid.

As far as I can see, the entire versioned code block here is wrong, and
> you should *only* be restarting squid.
>
>
The block (without my addition) is from debian itself. The original block
that's
present in debian:

update-rc.d squid defaults 30 >/dev/null

if test -d /etc/squid3 && dpkg --compare-versions "$2" lt '3.5'; then
    invoke-rc.d squid3 stop
    invoke-rc.d squid start
else
    invoke-rc.d squid restart
fi

However, this failed in our dist-upgrade from trusty scenario as the squid3
service file no longer
exists. Reading your analysis; it seems that /etc/init.d/squid3 should
still be on disk;
but it was not.

In the dist-upgrade from trusty case; I believe we can just use the squid
restart.

In the upgrade from wily case, it's not clear to me why the
/etc/init.d/squid3 service
file would still be on disk when it wasn't when upgrading from trusty?

On Tue, Mar 29, 2016 at 9:05 PM, Steve Langasek <
<email address hidden>> wrote:

> Ryan, for ease of review and sponsorship please provide an incremental
> debdiff against the package that's already in xenial-proposed.
>

I actually did in comment 18, but I didn't know that was sufficient and
subsequently uploaded
those larger debdiffs. Thanks for the tips.

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1473691
>
> Title:
> [FFe] squid: Update to latest upstream release (3.5)
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/squid3/+bug/1473691/+subscriptions
>

On Wed, Mar 30, 2016 at 04:41:12AM -0000, Ryan Harper wrote:
> As far as I can see, the entire versioned code block here is wrong, and
> > you should *only* be restarting squid.

> The block (without my addition) is from debian itself.

Yes, and I believe it's wrong there also (unless their previous squid3
package, unlike ours, didn't stop itself in the prerm?)

> update-rc.d squid defaults 30 >/dev/null

OK, someone's living in 2005... The '30' sequence number is obsolete, and
also why is this not being driven by debhelper?

> However, this failed in our dist-upgrade from trusty scenario as the squid3
> service file no longer
> exists. Reading your analysis; it seems that /etc/init.d/squid3 should
> still be on disk;
> but it was not.

No. I said that in an upgrade from trusty, there *was no*
/etc/init.d/squid3 init script, only an /etc/init/squid3.conf upstart job.
And your maintainer script is removing /etc/init/squid3.conf on upgrade;
which is why the Debian logic fails.

> In the dist-upgrade from trusty case; I believe we can just use the squid
> restart.

Yes.

> In the upgrade from wily case, it's not clear to me why the
> /etc/init.d/squid3 service
> file would still be on disk when it wasn't when upgrading from trusty?

Because the wily version of the package *has* an /etc/init.d/squid3 file,
whereas the trusty package does not. And nothing in the xenial package is
handling the removal of this stale conffile.

So:

 - fix the maintainer script to remove /etc/init.d/squid3 on upgrade
 - drop all the versioned upgrade logic and just call invoke-rc.d squid
  restart.

BTW, note that dpkg-maintscript-helper has to be called from several
different maintainer scripts with the same arguments, not just from the
preinst - the squid package already has several bugs because it's not doing
this correctly. This is a royal pain to get right by hand, and all
right-thinking maintainers would use debian/squid.maintscript as documented
in dh_installdeb, instead of hand-editing maintainer scripts and trying to
keep them all in sync. This *probably* will DTRT in the package, despite
the fact that squid3 is using the one-step-removed-from-obsolete debhelper
compat level 5.

--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
<email address hidden> <email address hidden>

Debian testing now has the maintenance release 3.5.15.

Changes to squid-3.5.15 (23 Feb 2016):

 - Bug 3870: assertion failed: String.cc: 'len_ + len <65536' in ESI::CustomParser
 - Fix multiple assertion on String overflows
 - Fix unit test errors on MacOS
 - Better handling of huge response headers. Fewer incorrect "Bug #3279" messages.
 - Log noise reduction for eCAP

Changes to squid-3.5.14 (16 Feb 2016):

 - Bug 4437: Fix Segfault on Certain SSL Handshake Errors
 - Bug 4431: C code is not compiled with CFLAGS
 - Bug 4418: FlexibleArray compile error with GCC 6
 - Bug 4378: assertion failed: DestinationIp.cc:60:
  'checklist->conn() && checklist->conn()->clientConnection != NULL'
 - Fix invalid FTP connection handling on blocked content
 - Fix handling of shared memory left over by Squid crashes or bugs
 - Fix mgr:config report 'qos_flows mark' output
 - Fix compile error in CPU affinity
 - Fix %un logging external ACL username
 - Avoid more certificate validation memory leaks
 - ... and some documentation updates

Changes to squid-3.5.13 (06 Jan 2016):

 - Bug 4397: DragonFly BSD, POSIX shared memory is implemented as filepath
 - Bug 4387: Kerberos build errors on Solaris
 - TLS: Support Ephemeral Elliptic Curve Diffie-Hellman (EECDH) key exchange
 - TLS: Complete certificate chains using external intermediate certificates
 - Avoid memory leaks when an X.509 certificate validator is used with SslBump
 - Fix connection retry and fallback after failed server TLS connections
 - Fix GnuTLS detection via pkg-config
 - Fix startup crash with a misconfigured (too-small) shared memory cache
 - ... and some documentation updates

squid3-3.5.12-1ubuntu3 uploaded already by stgraber handled using the dpkg-maintscript-helper to clean up the old /etc/init.d/squid3 file.

This debdiff handles the second task of removing the check on how to restart squid and just calls invoke-rc.d squid restart unconditionally.

I've tested this in a wily -> xenial upgrade, as well as a trusty -> xenial upgrade; squid, squidGuard, and squid-deb-proxy work as expected after upgrading to 1ubuntu4 version.

This bug was fixed in the package squid3 - 3.5.12-1ubuntu6

---------------
squid3 (3.5.12-1ubuntu6) xenial; urgency=medium

  * Attempt to migrate /var/log/squid3 -> /var/log/squid on upgrade.
  * Update apparmor profile for s/squid3/squid/ and /dev/shm access.

 -- Adam Conrad <email address hidden> Sun, 03 Apr 2016 21:34:50 -0600