squid3: segfault when ftp passive mode is not available

Bug #1560429 reported by Goacid
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Squid
Unknown
Unknown
squid3 (Debian)
Fix Released
Unknown
squid3 (Ubuntu)
Fix Released
Medium
Andreas Hasenack
Trusty
Won't Fix
Medium
Unassigned
Xenial
Fix Released
Medium
Andreas Hasenack
Yakkety
Won't Fix
Medium
Andreas Hasenack

Bug Description

[Impact]

Users who use squid as an FTP proxy and access sites that block ftp PASV mode will trigger a squid segfault. That means a brief service interruption, as upstart/systemd will restart it.

Since this is a crash, the backport seems justified. But there is an effective workaround, see below.

Upstream committed a fix, the same fix we are introducing here, which essentially adds a lot of NULL checks but at the same time disables the fallback ftp command EPRT should passive mode fail. Upstream states that this command doesn't work properly in squid yet.

This is also the recommended workaround: disable EPRT by setting the following in /etc/squid/squid.conf and restarting the service:

ftp_eprt off

[Test Case]

- setup a xenial machine/lxd with proftpd configured like this (/etc/proftpd/proftpd.conf):

http://pastebin.ubuntu.com/25039718/

- restart proftpd: sudo service proftpd restart

Alternatively, setup any anonymout ftp server to your liking with passive mode disabled/forbidden.

- Create a simple file under the anonymous area, for the ftp client to fetch later on:
echo hello | sudo tee /srv/ftp/readme.txt

- install the squid proxy under test in another machine/lxd.

- configure /etc/squid/squid.conf like this: http://pastebin.ubuntu.com/25233360/

- in the above, adjust localnet to your network, or replace the line "http_access allow localnet" with "http_access allow all" to accept everything

- restart squid: sudo service squid restart

- access the ftp server via the squid proxy:

$ ftp_proxy=http://xenial-squid-passive.lxd:3128/ wget ftp://xenial-proftpd.lxd/readme.txt -O /dev/null -t1

(replace the URLs with whatever you need to reach the squid server under test, and the ftp server you setup)

In the case of a vulnerable squid server you will get:
a) wget gives up:
andreas@nsn7:~$ ftp_proxy=http://xenial-squid-passive.lxd:3128/ wget ftp://xenial-proftpd.lxd/readme.txt -O /dev/null -t1
--2017-07-07 11:58:16-- ftp://xenial-proftpd.lxd/readme.txt
Resolving xenial-squid-passive.lxd (xenial-squid-passive.lxd)... 10.0.100.151
Connecting to xenial-squid-passive.lxd (xenial-squid-passive.lxd)|10.0.100.151|:3128... connected.
Proxy request sent, awaiting response... No data received.
Giving up.

b) /var/log/squid/cache.log shows a squid restart with a new PID:
2017/07/07 14:58:19 kid1| Starting Squid Cache version 3.5.12 for x86_64-pc-linux-gnu...
2017/07/07 14:58:19 kid1| Service Name: squid
2017/07/07 14:58:19 kid1| Process ID 1638

c) proftpd /var/log/proftpd/extended.log will show the passive ftp attempts being forbidden with a 501 error:
xenial-squid-passive.lxd UNKNOWN - [07/Jul/2017:14:58:16 +0000] "USER anonymous" 331 -
xenial-squid-passive.lxd UNKNOWN ftp [07/Jul/2017:14:58:16 +0000] "PASS (hidden)" 230 -
xenial-squid-passive.lxd UNKNOWN ftp [07/Jul/2017:14:58:16 +0000] "TYPE A" 200 -
xenial-squid-passive.lxd UNKNOWN ftp [07/Jul/2017:14:58:16 +0000] "MDTM readme.txt" 213 -
xenial-squid-passive.lxd UNKNOWN ftp [07/Jul/2017:14:58:16 +0000] "EPSV 1" 501 -
xenial-squid-passive.lxd UNKNOWN ftp [07/Jul/2017:14:58:16 +0000] "PASV" 501 -

In the case of the fixed squid server, you will get:
a) wget gets a 502 error instead of "no data":
andreas@nsn7:~$ ftp_proxy=http://xenial-squid-passive.lxd:3128/ wget ftp://xenial-proftpd.lxd/readme.txt -O /dev/null -t1
--2017-07-07 12:04:14-- ftp://xenial-proftpd.lxd/readme.txt
Resolving xenial-squid-passive.lxd (xenial-squid-passive.lxd)... 10.0.100.151
Connecting to xenial-squid-passive.lxd (xenial-squid-passive.lxd)|10.0.100.151|:3128... connected.
Proxy request sent, awaiting response... 502 Bad Gateway
2017-07-07 12:04:14 ERROR 502: Bad Gateway.

b) /var/log/squid/cache.log doesn't "blip", and access.log just logs the 502:
1499439854.710 18 10.0.100.1 TCP_MISS/502 4324 GET ftp://xenial-proftpd.lxd/readme.txt - HIER_DIRECT/10.0.100.134 text/html

[Regression Potential]
You won't be able to use squid to access FTP sites that block passive mode transfers. But that was the case already, except it was the segfault that was preventing this from working, and not an error message.

There are many more fixes in the 3.5 branch that are not being applied here, related to other problems. Debian upted to upgrade to 3.5.23 in their bug http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=793473

One could argue that updating to that version is "safer" than cherry picking a patch from their code tree.

[Other Info]
I don't have a patch for trusty, which is on an older version of squid (3.3.8-1ubuntu6.9). The code changed a lot and it's not just a matter of fixing conflicts.

Revision history for this message
Amos Jeffries (yadi) wrote :

Workaround is to configure "ftp_eprt off" for now.

A possible fix patch is available at upstream Squid Project in 3.5.23 release. Though we are not sure of completeness yet so the upstream bug report is staying open for now.

Changed in squid3 (Debian):
status: Unknown → Confirmed
Changed in squid3 (Debian):
status: Confirmed → Fix Released
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Debian Closed by new Upstream release 3.5.23-1 into unstable, so needing a merge to fix that.
For a backport to 14.04 it has to be identified what change exactly contains the fix.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

http://bugs.squid-cache.org/show_bug.cgi?id=4004#c16:

"Our changes are in v3.5 (r14115), v4 (r14935), and v5 (r14936, r14937). If they do not fix this bug fully, please reopen this bug report or file a new one."

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Easy to reproduce with proftpd and this config inside the default anonymous section:
  <Limit EPSV PASV>
    DenyAll
  </Limit>

Proftpd's extended log shows squid trying passive mode and getting back a 501:
trusty-squid-passive-ftp-1560429.lxd UNKNOWN - [04/Jul/2017:19:35:24 +0000] "USER anonymous" 331 -
trusty-squid-passive-ftp-1560429.lxd UNKNOWN ftp [04/Jul/2017:19:35:24 +0000] "PASS (hidden)" 230 -
trusty-squid-passive-ftp-1560429.lxd UNKNOWN ftp [04/Jul/2017:19:35:24 +0000] "TYPE A" 200 -
trusty-squid-passive-ftp-1560429.lxd UNKNOWN ftp [04/Jul/2017:19:35:24 +0000] "MDTM readme.txt" 213 -
trusty-squid-passive-ftp-1560429.lxd UNKNOWN ftp [04/Jul/2017:19:35:24 +0000] "EPSV 1" 501 -
trusty-squid-passive-ftp-1560429.lxd UNKNOWN ftp [04/Jul/2017:19:35:24 +0000] "PASV" 501 -

Which squid doesn't like and crashes:
Program received signal SIGSEGV, Segmentation fault.
Ip::Address::GetPort (this=0x8) at Address.cc:806
806 Address.cc: No such file or directory.
(gdb) bt
#0 Ip::Address::GetPort (this=0x8) at Address.cc:806
#1 0x0000559917ef0d68 in ftpSendEPRT (ftpState=ftpState@entry=0x5599192f9198) at ftp.cc:2925

I'll try the bzr commit diff from the upstream squid bug.

Revision history for this message
Andreas Hasenack (ahasenack) wrote :

This also affects:

xenial squid 3.5.12-1ubuntu7.3
yakkety 3.5.12-1ubuntu8.1

Zesty and Artful do not crash. There, squid responds with a "502 Bad Gateway" error instead.

Changed in squid3 (Ubuntu):
status: New → Confirmed
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Setting "ftp_eprt off" in squid.conf in xenial and yakkety produces the same result as with the zesty and artful version, i.e., the proxy returns a 502 instead of crashing.

Changed in squid3 (Ubuntu):
assignee: nobody → Andreas Hasenack (ahasenack)
status: Confirmed → In Progress
importance: Undecided → Medium
Changed in squid3 (Ubuntu Yakkety):
status: New → In Progress
assignee: nobody → Andreas Hasenack (ahasenack)
importance: Undecided → Medium
Changed in squid3 (Ubuntu Xenial):
status: New → In Progress
importance: Undecided → Medium
assignee: nobody → Andreas Hasenack (ahasenack)
Revision history for this message
Andreas Hasenack (ahasenack) wrote :

Test packages for yakkety and xenial: ppa:ahasenack/squid-passive-ftp-1560429

I'm not sure yet how to handle squid in trusty. The code is very different there.

description: updated
description: updated
description: updated
description: updated
description: updated
Changed in squid3 (Ubuntu Trusty):
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Robie Basak (racb) wrote :

Uploaded to Xenial unapproved.

Yakkety -> EOL.

Changed in squid3 (Ubuntu Yakkety):
status: In Progress → Won't Fix
Revision history for this message
Robie Basak (racb) wrote :

> Zesty and Artful do not crash. There, squid responds with a "502 Bad Gateway" error instead.

Changed in squid3 (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Goacid, or anyone else affected,

Accepted squid3 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/squid3/3.5.12-1ubuntu7.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in squid3 (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-xenial
description: updated
description: updated
description: updated
Revision history for this message
Andreas Hasenack (ahasenack) wrote :
Download full text (4.3 KiB)

xenial verification

Confirmed the crash and squid's restart:
(...)
2017/08/03 13:40:17| pinger: ICMPv6 socket opened
2017/08/03 13:40:18 kid1| storeLateRelease: released 0 objects
(issued wget here)
2017/08/03 13:40:28 kid1| cannot change current directory to /var/spool/squid3: (2) No such file or directory
2017/08/03 13:40:28 kid1| Current Directory is /
2017/08/03 13:40:28 kid1| Starting Squid Cache version 3.5.12 for x86_64-pc-linux-gnu...
2017/08/03 13:40:28 kid1| Service Name: squid
2017/08/03 13:40:28 kid1| Process ID 5725 <----- new PID
2017/08/03 13:40:28 kid1| Process Roles: worker

wget gave up:
$ ftp_proxy=http://xenial-squid-passive-ftp-1560429.lxd:3128/ wget ftp://proftpd.lxd/readme.txt -O /dev/null -t1
--2017-08-03 10:40:22-- ftp://proftpd.lxd/readme.txt
Resolving xenial-squid-passive-ftp-1560429.lxd (xenial-squid-passive-ftp-1560429.lxd)... 10.0.100.226
Connecting to xenial-squid-passive-ftp-1560429.lxd (xenial-squid-passive-ftp-1560429.lxd)|10.0.100.226|:3128... connected.
Proxy request sent, awaiting response... No data received.
Giving up.

Installed squid from proposed:
(...)
Get:1 http://br.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 squid amd64 3.5.12-1ubuntu7.4 [2,310 kB]
Get:2 http://br.archive.ubuntu.com/ubuntu xenial-proposed/main amd64 squid-common all 3.5.12-1ubuntu7.4 [176 kB]
Fetched 2,486 kB in 0s (11.7 MB/s)
(Reading database ... 27771 files and directories currently installed.)
Preparing to unpack .../squid_3.5.12-1ubuntu7.4_amd64.deb ...
Unpacking squid (3.5.12-1ubuntu7.4) over (3.5.12-1ubuntu7.3) ...
Preparing to unpack .../squid-common_3.5.12-1ubuntu7.4_all.deb ...
Unpacking squid-common (3.5.12-1ubuntu7.4) over (3.5.12-1ubuntu7.3) ...
Processing triggers for systemd (229-4ubuntu19) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for ufw (0.35-0ubuntu2) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up squid-common (3.5.12-1ubuntu7.4) ...
Setting up squid (3.5.12-1ubuntu7.4) ...
Skipping profile in /etc/apparmor.d/disable: usr.sbin.squid

wget now returns the expected 502 error:
$ ftp_proxy=http://xenial-squid-passive-ftp-1560429.lxd:3128/ wget ftp://proftpd.lxd/readme.txt -O /dev/null -t1
--2017-08-03 10:44:58-- ftp://proftpd.lxd/readme.txt
Resolving xenial-squid-passive-ftp-1560429.lxd (xenial-squid-passive-ftp-1560429.lxd)... 10.0.100.226
Connecting to xenial-squid-passive-ftp-1560429.lxd (xenial-squid-passive-ftp-1560429.lxd)|10.0.100.226|:3128... connected.
Proxy request sent, awaiting response... 502 Bad Gateway
2017-08-03 10:44:58 ERROR 502: Bad Gateway.

And squid just logs the attempt, with no crash:
==> /var/log/squid/access.log <==
1501767898.437 14 10.0.100.1 TCP_MISS/502 4356 GET ftp://proftpd.lxd/readme.txt - HIER_DIRECT/10.0.100.227 text/html

If I allow passive ftp on proftpd, then the file can be fetched:
$ ftp_proxy=http://xenial-squid-passive-ftp-1560429.lxd:3128/ wget ftp://proftpd.lxd/readme.txt -O /dev/null -t1
--2017-08-03 10:47:06-- ftp://proftpd.lxd/readme.txt
Resolving xenial-squid-passive-ftp-1560429.lxd (xenial-squid-passive-ftp-1560429.lxd)... 10.0.100.226
Connecting to xenial-squid-passive-ftp-1560429.lxd (xenial-squid-pa...

Read more...

tags: added: verification-done-xenial
removed: verification-needed-xenial
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package squid3 - 3.5.12-1ubuntu7.4

---------------
squid3 (3.5.12-1ubuntu7.4) xenial; urgency=medium

  * debian/patches/passive-ftp-segfault-1560429.patch: Fix for segfault
    when ftp passive mode is not available. Closes: #793473, LP:
    #1560429.

 -- Andreas Hasenack <email address hidden> Fri, 07 Jul 2017 09:39:40 -0300

Changed in squid3 (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for squid3 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Athos Ribeiro (athos-ribeiro) wrote :

Standard support has ended for 14.04

Changed in squid3 (Ubuntu Trusty):
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.