libc on 2016-05-25 causes Apache not to restart, libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Bug #1585614 reported by Michael Lueck
374
This bug affects 22 people
Affects Status Importance Assigned to Milestone
eglibc (Ubuntu)
Fix Released
Critical
Unassigned
glibc (Ubuntu)
Fix Released
Critical
Unassigned

Bug Description

This morning I applied the following updates to Ubuntu 12.04 webservers:

The following packages are currently pending an upgrade:

 apt 0.8.16~exp12ubuntu10.27
 apt-transport-https 0.8.16~exp12ubuntu10.27
 apt-utils 0.8.16~exp12ubuntu10.27
 libapache2-mod-php5 5.3.10-1ubuntu3.23
 libapt-inst1.4 0.8.16~exp12ubuntu10.27
 libapt-pkg4.12 0.8.16~exp12ubuntu10.27
 php5-cli 5.3.10-1ubuntu3.23
 php5-common 5.3.10-1ubuntu3.23
 php5-curl 5.3.10-1ubuntu3.23
 php5-dev 5.3.10-1ubuntu3.23
 php5-gd 5.3.10-1ubuntu3.23
 php5-mysql 5.3.10-1ubuntu3.23
 php-pear 5.3.10-1ubuntu3.23

Apache fails to restart, siting error:

apache2: Syntax error on line 212 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/php5.load: Cannot load /usr/lib/apache2/modules/libphp5.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Into file /var/log/apache2/error.log

Looks like I am trying to roll back all those php packages... Which I see several updates marked Security Update in the changelogs, thus checking the "security vulnerability" box as rolling back in this case is such. :-(

Revision history for this message
Michael Lueck (mlueck) wrote :

Rollback successful on all 12.04 servers to PHP build 5.3.10-1ubuntu3. Websites are again working.

description: updated
information type: Private Security → Public Security
Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

Could you please attach your apt upgrade log?

Revision history for this message
Michael Lueck (mlueck) wrote :

APT history.log

Revision history for this message
Michael Lueck (mlueck) wrote :

APT term.log

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in php5 (Ubuntu):
status: New → Confirmed
Revision history for this message
pepo (pepoviola) wrote :

This bug also affect to my servers, after the upgrade I see this error in apache

[Wed May 25 06:42:23 2016] [notice] Graceful restart requested, doing restart
apache2: Syntax error on line 211 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/php5.load: Cannot load /usr/lib/apache2/modules/libphp5.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Upgrade log

Start-Date: 2016-05-25 06:42:18
Upgrade: libc-bin:amd64 (2.15-0ubuntu10.13, 2.15-0ubuntu10.14), php5:amd64 (5.3.10-1ubuntu3.22, 5.3.10-1ubuntu3.23), libapache2-mod-php5:amd64 (5.3.10-1ubuntu3.22, 5.3.10-1ubuntu3.23), php5-curl:amd64 (5.3.10-1ubuntu3.22, 5.3.10-1ubuntu3.23), multiarch-support:amd64 (2.15-0ubuntu10.13, 2.15-0ubuntu10.14), libc6-dev:amd64 (2.15-0ubuntu10.13, 2.15-0ubuntu10.14), php5-mysql:amd64 (5.3.10-1ubuntu3.22, 5.3.10-1ubuntu3.23), php5-cli:amd64 (5.3.10-1ubuntu3.22, 5.3.10-1ubuntu3.23), libc-dev-bin:amd64 (2.15-0ubuntu10.13, 2.15-0ubuntu10.14), libc6:amd64 (2.15-0ubuntu10.13, 2.15-0ubuntu10.14), php5-common:amd64 (5.3.10-1ubuntu3.22, 5.3.10-1ubuntu3.23)
End-Date: 2016-05-25 06:42:23

---
But, in my case after manually restart it's works again.

Thx!

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

Ah, yes, looks like a manual restart of apache is necessary.

Revision history for this message
Jon Bach (jonbach) wrote :

Just adding another data point. Three web servers all went down for me about half hour ago. All 3 Ubuntu 12.04 + Apache2 + PHP5, with security updates auto-applying. All I had to do was start apache again.

Server 1 error.log:
[Wed May 25 06:43:08 2016] [notice] Graceful restart requested, doing restart
apache2: Syntax error on line 210 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/php5.load: Cannot load /usr/lib/apache2/modules/libphp5.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Server 2 error.log:
[Wed May 25 06:48:32 2016] [notice] Graceful restart requested, doing restart
apache2: Syntax error on line 211 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/auth_mysql.load: Cannot load /usr/lib/apache2/modules/mod_auth_mysql.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Revision history for this message
musashiXXX (musashi-nefaria) wrote :

Same thing happened to me this morning. A manual restart of apache fixed the problem though.

Revision history for this message
musashiXXX (musashi-nefaria) wrote :

Also, here's my term.log

Revision history for this message
musashiXXX (musashi-nefaria) wrote :

This seems to have affected perl as well, not just PHP. Here's a log snippet from apache's error.log:

apache2: Syntax error on line 210 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/perl.load: Cannot load /usr/lib/apache2/modules/mod_perl.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Again, a manual restart of Apache _seems_ to have fixed the problem.

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

I suspect this is caused by the eglibc update, not the php5 update. Reassigning bug.

affects: php5 (Ubuntu) → eglibc (Ubuntu)
Revision history for this message
Michael Lueck (mlueck) wrote :

For our servers (12.04) manually restarting the Apache service would cause the same error. Apache would not manually restart.

Revision history for this message
Joi Owen (jlellis) wrote :

This change in libm.so has also broken the pam_mysql.so library, thus my vsftpd service is also broken. My nagios started alarming about this breaking around 6:30 am this morning, dpkg.log shows only 5 packages were updated in this morning's automatic update: man-db, libc-bin, libc6, and multiarch-support (all from amd64 arch.)

The error in auth.log is:

vsftpd: PAM unable to dlopen(pam_mysql.so): /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference.
vsftpd: PAM adding faulty module: pam_mysql.so

(I had to manually type the above, as MS hyperv can't do simple clipboard operations like Xen has managed for over a decade now. So please forgive any typos that might be there.)

I don't think this is an apache or pam bug, as libm.so is provided by libc6.

Revision history for this message
Adam Conrad (adconrad) wrote :

@jlellis Does hard restarting vsftpd clear up the issue?

Revision history for this message
Joi Owen (jlellis) wrote :

@adconrad As it happens, it did. I just came back to update my comment to say so and found your question.

This suggests the libc6 update (not glibc, my server doesn't have glibc package installed) should have required a reboot? I don't recall if the server mentioned a reboot was required in the motd.

Revision history for this message
Trevor Bradley (ck-trevor) wrote :

Just a quick note that this bug affected both my Ubuntu 12.04 and 14.04 web servers today. My 16.04 servers appear to be unaffected.

Revision history for this message
BrandonTomlinson (druke) wrote :

We had around 60 12.04 servers impacted today, no 14.04 or newer web servers.

Specifically it looks like an update race condition where libapache2-mod-php (which restarts/reloads apache) is updated before the libc6 package.

We could prove this theory by doing 'apt-get install libc6 && apt-get update' to see, but I don't have the ability to test this myself.

Revision history for this message
BrandonTomlinson (druke) wrote :

no 14.04/16.04 servers were impacted, I should say.

Revision history for this message
Trevor Bradley (ck-trevor) wrote :

Verified. I swore one of my affected servers was 14.04, but it was actually 12.04.

Revision history for this message
Steve Beattie (sbeattie) wrote :

Yes, my apologies, the upstream libc fixes for CVE-2014-9761 did some reworking of functions to eliminate some repeated vulnerable code, using internal functions to do the work instead. Unfortunately, this did introduce new function references between libc and libm, causing the problems seen above. Unfortunately, these changes were applied to libc in Ubuntu 14.04 LTS and Ubuntu 15.10 as well, so I'm surprised the same problem was not seen there, too. There was no update for Ubuntu 16.04 LTS, so no issues should be seen there.

Joi: yes, the reboot motd notification was triggered with this update (and will be for future libc updates). That doesn't help you if you can't log in to see it. :(

summary: - PHP Update on 2016-05-25 causes Apache not to restart, libm.so.6: symbol
+ libc on 2016-05-25 causes Apache not to restart, libm.so.6: symbol
__strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with
link time reference
Revision history for this message
Andreas "Jimmy" Gredler (jimmyrax) wrote :

Steve: I think it does not happen with 14.04 because Apache is not restarted during the upgrade process. Not sure why but that's what my tests with 14.04 showed me.
My tests with 12.04 are not 100% reproducible which makes me wonder if there is a race condition in the postinst script of libapache2-mod-php5. Is there a chance that Apache is reloaded before ldconfig is completely finished? That would also explain why a simple restart of Apache solves the problem.
This is how I test:
# apt-get install libc6=2.15-0ubuntu10.13 libc-bin=2.15-0ubuntu10.13 libapache2-mod-php5=5.3.10-1ubuntu3 php5-common=5.3.10-1ubuntu3
# apt-get upgrade

In some cases Apache does not restart successfully, in some it does.

Revision history for this message
Anders Sandblad (arune) wrote :

I just had this on 14.04 this morning due to logrotate restarting apache. Manually starting apache worked.
This are the last lines in apache error.log.1:
[Thu May 26 06:45:09.064664 2016] [mpm_prefork:notice] [pid 62472] AH00171: Graceful restart requested, doing restart
apache2: Syntax error on line 140 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/php5.load: Cannot load /usr/lib/apache2/modules/libphp5.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Revision history for this message
Stephen Cox (stephen-cox) wrote :

This has affected one of my 14.04 web servers and a couple of 12.04, out of 40 Ubuntu servers.

All show the following in the Apache error log:
[Thu May 26 02:29:08.076578 2016] [mpm_prefork:notice] [pid 2199] AH00171: Graceful restart requested, doing restart
apache2: Syntax error on line 191 of /etc/apache2/apache2.conf: Syntax error on line 1 of /etc/apache2/mods-enabled/php5.load: Cannot load /usr/lib/apache2/modules/libphp5.so into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Manually restarting Apache fixed the problem.

Revision history for this message
Blinker (blinker1985) wrote :

This affects several Ubuntu 12.04 servers of mine as well.
I had the Apache error as stated above but also all the Cron services aren't running anymore.

The following error message is in dmesg:
[9065273.669402] cron[10944]: segfault at 968 ip 00007f29c2ba28f0 sp 00007fff2242f8a0 error 4 in libpthread-2.15.so[7f29c2b9c000+18000]

A restart of the Cron daemon fixes this.

Revision history for this message
William Grant (wgrant) wrote :

I've unpublished the affected libc6 versions (2.15-0ubuntu10.14, 2.19-0ubuntu6.8, 2.21-0ubuntu4.2).

For machines that have already upgraded, restarting affected processes or rebooting the whole system should resolve any problems.

Changed in eglibc (Ubuntu):
importance: Undecided → Critical
status: Confirmed → In Progress
Changed in glibc (Ubuntu):
importance: Undecided → Critical
status: New → In Progress
Revision history for this message
William Grant (wgrant) wrote :

Blinker, do those servers with crashed cron daemons use any unusual PAM modules?

Revision history for this message
Michael Lueck (mlueck) wrote :

@Marc #12, perhaps so it was actually eglibc package. I tested reapplying the PHP updates, IPL, and Apache stayed up. Evidently I missed that eglibc and the subsequently required IPL.

We are now good with the latest packages for 12.04. Phew! :-)

Revision history for this message
Timur Irmatov (irmatov) wrote :

It definitely affects 14.04. I had to restart apache2 on several 14.04 installations today.

Revision history for this message
Blinker (blinker1985) wrote :

@William #27. It is a virtual machine on VMWare with Plesk 12.5 installed.

I checked the /etc/pam.d folder and found a custom plesk librarie:

auth sufficient pam_plesk.so try_first_pass

Thus far all the machines with Cron daemon problems were Ubuntu 12.04 machines in combination with Plesk. So, yes. ;-)

Revision history for this message
B. (b-deactivatedaccount-deactivatedaccount) wrote :

Same issue

This morning, a security update for libc has been automatically
deployed on all our servers running 14.04 LTS.

unattended-upgrades-dpkg_2016-05-26_06:36:05.829399.log:Unpacking libc6-dev:amd64
unattended-upgrades-dpkg_2016-05-26_06:36:05.829399.log:Unpacking libc-dev-bin
unattended-upgrades-dpkg_2016-05-26_06:36:05.829399.log:Unpacking libc-bin
unattended-upgrades-dpkg_2016-05-26_06:36:05.829399.log:Unpacking libc6:amd64
unattended-upgrades-dpkg_2016-05-26_06:36:05.829399.log:Unpacking multiarch-support

On one of our web servers (only one??) Apache (2.4.7-1ubuntu4.9) has stopped working
after the unattended upgrades of libc6. When Apache received the signal SIGUSR1.

This did not impact any other web server with identical version of all softwares
(aligned with Ansible) ?! -- at least in the beginning of the day...

Later on at 12:52 we did "apachectl graceful" on all our web servers and all apache servers stopped responding until we restart them with "service apache2 restart".

[Thu May 26 06:35:36.222660 2016] [mpm_worker:notice] [pid 6318:tid 140737354041216]
AH00297: SIGUSR1 received. Doing graceful restart

[Thu May 26 06:35:36.231321 2016] [:alert] [pid 16293:tid 140737354041216]
(4)Interrupted system call: FastCGI: read() from pipe failed (0)

[Thu May 26 06:35:36.231395 2016] [:alert] [pid 16293:tid 140737354041216]
(4)Interrupted system call: FastCGI: the PM is shutting down, Apache seems to have disappeared - bye
apache2: Syntax error on line 140 of /etc/apache2/apache2.conf: Syntax error on line 2 of /etc/apache2/mods-enabled/security2.load: Cannot load libxml2.so.2 into server: /lib/x86_64-linux-gnu/libm.so.6: symbol __strtold_nan, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Revision history for this message
Steve Beattie (sbeattie) wrote :

Blinker (and anyone else), I have eglibc/glibc packages available in the ubuntu-security-proposed ppa https://launchpad.net/~ubuntu-security-proposed/+archive/ubuntu/ppa that revert the problematic fix that added references from the libm library to the new symbols, while keeping the added __strtol*_nan symbols in libc (so as not to break people who have already updated to the pulled packages). It'd be useful to know if this also addresses the cron/Plesk issue as well.

I've reproduced apache failing to soft restart after updating to the broken libc packages, and have verified that updating to the packages in the security-proposed ppa, a currently running apache2 will soft restart successfully when the upgrade is performed from both the restored packages and from the broken packages. But verification from others is appreciated as well.

(Note that the security-proposed ppa often gets packages for testing, so it's best not to leave it enabled after testing these specific libc packages.)

Thanks for your patience, and again, my apologies.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package eglibc - 2.15-0ubuntu10.15

---------------
eglibc (2.15-0ubuntu10.15) precise-security; urgency=medium

  * REGRESSION UPDATE: revert CVE-2014-9761 fix due to added symbol
    dependency from libm to libc (LP: #1585614)
    - debian/patches/any/CVE-2014-9761-2.diff: keep exporting
      __strto*_nan symbols added to libc.

 -- Steve Beattie <email address hidden> Thu, 26 May 2016 00:08:17 -0700

Changed in eglibc (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package glibc - 2.21-0ubuntu4.3

---------------
glibc (2.21-0ubuntu4.3) wily-security; urgency=medium

  * REGRESSION UPDATE: revert CVE-2014-9761 fix due to added symbol
    dependency from libm to libc (LP: #1585614)
    - debian/patches/any/CVE-2014-9761-2.diff: keep exporting
      __strto*_nan symbols added to libc.

 -- Steve Beattie <email address hidden> Thu, 26 May 2016 01:28:23 -0700

Changed in glibc (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.