Ubuntu
ganglia package

gmond crashes when an NFS mount is active

Bug #910678 reported by Dan Shick on 2012-01-02

132

This bug affects 24 people

	Status	Importance	Assigned to
ganglia (Ubuntu)	Fix Released	Undecided	Unassigned
Precise	Fix Released	Undecided	Reed Loden
Quantal	Fix Released	Undecided	Unassigned

Bug Description

[IMPACT]

* Current gmond crashes when an NFS mount is active due to stack corruption detection complaining about |
sscanf(procline, "%s %s %s %s ", device, mount, type, mode);| when used with too small buffers.

* Fixed three years ago upstream (https://github.com/ganglia/monitor-core/commit/883f169353bdd58df10f21fb4e78541f53bc0f17).

* Red Hat already backported a fix a year ago (https://bugzilla.redhat.com/show_bug.cgi?id=689483).

[TESTCASE]

* Make sure the ganglia moddisk.so module is loaded.

* Use NFS for some disk mount

* Start gmond (`gmond -f`)

* Ensure no crash for "stack smashing detected"

[Regression Potential]

* None known...

Previous description:

root@mymachine:/etc/init.d# lsb_release -rd
Description: Ubuntu 11.10
Release: 11.10

root@mymachine# apt-cache policy ganglia-monitor
ganglia-monitor:
  Installed: 3.1.7-2
  Candidate: 3.1.7-2
  Version table:
*** 3.1.7-2 0
        500 http://us.archive.ubuntu.com/ubuntu/ oneiric/universe i386 Packages
        100 /var/lib/dpkg/status

What I expected to happen: gmond should stay running regardless of whether I mount an NFS mount

What actually happened: gmond crashes when the system has an NFS mount active.

It looks like a manifestation of this behavior:

http://<email address hidden>/msg05118.html

With NFS mount:

root@mymachine# /usr/sbin/gmond -f *** stack smashing detected ***: /usr/sbin/gmond ======= Backtrace: =========
/lib/i386-linux-gnu/libc.so.linux-gnu/libc.so.ganglia/moddisk.ganglia/moddisk. ======= Memory map: ========
08048000-08055000 r-xp 00000000 08:06 659247 08055000-08056000 r--p 0000c000 08:06 659247 08056000-08057000 rw-p 0000d000 08:06 659247 08057000-08058000 rw-p 00000000 00:00 09495000-094b6000 rw-p 00000000 00:00 0 b72ad000-b72c9000 r-xp 00000000 08:06 1967034 b72c9000-b72ca000 r--p 0001b000 08:06 1967034 b72ca000-b72cb000 rw-p 0001c000 08:06 1967034 b72e1000-b72ec000 r-xp 00000000 08:06 1967054 b72ec000-b72ed000 r--p 0000a000 08:06 1967054 b72ed000-b72ee000 rw-p 0000b000 08:06 1967054 b72ee000-b72f8000 r-xp 00000000 08:06 1967058 b72f8000-b72f9000 r--p 00009000 08:06 1967058 b72f9000-b72fa000 rw-p 0000a000 08:06 1967058 b72fa000-b730f000 r-xp 00000000 08:06 1967048 b730f000-b7310000 r--p 00015000 08:06 1967048 b7310000-b7311000 rw-p 00016000 08:06 1967048 b7311000-b7313000 rw-p 00000000 00:00 b7321000-b7329000 rw-p 00000000 00:00 b7329000-b732f000 r-xp 00000000 08:06 787490 b732f000-b7330000 r--p 00005000 08:06 787490 b7330000-b7341000 rw-p 00006000 08:06 787490 b7341000-b7349000 rw-p 00000000 00:00 b7349000-b734f000 r-xp 00000000 08:06 787489 b734f000-b7350000 r--p 00005000 08:06 787489 b7350000-b7361000 rw-p 00006000 08:06 787489 b7361000-b7369000 rw-p 00000000 00:00 b7369000-b736f000 r-xp 00000000 08:06 787486 b736f000-b7370000 r--p 00005000 08:06 787486 b7370000-b7381000 rw-p 00006000 08:06 787486 b7381000-b7389000 rw-p 00000000 00:00 b7389000-b738f000 r-xp 00000000 08:06 787485 b738f000-b7390000 r--p 00005000 08:06 787485 b7390000-b73a1000 rw-p 00006000 08:06 787485 b73a1000-b73a9000 rw-p 00000000 00:00 b73a9000-b73af000 r-xp 00000000 08:06 787483 b73af000-b73b0000 r--p 00005000 08:06 787483 b73b0000-b73c1000 rw-p 00006000 08:06 787483 b73c1000-b73c9000 rw-p 00000000 00:00 b73c9000-b73cf000 r-xp 00000000 08:06 787484 b73cf000-b73d0000 r--p 00005000 08:06 787484 b73d0000-b73e1000 rw-p 00006000 08:06 787484 b73e1000-b73e9000 rw-p 00000000 00:00 b73e9000-b73ef000 r-xp 00000000 08:06 787482 b73ef000-b73f0000 r--p 00005000 08:06 787482 b73f0000-b7401000 rw-p 00006000 08:06 787482 b7401000-b740b000 rw-p 00000000 00:00 b740c000-b740d000 r--p 0043a000 08:06 662632 b740d000-b744d000 r--p 002bd000 08:06 662632 b744d000-b764d000 r--p 00000000 08:06 662632 b764d000-b764e000 rw-p 00000000 00:00 b764e000-b7651000 r-xp 00000000 08:06 1967024 b7651000-b7652000 r--p 00002000 08:06 1967024 b7652000-b7653000 rw-p 00003000 08:06 1967024 b7653000-b7657000 r-xp 00000000 08:06 1967092 b7657000-b7658000 r--p 00003000 08:06 1967092 b7658000-b7659000 rw-p 00004000 08:06 1967092 b7659000-b765a000 rw-p 00000000 00:00 b765a000-b7680000 r-xp 00000000 08:06 1967029 b7680000-b7681000 ---p 00026000 08:06 1967029 b7681000-b7683000 r--p 00026000 08:06 1967029 b7683000-b7684000 rw-p 00028000 08:06 1967029 b7684000-b77fa000 r-xp 00000000 08:06 1967013 b77fa000-b77fc000 r--p 00176000 08:06 1967013 b77fc000-b77fd000 rw-p 00178000 08:06 1967013 b77fd000-b7800000 rw-p 00000000 00:00 b7800000-b7817000 r-xp 00000000 08:06 1967073 b7817000-b7818000 r--p 00016000 08:06 1967073 b7818000-b7819000 rw-p 00017000 08:06 1967073 b7819000-b781b000 rw-p 00000000 00:00 b781b000-b784e000 r-xp 00000000 08:06 672170 b784e000-b784f000 r--p 00032000 08:06 672170 b784f000-b7850000 rw-p 00033000 08:06 672170 b7850000-b785a000 r-xp 00000000 08:06 672024 b785a000-b785b000 r--p 00009000 08:06 672024 b785b000-b785c000 rw-p 0000a000 08:06 672024 b785c000-b785d000 rw-p 00000000 00:00 b785d000-b789a000 r-xp 00000000 08:06 1967070 b789a000-b789b000 r--p 0003c000 08:06 1967070 b789b000-b789c000 rw-p 0003d000 08:06 1967070 b789c000-b78ae000 r-xp 00000000 08:06 668473 b78ae000-b78af000 r--p 00011000 08:06 668473 b78af000-b78b1000 rw-p 00012000 08:06 668473 b78b1000-b78b9000 r-xp 00000000 08:06 1967050 b78b9000-b78ba000 r--p 00007000 08:06 1967050 b78ba000-b78bb000 rw-p 00008000 08:06 1967050 b78bb000-b78c9000 rw-p 00000000 00:00 b78c9000-b78ca000 r-xp 00000000 00:00 0 b78ca000-b78e8000 r-xp 00000000 08:06 1967000 b78e8000-b78e9000 r--p 0001d000 08:06 1967000 b78e9000-b78ea000 rw-p 0001e000 08:06 1967000 bfa4b000-bfa6c000 rw-p 00000000 00:00 0 Aborted -c /etc/ganglia/gmond.conf
terminated
/>6(__fortify_fail+0x45)[0xb776b8d5]
/>6(+0xe7887)[0xb776b887]
/>so(+0x4844)[0xb73cd844]
/>so(+0x473c)[0xb73cd73c]
/usr/sbin/gmond
/usr/sbin/gmond
/usr/sbin/gmond
0
[heap]
/lib/i386-linux-gnu/libgcc_s.so.1
/lib/i386-linux-gnu/libgcc_s.so.1
/lib/i386-linux-gnu/libgcc_s.so.1
/lib/i386-linux-gnu/libnss_files-2.13.so
/lib/i386-linux-gnu/libnss_files-2.13.so
/lib/i386-linux-gnu/libnss_files-2.13.so
/lib/i386-linux-gnu/libnss_nis-2.13.so
/lib/i386-linux-gnu/libnss_nis-2.13.so
/lib/i386-linux-gnu/libnss_nis-2.13.so
/lib/i386-linux-gnu/libnsl-2.13.so
/lib/i386-linux-gnu/libnsl-2.13.so
/lib/i386-linux-gnu/libnsl-2.13.so
0
0
/usr/lib/ganglia/modsys.so
/usr/lib/ganglia/modsys.so
/usr/lib/ganglia/modsys.so
0
/usr/lib/ganglia/modproc.so
/usr/lib/ganglia/modproc.so
/usr/lib/ganglia/modproc.so
0
/usr/lib/ganglia/modnet.so
/usr/lib/ganglia/modnet.so
/usr/lib/ganglia/modnet.so
0
/usr/lib/ganglia/modmem.so
/usr/lib/ganglia/modmem.so
/usr/lib/ganglia/modmem.so
0
/usr/lib/ganglia/modload.so
/usr/lib/ganglia/modload.so
/usr/lib/ganglia/modload.so
0
/usr/lib/ganglia/moddisk.so
/usr/lib/ganglia/moddisk.so
/usr/lib/ganglia/moddisk.so
0
/usr/lib/ganglia/modcpu.so
/usr/lib/ganglia/modcpu.so
/usr/lib/ganglia/modcpu.so
0
/usr/lib/locale/locale-archive
/usr/lib/locale/locale-archive
/usr/lib/locale/locale-archive
0
/lib/i386-linux-gnu/libdl-2.13.so
/lib/i386-linux-gnu/libdl-2.13.so
/lib/i386-linux-gnu/libdl-2.13.so
/lib/i386-linux-gnu/libuuid.so.1.3.0
/lib/i386-linux-gnu/libuuid.so.1.3.0
/lib/i386-linux-gnu/libuuid.so.1.3.0
0
/lib/i386-linux-gnu/libexpat.so.1.5.2
/lib/i386-linux-gnu/libexpat.so.1.5.2
/lib/i386-linux-gnu/libexpat.so.1.5.2
/lib/i386-linux-gnu/libexpat.so.1.5.2
/lib/i386-linux-gnu/libc-2.13.so
/lib/i386-linux-gnu/libc-2.13.so
/lib/i386-linux-gnu/libc-2.13.so
0
/lib/i386-linux-gnu/libpthread-2.13.so
/lib/i386-linux-gnu/libpthread-2.13.so
/lib/i386-linux-gnu/libpthread-2.13.so
0
/usr/lib/libapr-1.so.0.4.5
/usr/lib/libapr-1.so.0.4.5
/usr/lib/libapr-1.so.0.4.5
/usr/lib/i386-linux-gnu/libconfuse.so.0.0.0
/usr/lib/i386-linux-gnu/libconfuse.so.0.0.0
/usr/lib/i386-linux-gnu/libconfuse.so.0.0.0
0
/lib/i386-linux-gnu/libpcre.so.3.12.1
/lib/i386-linux-gnu/libpcre.so.3.12.1
/lib/i386-linux-gnu/libpcre.so.3.12.1
/usr/lib/libganglia-3.1.7.so.0.0.0
/usr/lib/libganglia-3.1.7.so.0.0.0
/usr/lib/libganglia-3.1.7.so.0.0.0
/lib/i386-linux-gnu/libnss_compat-2.13.so
/lib/i386-linux-gnu/libnss_compat-2.13.so
/lib/i386-linux-gnu/libnss_compat-2.13.so
0
[vdso]
/lib/i386-linux-gnu/ld-2.13.so
/lib/i386-linux-gnu/ld-2.13.so
/lib/i386-linux-gnu/ld-2.13.so
[stack]

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: ganglia-monitor 3.1.7-2 [modified: usr/share/doc/ganglia-monitor/changelog.Debian.gz]
ProcVersionSignature: Ubuntu 3.0.0-14.23-generic-pae 3.0.9
Uname: Linux 3.0.0-14-generic-pae i686
ApportVersion: 1.23-0ubuntu4
Architecture: i386
Date: Sun Jan 1 16:45:03 2012
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release i386 (20111012)
ProcEnviron:
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: ganglia
UpgradeStatus: No upgrade log present (probably fresh install)

See original description

Tags:

Related branches

lp:ubuntu/precise-proposed/ganglia

Revision history for this message

Dan Shick (ubuntu-eleven) wrote on 2012-01-02:

Dependencies.txt Edit (1.0 KiB, text/plain; charset="utf-8")

Revision history for this message

Launchpad Janitor (janitor) wrote on 2012-01-18:

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ganglia (Ubuntu):
status:	New → Confirmed

Revision history for this message

Chris Adamson (addo81) wrote on 2012-03-23:

A fix for this bug was committed to Fedora

https://bugzilla.redhat.com/show_bug.cgi?id=689483

libmetrics/linux/metrics.c

line 1231:

char mount[128], device[128], type[32], mode[128];

change this to

char mount[2048], device[2048], type[32], mode[2048];

in order to accommodate the longer strings in /proc/mounts for NFS and automounts

Revision history for this message

teonghan (teonghan) wrote on 2012-04-17:

Hi all,

If I did an "apt-get upgrade" or specifically, apt-get install libc6, ganglia-monitor (gmond) can be started without problem, even if there is a NFS partition. Can anyone verify this?

Thanks.

Revision history for this message

teonghan (teonghan) wrote on 2012-04-17:

Hi all,

Sorry, should have stated that the version of libc6 I am using is 2.13-20ubuntu5.1.

Thanks.

Revision history for this message

dab (dabosuke) wrote on 2012-04-30:

I got the same problem with Ubuntu 12.04.

Revision history for this message

Teppo Sulavuori (sulate) wrote on 2012-05-15:

This has been fixed in upstream since version 3.2.0

https://github.com/ganglia/monitor-core/commit/883f169353bdd58df10f21fb4e78541f53bc0f17

(I'm a bit sad that Precise being an LTS version did not get a newer version of ganglia-monitor since current version 3.1.7 is already over two years old.)

Anthony Costa (anthony-costa) on 2012-07-17

description:

updated

Revision history for this message

Mark Asbach (mark-asbach) wrote on 2012-08-09:

A quick fix for users not wanting to wait for an update or build gmond manually is to change the gmond.conf to not load the disks module.

Comment out the respective sections of /etc/ganglia/gmond.conf as follows:

  module {
    name = "cpu_module"
    path = "/usr/lib/ganglia/modcpu.so"
  }
/*
  module {
    name = "disk_module"
    path = "/usr/lib/ganglia/moddisk.so"
  }
*/
  module {
    name = "load_module"
    path = "/usr/lib/ganglia/modload.so"
  }

[...]

/*
collection_group {
  collect_every = 1800
  time_threshold = 3600
  metric {
    name = "disk_total"
    value_threshold = 1.0
    title = "Total Disk Space"
  }
}

collection_group {
  collect_every = 40
  time_threshold = 180
  metric {
    name = "disk_free"
    value_threshold = 1.0
    title = "Disk Space Available"
  }
  metric {
    name = "part_max_used"
    value_threshold = 1.0
    title = "Maximum Disk Space Used"
  }
}

Revision history for this message

Niklas Edmundsson (niklas-edmundsson) wrote on 2012-08-24:

This should be fixed ASAP, Ganglia is an important component in many server environments which is the target of LTS releases.

Revision history for this message

Mike Johnson (mikejohnson) wrote on 2012-08-24:

#10

I'm having this problem in 12.04 as well.

Reed Loden (reed) on 2012-09-18

description:

updated

Revision history for this message

Exilist (sandro-grundmann) wrote on 2012-10-11:

#11

i can confirm this, ganglia/gmond is not starting on 12.04 and nfs-mounts

Revision history for this message

Trevor Robinson (scurrilous) wrote on 2012-10-15:

#12

I hit this issue as well. My workaround was to install the ganglia packages from quantal, which picked up a substantially newer version that includes the upstream fix.

IMHO, the quantal packages should be backported to precise, given its status as an LTS release. Ganglia is widely used to monitor server apps like Hadoop, and those environments often use NFS.

Revision history for this message

Reed Loden (reed) wrote on 2012-10-15:

#13

quantal has 3.3.8, so it's not affected by this issue.

Changed in ganglia (Ubuntu Quantal):
status:	Confirmed → Fix Released

Revision history for this message

Reed Loden (reed) wrote on 2012-10-15:

#14

debdiff Edit (2.3 KiB, text/plain)

My attempt at a debdiff for this fix, but I don't think I did it right. :(

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2012-10-15:

#15

Thanks for the debdiff! There are a few things that need updating:
* this should target precise-proposed, not precise
* the version should be 3.1.7-2ubuntu1.1 as per https://wiki.ubuntu.com/StableReleaseUpdates
* there is an undocumented change to debian/source/options. I see why you want to do this, but it doesn't seem to work because of the upstream Debian changes in debian-changes-3.1.7-2 that add config.guess and config.sub. I've updated debian-changes-3.1.7-2 to not add config.guess and config.sub and this seems to work.

Since the first two were simple and the third less so, I went ahead an made these changes and uploaded to precise-proposed.

Revision history for this message

Jamie Strandboge (jdstrand) wrote on 2012-10-15:

#16

Unsubscribing ubuntu-sponsors.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2012-11-14:

#17

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ganglia (Ubuntu Precise):
status:	New → Confirmed

Revision history for this message

Clint Byrum (clint-fewbar) wrote on 2012-11-14: Please test proposed package

#18

Hello Dan, or anyone else affected,

Accepted ganglia into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/ganglia/3.1.7-2ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in ganglia (Ubuntu Precise):
assignee:	nobody → Reed Loden (reed)
status:	Confirmed → In Progress
status:	In Progress → Fix Committed
tags:	added: verification-needed

Revision history for this message

Robert Dyer (psybers) wrote on 2012-11-15:

#19

Updated to 3.1.7-2ubuntu1.1 from precise-proposed. Re-enabled the disk module. Ganglia now starts up instead of segfaulting and appears to be working normal.

tags:

added: verification-done
removed: verification-needed

Revision history for this message

Joseph Holsten (i4-joseph) wrote on 2012-11-19:

#20

This is preventing my company from upgrading from 10.04 to 12.04. Thanks for working on this.

Revision history for this message

Joseph Holsten (i4-joseph) wrote on 2012-11-24:

#21

Confirming that the package in -proposed also resolved the issue for us.

Revision history for this message

Scott Kitterman (kitterman) wrote on 2012-11-26: Update Released

#22

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2012-11-26:

#23

This bug was fixed in the package ganglia - 3.1.7-2ubuntu1.1

---------------
ganglia (3.1.7-2ubuntu1.1) precise-proposed; urgency=low

  [ Reed Loden ]
  * Fix buffer overflow in moddisk.so. (LP: #910678)
  * debian/source/options: ignore autogenerated files

  [ Jamie Strandboge ]
  * debian/patches/debian-changes-3.1.7-2: remove addition of config.guess
    and config.sub
-- Reed Loden <email address hidden> Mon, 15 Oct 2012 11:22:10 -0700