Binutils corrupts Open MPI

On Sun, Jun 15, 2008 at 07:30:25PM -0000, Cesare Tirabassi wrote:
> ** Also affects: openmpi (Ubuntu)
> Importance: Undecided
> Status: New
>
> ** Changed in: binutils (Ubuntu)
> Status: New => Invalid
>
> ** Changed in: binutils (Ubuntu Hardy)
> Status: New => Invalid

Care to explain?

As my bug report stated, the _only_ change was the Gutsy/Hardy upgrade
as I compiled the _same sources_.

This is no Open MPI bug. This is a toolchain bug that happens to break Open MPI.

Dirk

>
> --
> Binutils corrupts Open MPI
> https://bugs.launchpad.net/bugs/234837
> You received this bug notification because you are a direct subscriber
> of the bug.

--
Three out of two people have difficulties with fractions.

I can add that the problem has been introduced somewhere
within the hardy development time. We have a machine
that has been installed around february/march with a then-current hardy.
This includes a working OpenMPI package. The update to the released hardy
broke openMPI.

Yours,
Steffen

(Whose bugs 210273 and 224706 have been marked a duplicate of this).
So if this bug is "invalid" I'd like to know if someone is working on any of
the other two.

Yours,
Steffen

On 15 June 2008 at 19:53, Steffen Neumann wrote:
| I can add that the problem has been introduced somewhere
| within the hardy development time. We have a machine
| that has been installed around february/march with a then-current hardy.
| This includes a working OpenMPI package. The update to the released hardy
| broke openMPI.

Yes, exactly what I observerd too.

| Yours,
| Steffen
|
| (Whose bugs 210273 and 224706 have been marked a duplicate of this).
| So if this bug is "invalid" I'd like to know if someone is working on any of
| the other two.

I am as stunned by this as Steffen and am eagerly awaiting an explanation.

Dirk
who happens to co-maintain Open MPI in Debian and suspects that Open MPI is
not at fault here but rather how the package was built for Hardy / whatever
changed for the compilers / linkers / ... in hardy as _the same source
package built on Gutsy_ works for me.

| Yours,
| Steffen
|
| --
| Binutils corrupts Open MPI
| https://bugs.launchpad.net/bugs/234837
| You received this bug notification because you are a direct subscriber
| of the bug.

--
Three out of two people have difficulties with fractions.

I still have not received any answer to my emails (Friday 13 June @12:22:46 CEST and Sunday 15 June @21:25:20 CEST).
In summary, this is apparently caused by compiling openmpi with -Wl,-Bsymbolic-functions (default from latest dpkg-buildpackage).
This could possibly be due to any of the three libopenmpi1 different interrelated libraries sharing symbols?
Can you and/or Manuel comment on this before we release a fix?

Cesare,

Thanks for your mail!

On 15 June 2008 at 20:53, Cesare Tirabassi wrote:
| I still have not received any answer to my emails (Friday 13 June @12:22:46 CEST and Sunday 15 June @21:25:20 CEST).

I replied to your email from Friday on Friday right when I got, ie before
leaving for work:

  From: Dirk Eddelbuettel <email address hidden>
  To: Cesare Tirabassi <email address hidden>
  Subject: Re: SUSPECT: Re: [Rd] Rmpi segfault after install on Ubuntu Hardy Heron
  Date: Fri, 13 Jun 2008 07:27:37 -0500

and exim4 had no issues as far as I can see

  2008-06-13 07:27:37 1K78NV-00078e-Jd <= <email address hidden> U=edd P=local S=1869 <email address hidden>
  2008-06-13 07:27:39 1K78NV-00078e-Jd => <email address hidden> R=smarthost T=remote_smtp_smarthost H=smtp.g.comcast.net [76.96.30.117] X=TLS1.0:RSA_AES_256_CBC_SHA1:32 DN="C=US,ST=Pennsylvania,L=Philadelphia,O=Comcast Cable Communications Management LLC,OU=Business Center,CN=smtp.comcast.net"
  2008-06-13 07:27:39 1K78NV-00078e-Jd Completed

Did you get that or not?

Today I only received (automated) Launchpad messages from you but no direct
mail. I did send you one given that I had not heard from you.

Looks like there may be email troubles at your end? Just to be sure, I added
both you email addresses as CC.

| In summary, this is apparently caused by compiling openmpi with -Wl,-Bsymbolic-functions (default from latest dpkg-buildpackage).

If it is dpkg-buildpackages, it would 'hit' us too as we're building in
unstable. So I think it may be something else.

Could it be that Ubuntu has a different GCC default in stripping symbols or
something? The first person I talked to about this was doko but is always so
overwhelmed that he told me to please file a bug report which I did.

Which promptly got ignored by everybody.

| This could possibly be due to any of the three libopenmpi1 different interrelated libraries sharing symbols?

That was my immediate gut reaction as well. But that is a design choice Open
MPI made upstream, and "we" (as in Debian's Open MPI maintainer) do not fight
it.

*If* you guys changed build options, I'd start by reverting to what we do.

But if you guys changed nothing, well then I am at a loss.

| Can you and/or Manuel comment on this before we release a fix?

[ CCing Manuel who you forgot to CC. ]

Sure. Let's hash this out for a moment to get it right. Manuel may want to
comment as I may well have overlooked or forgotten something.

Dirk

| ** Changed in: openmpi (Ubuntu Intrepid)
| Status: New => Incomplete
|
| --
| Binutils corrupts Open MPI
| https://bugs.launchpad.net/bugs/234837
| You received this bug notification because you are a direct subscriber
| of the bug.

--
Three out of two people have difficulties with fractions.

Well, by looking at your buildd build log your default LDFLAGS is "" while we do use -Wl,-Bsymbolic. I guess you never received my emails as I was explaining that in details. Anyhow, compiling unexporting LDFLAGS in debian/rules works as expected:

root@norsetto:/root/debian# echo 'library(Rmpi); cat("Still alive\n")' |
R --slave
[norsetto:13561] mca: base: components_open: component timer / linux open
function failed
[norsetto:13561] mca: base: component_find: unable to open osc pt2pt: file not
found (ignored)
libibverbs: Fatal: couldn't read uverbs ABI version.
--------------------------------------------------------------------------
[0,0,0]: OpenIB on host norsetto was unable to find any HCAs.
Another transport will be used instead, although this may result in
lower performance.
--------------------------------------------------------------------------
Still alive

The above is with the patched and rebuilt 1.2.5-1.

You seem to confirm that upstream decided to share symbols between libraries, it is therefore a design choice and not a bug; I'm therefore happy to release this fix for intrepid right now. For hardy we need an sru, for which we need cooperation from you and possibly others since we need at least 3 persons to test and confirm this works (and doesn't break anything else) before we can make it official.

I'll work on this first thing tomorrow morning.

Just back in from a long run...

On 15 June 2008 at 22:49, Cesare Tirabassi wrote:
| Well, by looking at your buildd build log your default LDFLAGS is ""
| while we do use -Wl,-Bsymbolic.

That'll do to break the package, I suppose!

| I guess you never received my emails as
| I was explaining that in details.

That email lossage is a side issue we should take up on the side. Any idea
why I am not getting yours / your not getting mine?

| Anyhow, compiling unexporting LDFLAGS
| in debian/rules works as expected:
|
| root@norsetto:/root/debian# echo 'library(Rmpi); cat("Still alive\n")' |
| R --slave
| [norsetto:13561] mca: base: components_open: component timer / linux open
| function failed
| [norsetto:13561] mca: base: component_find: unable to open osc pt2pt: file not
| found (ignored)
| libibverbs: Fatal: couldn't read uverbs ABI version.
| --------------------------------------------------------------------------
| [0,0,0]: OpenIB on host norsetto was unable to find any HCAs.
| Another transport will be used instead, although this may result in
| lower performance.
| --------------------------------------------------------------------------
| Still alive

Perfect. And if you uncomment the line saying "btl = ^openib" in
/etc/openmpi/openmpi-mca-params.conf as in

  # Disable the use of InfiniBand
  # btl = ^openib
  btl = ^openib

you will suppress the noise telling you that you have no IB hardware... This
has now been improved upstream, by the way.

| The above is with the patched and rebuilt 1.2.5-1.

Yes, look good! Thanks for that!

| You seem to confirm that upstream decided to share symbols between
| libraries, it is therefore a design choice and not a bug; I'm therefore
| happy to release this fix for intrepid right now. For hardy we need an
| sru, for which we need cooperation from you and possibly others since we
| need at least 3 persons to test and confirm this works (and doesn't
| break anything else) before we can make it official.

Let me know how I can help. As I said, same code / same everything but built
on Gutsy (ie without -Wl,-Bsymbolic) works.

| I'll work on this first thing tomorrow morning.

Much appreciated. This seems to have bitten a few people, and is too obscure
for most to figure out by themselves.

Dirk

| --
| Binutils corrupts Open MPI
| https://bugs.launchpad.net/bugs/234837
| You received this bug notification because you are a direct subscriber
| of the bug.

--
Three out of two people have difficulties with fractions.

I reported the bug marked as a duplicate of this (https://bugs.launchpad.net/ubuntu/+source/openmpi/+bug/224706), and have had problems with using Python on top of openmpi. We have been building our own packages for openmpi based on Debian. A fix for Hardy would be highly appreciated, so that we no longer have to make people using our software add third-party repositories.

Bottom line, I would be happy to test and confirm this. In fact, there are several people here who would like to test such a fix.

ilmar

This bug was fixed in the package openmpi - 1.2.6-2ubuntu1

---------------
openmpi (1.2.6-2ubuntu1) intrepid; urgency=low

  * debian/rules: unexport LDFLAGS
    fix SIGSEGV when initialising mpi (LP: #234837).
  * Modify Maintainer value to match the DebianMaintainerField
    specification.

 -- Cesare Tirabassi <email address hidden> Sun, 15 Jun 2008 21:39:00 +0200

motu-sru is subscribed to give clearance for uploading the fix to hardy-proposed:

1. The openmpi package is currently broken in Hardy, since it requires compilation without the -Wl,-Bsymbolic linker option. Any user attempt to initialise the library either directly or through 3rd party applications fails with a SIGSEGV.
2. The bug has been fixed in the release version by disabling dpkg-buildpackage LDFLAGS default.
3. A minimal patch applicable to the stable version of the package is attached.
4. TEST CASE:

A simple test case is defined here. It will be appreciated if users which are using the library through other applications can also confirm the fix, or otherwise.

4.1 In Hardy, install the r-cran-rmpi package with your preferred package manager. This should drag in all required dependencies, including openmpi.

4.2 Try the following simple command in a terminal:

echo 'library(Rmpi); cat("Still alive\n")' | R --slave

This should fail with a SIGSEGV (Address/Memory not mapped) signal.

4.3 Enable the hardy-proposed repository. For instance, you can do this by adding the following line to /etc/apt/sources.list:

  deb http://archive.ubuntu.com/ubuntu/ hardy-proposed universe

Update your local cache and upgrade. This should upgrade at least the following packages:

openmpi-bin, libopenmpi1, openmpi-common

4.4 Re-run the command from step 4.2 in a terminal. Initialisation should be successfull and you should see the message "Still alive" on the screen.

4.5 Disable the hardy-proposed repository.

5. There is obviously a potential for regression. I believe this is mitigated by the fact that in Debian (and Ubuntu before the dpkg-buildpackage change) this has not shown any ill effect.

WARNING: Any user willing to help with testing is more than welcome. Please be aware that there might be some time before the updated packages will be available in your mirror. The sru needs to be approved and the package manually copied by an archive admin, this may take from 1 to several days to be processed.

Where is LDFLAGS defined in the first place? We built the package from source without the patch, and this worked.

Also, should InfiniBand be disabled as Dirk suggested?

ilmar

Disabling IB was only about the "MEEP! You don't have IB hardware!" warning.

Andreas

I know, but I find it annoying to having to edit /etc/openmpi/openmpi-mca-params.conf, and since it is disabled in openmpi i newer versions it could as well be removed.

ilmar

ACK from motu-sru.

On Mon, Jun 16, 2008 at 01:31:53PM -0000, ilmarw wrote:
> I know, but I find it annoying to having to edit /etc/openmpi/openmpi-
> mca-params.conf, and since it is disabled in openmpi i newer versions it
> could as well be removed.

"Our" (as in Debian Open MPI maintainers in discussion with upstream)
consensus was that it is still preferable to have the 'meep' rather
than to disable IB for those who actually have the hardware.

Luckily that is all somewhat moot now as the code will behave better
going forward and deal with this autoMAGICally.

Dirk

--
Three out of two people have difficulties with fractions.

accepted into hardy-proposed, please test

Nice! I had the same problems as reported in bug #224706 and now they are gone.

Johannes

I can confirm that this indeed fixed the problems that I have been having (and that I reported in #224706).

ilmar

The fix works fine for me. Thanks!

Olaf

The fix worked for me too.

Thanks to all that tested.
We have 4 positives and no negatives, I'm therefore asking archive to publish this to hardy-updates.

Copied to hardy-updates.