s390x new ICEs due to not enough memory with parallel builds

Not ICE per-se, as no pre-processed source code is generated, nor e.g. it is retried. Just compiler is getting killed, probably due to out of memory as doko points out. Not reproducible in schroot on a zvm.

Apparently this happens because the containers see 8 CPUs (and thus set DEB_BUILD_OPTIONS=parallel=8) but only have 1.8 GB of RAM, which blows up with memory-hungry C++ code.

Try to change LXC limits to only make one or two CPUs visible in the container, otherwise tell adt-run to use a smaller parallel= value.

Added new adt-run option in http://anonscm.debian.org/cgit/autopkgtest/autopkgtest.git/commit/?id=6486fc5862

Now going to work on the glue to use this in the LXC runners.

I changed the worker to accept a new "extra_args" option in the config (https://git.launchpad.net/~ubuntu-release/+git/autopkgtest-cloud/commit/?id=86a510a), added "extra_args = --build-parallel=2" to the s390x config, and rolled it out.

A re-run now correctly shows "make -j2", but it still fails the same way:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/s390x/r/rocs/20160413_144415@/log.gz

I reproduced the ICE manually in a container with "apt-get build-dep rocs" and "DEB_BUILD_OPTIONS=parallel=2 dpkg-buildpackage -us -uc -b". rocs still ICEs with that, but works with parallel=1.

So I suppose if g++ is that memory hungry on s390x, I'll have to disable parallelism entirely on s390x. I did that now, retried rocs, and it passes now:

  https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-xenial/xenial/s390x/r/rocs/20160413_151809@/log.gz

This bug was fixed in the package autopkgtest - 3.20.4

---------------
autopkgtest (3.20.4) unstable; urgency=medium

  * setup-commands/setup-testbed: Fix kernel header installation for
    precise/armhf, which does not yet have linux-headers-generic.
  * Use "nproc" to determine the number of processors available, which is both
    more correct in a cgroup world, and also simpler.
  * Add new --build-parallel=N option to explicitly set the "parallel=N"
    $DEB_BUILD_OPTION for building packages, to override the default of
    "number of available CPUs". This is mostly useful in containers where you
    can restrict the available RAM, but not restrict the number of CPUs.
    (LP: #1569750)
  * setup-commands/setup-testbed: ubuntu-snappy got renamed to snapd, adjust
    package name.
  * adt-build-lxd: Set "distribution", "release", and "architecture"
    properties of generated images.
  * adt-build-lxd: Clean up all old images of the same distro/release/arch,
    not just the previous one. Also fix cleanup to work in non-English
    locales.
  * setup-commands/setup-testbed: Call apt-get purge only once with the list
    of all packages, instead of once per package. This is much faster.
  * SchrootClickRunner tests: Chown click dir in /opt after creating the user.
  * Bump Standards-Version to 3.9.8 (no changes necessary).
  * Update Vcs-* URLs.

 -- Martin Pitt <email address hidden> Mon, 18 Apr 2016 11:19:45 +0200