[PR40735] gcc-4.4 fails to build upstart 0.6 on armel due to an internal compiler error

different place of the error in a different system with a local build, it seems to be closely related to the hardware gcc is running on ...

gcc -std=gnu99 -DHAVE_CONFIG_H -DLOCALEDIR="\"/usr/share/locale\"" -I.. -I.. -iquote. -iquote. -I../intl -I/usr/include/dbus-1.0 -I/usr/lib/dbus-1.0/include -Wall -g -fstack-protector -fPIE -Os -MT test_node.o -MD -MP -MF .deps/test_node.Tpo -c -o test_node.o `test -f 'tests/test_node.c' || echo './'`tests/test_node.c
gcc: Internal error: Killed (program cc1)
Please submit a full bug report.
See <file:///usr/share/doc/gcc-4.4/README.Bugs> for instructions.
make[4]: *** [test_node.o] Error 1
make[4]: Leaving directory `/home/ogra/upstart-0.6.0/nih-dbus-tool'
make[3]: *** [check-am] Error 2
make[3]: Leaving directory `/home/ogra/upstart-0.6.0/nih-dbus-tool'
make[2]: *** [check] Error 2
make[2]: Leaving directory `/home/ogra/upstart-0.6.0/nih-dbus-tool'
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory `/home/ogra/upstart-0.6.0'

downgrading my local buildsystem to gcc-4.3 i see similar issues

Perhaps it was forced to swap or ran out of virtual memory. Sometimes
you will find some strange situation in some piece of code where gcc
seems to decide it needs a massive amount of memory and a very long time
to compile something optimized. For example, there was a time when
compiling the pci code in X.org for powerpc would require more than 1
gig of ram, but only to compile one very specific file, and only as long
as optimization was used. Maybe we are seeing something similar here?

Oliver Grawert wrote:
> Public bug reported:
>
> Binary package hint: gcc-4.4
>
> in
> https://edge.launchpad.net/ubuntu/+source/upstart/0.6.0-2/+build/1112005/+files/buildlog_ubuntu-karmic-armel.upstart_0.6.0-2_FAILEDTOBUILD.txt.gz
> we see that the buidd appears to time out (in reality its hiding the actual error).
> building that package on a local armel machine spits out the following:
>
>
> /bin/bash ../libtool --tag=CC --mode=link gcc -std=gnu99 -Wall -g -fstack-protector -fPIE -Os -static -Wl,-z,relro -Wl,-z,now -pie -Wl,-O1 -o test_file test_file.o libnih.la
> libtool: link: gcc -std=gnu99 -Wall -g -fstack-protector -fPIE -Os -Wl,-z -Wl,relro -Wl,-z -Wl,now -pie -Wl,-O1 -o test_file test_file.o ./.libs/libnih.a -lrt
> gcc -std=gnu99 -DHAVE_CONFIG_H -DLOCALEDIR="\"/usr/share/locale\"" -I.. -I.. -I../intl -Wall -g -fstack-protector -fPIE -Os -MT test_watch.o -MD -MP -MF .deps/test_watch.Tpo -c -o test_watch.o `test -f 'tests/test_watch.c' || echo './'`tests/test_watch.c
> tests/test_watch.c: In function ‘test_new’:
> tests/test_watch.c:590: internal compiler error: in df_ref_record, at df-scan.c:2845
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <file:///usr/share/doc/gcc-4.4/README.Bugs> for instructions.
> make[3]: *** [test_watch.o] Error 1
> make[3]: Leaving directory `/home/ogra/devel/upstart-0.6.0/nih'
> make[2]: *** [check-am] Error 2
> make[2]: Leaving directory `/home/ogra/devel/upstart-0.6.0/nih'
> make[1]: *** [check-recursive] Error 1
> make[1]: Leaving directory `/home/ogra/devel/upstart-0.6.0'
>
> ** Affects: gcc-4.4 (Ubuntu)
> Importance: High
> Status: New
>
>
> ** Tags: armel
>
> ** Changed in: gcc-4.4 (Ubuntu)
> Importance: Undecided => High
>
> ** Tags added: armel
>

David, please try rebuilding locally and see if you run out of memory; I doubt it's the case though.

Any of you reproducing the bug with latest compiler, please collect the generated file which triggers the crash as explained in the gcc debug instructions.

i doubt that its a memory issue and i see it on all SoCs here
i will try building gcc-snapshot and see if it occus with that too ...
for both SoCs i have here, i only have binary kernels, it might be cause by some ulimit issue ...

there should be enough swap here and the system is running (and swapping) from a SATA disk

ogra@dove:~/devel/upstart-0.6.0$ free
             total used free shared buffers cached
Mem: 514740 435284 79456 0 55104 306752
-/+ buffers/cache: 73428 441312
Swap: 3903784 168 3903616

So it's actually gcc using a load of memory indeed; it seems that it doesn't like the 16k lines of code in test_node.c and there are larger files to build after that.

19:06 < lool> cjwatson: I built with -O0 -fno-common -fno-foo ... disabling all
              enabled opts until the output would only mention one enabled opt
              which is the reverse of another one; the build still takes
              hundreds of megs of RAM and I stop it at around 80 % RAM after 3
              minutes
19:07 < lool> I guess I should have tried with an unpatched gcc though
...
19:08 < lool> Oh it passed
19:08 < lool> after 4 minutes or so
19:09 < lool> but using a big load of RAM
...
19:12 < lool> doko: I don't quite know how to attack the upstart build slowness
              issue
19:12 < lool> Basically, we're seeing underperformance, but it's hard to
              qualify how fast gcc should run, or how much memory it's allowed
              to use
19:12 < lool> doko: Do you think some patches we're using could be destroying
              performance?
19:12 < lool> e.g. that 64 bytes align patch
19:13 < lool> That's a binutils patch, but I can see how it could lead to much
              larger RAM usage
19:13 < doko> lool: where is the time used, in cc1, or as?
19:13 < lool> doko: cc1
19:14 < doko> well, then it's not the align patch, and we are pretty much using
              upstream, no fancy patches on our own
19:15 < lool> doko: Can I just report it upsteam opening a new bug saying that
              it uses a load of RAM/CPU to build a file?
19:15 < lool> ie Is that an acceptable bug?
19:16 < doko> lool: sure, better check it with 4.3 (or older) as well to see if
              it's a regression or not.
...
19:23 < lool> doko: It's about the same performance with gcc-4.3

This is so very much like my strange experience last year with just a
very specific file in X.org that would similarly balloon gcc memory
usage cross-compiling for powerpc...the solution chosen at the time was
to use 2G or larger build machines ;).

Loïc Minier wrote:
> 19:06 < lool> cjwatson: I built with -O0 -fno-common -fno-foo ... disabling all
> enabled opts until the output would only mention one enabled opt
> which is the reverse of another one; the build still takes
> hundreds of megs of RAM and I stop it at around 80 % RAM after 3
> minutes
> 19:07 < lool> I guess I should have tried with an unpatched gcc though
> ...
> 19:08 < lool> Oh it passed
> 19:08 < lool> after 4 minutes or so
> 19:09 < lool> but using a big load of RAM
> ...
> 19:12 < lool> doko: I don't quite know how to attack the upstart build slowness
> issue
> 19:12 < lool> Basically, we're seeing underperformance, but it's hard to
> qualify how fast gcc should run, or how much memory it's allowed
> to use
> 19:12 < lool> doko: Do you think some patches we're using could be destroying
> performance?
> 19:12 < lool> e.g. that 64 bytes align patch
> 19:13 < lool> That's a binutils patch, but I can see how it could lead to much
> larger RAM usage
> 19:13 < doko> lool: where is the time used, in cc1, or as?
> 19:13 < lool> doko: cc1
> 19:14 < doko> well, then it's not the align patch, and we are pretty much using
> upstream, no fancy patches on our own
> 19:15 < lool> doko: Can I just report it upsteam opening a new bug saying that
> it uses a load of RAM/CPU to build a file?
> 19:15 < lool> ie Is that an acceptable bug?
> 19:16 < doko> lool: sure, better check it with 4.3 (or older) as well to see if
> it's a regression or not.
> ...
> 19:23 < lool> doko: It's about the same performance with gcc-4.3
>

not building the file with -fPIE limits the memory usage to a maximum of 600MB, approx. 10min build time.

regression in gcc-4.3/gcc-4.4.

still unsure why these functions in the test cases cannot be split up into smaller functions which then are called sequentially.

18:24 < lool> Keybuk: Couldn't we split at least the
              test_path_valid/test_new/test_start_tag etc. into separate files?
18:24 < Keybuk> lool: no, the test suite doesn't work that way
18:24 < lool> test_proxy_functions is the huge function; 10k lines
18:25 < lool> Keybuk: What do you suggest?
18:25 < Keybuk> lool: fix gcc

This bug was fixed in the package upstart - 0.6.0-4

---------------
upstart (0.6.0-4) karmic; urgency=low

  * Don't build the testsuite with -fPIE on armel; LP: #398403.

 -- Loic Minier <email address hidden> Mon, 13 Jul 2009 22:12:34 +0200

libnih build fails in maverick on armel

This bug was fixed in the package libnih - 1.0.2-1ubuntu1

---------------
libnih (1.0.2-1ubuntu1) maverick; urgency=low

  * Rebuild with libc6-dev (>= 2.12~), after checking that
    __abort_msg is available with the same signature in eglibc 2.12.
  * Don't build the testsuite with -fPIE on armel; LP: #398403.
 -- Matthias Klose <email address hidden> Sun, 30 May 2010 02:54:56 +0200

I uploaded upstart back without -fPIE / -pie, just to see how that goes with Linaro toolchain; otherwise, I think we should fix that.

On 23.07.2010 17:07, Loïc Minier wrote:
> I uploaded upstart back without -fPIE / -pie, just to see how that goes
> with Linaro toolchain; otherwise, I think we should fix that.

it's enough to build the *testcases* without pie to pass the testsuite.

Yes, but someone changed the packaging to remove it everywhere instead of just for the testsuite, and also I'm curious of the gcc issue -- not the testsuite one, I think this is fine.

So the new upstart upload I did fails to build in the testsuite as on the other architectures, but most tests DID build with -fPIE, some 3 tests or so remained to be built. test_watch.c isn't in upstart anymore.

I tried removing -fPIE/-pie in libnih, and it failed to build with the maverick Linaro-based toolchain; I can't access the build log to check how it failed though.

Ah just as said that, I could access the log:
gcc -std=gnu99 -DHAVE_CONFIG_H -DLOCALEDIR="\"/usr/share/locale\"" -I.. -I.. -iquote. -iquote. -I../intl -I/usr/include/dbus-1.0 -I/usr/lib/dbus-1.0/include -Wall -g -fstack-protector -fPIE -Os -c -o demarshal_code.o `test -f 'tests/demarshal_code.c' || echo './'`tests/demarshal_code.c
  CCLD test_demarshal
  CC test_node.o
gcc -std=gnu99 -DHAVE_CONFIG_H -DLOCALEDIR="\"/usr/share/locale\"" -I.. -I.. -iquote. -iquote. -I../intl -I/usr/include/dbus-1.0 -I/usr/lib/dbus-1.0/include -Wall -g -fstack-protector -fPIE -Os -c -o test_node.o `test -f 'tests/test_node.c' || echo './'`tests/test_node.c

Session terminated, killing shell...make: *** [build] Terminatedmake[1]: *** wait: No child processes
. Stop.
make[1]: *** Waiting for unfinished jobs....
[...]
Build killed with signal 15 after 150 minutes of inactivity

It might be an issue with the cheer size of the file, but it didn't segfault at least:
  CC test_watch.o
gcc -std=gnu99 -DHAVE_CONFIG_H -DLOCALEDIR="\"/usr/share/locale\"" -I.. -I.. -I../intl -Wall -g -fstack-protector -fPIE -Os -c -o test_watch.o `test -f 'tests/test_watch.c' || echo './'`tests/test_watch.c
  CCLD test_watch

I've just built upstart_0.6.6-1ubuntu1.dsc on an IGEPv2 board.

The build appears to succeed and run as far as testing, but the tests fail:

....
FAIL: test_job_class
....
FAIL: test_job
....
FAIL: test_control
===============================================
3 of 13 tests failed
Please report to <email address hidden>
===============================================
make[4]: *** [check-TESTS] Error 1
make[4]: Leaving directory `/home/ams/lp398403/upstart-0.6.6/init'
make[3]: *** [check-am] Error 2
make[3]: Leaving directory `/home/ams/lp398403/upstart-0.6.6/init'
make[2]: *** [check] Error 2
make[2]: Leaving directory `/home/ams/lp398403/upstart-0.6.6/init'
make[1]: *** [check-recursive] Error 1
make[1]: Leaving directory `/home/ams/lp398403/upstart-0.6.6'
dh_auto_test: make -j1 check returned exit code 2
make: *** [build] Error 29
dpkg-buildpackage: error: debian/rules build gave error exit status 2

I added -fpie back in to a local build of libnih 1.0.2-1ubuntu1 and it built and tested correctly. Compiling test_node.c takes about 1 GB of RAM.

This is with maverick and gcc-4.4-4.4.4-8ubuntu1.

So apparently fixed in some newer gcc-4.4; thanks for testing

i've just uploaded libnih without the armel specific behavior of building the testsuite with -fPIE.