Comment 5 for bug 1707409

Revision history for this message
Christian Ehrhardt  (paelzer) wrote : Re: testsuite fails under qemu (SIGILL) works fine on real hw

TL;DR:
- you can use the pbuilder + static qemu setup to debug
- qemu/libvirt throw no error
- the tests do not "consider" the unsupported syscall
- I found to get just the same issues on Artful but with more context indicating that missing syscall
- You can use the setup described above (or sbuild) to debug further as there are 1-2 issues which seem to have other reasons e.g. "Xapian exception: read only files"
- the testcases might need several fixes, but one of them is surely to test not only if gdb is installed, but if it is working.

---

Hi Gianfranco,
Lacking a working arm system atm to test any further I tried this in qemu static.
Might be an odd setup, but I had it around from evrifying another bug.

pbuilder-dist artful armhf create
pbuilder-dist artful armhf login
apt install ubuntu-dev-tools vim-nox fakeroot
pull-lp-source notmuch
# enable deb-src in /etc/apt/sources.list
apt update
apt-get build-dep notmuch
cd notmuch
debuild -uc -us

(tried the same in Trusty with the old version of notmuch)
Both had a few testcase errors - but not exactly the same you had.

But after the build running dh_auto_test manually in that dir shows more output that can help.

With that approach I think I saw some issues that might be related as with local dh_autp_test you get mroe details.
- Xapian exception: read only files - that could be your DB errors

I quite often saw tests skipped for gdb missing, on an older build environment that was around.
So for a test I installed gdb and reran the tests and found what might be related.
Gdb in the virtual environment (at least my qemu-statis-armhf one) could not work due to an Unsupported syscall.
The retval of gdb in that was 255, and that reminded me of your log:
FAIL success exit with --keep when add_message returns READ_ONLY_DATABASE
 --- T070-insert.33.expected 2016-08-31 07:10:21.960346786 +0000
 +++ T070-insert.33.output 2016-08-31 07:10:21.960346786 +0000
 @@ -1 +1 @@
 -0
 +255
While I saw:
 FAIL success exit with --keep when add_message returns READ_ONLY_DATABASE
        exit code 255, expected 0 gdb --batch-silent --return-child-result -ex 'set args insert --keep < /usr/notmuch-0.25/test/tmp.T070-insert/mail/msg-018' -x index-file-READ_ONLY_DATABASE.gdb notmuch
qemu: Unsupported syscall: 26

Not exactly the same, but the question is what is throwing the 255 in your case, but as I said my setup seems insufficient for that. Maybe the "full" qemu running arm on arm supports that system call but fails differently?

I wondered, in your buildlog [1] I see shell syntax errors like:
./T380-atomicity.sh: line 79: ((: i < : syntax error: operand expected (error token is "< ")
But when running locally (before installing gdb) I saw:
 missing prerequisites: gdb(1)
 SKIP all tests in T380-atomicity
With gdb installed I get exactly your error on the "./T380-atomicity.sh: line 79:" case.
I also saw that at least back in Yakkety gdb was a build dep but with various arch restrictions:
Build-Depends: gdb-minimal, gdb [!s390x !ia64 !armel !ppc64el !mips !mipsel !mips64el]
Build-Conflicts: ruby1.8, gdb-minimal, gdb [s390x ia64 armel ppc64el mips mipsel mips64el]

Note - the same on a Trusty pbuilder target instead of Artful ran all but one test.
So maybe it is neither LP, nor qemu but some of the dependencies that broke this?

Your build log I compared to was actually Yakkety so I picked that finally to check if this is fully reproducible.
There gdb could be around by default (just as it was in trusty for me) due to dependencies.
I slimlined the repro to get less extra deps:
pbuilder-dist yakkety armhf create
pbuilder-dist yakkety armhf login
# enable deb-src in /etc/apt/sources.list
apt update
apt-get source notmuch
apt-get build-dep notmuch
apt install fakeroot devscripts
cd notmuch
debuild -uc -us

And here things confirmed - it was the unsupported system call and on yakktey the output matches your case:
T060-count: Testing "notmuch count" for messages and threads
 FAIL error message from query_search_messages
        --- T060-count.14.EXPECTED 2017-08-08 09:01:36.000000000 +0000
        +++ T060-count.14.OUTPUT.clean 2017-08-08 09:01:36.000000000 +0000
        @@ -1,3 +1 @@
        -notmuch count: A Xapian exception occurred
        -A Xapian exception occurred performing query
        -Query string was: *
        +qemu: Unsupported syscall: 26

T070-insert: Testing "notmuch insert"
 FAIL error exit when add_message returns OUT_OF_MEMORY
        --- T070-insert.26.expected 2017-08-08 09:01:46.000000000 +0000
        +++ T070-insert.26.output 2017-08-08 09:01:46.000000000 +0000
        @@ -1 +1 @@
        -1
        +255
qemu: Unsupported syscall: 26

Interesting might also be that despite the same issues your build log has it ran not into the final shell errors you had. So it continues to build despite the failed tests:

FATAL: ./T380-atomicity.sh: interrupted by signal -128
test/Makefile.local:64: recipe for target 'test' failed
make[2]: *** [test] Error 124
make[2]: Leaving directory '/root/notmuch-0.22.1'
dh_auto_test: make -j1 test VERBOSE=1 returned exit code 2
make[1]: Leaving directory '/root/notmuch-0.22.1'
 fakeroot debian/rules binary

And with that finally I found:
override_dh_auto_test:
ifeq ($(DEB_HOST_ARCH),armhf)
        TERM=vt100 dh_auto_test || true
Which means it is not meant/expected to work properly on archhf.

The difference that has to be found is why/how it breaks out of that || true
OTOH that is no more in latter versions like Artful, so the fix might be to actually fix that for virtual environments that are armhf, but do not support that syscall.

That now known it might also be reproducible via sbuild which I usually prefer, but I rarely hav cross compile setups and struggled with it trying for this case.

I lack the arm HW to check any further on arm64 host in particular, but as outlined above don't tihnk it is needed.
IMHO the testcases might need a few fixes, one of them is to test not only if gdb is installed, but if it is working.
You can use the setup described above to debug further - is that ok for you?

[1]: https://launchpadlibrarian.net/281860948/buildlog_ubuntu-yakkety-armhf.notmuch_0.22.1-2ubuntu1_BUILDING.txt.gz