128 bit float precision breaks mysql self-tests (stalls cosmic updates)

I checked the release notes of the .23 version that triggered this first - nothing in there that would explain.

Hrm, .. this is released in Bionic through the -security pocket.
https://launchpad.net/ubuntu/+source/mysql-5.7/5.7.23-0ubuntu0.18.04.1

But it has no entry in
http://autopkgtest.ubuntu.com/packages/m/mysql-5.7/bionic/ppc64el
since it skips the usual proposed migration.

I wonder if that would fail on Bionic as well (meaning the version in the archive is broken)

This could have been detected at build time IMHO.
The PPC Build of this has:
main.sp w1 [ fail ]
        Test ended at 2018-08-24 17:22:52
[...]
-6.300000071525574 1
+6.299999833106995 1
[...]
Cosmic build log from what is in -proposed atm:
=> https://launchpadlibrarian.net/385184440/buildlog_ubuntu-cosmic-ppc64el.mysql-5.7_5.7.23-0ubuntu2_BUILDING.txt.gz

But it is also "good" that it will show up in the build log:
=> https://launchpadlibrarian.net/380342345/buildlog_ubuntu-bionic-ppc64el.mysql-5.7_5.7.23-0ubuntu0.18.04.1_BUILDING.txt.gz

Thereby we know that Bionic is not affected, dspite being the same mysql source.

I forked bug 1791018 to let Rbasak/Otto check if this should have been breaking at build time instead of later at autopkgtest.

The is reproducible in a local build on ppc64 like:
$ sudo ./debian/rules clean && sudo ./debian/rules build
That sees main.sp failing the same way in the log.

But the build takes quite some time, no fast iteration on this one :-/

The fact that the sources should be mostly the same and Bionic working while Cosmic fails leaves me puzzled - maybe after all this is like a math library that changed.

I'll change my repro build setup to have Bionic and Cosmic builds of the same source (will take a while) ...

Essentially 4 LXD containers doing:
$ pull-lp-source mysql-5.7
$ apt build-dep mysql-5.7
$ sudo ./debian/rules clean && sudo ./debian/rules build

Sys-Bionic-SRC-Bionic: ok
Sys-Bionic-SRC-Cosmic: ok
Sys-Cosmic-SRC-Bionic: fail
Sys-Cosmic-SRC-Cosmic: fail

So it is one of the build dependencies, and not the mysql upgrade itself.
Very likely the same would also hit the former mysql that already is in -proposed (not worth to try wasting CPU power).

I think I need to bisect "apt upgrade" oO

1. Compiler
Container Sys-Bionic-SRC-Bionic:
The following NEW packages will be installed:
  cpp-8 g++-8 gcc-8 libasan5 libgcc-8-dev libquadmath0 libstdc++-8-dev libubsan1
The following packages will be upgraded:
  binutils binutils-common binutils-powerpc64le-linux-gnu cpp cpp-7 g++ g++-7 gcc gcc-7 gcc-7-base gcc-8-base libasan4 libatomic1 libbinutils libcc1-0 libgcc-7-dev libgcc1 libgomp1 libitm1
  liblsan0 libstdc++-7-dev libstdc++6 libtsan0 libubsan0

2. LIBC
Container Sys-Bionic-SRC-Cosmic - updated
The following packages will be upgraded:
  libc-bin libc-dev-bin libc6 libc6-dev locales multiarch-support

#1 failing
#2 ok

So it is the gcc update that changed the float precision/values.
If that is good/bad I don't know.
It could mean everything in cosmic might have a fun time rebuilding (on ppc64el)?

Subscribing Doko for his thoughts on this.

If none of the above then update to both and build once more on one.
Both plus next selection on the other

Trying to split the above case #1

#1a (just binutils)
The following packages will be upgraded:
  binutils binutils-common binutils-powerpc64le-linux-gnu libbinutils
=> ok

#1b (almost all but gcc-8)
The following packages will be upgraded:
  binutils binutils-common binutils-powerpc64le-linux-gnu cpp-7 g++-7 gcc-7 gcc-7-base libasan4 libbinutils libgcc-7-dev libstdc++-7-dev libubsan0
=> ok

#1c libquadmath?
The following NEW packages will be installed:
  cpp-8 gcc-8 libasan5 libgcc-8-dev libquadmath0 libubsan1
The following packages will be upgraded:
  binutils binutils-common binutils-powerpc64le-linux-gnu cpp gcc gcc-8-base libatomic1 libbinutils libcc1-0 libgcc1 libgomp1 libitm1 liblsan0 libstdc++6 libtsan0
# dpkg -r --force-depends libquadmath0
=> ok

That confirms that libquadmath is the reason.
So we get actually higher accuracy (64->128 bits) but the expected results are 64 bit only.
I'd assume that this should be brought upstream for gcc8/quadmath support and added as an alternate accepted test output.

I'd ask if Lars/Rbasak could do so being the usual maintainers.

typically, the test case should truncate the expected results, and do not assert on precision differences beyond 64bit.

Made a test patch and uploaded to https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mysql-5.7/+packages (also set test failures to break the build on all architectures, rather than just i386 and amd64)

Your upload seems to build fine on all architectures and checking ppc64 in particular the tests that formerly failed are good now.

  The servers were restarted 167 times
  Spent 1178.435 of 482 seconds executing testcases
  Completed: All 768 tests were successful.
  157 tests were skipped, 59 by the test itself.

That said I thought this looks good and you could go on coordinating with rbasak and tsimonq (who synced the last version) for the upload into cosmic that should finally unblock it from proposed migration.
But I realized that the build log in the PPA still has:
   Depends: [...] libmecab2
which should also be removed for 18.10 as the security review isn't complete and we have a component mismatch at the moment (the synced version currently in cosmic-proposed blocks on the lack of mecab in main).

On 12. sep. 2018 07:08,  Christian Ehrhardt  wrote:
> Your upload seems to build fine on all architectures and checking ppc64
> in particular the tests that formerly failed are good now.
>
> The servers were restarted 167 times
> Spent 1178.435 of 482 seconds executing testcases
> Completed: All 768 tests were successful.
> 157 tests were skipped, 59 by the test itself.
>
> That said I thought this looks good and you could go on coordinating with rbasak and tsimonq (who synced the last version) for the upload into cosmic that should finally unblock it from proposed migration.
> But I realized that the build log in the PPA still has:
> Depends: [...] libmecab2
> which should also be removed for 18.10 as the security review isn't complete and we have a component mismatch at the moment (the synced version currently in cosmic-proposed blocks on the lack of mecab in main).
>
Right, we have a routine for dropping the mecab dependency from Ubuntu.
I'll add the patch descriptions etc. and prepare the upload :)

--
Lars

This bug was fixed in the package mysql-5.7 - 5.7.23-2ubuntu1

---------------
mysql-5.7 (5.7.23-2ubuntu1) cosmic; urgency=medium

  * Merge from Debian.
  * Remaining changes:
    - Revert "Build with libmecab-dev" since libmecab-dev is in
      universe.
  * Drop changes:
    - debian/patches/disable_float_test.patch: disable test; this is
      fixed in Debian since 5.7.23-2 (LP: #1791010).
  * This reverts the previous direct sync from Debian, which was
    missing the above change.

mysql-5.7 (5.7.23-2) unstable; urgency=medium

  [ Robie Basak ]
  * Drop dependency on bsdutils
  * Update Vcs-* URLs to salsa following migration

  [ Lars Tangvald ]
  * d/tests: Disable tests failing on ppc64el
  * d/patches: Fix failing tests on ppc64el

mysql-5.7 (5.7.23-1) unstable; urgency=high (security fixes)

  * Imported upstream version 5.7.23 to fix security issues:
    - http://www.oracle.com/technetwork/security-advisory/cpujul2018-4258247.html
    - CVE-2018-0739 CVE-2018-2767 CVE-2018-3054 CVE-2018-3056
    - CVE-2018-3058 CVE-2018-3060 CVE-2018-3061 CVE-2018-3062
    - CVE-2018-3064 CVE-2018-3065 CVE-2018-3066 CVE-2018-3070
    - CVE-2018-3071 CVE-2018-3077 CVE-2018-3081
    (Closes: #904121)
  * Moved internal test binaries out of usr/bin
    The client binaries mysqltest and mysql_client_test are only
    meant to be used by the mysql test suite, and are no longer
    installed in /usr/bin

 -- Lars Tangvald <email address hidden> Thu, 20 Sep 2018 08:31:11 +0200