Percona crashes when doing a a 'larger' update

Bug #1657256 reported by Mario Splivalo
52
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Charm Test Infra
Confirmed
Low
Ryan Beisner
percona-xtradb-cluster-5.5 (Ubuntu)
Confirmed
Medium
Jorge Niedbalski
Trusty
Fix Released
High
Jorge Niedbalski
percona-xtradb-cluster-5.6 (Ubuntu)
Fix Released
High
Jorge Niedbalski
Xenial
Fix Released
High
Jorge Niedbalski
Zesty
Fix Released
High
Jorge Niedbalski
Artful
Fix Released
High
Jorge Niedbalski

Bug Description

[Impact]

 * Percona will segfault when exposed to medium load, almost imediately
 * This is because of a bug in upstream, which manifests on ARM and PPC

[Test Case]

 * Install and configure percona-xtradb-cluster-server
  * Run sysbench against configured node (only one node is needed, no need for proper cluster):
   # sysbench --test=oltp --oltp-test-mode=complex --max-time=60 --num-threads=110 run
 * mysqld will segfault seconds withing starting the test

[Regression Potential]

 * This is a cherry-pick from an upstream fix (https://jira.mariadb.org/browse/MDEV-6450)
 * This is not fixed in upstream Percona becasue Percona does not officially support non-intel archs.
 * Because code adds additional memory barriers there was a chance of performance degradation on i386/amd64. However, intensive sysbench syntetic loads proved this is not the case - there are no performance penalties.

* Zesty autopkgtest failures :
  Explanation --> https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.5/+bug/1657256/comments/94

[Other Info]

 * percona-xtradb-cluster-5.5 is only available for Trusty.
$ rmadison percona-xtradb-cluster-5.5
 percona-xtradb-cluster-5.5 | 5.5.34-25.9+dfsg-0ubuntu4 | trusty/universe | source
 percona-xtradb-cluster-5.5 | 5.5.37-25.10+dfsg-0ubuntu0.14.04.1 | trusty-security/universe | source
 percona-xtradb-cluster-5.5 | 5.5.37-25.10+dfsg-0ubuntu0.14.04.2 | trusty-updates/universe | source

* See comment #22 for more context about other releases that offers percona-xtradb-cluster-5.6 :
https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.5/+bug/1657256/comments/22

 * Upstream commit: https://github.com/MariaDB/server/commit/40497577ffd9f85557b15e08ad913f627b2e9530

[Original Description]

I'm trying to set up percona-xtradb-cluster-5.5 on PPC machine. While the package installs fine, as soon as I run sysbench oltp becnhmark against it, Percona dies (even when I start the benchmark with just one connection).

I can also crash mysql manually, by updating the sbtest table (which is created by the sysbench utility):

mysql> update sbtest set pad = 'mario1' limit 1000000;
ERROR 2013 (HY000): Lost connection to MySQL server during query

Sometimes I need to repeat this update (with different values for 'pad' field) few times. This happens regardless of whether I run the UPDATE inside the transaction or not.

This is the assertion found in the log file:

170117 21:10:55 InnoDB: Assertion failure in thread 70366668321152 in file buf0buf.ic line 1277
InnoDB: Failing assertion: block->page.buf_fix_count > 0

This is a single-node percona-xtradb-cluster server, without wsrep_provied configured, run inside 14.04 lxc container on 16.04 host.

I'm attaching the full log file, mysql configuration file and the core dumped.

The version of the package installed is this 5.5.37-25.10+dfsg-0ubuntu0.14.04.2.

Related branches

Revision history for this message
Mario Splivalo (mariosplivalo) wrote :
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :
description: updated
description: updated
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

I was able to reproduce this with percona-xtradb-cluster-server 5.6 from xenial too, on the same architecture (again inside xenial lxc container).

I'm attaching the error.log for percona-5.6 too here, as the assertion is different.

Revision history for this message
Mario Splivalo (mariosplivalo) wrote :
description: updated
Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

I could find:

https://jira.mariadb.org/browse/MDEV-7762

And commit:

commit 33589b25efe3283b748e43a54c42db2ed176c3e5
Author: Jan Lindström <email address hidden>
Date: Thu Dec 3 13:18:10 2015 +0200

    MDEV-7762 InnoDB: Failing assertion: block->page.buf_fix_count > 0 in buf0buf.ic line 730

    Analysis: debug only assertion I_S function (IS is XtraDB feature) is calling
    buf_block_get_frame on any page it reads, which debug-asserts that the page is
    buffer-fixed, which is not the case in I_S query.

    Fixed by holding the buffer page mutex while the fields are read directly.

which is NOT applied to percona code 5.5 for sure.

Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

Possibly upstream related code:

https://bugs.launchpad.net/percona-server/+bug/1217158

Which is likely related to this one.

Changed in percona-xtradb-cluster-5.5 (Ubuntu):
status: New → Confirmed
status: Confirmed → In Progress
assignee: nobody → Rafael David Tinoco (inaddy)
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "trusty_percona-xtradb-cluster-5.5_5.5.37-25.10+dfsg-0ubuntu0.14.04.3~fix~1.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

Hello, Rafael.

Thank you for the swift debdiff. Unfortunately, percona is still crashing with the same assertion. I'm attaching the error.log file, as well as the core dumped (in case you need it).

tags: removed: patch
no longer affects: percona-xtradb-cluster
Mathew Hodson (mhodson)
Changed in percona-xtradb-cluster-5.5 (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

I bisected the MariaDB source tree and found out that this commit fixed the issue:

40497577ffd9f85557b15e08ad913f627b2e9530 - it is related to this: https://jira.mariadb.org/browse/MDEV-6450

Revision history for this message
Rafael David Tinoco (rafaeldtinoco) wrote :

Possibly related commits:

commit 40497577ffd9f85557b15e08ad913f627b2e9530
Author: Sergey Vojtovich <email address hidden>
Date: Fri Aug 29 16:02:46 2014 +0400

    Backport from 10.0:

    MDEV-6450 - MariaDB crash on Power8 when built with advance tool
                chain

    InnoDB mutex_exit() function calls __sync_test_and_set() to release
    the lock. According to manual this function is supposed to create
    "acquire" memory barrier whereas in fact we need "release" memory
    barrier at mutex_exit().

    The problem isn't repeatable with gcc because it creates
    "acquire-release" memory barrier for __sync_test_and_set().
    ATC creates just "acquire" barrier.

commit 5569132ffebba3fd2e37964543f658ed24d8caaf
Author: Michael Widenius <email address hidden>
Date: Tue Aug 19 19:28:35 2014 +0300

    MDEV-6450 - MariaDB crash on Power8 when built with advance tool chain

    Part of this work is based on Stewart Smitch's memory barrier and lower priori
    patches for power8.

    - Added memory syncronization for innodb & xtradb for power8.
    - Added HAVE_WINDOWS_MM_FENCE to CMakeList.txt
    - Added os_isync to fix a syncronization problem on power
    - Added log_get_lsn_nowait which is now used srv_error_monitor_thread to ensur
      if log mutex is locked.

    All changes done both for InnoDB and Xtradb

commit 82ce2a2503a95e8bdc76c351a87e98c93445a433
Author: Sergey Vojtovich <email address hidden>
Date: Mon Aug 4 11:55:38 2014 +0400

    MDEV-6450 - MariaDB crash on Power8 when built with advance
                tool chain

    Reverted addition to the original patch:
    InterlockedExchangeAcquire and InterlockedAndRelease are
    supported only on Itanium-based systems.

commit 53360fd45f89e9b16aac69625820988702a339ee
Author: Sergey Vojtovich <email address hidden>
Date: Thu Jul 31 14:31:05 2014 +0400

    MDEV-6450 - MariaDB crash on Power8 when built with advance
                tool chain

    This is an addition to the original patch. On Windows
    InterlockedExchange implies full memory barrier, whereas
    only acquire/release barriers required.

commit c0ebb3f38811c6a0e3e2b49b3ae40b4ea0c2b0e9
Author: Sergey Vojtovich <email address hidden>
Date: Fri Jul 18 15:16:25 2014 +0400

    MDEV-6450 - MariaDB crash on Power8 when built with advance tool
                chain

    InnoDB mutex_exit() function calls __sync_test_and_set() to release
    the lock. According to manual this function is supposed to create
    "acquire" memory barrier whereas in fact we need "release" memory
    barrier at mutex_exit().

    The problem isn't repeatable with gcc because it creates
    "acquire-release" memory barrier for __sync_test_and_set().
    ATC creates just "acquire" barrier.

    Fixed by creating proper barrier at mutex_exit() by using
    __sync_lock_release() instead of __sync_test_and_set().

--------

Mario,

I provided you: http://inaddy.ddns.net:3333/~inaddy/trusty_percona-xtradb-cluster-5.5_5.5.37-25.10%2bdfsg-0ubuntu0.14.04.3.debdiff based on this last patch. This was the only patch backport from 10.0.

Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

Rafael!

I have built percona on ppc against your debdiff and the issue is gone.

I just concluded my testing - having hammered percona on ppc (and on amd64) for 6+ hours with sysbench yielded no errors.

I will get on doing this on xenial too now.

no longer affects: percona-xtradb-cluster
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

I have built test packages, for both trusty and xenial:

https://launchpad.net/~mariosplivalo/+archive/ubuntu/lp1657256

Changed in percona-xtradb-cluster-5.6 (Ubuntu):
assignee: nobody → Mario Splivalo (mariosplivalo)
status: New → In Progress
Changed in percona-xtradb-cluster-5.5 (Ubuntu):
assignee: Rafael David Tinoco (inaddy) → Mario Splivalo (mariosplivalo)
Victor Tapia (vtapia)
Changed in percona-xtradb-cluster-5.6 (Ubuntu):
importance: Undecided → Medium
Louis Bouchard (louis)
no longer affects: percona-xtradb-cluster-5.5 (Ubuntu Xenial)
no longer affects: percona-xtradb-cluster-5.5 (Ubuntu Yakkety)
no longer affects: percona-xtradb-cluster-5.6 (Ubuntu Trusty)
no longer affects: percona-xtradb-cluster-5.5 (Ubuntu Zesty)
Changed in percona-xtradb-cluster-5.5 (Ubuntu Trusty):
assignee: nobody → Mario Splivalo (mariosplivalo)
Changed in percona-xtradb-cluster-5.6 (Ubuntu Xenial):
assignee: nobody → Mario Splivalo (mariosplivalo)
Changed in percona-xtradb-cluster-5.6 (Ubuntu Yakkety):
assignee: nobody → Mario Splivalo (mariosplivalo)
Changed in percona-xtradb-cluster-5.5 (Ubuntu Trusty):
status: New → In Progress
Louis Bouchard (louis)
Changed in percona-xtradb-cluster-5.5 (Ubuntu Trusty):
importance: Undecided → Medium
Changed in percona-xtradb-cluster-5.6 (Ubuntu Xenial):
importance: Undecided → Medium
Changed in percona-xtradb-cluster-5.6 (Ubuntu Yakkety):
importance: Undecided → Medium
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in percona-xtradb-cluster-5.6 (Ubuntu Xenial):
status: New → Confirmed
Changed in percona-xtradb-cluster-5.6 (Ubuntu Yakkety):
status: New → Confirmed
Revision history for this message
Ryan Beisner (1chb1n) wrote :

Seeing the following with pxc 5.6 xenial ppc64el:

http://paste.ubuntu.com/24287991/

root@juju-c5895e-11:/var/log# dpkg -l | egrep 'mysql|percona'
ii libdbd-mysql-perl 4.033-1ubuntu0.1 ppc64el Perl5 database interface to the MySQL database
ii libmysqlclient20:ppc64el 5.7.17-0ubuntu0.16.04.1 ppc64el MySQL database client library
ii mysql-client 5.7.17-0ubuntu0.16.04.1 all MySQL database client (metapackage depending on the latest version)
ii mysql-client-5.7 5.7.17-0ubuntu0.16.04.1 ppc64el MySQL database client binaries
ii mysql-client-core-5.7 5.7.17-0ubuntu0.16.04.1 ppc64el MySQL database core client binaries
ii mysql-common 5.7.17-0ubuntu0.16.04.1 all MySQL database common files, e.g. /etc/mysql/my.cnf
ii percona-galera-3 3.19-0ubuntu0.16.04.1 ppc64el Galera replication framework for Percona XtraDB Cluster
ii percona-xtrabackup 2.3.7-0ubuntu0.16.04.1 ppc64el Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-server-5.6 5.6.34-26.19-0ubuntu0.16.04.1 ppc64el Percona XtraDB Cluster database server binaries
ii python-mysqldb 1.3.7-1build2 ppc64el Python interface to MySQL

Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

I'm attaching a debdiff that fixes the issue for trusty.

I am currently working on creating a patch for zesty, going down to xenial.

description: updated
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

I've also put the patched package for for trusty into this ppa:
https://launchpad.net/~mariosplivalo/+archive/ubuntu/lp1657256-sru

Revision history for this message
Robie Basak (racb) wrote :

This needs Zesty fixed before we can upload SRUs, so unsubscribing ~ubuntu-sponsors for now. Please resubscribe ~ubuntu-sponsors once you have a Zesty debdiff ready.

I realise they're two separate source packages, but from the POV of the underlying reason for the "fix development first" SRU requirement this is a technicality and the reasoning still applies to this case IMHO.

description: updated
no longer affects: percona-xtradb-cluster-5.6 (Ubuntu Zesty)
no longer affects: percona-xtradb-cluster-5.6 (Ubuntu)
no longer affects: percona-xtradb-cluster-5.6 (Ubuntu Xenial)
no longer affects: percona-xtradb-cluster-5.6 (Ubuntu Yakkety)
Eric Desrochers (slashd)
tags: added: sts-sru-needed
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

I have removed percona-xtradb-cluster-5.6 from this bug as in Trusty we have percona-xtradb-cluster-5.5, and in xenial (and above) we have percona-xtradb-cluster-5.6. I've created a separate bug for Percona in xenial (and above): #1693765.

The reason for separation is to speed up SRU for Trusty as the patch ready (it is attached to this bug) - the customer can resort to using Percona on Trusty when this fix lands.

Percona in trusty uses InnoDB 5.5 where in Xenial and above it is Innodb 5.6, with different codebase so the fix for the issue is different.

Eric Desrochers (slashd)
description: updated
Eric Desrochers (slashd)
tags: added: sts-sru-done
removed: sts-sru-needed
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Mario, or anyone else affected,

Accepted percona-xtradb-cluster-5.5 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.5/5.5.37-25.10+dfsg-0ubuntu0.14.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in percona-xtradb-cluster-5.5 (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :
Download full text (3.4 KiB)

I have verified the package from -proposed, and it indeed fixes the bug.

I first deployed percona-xtradb-cluster-server from trusty-updates:

root@mario-trusty-percona-sru-verification:~# apt-cache policy percona-xtradb-cluster-server
percona-xtradb-cluster-server:
  Installed: 5.5.37-25.10+dfsg-0ubuntu0.14.04.2
  Candidate: 5.5.37-25.10+dfsg-0ubuntu0.14.04.2
  Version table:
 *** 5.5.37-25.10+dfsg-0ubuntu0.14.04.2 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty-updates/universe ppc64el Packages
        100 /var/lib/dpkg/status
     5.5.37-25.10+dfsg-0ubuntu0.14.04.1 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty-security/universe ppc64el Packages
     5.5.34-25.9+dfsg-0ubuntu4 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty/universe ppc64el Packages
root@mario-trusty-percona-sru-verification:~# apt-cache policy percona-xtradb-cluster-server-5.5
percona-xtradb-cluster-server-5.5:
  Installed: 5.5.37-25.10+dfsg-0ubuntu0.14.04.2
  Candidate: 5.5.37-25.10+dfsg-0ubuntu0.14.04.2
  Version table:
 *** 5.5.37-25.10+dfsg-0ubuntu0.14.04.2 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty-updates/universe ppc64el Packages
        100 /var/lib/dpkg/status
     5.5.37-25.10+dfsg-0ubuntu0.14.04.1 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty-security/universe ppc64el Packages
     5.5.34-25.9+dfsg-0ubuntu4 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty/universe ppc64el Packages
root@mario-trusty-percona-sru-verification:~#

Then I run sysbench tests from the control node, and percona crashed, as expected (and explained in this bug):

http://paste.ubuntu.com/24739196/

I then enabled -proposed pocket and upgraded percona packages:

root@mario-trusty-percona-sru-verification:~# apt-cache policy percona-xtradb-cluster-server
percona-xtradb-cluster-server:
  Installed: 5.5.37-25.10+dfsg-0ubuntu0.14.04.3
  Candidate: 5.5.37-25.10+dfsg-0ubuntu0.14.04.3
  Version table:
 *** 5.5.37-25.10+dfsg-0ubuntu0.14.04.3 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty-proposed/universe ppc64el Packages
        100 /var/lib/dpkg/status
     5.5.37-25.10+dfsg-0ubuntu0.14.04.2 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty-updates/universe ppc64el Packages
     5.5.37-25.10+dfsg-0ubuntu0.14.04.1 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty-security/universe ppc64el Packages
     5.5.34-25.9+dfsg-0ubuntu4 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty/universe ppc64el Packages
root@mario-trusty-percona-sru-verification:~# apt-cache policy percona-xtradb-cluster-server-5.5
percona-xtradb-cluster-server-5.5:
  Installed: 5.5.37-25.10+dfsg-0ubuntu0.14.04.3
  Candidate: 5.5.37-25.10+dfsg-0ubuntu0.14.04.3
  Version table:
 *** 5.5.37-25.10+dfsg-0ubuntu0.14.04.3 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty-proposed/universe ppc64el Packages
        100 /var/lib/dpkg/status
     5.5.37-25.10+dfsg-0ubuntu0.14.04.2 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty-updates/universe ppc64el Packages
     5.5.37-25.10+dfsg-0ubuntu0.14.04.1 0
        500 http://ports.ubuntu.com/ubuntu-ports/ trusty-security/universe ppc64el Packages
     5.5.34-25.9...

Read more...

tags: added: verification-done
removed: verification-needed
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

I have also tested the package under Intel architectures to make sure no regression is introduced and, indeed, the sysbench test completes without an error.

Also, there is failed buidl for powerpc architecture, but this was not introduced by this change - one can check that this has been like this in previous releases too:

https://launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.5

If you expand previous versions (14.04.1 and 14.04.2) you can see that the build was failing for powerpc.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

It seems I made a mistake by publishing this to -proposed, not sure what happened in my head. This needs to be released *first* to the devel series *at least*. I request this being blocked in the -proposed pocket until all other series don't at least get released to artful (so bug LP: #1693765).

tags: added: block-proposed
Revision history for this message
Brian Murray (brian-murray) wrote :

"I have removed percona-xtradb-cluster-5.6 from this bug as in Trusty we have percona-xtradb-cluster-5.5, and in xenial (and above) we have percona-xtradb-cluster-5.6. I've created a separate bug for Percona in xenial (and above): #1693765."

Given that percona-xtradb-cluster-server-5.6 replaces percona-xtradb-cluster-server-5.5 and the same bug is being fixed in both packages (even though they happen to have different names) the work should be tracked in one bug report. Splitting the work up into two separate bug reports is unnecessary and creates confusion for both users and the Ubuntu Stable Release Updates team.

Eric Desrochers (slashd)
no longer affects: percona-xtradb-cluster-5.5 (Ubuntu Yakkety)
no longer affects: percona-xtradb-cluster-5.5 (Ubuntu Zesty)
no longer affects: percona-xtradb-cluster-5.5 (Ubuntu Xenial)
Changed in percona-xtradb-cluster-5.6 (Ubuntu):
importance: Undecided → Medium
Changed in percona-xtradb-cluster-5.6 (Ubuntu Xenial):
importance: Undecided → Medium
Changed in percona-xtradb-cluster-5.6 (Ubuntu Yakkety):
importance: Undecided → Medium
Changed in percona-xtradb-cluster-5.6 (Ubuntu Zesty):
importance: Undecided → Medium
Changed in percona-xtradb-cluster-5.6 (Ubuntu):
assignee: nobody → Mario Splivalo (mariosplivalo)
Changed in percona-xtradb-cluster-5.6 (Ubuntu Xenial):
assignee: nobody → Mario Splivalo (mariosplivalo)
Changed in percona-xtradb-cluster-5.6 (Ubuntu Yakkety):
assignee: nobody → Mario Splivalo (mariosplivalo)
Changed in percona-xtradb-cluster-5.6 (Ubuntu Zesty):
assignee: nobody → Mario Splivalo (mariosplivalo)
Changed in percona-xtradb-cluster-5.6 (Ubuntu Xenial):
status: New → Confirmed
Changed in percona-xtradb-cluster-5.6 (Ubuntu Yakkety):
status: New → Confirmed
Changed in percona-xtradb-cluster-5.6 (Ubuntu Zesty):
status: New → Confirmed
tags: added: verification-done-trusty
removed: verification-done
Revision history for this message
Mario Splivalo (mariosplivalo) wrote :

Hi, Brian, Łukasz.

I understand; the main reasoning for splitting up the SRU work is so that trusty version can be released to trusty-updates sooner so that end users can start using it.

But, I understand the implications of splitting the SRU - I will continue my work on fixing xenial-and-above, but this will take more time as percona-5.6 changed quite a bit.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

As per previous discussion on IRC and other places, I have removed the update from trusty-proposed to unblock other fixes to the package. The reason being: SRU postponed until the bug gets fixed in 5.6 versions of the package in the current devel series.

Feel free to re-upload the fix once at least the current development series has the bug fixed. Thanks and apologies for the bump-back!

Changed in percona-xtradb-cluster-5.5 (Ubuntu Trusty):
status: Fix Committed → Confirmed
tags: removed: block-proposed verification-done-trusty
Eric Desrochers (slashd)
tags: added: sts-sru-needed
removed: sts-sru-done
Changed in percona-xtradb-cluster-5.6 (Ubuntu):
status: New → In Progress
assignee: Mario Splivalo (mariosplivalo) → Jorge Niedbalski (niedbalski)
Changed in percona-xtradb-cluster-5.5 (Ubuntu):
assignee: Mario Splivalo (mariosplivalo) → Jorge Niedbalski (niedbalski)
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

Hello Sponsors,

Attached you can find a debdiff for the artful series which fixes the issue expressed
on this bug for powerpc architecture.

The patch cover multiple commits from upstream MariaDB,

* Commit 6ea41f1e84eb6b864cac17ad0b862bde9820dc33: Implements
the os_fast_mutex(unlock_full_barrier/trylock_full_barrier) which
implies a full memory barrier.

It also implements the os_mb OS memory barrier primitives for the
different architectures which is guaranteed to act as
full acquire-release barrier.

* Commit 5ad02062d928cccbd29c0a2db6f0f7ceb33195d1: Based on the
previous commit, it adds a memory barrier on the mutex_exit
to prevent reading the number of waiters before releasing the lock,
this is completed with a modification on mutex_set_waiters that adds
a os_mb to make sure waiters store won't pass over mutex_test_and_set.

* Commit 40497577ffd9f85557b15e08ad913f627b2e9530: The ib mutex set_and_test
function and acquires the lock by calling the os_atomic_test_and_set_byte_acquire
method which should be safe in PowerPC, at the same time the mutex unlock method
will call the os_atomic_test_and_set_byte_release which is implemented as a
__sync_lock_release + a __sync_synchronize call in PowerPC.

I've already performed the verification marked on this bug, with the following results:
http://pastebin.ubuntu.com/25286767/

Eric Desrochers (slashd)
tags: added: patch
Revision history for this message
Robie Basak (racb) wrote :

Thanks! Sponsored because I appreciate the delay already. But please file a bug with Percona upstream and link to it here. Then hopefully on future updates we can drop the patch. Otherwise we'll slowly build up patches to this package over time and it'll become an unmaintainable mess.

Revision history for this message
Ryan Beisner (1chb1n) wrote :

This is impacting >= Xenial ppc64el deployments and testing for OpenStack Charms. Please let us know if there is a PPA with the fixed package available. Thank you.

tags: added: ppc64el uosci
Changed in charm-test-infra:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Ryan Beisner (1chb1n)
Revision history for this message
Eric Desrochers (slashd) wrote :

Hi Ryan,

The package has been sponsored by Robie but didn't successfully build[1] due to a certain recent bump on the gcc version from what I heard.

Jorge and Robie will probably know more about the details and the next step.

[1] - https://launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/5.6.34-26.19-0ubuntu3

Thanks,
Eric

Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

Hey Ryan,

Yes .. after the latest update of gcc to 7.2, the package (with and without the proposed patch applied) fails to build. Any subsequent rebuild of percona on the archive will fail to build.

I created another bug for fixing the build (https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/+bug/1714554) and I am currently working on it.

Once that's fixed, I will provide you a PPA and robbie can re-upload this patch into proposed.

Thanks.

Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

Hello Beisner,

I built a PPA here https://launchpad.net/~niedbalski/+archive/ubuntu/lp1657256/ for
ppc64el with the fix applied. Could you please check if your tests passes with this PPA?

Thanks.

Changed in percona-xtradb-cluster-5.5 (Ubuntu):
status: In Progress → Confirmed
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :
Revision history for this message
Ryan Beisner (1chb1n) wrote :
Revision history for this message
Sean Feole (sfeole) wrote :
Download full text (4.5 KiB)

Deploying Openstack Ocata, w/ Juju on arm64 hosts, the mysql crashed with the following:

2017-09-20 21:38:39 17783 [Warning] 'user' entry 'root@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:38:39 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:38:39 17783 [Warning] 'user' entry 'sstuser@ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:38:39 17783 [Warning] 'proxies_priv' entry '@ root@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:02 17783 [Warning] 'user' entry 'root@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:02 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:02 17783 [Warning] 'user' entry 'sstuser@ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:39:02 17783 [Warning] 'proxies_priv' entry '@ root@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:20 17783 [Warning] 'user' entry 'root@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:20 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:20 17783 [Warning] 'user' entry 'sstuser@ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:39:20 17783 [Warning] 'proxies_priv' entry '@ root@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:41 17783 [Warning] 'user' entry 'root@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:41 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:39:41 17783 [Warning] 'user' entry 'sstuser@ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:39:41 17783 [Warning] 'proxies_priv' entry '@ root@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:53 17783 [Warning] 'user' entry 'root@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:53 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:53 17783 [Warning] 'user' entry 'sstuser@ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:40:53 17783 [Warning] 'proxies_priv' entry '@ root@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:56 17783 [Warning] 'user' entry 'root@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:56 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:56 17783 [Warning] 'user' entry 'sstuser@ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:40:56 17783 [Warning] 'proxies_priv' entry '@ root@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:58 17783 [Warning] 'user' entry 'root@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:58 17783 [Warning] 'user' entry '@juju-24a047-0-lxd-1' ignored in --skip-name-resolve mode.
2017-09-20 21:40:58 17783 [Warning] 'user' entry 'sstuser@ip6-localhost' ignored in --skip-name-resolve mode.
2017-09-20 21:40:58 17783 [Warning] 'proxies_pri...

Read more...

Revision history for this message
Sean Feole (sfeole) wrote :
tags: added: arm64
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

Hello Sean,

I built a PPA here https://launchpad.net/~niedbalski/+archive/ubuntu/lp1657256/ for
ppc64el with the fix applied. Could you please check if your tests passes with this PPA?

Thanks!

Revision history for this message
Sean Feole (sfeole) wrote :

Hey Jorge,

I need a build for arm64, i only see the pkg available for Builds

[FULLYBUILT] amd64
[FULLYBUILT] ppc64el

Revision history for this message
Manoj Kumar (manojnkumar) wrote :

I tried this on ppc64el and see this issue:

2017-09-21 20:07:06 DEBUG install Some packages could not be installed. This may mean that you have
2017-09-21 20:07:06 DEBUG install requested an impossible situation or if you are using the unstable
2017-09-21 20:07:06 DEBUG install distribution that some required packages have not yet been created
2017-09-21 20:07:06 DEBUG install or been moved out of Incoming.
2017-09-21 20:07:06 DEBUG install The following information may help to resolve the situation:
2017-09-21 20:07:06 DEBUG install
2017-09-21 20:07:06 DEBUG install The following packages have unmet dependencies:
2017-09-21 20:07:06 DEBUG install percona-xtradb-cluster-server-5.6 : Depends: libevent-2.1-6 (>= 2.1.8-stable) but it is not installable
2017-09-21 20:07:06 DEBUG install E: Unable to correct problems, you have held broken packages.
2017-09-21 20:07:06 DEBUG worker.uniter.jujuc server.go:178 running hook tool "juju-log"
2017-09-21 20:07:06 INFO juju-log Couldn't acquire DPKG lock. Will retry in 10 seconds

Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

Which series? the PPA just covers artful. fyi,

Revision history for this message
Manoj Kumar (manojnkumar) wrote :

Sorry, xenial is what I need.

Revision history for this message
Sean Feole (sfeole) wrote :

Hey Jorge,

I applied the updated deb last night to openstack env, mysql did not crash overnight and I was able to get 10+ sysbench runs on my test machine without a crash. I'll keep sysbench running throughout the day to ensure the crash does not occur again.

looks ok so far.

no longer affects: percona-xtradb-cluster-5.5 (Ubuntu Artful)
Eric Desrochers (slashd)
no longer affects: percona-xtradb-cluster-5.6 (Ubuntu Yakkety)
Revision history for this message
dann frazier (dannf) wrote :

fyi, looks like the same issue we fixed in MySQL/MariaDB a couple years ago - LP: #1427406

Revision history for this message
Manoj Kumar (manojnkumar) wrote :

Can you build a PPA for Xenial?

Revision history for this message
Jorge Niedbalski (niedbalski) wrote :
Revision history for this message
Jason Furmanek (furmanek) wrote :

Jorge,
I tied this install now on Xenial, but am getting the same error about libevent being not installable:

2017-09-28 23:08:14 INFO install Some packages could not be installed. This may mean that you have
2017-09-28 23:08:14 INFO install requested an impossible situation or if you are using the unstable
2017-09-28 23:08:14 INFO install distribution that some required packages have not yet been created
2017-09-28 23:08:14 INFO install or been moved out of Incoming.
2017-09-28 23:08:14 INFO install The following information may help to resolve the situation:
2017-09-28 23:08:14 INFO install
2017-09-28 23:08:14 INFO install The following packages have unmet dependencies:
2017-09-28 23:08:14 INFO install percona-xtradb-cluster-server-5.6 : Depends: libevent-2.1-6 (>= 2.1.8-stable) but it is not installable
2017-09-28 23:08:14 INFO install E: Unable to correct problems, you have held broken packages.

Xenial has libevent 2.0-5. Has this package been created properly for Xenial, or am I doing something dumb?

Thanks!

Revision history for this message
Manoj Kumar (manojnkumar) wrote :

I hope not Jason. I am running into the same issue on Xenial:

2017-09-27 21:48:07 DEBUG install percona-xtradb-cluster-server-5.6 : Depends: libevent-2.1-6 (>= 2.1.8-stable) but it is not installable
2017-09-27 21:48:07 DEBUG install E: Unable to correct problems, you have held broken packages.

Revision history for this message
Eric Desrochers (slashd) wrote :

@racb,

Could you please do a 2nd review of Jorge's newest debdiff attached in comment #38.

Thanks in advance

- Eric

Revision history for this message
Manoj Kumar (manojnkumar) wrote :

The existing PPA does not work on ppc64/xenial because of the libevent dependancy? Can we get that resolved?

Revision history for this message
Robie Basak (racb) wrote :

There seems to be a ton of confusion here. I have commented to various people on IRC as they have asked me. But the messages don't seem to have been received. I'll try and summarize my opinion here.

0. I asked for a link to document upstream's rejection of this patch in comment 33 and on IRC[1].

1. It seems inappropriate for an SRU affecting users to be blocked on an unrelated FTBFS fix for the development release, such as in this case where the build for the actual fix in the development release has got caught up with the transition to gcc-7. I would not block the progress of the SRU on this.

2. However, if the fixing of the development release is trivial, then I'd expect it to be done to minimise risk to users in the future, in the spirit of the SRU policy. In other words, if it takes just half an hour of developer time, can we just do it? If what I think is the most appropriate fix for the development release turns out to be non-trivial, then sure, we can proceed with the SRU without it.

3. To fix this in the development release, I suggested a minimal distro patch that stops warning->error for non-bugs is OK to fix this and suggested that you could look at a recent and similar fix in squid3 for an example[2].

4. In response to the debdiff in comment 38, I said that I didn't understand why you didn't follow my previous suggestion, and that the approach there is not suitable for the development release since it would hold back the gcc-7 transition rather than progressing it[3].

[1] https://irclogs.ubuntu.com/2017/09/01/%23ubuntu-devel.html#t15:59
[2] https://irclogs.ubuntu.com/2017/09/01/%23ubuntu-devel.html#t16:20
[3] https://irclogs.ubuntu.com/2017/09/19/%23ubuntu-devel.html#t16:14

To make progress, in the first instance, please work on 3. If it turns out to be non-trivial, please talk to me again. I'd also appreciate you resolving 0 but I don't think that's a blocker right now.

Revision history for this message
Robie Basak (racb) wrote :

Unsubscribing ~ubuntu-sponsors for now as I have declined the current debdiff for artful and so there is nothing left to sponsor.

Changed in percona-xtradb-cluster-5.6 (Ubuntu Artful):
importance: Medium → High
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

Hello Robie,

>3. To fix this in the development release, I suggested a minimal distro patch that stops warning->error for non-bugs is OK to fix this and suggested that you could look at a recent and similar fix >in squid3 for an example[2].

The definitive fix for all the warnings (>10k) manifested at build time
is a non-trivial change that I don't have time capacity to address in all its extention
and will require attention from the percona team itself to be fixed.

As you mentioned on [3] I am attaching a new patch which basically hides the warnings triggered at compile time by removing the -Werror flags on CXX_FLAGS and C_FLAGS and passing the -fpermissive flag, which fixes the FTBFS manifested on the current -proposed package and allows
the package to build and work on the affected architectures.

Please review it and let me know if you have any concern or question.

Thank you for your time, review and sponsorship.

Revision history for this message
Robie Basak (racb) wrote :

21:57 <rbasak> niedbalski: thanks! Quick glance. 5.6.34-26.19-0ubuntu3 is already taken. Can you prepare a debdiff against the already uploaded 5.6.34-26.19-0ubuntu3 to 5.6.34-26.19-0ubuntu4? I expect that debdiff to include only the FTBFS fix, as I think the other necessary fixes are already present in 5.6.34-26.19-0ubuntu3?

Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

<niedbalski>rbasak, ok, I don't think is going to be possible, I incorporated a small nitpick on the original patch (that prevents a warning), thus the ibuf* patch actually differs from ubuntu3.

<rbasak>niedbalski: if you want to update what you did in ubuntu3, that's fine. Just include it in ubuntu4 and bullet point that additional change separately in the changelog.
<niedbalski>rbasak, perfect.
<rbasak>niedbalski: but ubuntu4's changelog *shouldn't* include things that are already covered in ubuntu3's changelog. Does that make sense?
<niedbalski>rbasak, yes, it does.

---

Robbie,

According to what we discussed on #ubuntu-devel, I am attaching a new
version of the patch on top of the current -proposed version for your review.

Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

I've built a Xenial PPA with the fix (ppa:niedbalski/xenial-lp1657256) for testing.

Revision history for this message
Manoj Kumar (manojnkumar) wrote :

Jorge: The latest did not install (I am using a juju charm bundle to deploy openstack):

2017-10-08 15:41:09 DEBUG install WARNING: The following packages cannot be authenticated!
2017-10-08 15:41:09 DEBUG install percona-xtradb-cluster-server-5.6
2017-10-08 15:41:09 DEBUG install E: There were unauthenticated packages and -y was used without --allow-unauthenticated

Revision history for this message
Robie Basak (racb) wrote :

I found this quite complicated to review, so to break it down I imported the package into git and then split your debdiff up into multiple git commits and then applied the relevant quilt patches as commits at the end so that I could see what was going on. I've pushed this to https://git.launchpad.net/~racb/ubuntu/+source/percona-xtradb-cluster-5.6/log/?h=niedbalski&id=niedbalski.c59 and I'll refer to commits from there in my review as I disect your debdiff.

1. Whitespace cleanups in 6624138 from your previous upload are fine I suppose, though they did make the debdiff more difficult to review by creating debdiff noise. But if you're going to clean up like this, please could you also take control of your trailing whitespace in debian/changelog in your latest changes in 371f0bc?

2. Quilt noise in 75439ba is fine, though the did make the debdiff more difficult to review by creating debdiff noise. Please configure your quilt according to https://wiki.debian.org/UsingQuilt (-p ab --no-timestamps --no-index) so that quilt patches your generate are always normalised. This will stop this happening and speed up my reviews.

With those cleaned up, I could then see through the noise through to the substance of your proposed changes.

3. Commit 18f879d. I thought the purpose of this upload was to stop the
warnings being fatal? Why in addition are we fixing up warnings? I thought this
patch was cherry-picked from upstream? Is this additional change upstream? And
if os_mb is now defined for all architectures, presumably all the calls to
os_mb elsewhere in this patch no longer want to be guarded conditionally on
architecture? I'm not yet sure what I'm asking for here. I think I need to give
the full patch more thorough review (which might impact what's going into the
SRU). If the entire patch can demonstrably only affect ppc64el, then I'd be
happier. I need to check if this is the case.

4. Commit 6842cbe. I understand the disabling of -Werror. Is adding -fpermissive strictly required or did the build work without? Why is dropping -Wextra required? And in the diff, why are you dropping -Wall, -Wextra, -Wunused, -Wwrite-strings, etc? Won't the build still succeed with those warnings but not being converted to errors with just a drop of -Werror, or am I missing something?

5. Commit 86ca582. I don't understand the purpose of this change and debian/changelog doesn't explain it either. Please advise.

6. Commit cc4a8b7. I don't understand the purpose of this change and debian/changelog doesn't explain it either. Please advise.

7. Commit 371f0bc5. debian/changelog has trailing whitespace. I really don't care when reviewing others' work, but if you also don't care, then why clean up trailing whitespace in your quilt patch, creating extra diff noise and review pain?

Revision history for this message
Manoj Kumar (manojnkumar) wrote :

Jorge: When I installed it manually, the openstack cluster came up fine.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Download full text (4.4 KiB)

I was asked to help with the review and on a first glimpse agree with Robie c#62.

TL;DR:
1. munching various changes together makes it hard to review
2. several changes are not clear why they were added to the debdiff
2. We are in Final Freeze already, so SRU Rules apply -> lets fix the FTBFS and drop all not needed changes which make the change easier to grasp, better to review and safer to apply.

If you could adapt a git based workflow for complex uploads that would help review and changes as well.

I'll rework the patch in a way I assume it might work.
Then I ask Jorge to review if this is still ok with him.

I can recreate the build errors on the last upload.
It matches those in [1].

I checked Robies comments above and referencing his changes I did:
1. really not needed atm - dropping
2. really not needed atm - dropping
3. the effect of this depends on defines like IB_STRONG_MEMORY_MODEL
   For Intel this is still a no-op (as the other defines), just now a defined
   instead of an undefined no-op define.
   => keep this change
4. Yes this is like "compiler please check nothing" which I think is too much
   The code actually already takes a very detailed list of ignores to run with -Werror despite
   some warnings. If you look at [1] you see e.g. -Wno-error=nonnull-compare and
   -Wno-error=unused-result in combination with -Werror. That way it is safe, but just allows
   a few.
   Dropping -Werror in general and even setting fpermissive [2] is much more than needed
   This is new in gcc7 [3] and that is what we look for.
   => use -Wno-error=implicit-fallthrough which will keep the warning, but make it non fatal
   => tests showed a more cases that are needed like format-overflow, ... (added as needed)
   => See below for details on permissive
5. nice cleanup but not needed
   => drop
6. This might be to make the defines available earlier?
   I tested to build without and it was fine on x86.
7. I'll have to rewrite the changelog anyway, so that will be cleaned

This exercise (other than review) of rewriting the fix was mostly disabling the new errors, but one was special. It now better detects ISO standard violations.
That causes:
  "error: ISO C++ forbids comparison between pointer and integer [-fpermissive]"#
Now one might think, lets add that because the doc says:
 -fpermissive
     Downgrade some diagnostics about nonconformant code from errors
     to warnings. Thus, using -fpermissive allows some nonconforming
     code to compile.

But that makes something that is an error a warning, which still is an error with -Werror.
You can't further downgrade this with -Wno-... [4]
Instead these errors need to be fixed in code - otherwise we would have to drop -Werrror which I would not like.

Test Program:
#include <iostream>
using namespace std;

int main()
{
    const char *str = "Foo";
    if (str != '\0')
    {
        cout << "Hello, World!";
    }
    return 0;
}

I fixed it with a in-source fix "fix-gcc7-compiler-errors.patch".
I'd ask you Jorge, to please discuss to upstream this.

After quite some iterations I was successful with that on a local build.
Switching to cross arch build on Launchpad to be entirely sure - this is s...

Read more...

Ryan Beisner (1chb1n)
Changed in charm-test-infra:
importance: High → Low
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :
Download full text (3.7 KiB)

Christian, Robbie:

Thanks for taking the time to review the patch that I've submitted. I think that
the latest patch that Christian worked on covers and clean most of the observations made by Robbie
earlier in the bug and I can confirm that it doesn't modifies the original patch with any substantial changes to what I am proposing to fix via this bug.

I've 2 observations in regards to comment #64.

1) I can definitively work with upstream to the fix made in "fix-gcc7-compiler-errors.patch",
I will make a reference to this bug in order to keep all the further work done in upstream in sync with this bug.

2) I have performed testing on top of the PPA[0] and the results seems to work and confirm
the original results of the patch, that means:

a) The sysbench OLTP completes with 50,100 and 200 concurrent threads.
b) No further segfaults are being trapped on the machine for the mysqld process.

A detail on the tests performed:

root@niedbalski-artful:~# dpkg -l | grep percona
ii percona-galera-3 3.19-0ubuntu1 ppc64el Galera replication framework for Percona XtraDB Cluster
ii percona-xtrabackup 2.3.7-0ubuntu2 ppc64el Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-5.6-dbg 5.6.34-26.19-0ubuntu4~ppa11 ppc64el Debugging package for Percona XtraDB Cluster
ii percona-xtradb-cluster-server 5.6.34-26.19-0ubuntu4~ppa11 all Percona XtraDB Cluster database server
ii percona-xtradb-cluster-server-5.6 5.6.34-26.19-0ubuntu4~ppa11 ppc64el Percona XtraDB Cluster database server binaries
ii percona-xtradb-cluster-source-5.6 5.6.34-26.19-0ubuntu4~ppa11 ppc64el Percona XtraDB Cluster 5.6 source

Version: '5.6.34-79.1-79.1' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona XtraDB Cluster (GPL), Release 5.6.34-26.19.4c779b7, wsrep_26.19

~# sysbench --test=oltp --mysql-host=10.75.206.156 --mysql-user=test --mysql-db=dbtest --oltp-test-mode=complex --max-requests=0 --max-time=130 --num-threads=100 run
sysbench 0.4.12: multi-threaded system evaluation benchmark

No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 100

Doing OLTP test.
Running mixed OLTP test
Using Special distribution (12 iterations, 1 pct of values are returned in 75 pct cases)
Using "BEGIN" for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting...
(last message repeated 99 times)
Done.

OLTP test statistics:
    queries performed:
        read: 5309948
        write: 1794076
        other: 719120
        total: 7823144
    transactions: 339838 (2613.49 per sec.)
    deadlocks: 39444 (303.34 per sec.)
    read/write requests: 7104024 (54632.84 per sec.)
    other operations: 719120 (5530.33 per sec.)

Test execution summary:
    total time: 130.0321s
    total number of events: 339838
    total time taken b...

Read more...

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI - further work on the FTBFS on bug 1714554 (the one about the FTBFS) - if/once this gets in it will unblock this bug here as well.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package percona-xtradb-cluster-5.6 - 5.6.34-26.19-0ubuntu4

---------------
percona-xtradb-cluster-5.6 (5.6.34-26.19-0ubuntu4) artful; urgency=high

  [ Jorge Niedbalski ]
  * Fix FTBFS with gcc7 (LP: #1714554)
    - d/p/ibuf-uses-full-memory-barrier-ppc64.patch: Modified to prevent a
      warning at compile time due to the inexistence of the default os_mb.

  [ Christian Ehrhardt ]
  * Fix FTBFS with gcc7 - further fixes to LP 1714554
    - d/rules: make warnings new/enhanced in gcc7 non-fatal in regard to Werror:
      format-overflow, implicit-fallthrough, int-in-bool-context, nonnull,
      aligned-new (only c++), format-truncation
    - d/p/fix-gcc7-compiler-errors.patch: fix gcc-7 errors not downgradable
      to warnings (ISO C++ forbids comparison between pointer and integer)

percona-xtradb-cluster-5.6 (5.6.34-26.19-0ubuntu3) artful; urgency=high

  * d/p/ibuf-uses-full-memory-barrier-ppc64.patch: This patch implements
    a full memory barrier for InnoDB mutex entry/exit on PowerPC.
    (LP: #1657256).

  * d/p/debian/patches/weak-memory-compat.patch: Removed as is already
    covered by the newer patch.

 -- Christian Ehrhardt <email address hidden> Tue, 17 Oct 2017 10:16:36 +0200

Changed in percona-xtradb-cluster-5.6 (Ubuntu Artful):
status: In Progress → Fix Released
Changed in percona-xtradb-cluster-5.6 (Ubuntu Xenial):
status: Confirmed → In Progress
importance: Medium → High
assignee: Mario Splivalo (mariosplivalo) → Jorge Niedbalski (niedbalski)
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

Hello @paelzer,

I linked the MP with the proposal against ubuntu/xenial for your review and
validation.

Thanks.

Changed in percona-xtradb-cluster-5.5 (Ubuntu Trusty):
status: Confirmed → In Progress
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

Hello @paelzer,

I reviewed/tested the SRU made by Mario earlier on this bug and considering
that the devel series has the fix included already, I think there is no
other blocker to proceed with the Trusty SRU.

I am re-attaching the debdiff for your review.

Thanks.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi,
I reviewed the MP and it LGTM.
Also I'm ok with the debdiff for trusty given that it was already reviewed before.

The one thing missing is to also fix it in zesty.
Just as much as in Artful the point is to avoid users regressing when upgrading.
So the usual rule of thumb is:
1. current Development release
2. all affected supported releases

Without zesty one could have it working through T->X, but -> Z he would fail.
Also I have some comments on the MP that need fixing, but it LGTM so we will get that done soon.
See the MP comments for Details.

Changed in percona-xtradb-cluster-5.5 (Ubuntu Trusty):
assignee: Mario Splivalo (mariosplivalo) → Jorge Niedbalski (niedbalski)
Changed in percona-xtradb-cluster-5.6 (Ubuntu Zesty):
assignee: Mario Splivalo (mariosplivalo) → Jorge Niedbalski (niedbalski)
status: Confirmed → In Progress
importance: Medium → High
Changed in percona-xtradb-cluster-5.5 (Ubuntu Trusty):
importance: Medium → High
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

FYI:
- Done reviewing branches in general
- Niedbalski and I fixed several things already (whitespace, versions, correct author attribution, rebase to correct base versions, ...)
- X branch looks good and builds/tests fine
- T branch FTBFS on all !x86 in [1]
- Z branch FTBFS on all !x86 in [1]

TODO:
- Niedbalski to look into the Zesty build issues
- Niedbalski to look into the Trusty build issues
- Niedbalski to test the Zesty&Trusty builds once working

Note: please look in the MPs for more details (build logs, fixups and so on)

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3005

Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

@paelzer,

- For the Trusty build failure I've submitted a MP on top of your branch , that proposal is intended to fix the FTBFS problem on ppc64 [0],
there is PPA[1] for testing it. I'd appreciate if you can review it and merge if appropriate.

- I am currently investigating the origin of the Z build issues in most of the
architectures, once I've this done and fixed I'll ping you back.

[0] https://code.launchpad.net/~niedbalski/ubuntu/+source/percona-xtradb-cluster-5.5/+git/percona-xtradb-cluster-5.5/+merge/332794
[1] https://launchpad.net/~niedbalski/+archive/ubuntu/fix-1657256-t/

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

After a few rounds of cleanup we made it.
There was enough cleanup to be worth a retest - other than that we are ready IMO.

@Jorge - could you do a final run on the T/X/Z builds in [1] to be sure before I sponsor.

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3005

Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

@paelzer,

Thanks for providing a PPA.

I've verified your PPA for Zesty [0] Xenial [1] and Trusty[2] on ppc64el and
the problem isn't longer reproduced.

[0] http://pastebin.ubuntu.com/25825884/
[1] https://pastebin.com/raw/QpQScU7r
[2] http://pastebin.ubuntu.com/25825856/

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Ok, then we have
- code review and fixups done
- Fixed in latest Dev release
- SRU Template ready
- Tested build from PPAs
- Tested fix from PPAs
=> I'm merging the MPs and sponsoring the fix now for SRU.

I see it in the -unapproved queues - from now on it is up to the SRU Team.

@Jorge - Please help tracking proposed migration and verification in all three releases after acceptance.

dann frazier (dannf)
description: updated
Revision history for this message
Chris Halse Rogers (raof) wrote :

Hello Mario, or anyone else affected,

Accepted percona-xtradb-cluster-5.6 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/5.6.34-26.19-0ubuntu0.16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in percona-xtradb-cluster-5.6 (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-xenial
Revision history for this message
Chris Halse Rogers (raof) wrote :

Hello Mario, or anyone else affected,

Accepted percona-xtradb-cluster-5.6 into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.6/5.6.34-26.19-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-zesty to verification-done-zesty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-zesty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in percona-xtradb-cluster-5.6 (Ubuntu Zesty):
status: In Progress → Fix Committed
tags: added: verification-needed-zesty
Revision history for this message
Manoj Iyer (manjo) wrote :

Manoj, can you please verify the package in Proposed by enabling the proposed repo and tag this bug as verification-done-xenial ?

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Regression test results for zesty-ocata-proposed with stable charms was successful:

======
Totals
======
Ran: 102 tests in 1559.9808 sec.
 - Passed: 93
 - Skipped: 9
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 878.9997 sec.

Regression test results for zesty-ocata-proposed with development charms was successful:

======
Totals
======
Ran: 102 tests in 2073.5986 sec.
 - Passed: 93
 - Skipped: 9
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 925.9420 sec.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Regression testing for xenial-mitaka-proposed is hitting 2 heat test failures due to authorization failures. I've not yet dug into them but they exist on xenial-proposed and xenial-updates so likely not an issue related to this bug.

======
Totals
======
Ran: 103 tests in 1029.7887 sec.
 - Passed: 92
 - Skipped: 9
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 2
Sum of execute time for each test: 554.6188 sec.

Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

Verified proposed Xenial with HA in 3 units (http://paste.ubuntu.com/25903553/)

Tempest results, 10 failed tests, unlikely to be related to this change.

http://paste.ubuntu.com/25903540/

Juju status:

http://paste.ubuntu.com/25903545/

Revision history for this message
dann frazier (dannf) wrote :

@Jorge what else needs to be tested before this can get a verified tag?
Note: I've tested on arm64, though I was never able to recreate the problem myself w/ the unfixed package and percona.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

The regression test failures above were config issues on my end. They've since passed successfully.

I also ran tempest regression successfully with 3 units of percona in an HA configuration with corosync, pacemaker and haproxy.

Note: all the regression tests I've run are amd64.

Regression test results for xenial-mitaka-proposed with stable charms:

======
Totals
======
Ran: 102 tests in 1115.9995 sec.
 - Passed: 93
 - Skipped: 9
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 623.3908 sec.

Regression test results for zesty-ocata-proposed with stable charms:

======
Totals
======
Ran: 102 tests in 1622.8988 sec.
 - Passed: 93
 - Skipped: 9
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 830.7640 sec.

Revision history for this message
dann frazier (dannf) wrote :

OK - look liks arm64/amd64 are covered, but it seems to me that we still need test results for xenial-proposed and zesty-proposed on ppc64el.

Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

Summary of the verification:

* Xenial has been verified on the following scenarios:
 ** OpenStack cloud on amd64 with tempest with 1 node and 3 nodes cluster (https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.5/+bug/1657256/comments/83)
 ** ppc64 through sysbench on a single machine (http://pastebin.ubuntu.com/25825856/)

* Zesty has been verified on the following scenarios:
 ** OpenStack cloud on amd64 with tempest with 1 node and 3 nodes cluster (https://bugs.launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.5/+bug/1657256/comments/79)
 ** ppc64 through sysbench on a single machine (http://pastebin.ubuntu.com/25825884/)

I am marking the verification of zesty and xenial as done as the results
show conformity with the test scenario exposed on this bug + additional testing performed
on top of OpenStack shows confidence.

tags: added: verification-done-xenial verification-done-zesty
removed: verification-needed-xenial verification-needed-zesty
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Mario, or anyone else affected,

Accepted percona-xtradb-cluster-5.5 into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/percona-xtradb-cluster-5.5/5.5.37-25.10+dfsg-0ubuntu0.14.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in percona-xtradb-cluster-5.5 (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed-trusty
Revision history for this message
Jorge Niedbalski (niedbalski) wrote :

Hello Brian,

I've performed a validation on top of amd64[0] and ppc64el[1] reproducing
the test steps described here. The verification seems successful in both platforms.

[0] http://pastebin.ubuntu.com/25960515/
[1] http://pastebin.ubuntu.com/25960518/

tags: added: verification-done verification-done-trusty
removed: verification-needed verification-needed-trusty
Revision history for this message
Eric Desrochers (slashd) wrote :

I have identified 4 autopkgtests failure for Zesty in pending-sru page as follow :

[autopkgtest failures]
#https://people.canonical.com/~ubuntu-archive/pending-sru.html
Regression in autopkgtest for diaspora-installer (i386): test log
Regression in autopkgtest for diaspora-installer (s390x): test log
Regression in autopkgtest for diaspora-installer (armhf): test log
Regression in autopkgtest for diaspora-installer (amd64): test log

(Note that Trusty and Xenial are good based on pending-sru page)

niedbalski, can you investigate for Zesty and provide feedbacks before asking SRU team to release it ?

[build log]
#https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-zesty/zesty/amd64/d/diaspora-installer/20171101_012354_2cb3d@/log.gz
NoMethodError: undefined method `with' for #<Bundler::Settings:0x0055ab62849c30>
/usr/share/diaspora/config/application.rb:4:in `<top (required)>'
/usr/share/diaspora/Rakefile:8:in `require'
/usr/share/diaspora/Rakefile:8:in `<top (required)>'
/usr/share/diaspora/vendor/bundle/ruby/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'
(See full trace by running task with --trace)
dpkg: error processing package diaspora-installer (--configure):
 subprocess installed post-installation script returned error exit status 1
dpkg: dependency problems prevent configuration of autopkgtest-satdep:
 autopkgtest-satdep depends on diaspora-installer; however:
  Package diaspora-installer is not configured yet.

dpkg: error processing package autopkgtest-satdep (--configure):
 dependency problems - leaving unconfigured
No apport report written because the error message indicates its a followup error from a previous failure.
Errors were encountered while processing:
 diaspora-installer
 autopkgtest-satdep
E: Sub-process /usr/bin/dpkg returned an error code (1)
Exit request sent.
Creating nova instance adt-zesty-amd64-diaspora-installer-20171101-010718 from image ubuntu/ubuntu-zesty-17.04-amd64-server-20171031-disk1.img (UUID 9ecdf5f0-f4e2-40f5-a5f9-47fd083bccc3)...
blame: diaspora-installer
badpkg: Test dependencies are unsatisfiable. A common reason is that your testbed is out of date with respect to the archive, and you need to use a current testbed or run apt-get update or use -U.
autopkgtest [01:23:52]: ERROR: erroneous package: Test dependencies are unsatisfiable. A common reason is that your testbed is out of date with respect to the archive, and you need to use a current testbed or run apt-get update or use -U.

Revision history for this message
Eric Desrochers (slashd) wrote :

By precaution, I have restarted the autopkgtest for the above 4 failures.

Let's see the outcome.

- Eric

Revision history for this message
Eric Desrochers (slashd) wrote :

The autopkgtest still fails after an attempt to restart them.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package percona-xtradb-cluster-5.5 - 5.5.37-25.10+dfsg-0ubuntu0.14.04.5

---------------
percona-xtradb-cluster-5.5 (5.5.37-25.10+dfsg-0ubuntu0.14.04.5) trusty; urgency=medium

  [ Mario Splivalo ]
  * d/p/fix-power8-crash-under-load.patch: Create proper barriers at
    mutex_exit() to fix crashes on ppc64el (LP: #1657256)

  [ Jorge Niedbalski ]
  * d/p/fix_time_collector_lock_type.patch: Fix FTBFS on multiple
    architectures by making time_collector_lock mutable (LP: #1002848).

 -- Christian Ehrhardt <email address hidden> Tue, 24 Oct 2017 09:24:33 +0200

Changed in percona-xtradb-cluster-5.5 (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for percona-xtradb-cluster-5.5 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Note: Xenial would be good already in regard to verification (done), as well as autopkgtests (all good)

On Zesty:
- arm64 / ppc64 essentially always failed - pcp is detected as such, arm was never executed before
  "configure: error: cannot guess build type; you must specify one"
- on amd64/i386/s390x is different
  But here [1] had just the same issue in May a few times to then resolve without other package
  changes.
  - amd64/i386 seemed less affected so far there was one breakage in the past
   "https://rubygems.org/Net::HTTPNotFound"
  - but the new error is different
    "NoMethodError: undefined method `with' for #<Bundler::Settings:0x00560c0bd5a558>"
  - It seems to me this fetches code from rubygems.org and due to that this is now broken on all
    archs - that is an issue, but not one of the percona update we have here.

@Slashd - I'd ask you to file a new bug against disapora-installer and maybe ruby - and once done ask the SRU team to ignore these issues on this SRU as unrelated.

[1]: http://autopkgtest.ubuntu.com/packages/diaspora-installer/zesty/s390x

Revision history for this message
Eric Desrochers (slashd) wrote :

## FOR SRU VERIFICATION TEAM ##

After some investigation and more testing from my part, I confirm the 4 autopkgtest failures for Zesty[1] release are unrelated to the current percona SRU, and is a separate issue specifically in diaspora-installer pkg. I have reported a bug about it[2].

[1] - autopkgtest failures
#https://people.canonical.com/~ubuntu-archive/pending-sru.html
Regression in autopkgtest for diaspora-installer (i386): test log
Regression in autopkgtest for diaspora-installer (s390x): test log
Regression in autopkgtest for diaspora-installer (armhf): test log
Regression in autopkgtest for diaspora-installer (amd64): test log

[2] - LP: #1732520

- Eric

description: updated
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package percona-xtradb-cluster-5.6 - 5.6.34-26.19-0ubuntu0.16.04.2

---------------
percona-xtradb-cluster-5.6 (5.6.34-26.19-0ubuntu0.16.04.2) xenial; urgency=high

  * d/p/ibuf-uses-full-memory-barrier-powerpc.patch: This patch implements
    a full memory barrier for InnoDB mutex entry/exit on PowerPC.
    (LP: #1657256).

  * d/p/weak-memory-compat.patch: Removed as is covered by the new patch
    ibuf-uses-full-memory-barrier-ppc64.patch.

 -- Jorge Niedbalski <email address hidden> Mon, 23 Oct 2017 20:56:17 -0300

Changed in percona-xtradb-cluster-5.6 (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package percona-xtradb-cluster-5.6 - 5.6.34-26.19-0ubuntu1.1

---------------
percona-xtradb-cluster-5.6 (5.6.34-26.19-0ubuntu1.1) zesty; urgency=high

  [ Jorge Niedbalski ]
  * d/p/ibuf-uses-full-memory-barrier-powerpc.patch: This patch implements
    a full memory barrier for InnoDB mutex entry/exit on PowerPC.
    (LP: #1657256).
  * d/p/weak-memory-compat.patch: Removed as is covered by the new patch
    ibuf-uses-full-memory-barrier-ppc64.patch.

  [ Christian Ehrhardt ]
  * add ibuf-mutex-use-full-memory-barrier-powerpc.patch to d/p/series
  * remove unused d/p/s390-compat.patch

 -- Christian Ehrhardt <email address hidden> Thu, 26 Oct 2017 13:31:35 +0200

Changed in percona-xtradb-cluster-5.6 (Ubuntu Zesty):
status: Fix Committed → Fix Released
tags: added: sts-sru-done
removed: sts-sru-needed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.