[stein] 'Percona-XtraDB-Cluster prohibits use of DML command on a table (octavia.spares_pool) without an explicit primary key with pxc_strict_mode = ENFORCING or MASTER')

Bug #1826875 reported by Dmitrii Shcherbakov
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Fix Released
Medium
Unassigned
Stein
Fix Released
Medium
Unassigned
octavia
Fix Released
Undecided
Sahid Orentino
octavia (Ubuntu)
Fix Released
Medium
Unassigned
Disco
Fix Released
Medium
Unassigned
Eoan
Fix Released
Medium
Unassigned

Bug Description

Installed Octavia via charm-octavia using bionic-stein.

When the charm tried to execute a database migration

sudo octavia-db-manage upgrade head

it failed with the following error:

2019-04-29 11:49:20.096 22569 ERROR octavia-db-manage sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1105, 'Percona-XtraDB-Cluster prohibits use of DML command on a table (octavia.spares_pool) without an explicit primary key with pxc_strict_mode = ENFORCING or MASTER') [SQL: 'INSERT INTO spares_pool (updated_at) VALUES (%(updated_at)s)'] [parameters: {'updated_at': datetime.datetime(2019, 4, 29, 11, 49, 20, 94405)}] (Background on this error at: http://sqlalche.me/e/2j85)

It looks like the spares_pool table doesn't have a primary key.

mysql> DESCRIBE spares_pool;
+------------+----------+------+-----+-------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+----------+------+-----+-------------------+-------+
| updated_at | datetime | YES | | CURRENT_TIMESTAMP | |
+------------+----------+------+-----+-------------------+-------+
1 row in set (0.00 sec)

mysql> show variables like 'pxc_strict_mode';
+-----------------+-----------+
| Variable_name | Value |
+-----------------+-----------+
| pxc_strict_mode | ENFORCING |
+-----------------+-----------+
1 row in set (0.00 sec)

This spares_pool table was added quite recently (in Stein):

git --no-pager log -S spares_pool | grep -v Author
commit 0b468090e6eef1483f6f7c53708f500802a57d51
Date: Tue Apr 2 18:05:52 2019 +0200

    Fix spare amphora check and creation

    This patch fixes an issue when the SparesPool table is empty that blocks
    spare amphorae creation. It creates a new spares pool entry if the table
    is empty.

    Story 2005352
    Task 30306

    Change-Id: I0ce2778277640ee9e509c709bf8621b8b025d6d3

commit a205ab3ebe4ae14668f25328c062f9383d6c69cb
Date: Mon Mar 11 17:06:15 2019 -0700

    Limit spares pool to the spare_amphora_pool_size

    This patch fixes a bug where the housekeeping controllers may launch
    more spare amphora than the spare_amphora_pool_size setting.

    Story: 2003094
    Task 23186

    Change-Id: I4c98b3442d2471662488184fa7e91ac64ec33279

➜ octavia git:(master) git --no-pager branch --contains=a205ab3ebe4ae14668f25328c062f9383d6c69cb
* master
  stable/stein

Upstream bug: https://storyboard.openstack.org/#!/story/2005531

Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :
description: updated
description: updated
Revision history for this message
Sahid Orentino (sahid-ferdjaoui) wrote :

Based on the table definition I guess we can argue about the usage of a PK.

Beside of that, the model well define the primary_key on 'updated_at'. I proposed a patch upstream [0] to fix the table definition, let see what the community is saying.

[0] https://review.opendev.org/#/c/656459/

Changed in octavia:
assignee: nobody → Sahid Orentino (sahid-ferdjaoui)
Revision history for this message
Dmitrii Shcherbakov (dmitriis) wrote :

Meanwhile, here is a workaround (not ideal):

juju config percona-cluster pxc-strict-mode=PERMISSIVE

Changed in octavia (Ubuntu Disco):
status: New → Triaged
importance: Undecided → Medium
Changed in octavia (Ubuntu Eoan):
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Thanks Sahid! I've uploaded a new version to eoan and disco unapproved queue [1] where it is awaiting SRU team review.

[1] https://launchpad.net/ubuntu/disco/+queue?queue_state=1&queue_text=octavia

Sahid, please add SRU details and subscribe ubuntu-sru when you get a chance.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package octavia - 4.0.0-0ubuntu2

---------------
octavia (4.0.0-0ubuntu2) eoan; urgency=medium

  * d/p/bug1826875.patch: Cherry-picked from upstream to prevent
    issue when percona is configured with pxc-strict-mode=ENFORCING
    (LP: #1826875).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 17:23:52 +0200

Changed in octavia (Ubuntu Eoan):
status: Triaged → Fix Released
Revision history for this message
Sahid Orentino (sahid-ferdjaoui) wrote :

[Impact]

With Percona Cluster in pxc-strict-mode=ENFORCING, When the charm tried to execute a database migration with the command:

 $ sudo octavia-db-manage upgrade head

It failed with the following error:

2019-04-29 11:49:20.096 22569 ERROR octavia-db-manage sqlalchemy.exc.InternalError: (pymysql.err.InternalError) (1105, 'Percona-XtraDB-Cluster prohibits use of DML command on a table (octavia.spares_pool) without an explicit primary key with pxc_strict_mode = ENFORCING or MASTER') [SQL: 'INSERT INTO spares_pool (updated_at) VALUES (%(updated_at)s)'] [parameters: {'updated_at': datetime.datetime(2019, 4, 29, 11, 49, 20, 94405)}] (Background on this error at: http://sqlalche.me/e/2j85)

[Test Case]

 * After to have installed the new package, execute a db migration upgrade.

[Fix]

 * The fix accepted upstream and cherry-picked set a primary key to the table 'spares_pool'.

[Regression Potential]

 * n/a

[Other Info]

 * n/a

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Dmitrii, or anyone else affected,

Accepted octavia into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/octavia/4.0.0-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in octavia (Ubuntu Disco):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-disco
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hello Dmitrii, or anyone else affected,

Accepted octavia into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed
Revision history for this message
Dongwon Cho (dongwoncho) wrote :

Hi,

I had this issue on my test Env as well, applied the fix with following option and confirmed it fixes the issue.

openstack-origin: cloud:bionic-stein/proposed

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Hi Dmitrii, Dongwon, thanks for testing. If you've verified the fix in any of the -proposed pockets, can you tag this bug accordingly (ie. verification-done-*, etc)? It will have to be tested according to the [Test Case] section documented in this bug. Thanks!

Revision history for this message
Przemyslaw Hausman (phausman) wrote :

Hi, I have verified that the fix works on bionic-stein, with -proposed packages.

tags: added: verification-stein-done
Revision history for this message
David Coronel (davecore) wrote :

I also have verified that this fix works on bionic-stein, removing the verification-stein-needed tag.

tags: removed: verification-stein-needed
Revision history for this message
David Coronel (davecore) wrote :

I subscribed field-critical to this bug as this is affecting a customer deployment.

Revision history for this message
David Coronel (davecore) wrote :

Unsubscribed field-critical since there's a workaround to use package in stein-proposed and fix seems to be on the way.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Przemyslaw/David, you wouldn't happen to be able to test this on disco as well would you? The SRU team will have to land this in disco before it can land in stein. If not please let me know. Thanks!

Revision history for this message
David Coronel (davecore) wrote :

Hi Corey, I don't have access to a Disco cloud to test. Can we still get this fix in stein? This is a blocker for a customer deployment.

Revision history for this message
Frode Nordahl (fnordahl) wrote :
tags: added: verification-done verification-done-disco
removed: verification-needed verification-needed-disco
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Have the disco-proposed packages been tested here? I see comment #17 mentioning testing proposed packages on disco, but then the two attached links ([0] and [1]) don't look like they're verifications using the actual packages in disco-proposed (octavia 4.0.0-0ubuntu1.1). Did I read it wrong?
I just want to make sure the actual binaries from our Ubuntu -proposed pocket have been used in the verification.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package octavia - 4.0.0-0ubuntu1.1

---------------
octavia (4.0.0-0ubuntu1.1) disco; urgency=medium

  [ Corey Bryant ]
  * d/gbp.conf: Create stable/stein branch.

  [ Sahid Orentino Ferdjaoui ]
  * d/p/0001-db-add-missing-primary-key-in-spares_pool-table.patch: Cherry-picked
    from upstream to prevent issue when running percona in strict mode
    (LP: #1826875).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 17:18:36 +0200

Changed in octavia (Ubuntu Disco):
status: Fix Committed → Fix Released
Revision history for this message
Steve Langasek (vorlon) wrote : Update Released

The verification of the Stable Release Update for octavia has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
James Page (james-page) wrote :

The verification of the Stable Release Update for octavia has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
James Page (james-page) wrote :

This bug was fixed in the package octavia - 4.0.0-0ubuntu1.1~cloud0
---------------

 octavia (4.0.0-0ubuntu1.1~cloud0) bionic-stein; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 octavia (4.0.0-0ubuntu1.1) disco; urgency=medium
 .
   [ Corey Bryant ]
   * d/gbp.conf: Create stable/stein branch.
 .
   [ Sahid Orentino Ferdjaoui ]
   * d/p/0001-db-add-missing-primary-key-in-spares_pool-table.patch: Cherry-picked
     from upstream to prevent issue when running percona in strict mode
     (LP: #1826875).

Revision history for this message
Mathew Hodson (mhodson) wrote :

Change 656459 merged by Zuul:
db: add missing primary key in spares_pool table

https://review.opendev.org/656459

Changed in octavia:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.