Merge ~lvoytek/ubuntu/+source/mysql-8.0:mysql-lp1899248-increase-shutdown-timeout-sru-focal into ubuntu/+source/mysql-8.0:ubuntu/focal-devel

[Lena Voytek]

The increase of the timer to 3 minutes is based on the shudown times seen in LP #1899248 and its duplicates. The average shutdown time seen here is 1:31, with a maximum reasonable shutdown time of 2:42.

ppa:lvoytek/mysql-8.0-increase-shutdown-timeout-sru-focal

Steps to test:

# lxc launch images:ubuntu/focal test-failure-focal
# lxc exec test-failure-focal bash

# apt update && apt dist-upgrade -y

# apt install -y software-properties-common

- This ppa contains a shutdown timeout of 0, causing the error to be reproduced every time
# add-apt-repository ppa:lvoytek/mysql-autofail-install-focal

# apt install -y mysql-server

- A shutdown error will show up in the terminal:

mysqld is running as pid 2569
Error: Unable to shut down server with process id 2569
dpkg: error processing package mysql-server-8.0 (--configure):
 installed mysql-server-8.0 package post-installation script subprocess returned
 error exit status 1

- After the initial install mysqlx errors will also show up in the error log
# grep mysqlx /var/log/mysql/error.log

2022-03-14T17:22:37.386262Z 0 [ERROR] [MY-011292] [Server] Plugin mysqlx reported: 'Preparation of I/O interfaces failed, X Protocol won't be accessible'
2022-03-14T17:22:37.386356Z 0 [ERROR] [MY-011300] [Server] Plugin mysqlx reported: 'Setup of socket: '/var/run/mysqld/mysqlx.sock' failed, can't create lock file /var/run/mysqld/mysqlx.sock.lock'

# exit

# lxc launch images:ubuntu/focal test-success-focal
# lxc exec test-success-focal bash

# apt update && apt dist-upgrade -y

# apt install -y software-properties-common

# add-apt-repository ppa:lvoytek/mysql-8.0-increase-shutdown-timeout-sru-focal

# apt install -y mysql-server

- mysql will install successfully without producing any shutdown errors
- After the initial install confirm no mysqlx errors show up in the error log
# grep mysqlx /var/log/mysql/error.log

2022-03-14T17:28:52.549965Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock
2022-03-14T17:28:55.249114Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-03-14T17:28:57.807353Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock

[Bryce Harrington (bryce)]

Glad to see the work done on this, and I think your approach is spot on. This'll fix a couple of the big issues resulting in apport bugs with mysql.

A few course corrections, however.

First, policy in Ubuntu regarding SRUs to stable releases is to make sure the changes are landed to jammy beforehand. So, please re-implement these changes against jammy's mysql package, and resubmit the MP against jammy, in favor of this one. Once the changes have landed to jammy, then effort can be put towards backporting those changes to focal (and impish) as well.

Second, it looks like there's two distinct problems being addressed here, namely the timeout and the permissions error on the socket files. It's not a problem to include multiple fixes in an MP to the -devel release, but for changes that are going to be SRU'd to focal, et al, each distinct fix is going to need its own bug report.

I agree that LP: #1899248 is an instance of both problems, however given that the jammy bug task was already closed (by me! sorry) that might make it a bit more confusing. You'll likely have best luck with SRU approvals if you locate two other bug reports (you could file brand new ones, or repurpose a couple of 1899248's dupes), and transform them into reports that focus on each of the two issues. Edit the bug descriptions and provide a discussion of the symptom or error, analysis of the problem (and identification of root-cause), and rationale for your chosen solution; later when you get to the SRU you'll also need a test-case and discussion of regression potential.

For the rationale for your chosen solutions, in the case of the increased timeout it sounds like you have good justification in considering the average / maximum values seen in the wild. You mentioned the maximal reasonable time was 2:42 min, were there many cases where the time was greater than that? (I vaguely recall once seeing one that was closer to 10 min, but I could be wrong).

For the case of the socket permissions, the rationale for the solution should probably explain what ownership / permissions the code will set the files to, and discuss the security implications / safety. E.g. if we're already setting other mysql socket files to these permissions, and/or if they're the same permissions that mariadb, postgres, or other services also use, then that'd suggest they're safe here too.

Also, I notice you're setting the sleep value from 1 to 0.9. I don't see a problem doing so, but you may want to explain the rationale for that change, such as if it resolves some problem or makes debugging easier.

Finally, thanks for putting in the good detailed steps to reproduce the problem and solution. That looks like it'll make a great [test case] section for the SRU bug reports. And again I'm really glad to see this work done on the apport reported mysql bugs, this should help quell a lot of our incoming mysql bugs. :-)

review: Needs Fixing

[Bryce Harrington (bryce)]

Oh, one other remark, a question I'm sure will come up in SRU review will be, "What causes the long installation times that are triggering the timeout?" In particular, they're going to be wondering if increasing the timeout is going to result in a successful installation or not.

For example, if there's evidence proving it to be resource limitations (especially if the conditions can be replicated synthetically), that would give a very solid rationale. Otherwise, the explanation should say "our suspicion is the delay is caused by ..." and point to the clues collected so far.

Btw, sorry for so much feedback on this MP; in truth the timeout change is probably a no-brainer. However, the general process outlined here for identifying and justifying the changes will be of value to follow for other mysql fixes you'll develop in the future. Getting the paperwork process into good practice will make SRU filing go smoothly. With larger packages like mysql, where there may sometimes be multiple SRUs in flight, being organized will pay off.

[Lena Voytek (lvoytek)]

> Glad to see the work done on this, and I think your approach is spot on.
> This'll fix a couple of the big issues resulting in apport bugs with mysql.
>
> A few course corrections, however.
>
> First, policy in Ubuntu regarding SRUs to stable releases is to make sure the
> changes are landed to jammy beforehand. So, please re-implement these changes
> against jammy's mysql package, and resubmit the MP against jammy, in favor of
> this one. Once the changes have landed to jammy, then effort can be put
> towards backporting those changes to focal (and impish) as well.
>
> Second, it looks like there's two distinct problems being addressed here,
> namely the timeout and the permissions error on the socket files. It's not a
> problem to include multiple fixes in an MP to the -devel release, but for
> changes that are going to be SRU'd to focal, et al, each distinct fix is going
> to need its own bug report.
>
> I agree that LP: #1899248 is an instance of both problems, however given that
> the jammy bug task was already closed (by me! sorry) that might make it a bit
> more confusing. You'll likely have best luck with SRU approvals if you locate
> two other bug reports (you could file brand new ones, or repurpose a couple of
> 1899248's dupes), and transform them into reports that focus on each of the
> two issues. Edit the bug descriptions and provide a discussion of the symptom
> or error, analysis of the problem (and identification of root-cause), and
> rationale for your chosen solution; later when you get to the SRU you'll also
> need a test-case and discussion of regression potential.
>
> For the rationale for your chosen solutions, in the case of the increased
> timeout it sounds like you have good justification in considering the average
> / maximum values seen in the wild. You mentioned the maximal reasonable time
> was 2:42 min, were there many cases where the time was greater than that? (I
> vaguely recall once seeing one that was closer to 10 min, but I could be
> wrong).
>
> For the case of the socket permissions, the rationale for the solution should
> probably explain what ownership / permissions the code will set the files to,
> and discuss the security implications / safety. E.g. if we're already setting
> other mysql socket files to these permissions, and/or if they're the same
> permissions that mariadb, postgres, or other services also use, then that'd
> suggest they're safe here too.
>
> Also, I notice you're setting the sleep value from 1 to 0.9. I don't see a
> problem doing so, but you may want to explain the rationale for that change,
> such as if it resolves some problem or makes debugging easier.
>
> Finally, thanks for putting in the good detailed steps to reproduce the
> problem and solution. That looks like it'll make a great [test case] section
> for the SRU bug reports. And again I'm really glad to see this work done on
> the apport reported mysql bugs, this should help quell a lot of our incoming
> mysql bugs. :-)

Thanks for the review, that helps a lot. I'll get the mps up for Jammy and Impish up soon, just have to finish the builds for their respective ppas. I'll file a new bug for the mys...

[Utkarsh Gupta (utkarsh)]

review: Needs Information

[Utkarsh Gupta (utkarsh)]

review: Approve

[Bryce Harrington (bryce)]

I've re-read the SRU text on both bugs - great work. It looks like all required information is presented clearly in the SRU text, along with contextual info on the solution.

I'll proceed with sponsoring the upload.

review: Approve

[Bryce Harrington (bryce)]

-rw-r--r-- 1 bryce bryce 1942 Mar 30 17:35 ../mysql-8.0_8.0.28-0ubuntu0.20.04.4_source.changes
../mysql-8.0_8.0.28-0ubuntu0.20.04.4_source.changes
Vcs-Git: https://git.launchpad.net/~bryce/ubuntu/+source/mysql-8.0
Vcs-Git-Commit: c257927ed86746c06474721fdf363ff28a0f6af8
Vcs-Git-Ref: refs/heads/mysql-lp1899248-increase-shutdown-timeout-sru-focal

 fixup_changes dsc ../mysql-8.0_8.0.28-0ubuntu0.20.04.4.dsc ../mysql-8.0_8.0.28-0ubuntu0.20.04.4_source.changes
 fixup_changes buildinfo ../mysql-8.0_8.0.28-0ubuntu0.20.04.4_source.buildinfo ../mysql-8.0_8.0.28-0ubuntu0.20.04.4_source.changes
 signfile changes ../mysql-8.0_8.0.28-0ubuntu0.20.04.4_source.changes A661100B3DAC1D4F2CAD8A54E603B2578FB8F0FB

Checking signature on .changes
gpg: ../mysql-8.0_8.0.28-0ubuntu0.20.04.4_source.changes: Valid signature from E603B2578FB8F0FB
Checking signature on .dsc
gpg: ../mysql-8.0_8.0.28-0ubuntu0.20.04.4.dsc: Valid signature from E603B2578FB8F0FB
Uploading to ubuntu (via ftp to upload.ubuntu.com):
  Uploading mysql-8.0_8.0.28-0ubuntu0.20.04.4.dsc: done.
  Uploading mysql-8.0_8.0.28-0ubuntu0.20.04.4.debian.tar.xz: done.
  Uploading mysql-8.0_8.0.28-0ubuntu0.20.04.4_source.buildinfo: done.
  Uploading mysql-8.0_8.0.28-0ubuntu0.20.04.4_source.changes: done.
Successfully uploaded packages.

diff --git a/debian/changelog b/debian/changelog
index b8b1236..b757b89 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,13 @@
+mysql-8.0 (8.0.28-0ubuntu0.20.04.4) focal; urgency=medium
+
+  * d/mysql-server-8.0.postinst:
+    - Increase stop_server timeout so shutdowns that last up to 3 minutes do not
+      trigger apport (LP: #1899248)
+    - Create and add correct permissions to the mysql socket directory before
+      running mysqld post-update. (LP: #1964969)
+
+ -- Lena Voytek <[e-mail]>  Thu, 10 Mar 2022 13:45:25 -0700
+
 mysql-8.0 (8.0.28-0ubuntu0.20.04.3) focal-security; urgency=medium
 
   * SECURITY UPDATE: Update to 8.0.28 to fix security issues
diff --git a/debian/mysql-server-8.0.postinst b/debian/mysql-server-8.0.postinst
index 05e2122..4aa0b80 100755
--- a/debian/mysql-server-8.0.postinst
+++ b/debian/mysql-server-8.0.postinst
@@ -40,18 +40,21 @@ start_server() {
 # Usage: stop_server <tmpdir>
 # Params:
 # tmpdir - Location of temporary pid-file
+# Returns:
+# 0 - Shutdown successful
+# 1 - Shutdown took longer than 3 minutes
 stop_server(){
   local tmpdir=$1
   # Send kill signal
   server_pid=$(cat "$tmpdir/mysqld.pid")
   kill "$server_pid"
 
-  for i in $(seq 1 60); do
+  for i in $(seq 1 180); do
     sleep 0.1 # A full second is too long, but we need to give the server _some_ time.
     if ! $(ps $server_pid >/dev/null 2>&1); then
       return 0
     fi
-    sleep 1
+    sleep 0.9
   done
   # The server hasn't shut down in a timely manner
   echo "Error: Unable to shut down server with process id $server_pid" >&2
@@ -190,9 +193,9 @@ case "$1" in
     # variants in Ubuntu.
     /usr/share/mysql-common/configure-symlinks install mysql "$mysql_cfgdir/mysql.cnf"
 
-    # Ensure the existence and right permissions for the database and
-    # log files.
-    for d in $mysql_statedir $mysql_filesdir $mysql_keyringdir $mysql_logdir
+    # Ensure the existence and right permissions for the database, socket
+    # folder, and log files.
+    for d in $mysql_statedir $mysql_filesdir $mysql_keyringdir $mysql_logdir $mysql_rundir
     do
       if [ ! -d "$d" -a ! -L "$d" ]; then mkdir "$d"; fi
       chown -R mysql:mysql $d