Ubuntu
systemd package

mysql in lxd fails to start with systemd 233, 234: failed at step KEYRING

Bug #1691096 reported by Andreas Hasenack on 2017-05-16

This bug affects 3 people

	Status	Importance	Assigned to	Milestone
systemd	Fix Released	Unknown	auto-github-systemd-systemd #6871
systemd (Ubuntu)	Fix Released	High	Dimitri John Ledkov	Ubuntu ubuntu-17.08
Artful	Fix Released	High	Dimitri John Ledkov	Ubuntu ubuntu-17.08

Bug Description

artful unprivileged container on a xenial host

xnox hinted that https://github.com/systemd/systemd/commit/b3415f5daef49642be3d5f417b8880c078420ff7 might be related

With systemd 233, mysql-server-5.7 in an artful lxd unprivileged container fails to start:

Process: 6460 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=237/KEYRING)

May 16 12:20:08 artful-mysql systemd[1]: mysql.service: Failed to set invocation ID on control group /system.slice/mysql.service, ignoring: Operation not permitted
(...)
May 16 12:20:08 artful-mysql systemd[6460]: mysql.service: Failed at step KEYRING spawning /usr/share/mysql/mysql-systemd-start: Permission denied

Reproducing it on a fresh xenial kvm as host, fresh artful lxd image. We start with systemd-232 in the artful lxd:
ubuntu@intense-sunbeam:~$ apt-cache policy systemd
systemd:
  Installed: 232-21ubuntu3
  Candidate: 233-6ubuntu1
  Version table:
     233-6ubuntu1 500
        500 http://br.archive.ubuntu.com/ubuntu artful/main amd64 Packages
*** 232-21ubuntu3 100
        100 /var/lib/dpkg/status

Then we install mysql-server-5.7:
ubuntu@intense-sunbeam:~$ sudo apt install mysql-server-5.7
...
Setting up mysql-server-5.7 (5.7.17-0ubuntu1) ...
update-alternatives: using /etc/mysql/mysql.cnf to provide /etc/mysql/my.cnf (my.cnf) in auto mode
Renaming removed key_buffer and myisam-recover options (if present)
Created symlink /etc/systemd/system/multi-user.target.wants/mysql.service → /lib/systemd/system/mysql.service.
Processing triggers for libc-bin (2.24-9ubuntu2) ...
Processing triggers for systemd (232-21ubuntu3) ...

Which starts just fine:
ubuntu@intense-sunbeam:~$ mysql -uroot -psecret
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
...

We then upgrade systemd:
ubuntu@intense-sunbeam:~$ sudo apt install systemd
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libpam-systemd libsystemd0
(...)
Setting up libpam-systemd:amd64 (233-6ubuntu1) ...

mysql is still running:
ubuntu@intense-sunbeam:~$ mysql -uroot -psecret
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor. Commands end with ; or \g.
(...)

But restarting mysql fails:
ubuntu@intense-sunbeam:~$ sudo service mysql restart
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.
ubuntu@intense-sunbeam:~$ mysql -uroot -psecret
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

More logs attached.

See original description

Tags:

Revision history for this message

Andreas Hasenack (ahasenack) wrote on 2017-05-16:

syslog Edit (47.3 KiB, text/plain)

/var/log/syslog of the container

Revision history for this message

Andreas Hasenack (ahasenack) wrote on 2017-05-16:

journalctl-xe.log Edit (114.7 KiB, text/plain)

journalctl -xe of the container

Revision history for this message

Andreas Hasenack (ahasenack) wrote on 2017-05-16:

dmesg.txt Edit (43.9 KiB, text/plain)

dmesg from the xenial host

description:	updated
description:	updated
description:	updated

Revision history for this message

Robie Basak (racb) wrote on 2017-05-16:

Importance: High, as it stops mysqld from working in a container, and MySQL is a fairly essential package in main in Ubuntu (from an ~ubuntu-server perspective).

Changed in systemd (Ubuntu):
importance:	Undecided → High
milestone:	none → ubuntu-17.10

Andreas Hasenack (ahasenack) on 2017-05-16

summary:

- mysql fails to start with systemd 233: failed at step KEYRING
+ mysql in lxd fails to start with systemd 233: failed at step KEYRING

Brian Murray (brian-murray) on 2017-06-09

Changed in systemd (Ubuntu Artful):
assignee:	nobody → Dimitri John Ledkov (xnox)

Steve Langasek (vorlon) on 2017-06-09

Changed in systemd (Ubuntu Artful):
milestone:	ubuntu-17.10 → ubuntu-17.07
status:	New → Triaged

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2017-08-15: Re: mysql in lxd fails to start with systemd 233: failed at step KEYRING

It looks like I'm hitting the same error with murano-cfapi in an artful unprivileged container:

$ sudo apt install murano-cfapi
$ sudo tail /var/log/syslog
Aug 15 12:25:25 a1 systemd[1]: murano-cfapi.service: Failed to set invocation ID on control group /system.slice/murano-cfapi.service, ignoring: Operation not permitted
Aug 15 12:25:25 a1 systemd[1]: Starting OpenStack Murano CF API...
Aug 15 12:25:25 a1 systemd[19791]: murano-cfapi.service: Failed at step KEYRING spawning /bin/mkdir: Permission denied
Aug 15 12:25:25 a1 systemd[1]: murano-cfapi.service: Control process exited, code=exited status=237

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2017-08-15:

I tried systemd from proposed as well but still hit the failure:

root@a1:~# apt policy systemd
systemd:
  Installed: 234-2ubuntu3
  Candidate: 234-2ubuntu3
  Version table:
*** 234-2ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu artful-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     233-8ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu artful/main amd64 Packages

I was able to reproduce this with several openstack services (cinder-scheduler, nova-scheduler, nova-conductor) and this is blocking autopkgtests so it appears to be fairly pervasive.

Revision history for this message

Corey Bryant (corey.bryant) wrote on 2017-08-15:

For good measure I also installed all of the above packages on a kvm artful instance and services came up fine. systemd version 233-8ubuntu3.

Dimitri John Ledkov (xnox) on 2017-08-16

Changed in systemd (Ubuntu Artful):
milestone:	ubuntu-17.07 → ubuntu-17.08

Revision history for this message

Robie Basak (racb) wrote on 2017-08-18:

This also affects Artful running a sid container in lxd. Nothing is particularly MySQL specific I here I think, since the failure happens before MySQL runs at all. The only special systemd things I see in the service unit are the use of ExecStartPre on a bash script and PermissionsStartOnly=true. The full service unit definition is:

# MySQL systemd service file

[Unit]
Description=MySQL Community Server
After=network.target

[Install]
WantedBy=multi-user.target

[Service]
User=mysql
Group=mysql
PermissionsStartOnly=true
ExecStartPre=/usr/share/mysql/mysql-systemd-start pre
ExecStart=/usr/sbin/mysqld
ExecStartPost=/usr/share/mysql/mysql-systemd-start post
TimeoutSec=600
Restart=on-failure
RuntimeDirectory=mysqld
RuntimeDirectoryMode=755

Revision history for this message

Robie Basak (racb) wrote on 2017-08-18:

AFAICT, /usr/share/mysql/mysql-systemd-start is never run:

Aug 18 10:53:51 autopkgtest-lxd-mnbewo systemd[1]: Starting MySQL Community Server...
-- Subject: Unit mysql.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit mysql.service has begun starting up.
Aug 18 10:53:51 autopkgtest-lxd-mnbewo systemd[1649]: mysql.service: Failed at step KEYRING spawning /usr/share/mysql/mysql-systemd-start: Permission denied
-- Subject: Process /usr/share/mysql/mysql-systemd-start could not be executed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The process /usr/share/mysql/mysql-systemd-start could not be executed and failed.
--
-- The error number returned by this process is 13.

Revision history for this message

Andreas Hasenack (ahasenack) wrote on 2017-09-19:

#10

This still happens with systemd 234

summary:

- mysql in lxd fails to start with systemd 233: failed at step KEYRING
+ mysql in lxd fails to start with systemd 233, 234: failed at step
+ KEYRING

Bug Watch Updater (bug-watch-updater) on 2017-09-19

Changed in systemd:
status:	Unknown → New

Bug Watch Updater (bug-watch-updater) on 2017-09-20

Changed in systemd:
status:	New → Fix Released

Francis Ginther (fginther) on 2017-09-29

tags:

added: id-597a8312028fa1cd2c156565

Dimitri John Ledkov (xnox) on 2017-10-04

Changed in systemd (Ubuntu Artful):
status:	Triaged → Fix Committed

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-10-06:

#11

This bug was fixed in the package systemd - 234-2ubuntu12

---------------
systemd (234-2ubuntu12) artful; urgency=medium

  [ Dimitri John Ledkov ]
  * debian/rules: do not strip test-copy.
    This insures test-copy is large enough for test-copy tests to pass.
    (LP: #1721203)

  [ Michael Biebl ]
  * Drop systemd-timesyncd.service.d/disable-with-time-daemon.conf.
    All major NTP implementations ship a native service file nowadays with a
    Conflicts=systemd-timesyncd.service so this drop-in is no longer
    necessary. (Closes: #873185) (LP: #1721204)

-- Dimitri John Ledkov <email address hidden> Wed, 04 Oct 2017 13:28:34 +0100