Ubuntu
multipath-tools package

multipathd.socket unit fails to start ("Failed to listen on sockets: Address already in use")

Bug #1682178 reported by Mauricio Faria de Oliveira on 2017-04-12

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	multipath-tools (Ubuntu)	Fix Released	Medium	Mathieu Trudel-Lapierre
	Zesty	Fix Released	Medium	Unassigned

Bug Description

[Impact]
Multipath users on Ubuntu Server will see error messages from multipathd at boot. While the system continues to boot and is in a stable state, multipathd should be appropriately handled via its systemd unit after the system is booted.

[Test case]
1) Install with multipath enabled (disk-detect/multipath/enable=true).
2) Boot the installed system
3) Verify that there are no error messages from multipathd at boot.

[Regression potential]
This update changes the startup/shutdown login in initramfs at an early boot state, this could lead to failures to complete the booting process for systems that depend on multipath being available to bring up the rootfs, or failure to correctly drive multipathd/multipath-tools from a booted system. Furthermore, failures or delays shutting down a system due to multipathd / multipath-tools remaining up should also be considered as possible regressions issued from this update.

---

This bug is used to include one more patch to completely fix LP 1670811.

That patch is mentioned in LP 1670811 comment 15 [1].

It addresses the case in which the initramfs instance of multipathd takes a while to actually shutdown after its receives the signal from initramfs/scripts/local-bottom/multipath.

That shutdown is asynchronous, and keeps the multipathd socket open.
If it is still open by the time the root fs is mounted and its units are started, the multipathd.socket will fail to start.

Patch attached.

Logs:
----

[ 36.465332] systemd[1]: Failed to listen on multipathd control socket.
[FAILED] Failed to listen on multipathd control socket.
See 'systemctl status multipathd.socket' for details.

# systemctl status multipathd.socket --no-pager -l
* multipathd.socket - multipathd control socket
   Loaded: loaded (/lib/systemd/system/multipathd.socket; static; vendor preset: enabled)
   Active: failed (Result: resources)
   Listen: @/org/kernel/linux/storage/multipathd (Stream)

# journalctl -b -x
...
Mar 07 11:02:04 ltciofvtr-s824-lp8 systemd[1]: multipathd.socket: Failed to listen on sockets: Address already in use
Mar 07 11:02:04 ltciofvtr-s824-lp8 systemd[1]: Failed to listen on multipathd control socket.
Mar 07 11:02:04 ltciofvtr-s824-lp8 systemd[1]: multipathd.socket: Unit entered failed state.
...

Links:
-----

[1] https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1670811/comments/15

See original description

Tags:

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2017-04-12:

multipath-tools_wait-socket.debdiff Edit (1.6 KiB, text/plain)

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2017-04-12:

Additional bug reported by @cyphermox request.

Mathieu Trudel-Lapierre (cyphermox) on 2017-04-12

description:

updated

Revision history for this message

Andy Whitcroft (apw) wrote on 2017-04-21: Please test proposed package

Hello Mauricio, or anyone else affected,

Accepted multipath-tools into zesty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/multipath-tools/0.6.4-3ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in multipath-tools (Ubuntu Zesty):
status:	New → Fix Committed
tags:	added: verification-needed

Mathew Hodson (mhodson) on 2017-04-25

Changed in multipath-tools (Ubuntu):
importance:	Undecided → Medium
Changed in multipath-tools (Ubuntu Zesty):
importance:	Undecided → Medium

Revision history for this message

Brian Murray (brian-murray) wrote on 2017-05-01:

This new version of multipath-tools is failing to migrate in artful, due to autopkgtest failures.

http://people.canonical.com/~ubuntu-archive/proposed-migration/update_excuses.html#multipath-tools

Changed in multipath-tools (Ubuntu):
status:	New → Fix Committed
assignee:	nobody → Mathieu Trudel-Lapierre (cyphermox)

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-05-02:

Thank Brian for the ping here,
Of the issues found in the logs none seem all too multipath specific, seems mostly:
- ERROR: InterpreterNotFound: python3.6
- sha3-384 mismatch for "pc-kernel"
- and a bunch of file not found for various .img files

I wonder if the latter issues are due to the missing python3.6.
Overall - so far - it appears to be more issues in the ubuntu-image tests than multipath issues.
Are these known issues in ubuntu-image or is the assumption of anyone that multipath really broke something here?

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-05-03:

On just rerunning the tests s390 and ppc worked this time.
Arm failed again but at another test - retrying again.
I really start to assume ubuntu-image has a noisy test even not knowing it.

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-05-10:

This bug was fixed in the package multipath-tools - 0.6.4-3ubuntu4

---------------
multipath-tools (0.6.4-3ubuntu4) zesty; urgency=medium

* debian/initramfs/local-bottom: wait for the multipathd unix socket
to close, so to avoid multipathd.socket unit failure. (LP: #1682178)

-- Mauricio Faria de Oliveira <email address hidden> Wed, 12 Apr 2017 14:30:13 -0400

Changed in multipath-tools (Ubuntu):
status:	Fix Committed → Fix Released

Revision history for this message

Łukasz Zemczak (sil2100) wrote on 2017-06-27: Change of SRU verification policy

As part of a recent change in the Stable Release Update verification policy we would like to inform that for a bug to be considered verified for a given release a verification-done-$RELEASE tag needs to be added to the bug where $RELEASE is the name of the series the package that was tested (e.g. verification-done-xenial). Please note that the global 'verification-done' tag can no longer be used for this purpose.

Thank you!

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2017-06-27:

Which reminds me - Mauricio can you test and verify this?

Revision history for this message

Mauricio Faria de Oliveira (mfo) wrote on 2017-06-27:

#10

Sure, thanks for the reminder. The verification is okay! Updating the (new) tags.

proposed:
--------

# dpkg -s multipath-tools-boot | grep ^Version:
Version: 0.6.4-3ubuntu4

...
Begin: Running /scripts/local-bottom ... Begin: Stopping multipathd ... done.
[... it takes a few seconds at this point ...]
done.
...
[ OK ] Listening on multipathd control socket.
...
[ OK ] Started Device-Mapper Multipath Device Controller.
...

current:
-------

# dpkg -s multipath-tools-boot | grep ^Version:
Version: 0.6.4-3ubuntu3

...
Begin: Running /scripts/local-bottom ... Begin: Stopping multipathd ... done.
done.
...
[FAILED] Failed to listen on multipathd control socket.
See 'systemctl status multipathd.socket' for details.
...
[FAILED] Failed to start Device-Mapper Multipath Device Controller.
See 'systemctl status multipathd.service' for details.
...

# systemctl status multipathd.socket --no-pager
  multipathd.socket - multipathd control socket
   Loaded: loaded (/lib/systemd/system/multipathd.socket; static; vendor preset: enabled)
   Active: failed (Result: resources)
   Listen: @/org/kernel/linux/storage/multipathd (Stream)

# systemctl status multipathd.service --no-pager -l
  multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2017-06-27 13:01:04 CDT; 2min 18s ago
Main PID: 782 (code=exited, status=1/FAILURE)
      CPU: 10ms

Jun 27 13:01:04 ubuntu systemd[1]: Starting Device-Mapper Multipath Device Controller...
Jun 27 13:01:04 ubuntu multipathd[782]: process is already running
Jun 27 13:01:04 ubuntu systemd[1]: multipathd.service: Main process exited, code=exited, status=1/FAILURE
Jun 27 13:01:04 ubuntu systemd[1]: Failed to start Device-Mapper Multipath Device Controller.
Jun 27 13:01:04 ubuntu systemd[1]: multipathd.service: Unit entered failed state.
Jun 27 13:01:04 ubuntu systemd[1]: multipathd.service: Failed with result 'exit-code'.

Sure, thanks for the reminder. The verification is okay! Updating the (new) tags.

proposed:
--------

# dpkg -s multipath-tools-boot | grep ^Version:
Version: 0.6.4-3ubuntu4

...
Begin: Running /scripts/local-bottom ... Begin: Stopping multipathd ... done.
[... it takes a few seconds at this point ...]
done.
...
[  OK  ] Listening on multipathd control socket.
...
[  OK  ] Started Device-Mapper Multipath Device Controller.
...

current:
-------

# dpkg -s multipath-tools-boot | grep ^Version:
Version: 0.6.4-3ubuntu3

# systemctl status multipathd.service --no-pager -l
  multipathd.service - Device-Mapper Multipath Device Controller
   Loaded: loaded (/lib/systemd/system/multipathd.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2017-06-27 13:01:04 CDT; 2min 18s ago
 Main PID: 782 (code=exited, status=1/FAILURE)
      CPU: 10ms

tags:

added: verification-done-zesty
removed: verification-needed

Revision history for this message

Launchpad Janitor (janitor) wrote on 2017-07-05:

#11

This bug was fixed in the package multipath-tools - 0.6.4-3ubuntu4

---------------
multipath-tools (0.6.4-3ubuntu4) zesty; urgency=medium

* debian/initramfs/local-bottom: wait for the multipathd unix socket
to close, so to avoid multipathd.socket unit failure. (LP: #1682178)

-- Mauricio Faria de Oliveira <email address hidden> Wed, 12 Apr 2017 14:30:13 -0400

Changed in multipath-tools (Ubuntu Zesty):
status:	Fix Committed → Fix Released

Revision history for this message

Chris Halse Rogers (raof) wrote on 2017-07-05: Update Released

#12

The verification of the Stable Release Update for multipath-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.