iscsid does not run if there are only initramfs initiated targets

Bug #1802354 reported by Scott Moser
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
open-iscsi (Ubuntu)
Fix Released
High
Scott Moser
Bionic
Fix Released
High
Scott Moser
Cosmic
Fix Released
High
Scott Moser

Bug Description

=== Begin SRU Template ===
[Impact]
A previous change to open-iscsi made under bug 1755858 caused a regression
that is seen when a system has only a iscsi mounts that were made
in the initramfs. The most likely scenario would be iscsi root.

The problem seen is that in this scenario the iscsid.service will not be
started. That generally works, but will have unexpected issues if
service is done to the iscsi target (external system). That could occur
if the remote system needed to be serviced for any reason. If iscsid
is running, the client will handle this scenario properly.

The change made here is to add a udev rule installed into
/lib/udev/rules.d/70-iscsi-disk.rules with the content:
   SUBSYSTEM=="block", ACTION=="add", ENV{ID_PATH}=="*-iscsi-*",
       ENV{SYSTEMD_WANTS}="iscsid.service"

The key things to see in that rule are:
a.) only matches on add of a block device.
We will not currently stop the iscsid.service ever via udev.

b.) only matches a PATH with '-iscsi-' in it.
tests show that ID_PATH will come up with something like:
   ip-<ipv4-dotted-quad>:<port>-iscsi-<target>-lun-<lun>
example:
   ip-192.168.1.1:3270-iscsi-mydisk1-lun-2

See other info section below for a 'udevadm info' output.

c.) uses SYSTEMD_WANTS.
More information can be seen on that at
 https://www.freedesktop.org/software/systemd/man/systemd.device.html
doc states:

  "[systemd_wants] may be used to activate arbitrary units when a specific
   device becomes available."

[Test Case]
A check for iscsid.service to be running has been added to the dep8
test of the open-iscsi package. The simplist check is to have
the dep8 test run. The test will gate entry into -proposed so
we can have confidence the bug is fixed.

Since this bug was originally reported on Oracle public cloud instances
we should also test that platform.

To do that:
 * launch a hardware instance on oracle of 16.04.
 * verify iscsid.service is *not* running.
   If iscsid.service is already running, it may have been started by
   an image modification. To make this a valid test, you will
   then need to disable whatever change made that occur.
 * enable proposed and upgrade
 * reboot
 * verify that iscsid.service is running.

[Regression Potential]
The most likely chance for regression here would be the iscsid.service
running when it should not be running. Saving a udev rule execution
error, a false-positive on the 'match' above would trigger this.
It seems relatively unlikely that a non-iscsi device would have a ID_PATH
containing '-iscsi-', but is possible.

**Note**: This change caused regression described in bug 1807978.

[Other Info]
Here is output of 'udevadm info' on a iscsi root device:
    http://paste.ubuntu.com/p/dj89ScVR82/

There is autopkgtest run of proposed
  bionic: http://paste.ubuntu.com/p/FqPPtD8jV6/
  cosmic: http://paste.ubuntu.com/p/Fb3rYP7Xhs/

=== End SRU Template ===

It was reported that after the changes made in bug 1755858, the iscsid
service was not running when initramfs (via iscsi_* params or iBft)
set up an iscsi mount.

This seems to not be a problem until there is a restart of the iscsi
*host* service or some other hiccup. Thus, in normal testing you will
not see this issue.

--
Related bugs:
 * bug 1755858: iscsid autostarts on all servers when it has nothing to do.
 * bug 1807978: vmtest: iscsi/LVM tests failing on cosmic/disco

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: open-iscsi 2.0.874-5ubuntu2.3 [modified: lib/systemd/system/open-iscsi.service]
ProcVersionSignature: Ubuntu 4.15.0-36.39-generic 4.15.18
Uname: Linux 4.15.0-36-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.4
Architecture: amd64
Date: Thu Nov 8 17:42:46 2018
ProcEnviron:
 TERM=screen.xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
 SHELL=/bin/bash
SourcePackage: open-iscsi
UpgradeStatus: No upgrade log present (probably fresh install)
mtime.conffile..etc.iscsi.iscsid.conf: 2018-10-12T16:51:00.187144

Related branches

Revision history for this message
Scott Moser (smoser) wrote :
Scott Moser (smoser)
Changed in open-iscsi (Ubuntu):
assignee: nobody → Scott Moser (smoser)
importance: Undecided → High
status: New → Confirmed
Scott Moser (smoser)
description: updated
Revision history for this message
Scott Moser (smoser) wrote :

The goal here should be simple.
 If there are iscsi sessions, then iscsid should be started/running.

You can tell if there are iscsi sessions by either:
 - 'iscsiadm --mode=session' exits 0
 - /sys/class/iscsi_session is non-empty.

The current open-iscsi.service implementation will not result in a
iscsid.service running if it did not create sessions (either because
they already existed or because there was nothing to do).

I'm trying to work out the best way to implement that logic in the
systemd services that we have now.

Would it be acceptable for the open-iscsi.service file to simply execute
 'systemctl start iscsid.service'
if there was content in /sys/class/iscsi_session ?

My testing shows that 'systemctl start SERVICE' for a service that
is already running exits 0 and is not harmful.

Revision history for this message
Steve Langasek (vorlon) wrote :

Instead of manually invoking systemctl here which seems like an abstraction violation to me, I would suggest that open-iscsi.service should issue an iscsiadm command that talks to the socket and triggers the daemon to be started.

In effect, '/sbin/iscsiadm -m node --loginall=automatic' already does this but only for the case that /etc/iscsi/nodes is non-empty.

I would suggest 'iscsiadm -m session -s'.

Revision history for this message
Scott Moser (smoser) wrote : Re: [Bug 1802354] Re: iscsid does not run if there are only initramfs initiated targets

@Steve,

That feels overly indirect.
If open-iscsi.service (or anything) knows that iscsid should be running,
then why be sly?

Thats essentially what we were trying to do, and in some cases it didn't
work. Starting a persistent daemon by requesting to list the current
sessions seems like an arbitrary side-effect.

fwiw, in my testing here 'iscsiadm -m session -s' ends up connecting to
the socket (thus starting the service) and writing:
  iscsiadm: Could not execute operation on all sessions: session not found
and then exiting 2.
That is all working "as designed".

If we want to start the service, or make sure that its running. It seems
better just to start the service. If we want to do that by the indirect
mechanism of connecting to a socket, then I suggest we do *that* directly
like in 'connect-abstract-socket.py':

  #!/usr/bin/python3
  from socket import socket, AF_UNIX, SOCK_STREAM
  import sys
  name = "ISCSIADM_ABSTRACT_NAMESPACE" if len(sys.argv) < 2 else sys.argv[1]
  print("Connecting to @%s" % name)
  socket(AF_UNIX, SOCK_STREAM).connect(('\0' + name))
  print("Connected to @%s" % name)
On Thu, Nov 8, 2018 at 6:20 PM Steve Langasek
<email address hidden> wrote:
>
> Instead of manually invoking systemctl here which seems like an
> abstraction violation to me, I would suggest that open-iscsi.service
> should issue an iscsiadm command that talks to the socket and triggers
> the daemon to be started.
>
> In effect, '/sbin/iscsiadm -m node --loginall=automatic' already does
> this but only for the case that /etc/iscsi/nodes is non-empty.
>
> I would suggest 'iscsiadm -m session -s'.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1802354
>
> Title:
> iscsid does not run if there are only initramfs initiated targets
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1802354/+subscriptions

Revision history for this message
Scott Moser (smoser) wrote :

in IRC, xnox suggested a SYSTEMD_WANTS as a udev rule.
I'm looking at that solution.
 https://irclogs.ubuntu.com/2018/11/09/%23ubuntu-devel.html

tags: added: id-5be36a3244fbc018f0cc1591
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package open-iscsi - 2.0.874-5ubuntu11

---------------
open-iscsi (2.0.874-5ubuntu11) disco; urgency=medium

  * debian/tests/test-open-iscsi.py: Fix called process error.
    If no image was available for a release, a NameError exception would be
    raised.

 -- Scott Moser <email address hidden> Wed, 14 Nov 2018 16:50:29 -0500

Changed in open-iscsi (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
Scott Moser (smoser) wrote :

Just for reference, this was actually fixed in disco at 2.0.874-5ubuntu10.
But that never made it into the release pocket before 5ubuntu11 went in.
The relevant changelogs:

open-iscsi (2.0.874-5ubuntu11) disco; urgency=medium

  * debian/tests/test-open-iscsi.py: Fix called process error.
    If no image was available for a release, a NameError exception would be
    raised.

 -- Scott Moser <email address hidden> Wed, 14 Nov 2018 16:50:29 -0500

open-iscsi (2.0.874-5ubuntu10) disco; urgency=medium

  * d/iscsi-disk.rules, d/tests: Add a udev rule so that iscsid.service
    will be run when udev disks are attached. (LP: #1802354)
  * debian/tests: improve tgt boot test
     - disable snapd and snap.seeded services to avoid unnecessary
       resource consumption during tests.
     - save artifacts from the test run.

 -- Scott Moser <email address hidden> Wed, 14 Nov 2018 16:05:46 -0500

description: updated
Changed in open-iscsi (Ubuntu Bionic):
status: New → Confirmed
Changed in open-iscsi (Ubuntu Cosmic):
status: New → Confirmed
Changed in open-iscsi (Ubuntu Bionic):
importance: Undecided → High
Changed in open-iscsi (Ubuntu Cosmic):
importance: Undecided → High
Changed in open-iscsi (Ubuntu Bionic):
assignee: nobody → Scott Moser (smoser)
Changed in open-iscsi (Ubuntu Cosmic):
assignee: nobody → Scott Moser (smoser)
Changed in open-iscsi (Ubuntu Bionic):
status: Confirmed → In Progress
Changed in open-iscsi (Ubuntu Cosmic):
status: Confirmed → In Progress
Scott Moser (smoser)
description: updated
Scott Moser (smoser)
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Scott, or anyone else affected,

Accepted open-iscsi into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-iscsi/2.0.874-5ubuntu9.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in open-iscsi (Ubuntu Cosmic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-cosmic
Changed in open-iscsi (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Scott, or anyone else affected,

Accepted open-iscsi into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-iscsi/2.0.874-5ubuntu2.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :
Download full text (3.5 KiB)

I'm marking bionic as verified.

In order to test this I did the following:
a.) have attached the log and artifacts from the open-iscsi dep8 run
 for open-iscsi/2.0.874-5ubuntu2.4 on 2018-11-20 18:42:40 UTC
 as 'bionic-autopkgtest.log' and 'bionic-autopkgtest-artifacts.tar.gz'

b.) tested on oracle bare metal.
Launched an instance of image '2018.10.16-0'
There was a more recent '2018.11.17-0', but I wanted to avoid any
workarounds that might have been put into it.

Upon connection via ssh:

## verify not running and show old version.
$ dpkg-query --show open-iscsi
open-iscsi 2.0.874-5ubuntu2.3

$ systemctl is-active iscsid.service
inactive

$ systemctl status iscsid.service --full --no-pager
● iscsid.service - iSCSI initiator daemon (iscsid)
   Loaded: loaded (/lib/systemd/system/iscsid.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:iscsid(8)

# show there is a iscsi mount
$ ls /dev/disk/by-path/
ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1
ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1-part1
ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1-part14
ip-169.254.0.2:3260-iscsi-iqn.2015-02.oracle.boot:uefi-lun-1-part15

## enable -proposed and upgrade and reboot
$ echo "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main" | sudo tee -a /etc/apt/sources.list.d/proposed.list
deb http://archive.ubuntu.com/ubuntu bionic-proposed main

$ sudo apt-get update -q
$ sudo apt-get install open-iscsi
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
  grub-pc-bin
Use 'sudo apt autoremove' to remove it.
The following packages will be upgraded:
  open-iscsi
1 upgraded, 0 newly installed, 0 to remove and 104 not upgraded.
Need to get 280 kB of archives.
After this operation, 1024 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 open-iscsi amd64 2.0.874-5ubuntu2.4 [280 kB]
...
Unpacking open-iscsi (2.0.874-5ubuntu2.4) over (2.0.874-5ubuntu2.3) ...
...
Setting up open-iscsi (2.0.874-5ubuntu2.4) ...
...
# the upgrade requires a reboot to fix.
$ systemctl is-active iscsid.service
inactive

$ sudo reboot

## ssh back in to see fix.
$ dpkg-query --show open-iscsi
open-iscsi 2.0.874-5ubuntu2.4
$ systemctl is-active iscsid.service
active
$ systemctl status iscsid.service --full --no-pager
● iscsid.service - iSCSI initiator daemon (iscsid)
   Loaded: loaded (/lib/systemd/system/iscsid.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-11-27 20:04:06 UTC; 1min 22s ago
     Docs: man:iscsid(8)
  Process: 1885 ExecStart=/sbin/iscsid (code=exited, status=0/SUCCESS)
  Process: 1873 ExecStartPre=/lib/open-iscsi/startup-checks.sh (code=exited, status=0/SUCCESS)
 Main PID: 1890 (iscsid)
    Tasks: 2 (limit: 11059)
   CGroup: /system.slice/iscsid.service
           ├─1888 /sbin/iscsid
           └─1890 /sbin/iscsid

Nov 27 20:04:01 smoser-b1 iscsid[1885]: iSCSI logger with pid=1888 started!
Nov 27 20:04:02 smoser-b1 iscsid[1888]: iSCSI daemon with pid=1890 started!
Nov 2...

Read more...

tags: added: verification-done-bionic
removed: verification-needed-bionic
Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Scott Moser (smoser) wrote :
tags: added: verification-done verification-done-cosmic
removed: verification-needed verification-needed-cosmic
Revision history for this message
Scott Moser (smoser) wrote :

I've attachd cosmic logs of a (second) run of the autopkg test.
this was successful and the following shows iscsi was running

$ grep open-iscsi manifest.txt
open-iscsi 2.0.874-5ubuntu9.1

$ cat ./iscsid-status.txt
active
● iscsid.service - iSCSI initiator daemon (iscsid)
   Loaded: loaded (/lib/systemd/system/iscsid.service; disabled; vendor preset: enabled)
   Active: active (running) since Tue 2018-11-27 19:50:33 UTC; 2min 28s ago
     Docs: man:iscsid(8)
  Process: 731 ExecStart=/sbin/iscsid (code=exited, status=0/SUCCESS)
  Process: 697 ExecStartPre=/lib/open-iscsi/startup-checks.sh (code=exited, status=0/SUCCESS)
 Main PID: 736 (iscsid)
    Tasks: 2 (limit: 547)
   Memory: 3.5M
   CGroup: /system.slice/iscsid.service
           ├─734 /sbin/iscsid
           └─736 /sbin/iscsid

Nov 27 19:50:15 ubuntu systemd[1]: Starting iSCSI initiator daemon (iscsid)...
Nov 27 19:50:30 ubuntu iscsid[731]: iSCSI logger with pid=734 started!
Nov 27 19:50:31 ubuntu iscsid[734]: iSCSI daemon with pid=736 started!
Nov 27 19:50:33 ubuntu systemd[1]: Started iSCSI initiator daemon (iscsid).
Nov 27 19:50:34 ubuntu iscsid[734]: Could not read data from db. Using default and currently negotiated values
Nov 27 19:50:38 ubuntu iscsid[734]: connection1:0 is operational after recovery (1 attempts)

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package open-iscsi - 2.0.874-5ubuntu9.1

---------------
open-iscsi (2.0.874-5ubuntu9.1) cosmic; urgency=medium

  * d/iscsi-disk.rules, d/rules: Add a udev rule so that iscsid.service
    will be started when iscsi disks are attached. (LP: #1802354)
  * debian/tests: sync tests with those in disco.
     - add check that iscsid.service is run.
     - save artifacts from the test run.
     - when running vm for iscsi root test, disable snapd.service
       and snapd.seeded.service to avoid unnecessary cpu and IO.

 -- Scott Moser <email address hidden> Thu, 15 Nov 2018 12:57:24 -0500

Changed in open-iscsi (Ubuntu Cosmic):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for open-iscsi has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Scott Moser (smoser) wrote :

This is currently held up from getting into 18.04 due to a failing
autopkgest for resource-agents on armhf.
   https://autopkgtest.ubuntu.com/packages/resource-agents/bionic/armhf

It seems this test fails transiently. It was quite reliable from
 2017-11-27 to 2018-08-09
with only a single failure in 44 runs.
But 2018-08-22 and after there have been only 3 successful runs in 18.

The point of this is to show that there may well have been a regression
that caused resource-agents to fail, but it is not likely this change.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package open-iscsi - 2.0.874-5ubuntu2.4

---------------
open-iscsi (2.0.874-5ubuntu2.4) bionic; urgency=medium

  * d/iscsi-disk.rules, d/rules: Add a udev rule so that iscsid.service
    will be started when iscsi disks are attached. (LP: #1802354)
  * debian/tests: sync tests with those in disco.
     - add check that iscsid.service is run.
     - save artifacts from the test run.

 -- Scott Moser <email address hidden> Thu, 15 Nov 2018 12:59:24 -0500

Changed in open-iscsi (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Scott Moser (smoser) wrote :

The change here seems to have regressed at least some non-root iscsi volume use cases. The curtin 'vmtest' test harness is showing failure of its LVM on iscsi tests.

See bug 1802354 for more information.

Revision history for this message
Steve Langasek (vorlon) wrote :

"more information" bug is bug #1807978.

Scott Moser (smoser)
description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.