fails on install due to systemd-service/sysv-service mismatch and lack of mkdir

P.S. I'm not sure how this evaded our detection on Sergio's merge in January nor in Groovy by Rafael. Maybe I'm mis-interpreting all the data ?

This is strange, indeed. The MP for the merge is:

https://code.launchpad.net/~sergiodj/ubuntu/+source/python-rtslib-fb/+git/python-rtslib-fb/+merge/397174

As you can read there, I did check if the package installed/uninstalled correctly. Bryce also double-checked it. I don't know if the problem is that we forgot to check if the systemd service file was up or not, or something else. But then again, the error you posted happens right at the installation step, so we would have caught that. I'm puzzled, and I apologize for not having seen this before. Thanks for taking care of this issue.

This bug was fixed in the package python-rtslib-fb - 2.1.74-0ubuntu2

---------------
python-rtslib-fb (2.1.74-0ubuntu2) hirsute; urgency=medium

  * Fix service fail on install due to name mismatch and lost mkdir (LP: #1916715)
    - d/python3-rtslib-fb.rtslib-fb-targetctl.service: add back debianized
      systemd service file
    - d/rules: install rtslib-fb-targetctl.service as it was in Focal

 -- Christian Ehrhardt <email address hidden> Wed, 24 Feb 2021 09:32:35 +0100

We'd want to fix groovy as well I guess, adding server-next tag.

This change didn't work for me in bionic
because of this:

/lib/systemd/system/rtslib-fb-targetctl.service:9: Executable path is not absolute: mkdir -p /etc/rtslib-fb-target

i had to change this:
ExecStart=mkdir -p /etc/rtslib-fb-target

to this:
ExecStart=/bin/mkdir -p /etc/rtslib-fb-target

Thanks Adam for your input, this is absolutely worth to consider where we apply the change.

But you mentioned Bionic where we only have old style init in the form of /etc/init.d/rtslib-fb-targetctl and IIRC that wasn't affected by the same issue.

@Utkarsh is this still on your list of things to look at (at least for groovy)? If you do so could you double check if bionic (or Focal) needs a helping hand as well?

Hi Christian,

> @Utkarsh is this still on your list of things to look at (at least
> for groovy)? If you do so could you double check if bionic (or Focal)
> needs a helping hand as well?

Absolutely, shall do. I intend to get this done by this week.

Hi Sergio,

> As you can read there, I did check if the package installed/uninstalled
> correctly. Bryce also double-checked it. I don't know if the problem
> is that we forgot to check if the systemd service file was up or not,
> or something else. But then again, the error you posted happens right
> at the installation step, so we would have caught that. I'm puzzled [..]

If you do a fresh install of this package in a fresh LXC container, you'll not be able to reproduce this. It needs the configfs kernel module, which is provided by linux-modules-extra-5.8.0-44-generic (in groovy; versions differ and so would the package name in other releases).

Once you install that and then try to install python3-rtslib-fb, you'd then be able to reproduce this error. So I believe you were missing the configfs module and hence the install appeared to be fine.

I am just letting you know so maybe you can check these things the next time or so! \o/
(but very likely I'd have missed this myself; only this bug report made me look deeper)

Fix for Groovy is up already and should soon be uploaded (hopefully!). And this isn't really reproducible on Bionic or Focal. They seem to be installing just fine with `systemctl status rtslib-fb-targetctl.service` looking alright as well.

Hello Christian, or anyone else affected,

Accepted python-rtslib-fb into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-rtslib-fb/2.1.73-1ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

#2 Container without modules installed

old:
● rtslib-fb-targetctl.service - LSB: Start LIO targets
     Loaded: loaded (/etc/init.d/rtslib-fb-targetctl; generated)
     Active: active (exited) since Wed 2021-04-07 06:14:25 UTC; 8s ago
       Docs: man:systemd-sysv-generator(8)
      Tasks: 0 (limit: 38268)
     Memory: 0B
     CGroup: /system.slice/rtslib-fb-targetctl.service

Apr 07 06:14:25 python-rtslib-fb-lp1916715-groovy systemd[1]: Starting LSB: Start LIO targets...
Apr 07 06:14:25 python-rtslib-fb-lp1916715-groovy rtslib-fb-targetctl[1193]: modprobe: FATAL: Module configfs not found in directory /lib/modules/5.4.0-70-generic
Apr 07 06:14:25 python-rtslib-fb-lp1916715-groovy rtslib-fb-targetctl[1187]: Could not load configfs module: exiting!
Apr 07 06:14:25 python-rtslib-fb-lp1916715-groovy systemd[1]: Started LSB: Start LIO targets.

new:
root@python-rtslib-fb-lp1916715-groovy-fixed:~# systemctl status rtslib-fb-targetctl.service
● rtslib-fb-targetctl.service - Restore LIO kernel target configuration
     Loaded: loaded (/lib/systemd/system/rtslib-fb-targetctl.service; enabled; vendor preset: enabled)
     Active: inactive (dead)

Apr 07 06:14:26 python-rtslib-fb-lp1916715-groovy-fixed systemd[1]: Condition check resulted in Restore LIO kernel target configuration being skipped.

=> better error message, no degradation

#2 VM with kernel&modules properly in place for LIO:

old:
ubuntu@g-1916715:~$ sudo apt install python3-rtslib-fb
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  python3-pyudev
The following NEW packages will be installed:
  python3-pyudev python3-rtslib-fb
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 86.0 kB of archives.
After this operation, 639 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu groovy/main amd64 python3-pyudev all 0.21.0-3ubuntu1 [37.1 kB]
Get:2 http://archive.ubuntu.com/ubuntu groovy/main amd64 python3-rtslib-fb all 2.1.73-1ubuntu4 [48.9 kB]
Fetched 86.0 kB in 0s (546 kB/s)
Selecting previously unselected package python3-pyudev.
(Reading database ... 73160 files and directories currently installed.)
Preparing to unpack .../python3-pyudev_0.21.0-3ubuntu1_all.deb ...
Unpacking python3-pyudev (0.21.0-3ubuntu1) ...
Selecting previously unselected package python3-rtslib-fb.
Preparing to unpack .../python3-rtslib-fb_2.1.73-1ubuntu4_all.deb ...
Unpacking python3-rtslib-fb (2.1.73-1ubuntu4) ...
Setting up python3-pyudev (0.21.0-3ubuntu1) ...
Setting up python3-rtslib-fb (2.1.73-1ubuntu4) ...
Job for rtslib-fb-targetctl.service failed because the control process exited with error code.
See "systemctl status rtslib-fb-targetctl.service" and "journalctl -xe" for details.
invoke-rc.d: initscript rtslib-fb-targetctl, action "start" failed.
● rtslib-fb-targetctl.service - LSB: Start LIO targets
     Loaded: loaded (/etc/init.d/rtslib-fb-targetctl; generated)
     Active: failed (Result: exit-code) since Wed 2021-04-07 06:25:13 UTC; 8ms ago
       Docs: man:systemd-sysv-generator(8)
    Process: 1327 ExecSt...

The verification of the Stable Release Update for python-rtslib-fb has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package python-rtslib-fb - 2.1.73-1ubuntu4.1

---------------
python-rtslib-fb (2.1.73-1ubuntu4.1) groovy; urgency=medium

  * Fix service fail on install due to name mismatch and lost mkdir
    (LP: #1916715)
    - d/python3-rtslib-fb.rtslib-fb-targetctl.service: add back debianized
      systemd service file.
    - d/python3-rtslib-fb.rtslib-fb-targetctl.service: Add check to skip
      starting this service when running in a container (LP: #1865037).
    - d/rules: install rtslib-fb-targetctl.service as it was in Focal.

 -- Utkarsh Gupta <email address hidden> Fri, 12 Mar 2021 15:52:34 +0530

This also affects Focal installs and it affects openstack upgrades where the cinder services are running in a container (see bug #1930455), and I think we should backport this patch to Focal as well. I'm going to nominate the series as such.

Status changed to 'Confirmed' because the bug affects multiple users.

Hi Billy,
thanks for the ping!

Before in comment #9 it was reported that our current reproducer/testcase wasn't present on Bionic/Focal. Also some of the fix was "make it as it was in focal" so soemthing is odd here.

Do you happen to have a non-charm based way to trigger the issue on Focal that we could use to drive the fix into an SRU?

Hi Christian,

Thanks for the feedback, I see the error I made here. I confused this bug with bug #1865037, which actually adds the ConditionVirtualization=!container directive to the service file but was wrapped in the changelog. That's the change that needs to be made for focal so I'll do some bug tidying here.

The recreate is simple - launch a new focal container and try to install python3-rtslib-fb inside it, but I'll move this over to the proper bug.

I just hit this issue on the UCA bionic-ussuri version, as part of testing automated OpenStack upgrades. This library is pulled in as a cinder "suggested" package, and under my testing it broke on the upgrade from the UCA bionic-train version to UCA bionic-ussuri.

The fix for the issue was exactly as Adam said in comment #5. While this is on Bionic, systemd is in play here and the invalid definition in /lib/systemd/system/rtslib-fb-targetctl.service is impacting in this case. The bionic-train version doesn't have the offending mkdir call at all; the bionic-ussuri version introduces the bugged version (i.e. "ExecStart=mkdir -p /etc/rtslib-fb-target").

I ran into the same exact issue as Paul in comment #18 while upgrading from UCA bionic-train to UCA bionic-ussuri.