zfs services fail on firstboot if zfs-utils is integrated into the deployment image

Personally I think we should modify MNTTAB in zfs-linux to point to /proc/self/mounts. This may not be completely straightforward *(there are a number of places that point to /etc/mnttab directly, or have logic around it and /proc/mounts), but it should be doable. I like this solution most because it doesn't require us to increase the number of dependencies in the zfs systemd scripts.

No reason to fix this via sysvinit, which should be considered deprecated at this point.

chiluk is looking to work on this for upstream. I might jump in too. https://github.com/zfsonlinux/zfs/issues/4680

Confirmed. /etc/mtab is dead, dead, dead, and an ill-conceived and broken concept. Don't ever use it please ☺

I jointly worked with the zfsonlinux maintainer on a patch that will be merged in 0.7.0 milestone
https://github.com/zfsonlinux/zfs/milestone/11?closed=1

#4680, has been tackled in two parts. the first part (patch that will be merge in 0.7.0), change from /etc/mtab to /proc/self/mounts, and keep the existing mtab update functions alone in the mount helper.

Eventually, a second part would be for ZoL contributor/maintainers to do the integration with libmount. libmount didn't exist on Linux when the mount helper code was originally written. That could allow ZoL to drop their custom code.

The fix was merged upstream here:
https://github.com/zfsonlinux/zfs/commit/792517389fad5c495a2738b61c2e9c65dedaaa9a

I tested this using my reproduction method in AWS EC2 with a test package/PPA provided by Eric@Ubuntu (slashd) containing the commit referenced in comment #6.

All looks good now - systemd units for ZFS no longer fail. Thanks for moving this along!

Thanks scotte for the feedbacks, I will now start the SRU process.

Eric

debdiff for Xenial (16.04 LTS)

debdiff for devel release : Yakkety (16.10)

Debian bug :
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839071

@scotte,

After some verification, I found out that cking uploaded a patch on Sept 26 on Xenial for another zfs-linux bug[1] using version "0.6.5.6-0ubuntu13", which interfere with my debdiff.

"pull-lp-source" and "rmadison" didn't shown the existence of "0.6.5.6-0ubuntu13".
I found it by looking in the upload queue : https://launchpad.net/ubuntu/xenial/+queue?queue_state=1

With that being said, I will need to wait for the current zfs-linux SRU on Xenial to be completed and then I'll re-do the debdiff in order to increment the version for this bug to accordingly "0.6.5.6-0ubuntu14".

The Xenial SRU might take a little bit more time than expected to land in stable (-update), but
the devel release debdiff (Yakkety) is still fine.

I'll keep you posted.

[1] - https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1627909

Eric
Eric

I removed the Xenial debdiff attached to this bug and I will re-do it when current zfs-linux SRU for Xenial by cking will be completed.

Until then, the Yakkety debdiff is still fine and can be sponsored.

Eric

I pulled the source code per version and rebase my debdiff on top of "0.6.5.6-0ubuntu13".

Will attach the new debdiff in a couple of minutes.

Eric

Rebased debdiff for Xenial (16.04 LTS)

This bug was fixed in the package zfs-linux - 0.6.5.8-0ubuntu3

---------------
zfs-linux (0.6.5.8-0ubuntu3) yakkety; urgency=medium

  [ Eric Desrochers ]
  * Fix zfs services to fail on first boot due to zfs services starting up
    before /etc/mtab has a chance to be symlinked to /proc/mounts. (LP: #1607920)
    (upstream commit 792517389fad5c495a2738b61c2e9c65dedaaa9a)

 -- Eric Desrochers <email address hidden> Thu, 29 Sep 2016 16:13:38 +0200

As per irc conversation with apw, I'm merging both LPs #1607920 & #1628279 into one debdiff.

@Eric -- I am concerned about the effect of some of these changes on translations. For example the below will make the format string a new translatable and therefore will not be translated for non english locales. If we simply replaces MNTTAB with "/etc/mtab" the translatable string would be unchanged:

+ (void) fprintf(stderr, gettext(
+- "filesystem '%s' was mounted, but %s "
++ "filesystem '%s' was mounted, but /etc/mtab "
+ "could not be opened due to error %d\n"),
+- dataset, MNTTAB, errno);
++ dataset, errno);

There are a few like that.

Also as you are changing MNTTAB is there any reason not to be using that here?

+- (void) fprintf(stderr, gettext("warning: %s not in mtab\n"),
+- path);
++ (void) fprintf(stderr, gettext("warning: %s not in"
++ "/proc/self/mounts\n"), path);

Does ZFS actually have any translations? There are no .po files in the source and no .mo files in the binary package.

@apw

To answer your question:
"Also as you are changing MNTTAB is there any reason not to be using that here?"

The "reading" is redirected to /proc/self/mounts and the "writing" to /etc/mtab, but if it doesn't exist, that's fine. ZFS just won't update /etc/mtab.

Ideally, ZFS should be integrated with libmount in a second phase (ZoL maintainers are aware that eventually a 2nd is needed to integrate libmount), until then /proc/self/mounts is a viable solution to address issues cause by mtab.

Eric

@apw

With regard to the translation ....

The commit has been intensively tested in different scenarios/env/... and of course with the "ZFS Test Suite"[1] which include functionality, perf, and stress tests.

All the tests were reported as successful.

I hope this answer your questions and addresses your concern.

Let rlaager and I know if you need more clarifications.

[1] - https://github.com/zfsonlinux/zfs/tree/master/tests

Eric

Hello Dave, or anyone else affected,

Accepted zfs-linux into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/zfs-linux/0.6.5.6-0ubuntu14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I tested again with xenial-proposed and confirmed that everything looks good as far as systemd no longer failing ZFS units due to no /etc/mtab.

ii libzfs2linux 0.6.5.6-0ubuntu14 amd64 Native OpenZFS filesystem library for Linux
ii zfs-dkms 0.6.5.6-0ubuntu14 amd64 Native OpenZFS filesystem kernel modules for Linux
ii zfs-doc 0.6.5.6-0ubuntu14 all Native OpenZFS filesystem documentation and examples.
ii zfs-initramfs 0.6.5.6-0ubuntu14 all Native OpenZFS root filesystem capabilities for Linux
ii zfs-zed 0.6.5.6-0ubuntu14 amd64 OpenZFS Event Daemon (zed)
ii zfsutils-linux 0.6.5.6-0ubuntu14 amd64 Native OpenZFS management utilities for Linux

# systemctl --failed
0 loaded units listed. [...]

Thanks scotte for the feedbacks.

This bug was fixed in the package zfs-linux - 0.6.5.6-0ubuntu14

---------------
zfs-linux (0.6.5.6-0ubuntu14) xenial; urgency=medium

  * Fix zfs services to fail on first boot due to zfs services starting up
    before /etc/mtab has a chance to be symlinked to /proc/mounts. (LP: #1607920)
    (upstream commit 792517389fad5c495a2738b61c2e9c65dedaaa9a)

  * Symlink utilities in /usr/sbin to remove suffix as per policy 10.4 Scripts (LP: #1628279)
    Users can rely on the new names, but relying on the old names won't break before
    release upgrade.

 -- Eric Desrochers <email address hidden> Tue, 04 Oct 2016 15:07:52 +0200

The verification of the Stable Release Update for zfs-linux has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Hi scotte,

The fix is now released as shown in comment#25 & #26

Eric

Thanks Eric, we will remove our local workaround.