Cannot install snaps on Ubuntu 14.04 with /var on its own partition

Additional information:

root@ubuntu:~# systemctl status systemd-fsck@dev-mapper-vg00\x2dvarvol.service
<email address hidden>
   Loaded: error (Reason: No such file or directory)
   Active: inactive (dead)

root@ubuntu:~# systemctl status systemd-fsck@dev-mapper-vg00\varvol.service
<email address hidden>
   Loaded: error (Reason: No such file or directory)
   Active: inactive (dead)

root@ubuntu:~# systemctl status --all | grep fsck
root@ubuntu:~#

Trusty doesn't have the full implementation of systemd like xenial has.

xenial:

$ ls -l /lib/systemd/system/systemd-fsck*
-rw-r--r-- 1 root root 551 Jul 18 19:56 /lib/systemd/system/systemd-fsckd.service
-rw-r--r-- 1 root root 540 Jul 18 19:56 /lib/systemd/system/systemd-fsckd.socket
-rw-r--r-- 1 root root 674 Jul 18 19:56 /lib/systemd/system/systemd-fsck-root.service
-rw-r--r-- 1 root root 648 Jul 18 19:56 /lib/systemd/system/systemd-fsck@.service

trusty:

# ls -l /lib/systemd/system/systemd-fsck*
ls: cannot access /lib/systemd/system/systemd-fsck*: No such file or directory

The system tries to call the fsck check but fails because the method or file doesn't exist:

Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck@dev-mapper-vg00\x2dvarvol.service failed to load: No such file or directory.

On a xenial system, that file would be this:
==========================
$ cat /lib/systemd/system/systemd-fsck@.service
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.

[Unit]
Description=File System Check on %f
Documentation=man:systemd-fsck@.service(8)
DefaultDependencies=no
BindsTo=%i.device
Wants=systemd-fsckd.socket
After=%i.device systemd-fsck-root.service local-fs-pre.target systemd-fsckd.socket
Before=shutdown.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/lib/systemd/systemd-fsck %f
TimeoutSec=0
==========================

And the systemctl services and units would be there on xenial:
==========================
$ systemctl status --all | grep fsck | grep ^●
● systemd-fsck-root.service - File System Check on Root Device
● systemd-fsck@dev-disk-by\x2duuid-0f1a47d4\x2d67c5\x2d4c18\x2d9159\x2dfd1d4409f30d.service - File System Check on /dev/disk/by-uuid/0f1a47d4-67c5-4c18-9159-fd1d4409f30d
● systemd-fsck@dev-disk-by\x2duuid-D233\x2dB0AC.service - File System Check on /dev/disk/by-uuid/D233-B0AC
● systemd-fsckd.service - File System Check Daemon to report status
● system-systemd\x2dfsck.slice
● systemd-fsckd.socket - fsck to fsckd communication Socket
==========================

But not on Trusty:

root@trustysnap:~# systemctl status --all | grep fsck
root@trustysnap:~#

I don't see any relevant logs in /var/log.

I was also able to reproduce this, here are my notes for now:

## /etc/fstab

LABEL=LVROOT / ext4 errors=remount-ro 0 1
LABEL=LVVAR /var ext4 defaults 0 1
LABEL=TESTE /teste ext4 defaults 0 1

Right after boot:

inaddy@trustylivepatch:~$ systemctl list-units --all | grep fsck
systemd-...el-LVVAR.service error inactive dead systemd-fsck@dev-disk-by\x2dlabel-LVVAR.service
systemd-...el-TESTE.service error inactive dead systemd-fsck@dev-disk-by\x2dlabel-TESTE.service

This indicates that UPSTART is the one mounting the block devices, NOT SYSTEMD using its mount units. SNAPS are mounting the SQUASH filesystems using SYSTEMD UNITS, despite UPSTART scripts.

It is likely that this wasn't noticed, on systems mounting "/" only, because the "-.mount" SYSTEMD UNIT doesn't depend on "systemd-fsck@.service" unit, it depends only on "systemd-fsck-root.service", non existent in TRUSTY's SYSTEMD version. Probably this made SYSTEMD to act like no error existed.

For SYSTEMD mount units to work, it is needed that no fsck unit error exists - like when having /var or any other mounting besides root filesystem - allowing all SYSTEMD units created by snappy to work.

Comparing default setups for TRUSTY and ZESTY:

--------
## TRUSTY

$ dpkg -L systemd | grep fsck
/lib/systemd/systemd-fsck

$ systemctl list-units --all | grep fsck
systemd-...el-LVVAR.service error inactive dead systemd-fsck@dev-disk-by\x2dlabel-LVVAR.service
systemd-...el-TESTE.service error inactive dead systemd-fsck@dev-disk-by\x2dlabel-TESTE.service

$ systemctl list-units --all | grep mount
-.mount loaded active mounted /
teste.mount loaded active mounted /teste
var.mount loaded active mounted /var
umount.target loaded inactive dead Unmount All Filesystems

## ZESTY

$ dpkg -L systemd | grep fsck
/lib/systemd/system/systemd-fsck-root.service
/lib/systemd/system/systemd-fsck@.service
/lib/systemd/system/systemd-fsckd.service
/lib/systemd/system/systemd-fsckd.socket
/lib/systemd/systemd-fsck
/lib/systemd/systemd-fsckd

$ systemctl list-unit-files | grep fsck
systemd-fsck-root.service static
systemd-fsck@.service static
systemd-fsck@dev-disk-by\x2dlabel-TESTE.service static
systemd-fsckd.service static
systemd-fsckd.socket static

$ systemctl list-unit-files | grep mount
-.mount generated
home-inaddy-work.mount generated
mnt.mount static
mountall.service masked
umountfs.service masked
umountroot.service masked
umount.target static
--------

SYSTEMD in TRUSTY was treated differently for FSCK. TRUSTY's version contains systemd-fsck but not systemd-fsckd, the daemon responsible for consolidating all fsck information for SYSTEMD journal. It is also clear that TRUSTY did not include any unit file for systemd-fsck@.service, that might still be considered for the automatically generated mount unit files.

You can reproduce this by trying to use TRUSTY SYSTEMD mount units:

--------
## TRUSTY...

Status changed to 'Confirmed' because the bug affects multiple users.

Ok so in systemd-204, in TRUSTY, you will find the following logic to create the mount units:

- manager_loop
  - process-event (WATCH_MOUNT)
    - mount_fd_event
      - mount_load_proc_self_mountinfo
        - mount_add_one (if load_extras == true)
          - mount_add_extras
            - mount_add_device_links

The last function will add: device/mount/socket/swap/path/requires/... dependencies in the mount unit to be created. The first one "device" is handled by "mount_add_device_links" and, according to logic:

        if (p->passno > 0 &&
            UNIT(m)->manager->running_as == SYSTEMD_SYSTEM) {
                char *name;
                Unit *fsck;
                /* Let's add in the fsck service */

                /* aka SPECIAL_FSCK_SERVICE */
                name = unit_name_from_path_instance("systemd-fsck", p->what, ".service");
                if (!name)
                        return -ENOMEM;

If filesystem option passno is 0, it won't have the systemd-fsck unit as a requirement, meaning that no error will be given, allowing the mount unit, created by snappy, to work.

----

Check it out:

1)

## /etc/fstab

LABEL=LVROOT / ext4 errors=remount-ro 0 1
LABEL=LVVAR /var ext4 defaults 0 1
LABEL=TESTE /teste ext4 defaults 0 1

$ sudo snap install hello-world

error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck@dev-disk-by\x2dlabel-LVVAR.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck@dev-disk-by\x2dlabel-LVVAR.service' for details.

2)

## /etc/fstab

LABEL=LVROOT / ext4 errors=remount-ro 0 1
LABEL=LVVAR /var ext4 defaults 0 0
LABEL=TESTE /teste ext4 defaults 0 0

$ sudo snap install hello-world
hello-world 6.3 from 'canonical' installed

----

Now I'm checking why debian/rules in systemd-204 didn't include fsck@.service files in the final .deb for "systemd" package. Possibly this wasn't noticed because at the time systemd was running in Trusty, it wasn't imagined that mount units would be managed by systemd (like snappy is doing now). I'll check if adding the units for fsck (plus having systemd-fsck binary, already there) is enough for the mount logic to work.

For now, use the workaround above ^.

I confirm the workaround works for me:

Before:

root@ubuntu:~# cat /etc/fstab
/dev/mapper/vg00-rootvol / ext4 errors=remount-ro 0 1
/dev/mapper/vg00-varvol /var ext4 defaults 0 2
/dev/mapper/vg00-swap none swap sw 0 0

root@ubuntu:~# snap install hello-world
error: cannot perform the following tasks:
- Mount snap "core" (2898) ([start snap-core-2898.mount] failed with exit status 6: Failed to issue method call: Unit systemd-fsck@dev-mapper-vg00\x2dvarvol.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck@dev-mapper-vg00\x2dvarvol.service' for details.
)

After:

root@ubuntu:~# cat /etc/fstab
/dev/mapper/vg00-rootvol / ext4 errors=remount-ro 0 1
/dev/mapper/vg00-varvol /var ext4 defaults 0 0
/dev/mapper/vg00-swap none swap sw 0 0

root@ubuntu:~# snap install hello-world
hello-world 6.3 from 'canonical' installed

root@ubuntu:~# snap install canonical-livepatch
canonical-livepatch 7 from 'canonical' installed

root@ubuntu:~# sudo canonical-livepatch enable <redacted>

root@ubuntu:~# canonical-livepatch status
kernel: 4.4.0-96.119~14.04.1-generic
fully-patched: true
version: ""

Thanks Rafael!

Okay, so another quick workaround, now keeping the fsck enabled for the volumes:

Create a file called "/lib/systemd/upstart/systemd-fsck@.service" with:

----
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.

[Unit]
Description=File System Check on %f
Documentation=man:systemd-fsck@.service(8)
DefaultDependencies=no
BindsTo=%i.device
After=systemd-readahead-collect.service systemd-readahead-replay.service %i.device
Before=shutdown.target

[Service]
Type=oneshot
RemainAfterExit=no
ExecStart=/lib/systemd/systemd-fsck %f
StandardOutput=journal+console
TimeoutSec=0
----

and then do:

$ sudo ln -s /lib/systemd/upstart/systemd-fsck@.service /lib/systemd/upstart/multi-user.target.wants/systemd-fsck@.service
$ sudo systemctl daemon-reload

This will allow the mount units to work using systemd in Trusty:

inaddy@trustylivepatch:~$ sudo systemctl start teste.mount
inaddy@trustylivepatch:~$ df -kh | grep teste
/dev/sdb2 976M 1.3M 908M 1% /teste
inaddy@trustylivepatch:~$ sudo systemctl stop teste.mount
inaddy@trustylivepatch:~$ cat /etc/fstab| grep teste
LABEL=TESTE /teste ext4 defaults 0 1

While still having fsck enabled in /etc/fstab.

I'll provide a SRU patch for systemd in Trusty and it shall solve this issue.

I've made a small change creating a unit file slightly different from the one already present in systemd-204, considering different dependencies for this usage (mounting/umounting volumes when upstart has already taken part on it).

The result is here: http://pastebin.ubuntu.com/25663762/

As you can see, after installing the new packages, being proposed in this fix, snapd can work again mounting all new squash filesystems it depends on.

- Attaching the debdiff for SRU.
- Requesting SRU sponsoring.
- Will verify once it gets uploaded to -proposed.

Meanwhile I have created a PPA containing the fix for all those affected users:

https://launchpad.net/~inaddy/+archive/ubuntu/lp1718966

$ sudo add-apt-repository ppa:inaddy/lp1718966
$ sudo apt-get update
$ sudo apt-get dist-upgrade

Is likely enough for you to have a temporary fix.

The attachment "trusty_systemd_204-5ubuntu20.25.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

I tested the fix from ppa:inaddy/lp1718966 and I am now able to install snaps on my two test systems without using the /etc/fstab workaround, one with LVM and one without. Both systems have /var isolated. Thanks!

Sponsored, the package is now in the Trusty upload queue, waiting for the SRU verification team to approve the package to start building in trusty-proposed.

- Eric

After fixing this BUG (for canonical-livepatch snap to work, but being affected by broken systemd in Trusty due to the lack of systemd-fsck@.service unit file, I'm facing the following opened BUG: https://bugs.launchpad.net/snapd/+bug/1721518 for canonical-livepatch (or any other snap) to work).

Hello David, or anyone else affected,

Accepted systemd into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/204-5ubuntu20.25 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

## PENDING SRU / REGRESSION EXPLANATIONS **

All the current systemd (trusty) autopktest failure are not related to the current SRU.
Seems like they are failing for a while now and should be considered as known test failure already.

Regression in autopkgtest for network-manager (ppc64el): test log
 - Always fails the same way since at least "systemd/204-5ubuntu20.17" : http://autopkgtest.ubuntu.com/packages/n/network-manager/trusty/ppc64el

Regression in autopkgtest for linux-lts-wily (armhf): test log
 - Always timeout the same way since at least "systemd/204-5ubuntu20.17" : http://autopkgtest.ubuntu.com/packages/l/linux-lts-wily/trusty/armhf

Regression in autopkgtest for linux-lts-vivid (ppc64el): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-vivid (i386): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-vivid (amd64): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-vivid (armhf): test log
 - Always fails the same way since at least "systemd/204-5ubuntu20.14" : http://autopkgtest.ubuntu.com/packages/l/linux-lts-vivid/trusty/armhf

Regression in autopkgtest for linux-lts-utopic (ppc64el): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-utopic (i386): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-utopic (amd64): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-utopic (armhf): test log
 - Always fails the same way since at least "systemd/204-5ubuntu20.15" : http://autopkgtest.ubuntu.com/packages/l/linux-lts-utopic/trusty/armhf

Regression in autopkgtest for linux-lts-xenial (ppc64el): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-xenial (i386): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-xenial (amd64): test log
 - ERROR: running version does not match source package

Regression in autopkgtest for linux-lts-xenial (armhf): test log
 - Always fails the same way since "systemd/204-5ubuntu20.19" : http://autopkgtest.ubuntu.com/packages/l/linux-lts-xenial/trusty/armhf

** ERROR: running version does not match source package **
Source Package Version: 3.16.0-77.99~14.04.1
Running Kernel Version: 3.13.0-133.182

- Eric

Verifying...

http://pastebin.ubuntu.com/25753116/

After using -proposed version:

http://paste.ubuntu.com/25753146/

Considering this fixed. Verified!

This bug was fixed in the package systemd - 204-5ubuntu20.25

---------------
systemd (204-5ubuntu20.25) trusty; urgency=medium

  * rules: introduce fsck@.service for snappy (LP: #1718966)

 -- Rafael David Tinoco <email address hidden> Mon, 02 Oct 2017 21:39:38 +0000

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.