Comment 2 for bug 1718966

I was also able to reproduce this, here are my notes for now:

## /etc/fstab

LABEL=LVROOT / ext4 errors=remount-ro 0 1
LABEL=LVVAR /var ext4 defaults 0 1
LABEL=TESTE /teste ext4 defaults 0 1

Right after boot:

inaddy@trustylivepatch:~$ systemctl list-units --all | grep fsck
systemd-...el-LVVAR.service error inactive dead systemd-fsck@dev-disk-by\x2dlabel-LVVAR.service
systemd-...el-TESTE.service error inactive dead systemd-fsck@dev-disk-by\x2dlabel-TESTE.service

This indicates that UPSTART is the one mounting the block devices, NOT SYSTEMD using its mount units. SNAPS are mounting the SQUASH filesystems using SYSTEMD UNITS, despite UPSTART scripts.

It is likely that this wasn't noticed, on systems mounting "/" only, because the "-.mount" SYSTEMD UNIT doesn't depend on "systemd-fsck@.service" unit, it depends only on "systemd-fsck-root.service", non existent in TRUSTY's SYSTEMD version. Probably this made SYSTEMD to act like no error existed.

For SYSTEMD mount units to work, it is needed that no fsck unit error exists - like when having /var or any other mounting besides root filesystem - allowing all SYSTEMD units created by snappy to work.

Comparing default setups for TRUSTY and ZESTY:

--------
## TRUSTY

$ dpkg -L systemd | grep fsck
/lib/systemd/systemd-fsck

$ systemctl list-units --all | grep fsck
systemd-...el-LVVAR.service error inactive dead systemd-fsck@dev-disk-by\x2dlabel-LVVAR.service
systemd-...el-TESTE.service error inactive dead systemd-fsck@dev-disk-by\x2dlabel-TESTE.service

$ systemctl list-units --all | grep mount
-.mount loaded active mounted /
teste.mount loaded active mounted /teste
var.mount loaded active mounted /var
umount.target loaded inactive dead Unmount All Filesystems

## ZESTY

$ dpkg -L systemd | grep fsck
/lib/systemd/system/systemd-fsck-root.service
/lib/systemd/system/systemd-fsck@.service
/lib/systemd/system/systemd-fsckd.service
/lib/systemd/system/systemd-fsckd.socket
/lib/systemd/systemd-fsck
/lib/systemd/systemd-fsckd

$ systemctl list-unit-files | grep fsck
systemd-fsck-root.service static
systemd-fsck@.service static
systemd-fsck@dev-disk-by\x2dlabel-TESTE.service static
systemd-fsckd.service static
systemd-fsckd.socket static

$ systemctl list-unit-files | grep mount
-.mount generated
home-inaddy-work.mount generated
mnt.mount static
mountall.service masked
umountfs.service masked
umountroot.service masked
umount.target static
--------

SYSTEMD in TRUSTY was treated differently for FSCK. TRUSTY's version contains systemd-fsck but not systemd-fsckd, the daemon responsible for consolidating all fsck information for SYSTEMD journal. It is also clear that TRUSTY did not include any unit file for systemd-fsck@.service, that might still be considered for the automatically generated mount unit files.

You can reproduce this by trying to use TRUSTY SYSTEMD mount units:

--------
## TRUSTY

$ mount /teste

$ mount | grep teste
/dev/sdb2 on /teste type ext4 (rw)

$ systemctl status teste.mount
teste.mount - /teste
   Loaded: loaded (/etc/fstab)
   Active: active (mounted) since Wed 2017-09-27 17:27:21 BRT; 10s ago
    Where: /teste
     What: /dev/sdb2
  Process: 1754 ExecUnmount=/bin/umount /teste (code=exited, status=0/SUCCESS)

$ systemctl stop teste.mount

$ systemctl status teste.mount
teste.mount - /teste
   Loaded: loaded (/etc/fstab)
   Active: inactive (dead) since Wed 2017-09-27 17:27:33 BRT; 2s ago
    Where: /teste
     What: /dev/disk/by-label/TESTE
  Process: 1778 ExecUnmount=/bin/umount /teste (code=exited, status=0/SUCCESS)

$ mount | grep teste

$ systemctl start teste.mount
Failed to issue method call: Unit systemd-fsck@dev-disk-by\x2dlabel-TESTE.service failed to load: No such file or directory. See system logs and 'systemctl status systemd-fsck@dev-disk-by\x2dlabel-TESTE.service' for details.
--------

Also, in TRUSTY, SYSTEMD creates the mount units based on /etc/fstab entries, just like the recent SYSTEMD does, but on them THERE IS NO setting for fsck dependencies (Requires/After), and, still, it appears to be considering those fsck dependencies when you try to mount a ".mount" unit, based on errors I showed above for "systemctl start XXX.mount".

--------
## TRUSTY

$ cat var.mount

# Automatically generated by systemd-fstab-generator

[Unit]
SourcePath=/etc/fstab
DefaultDependencies=no
After=local-fs-pre.target
Conflicts=umount.target
Before=umount.target
Before=local-fs.target

[Mount]
What=/dev/disk/by-label/LVVAR
Where=/var
Type=ext4
FsckPassNo=1

## ZESTY

$ systemctl edit --full mnt.mount
[Unit]
SourcePath=/etc/fstab
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
Before=local-fs.target
Requires=systemd-fsck@dev-disk-by\x2dlabel-TESTE.service
After=systemd-fsck@dev-disk-by\x2dlabel-TESTE.service

[Mount]
What=/dev/disk/by-label/TESTE
Where=/mnt
Type=ext4
--------

Unfortunately creating the fsck service unit didn't seem to help for TRUSTY, and, according to this:

$ for file in `dpkg -L systemd`; do [ -f $file ] && grep systemd-fsck $file; done
Binary file /lib/systemd/systemd matches
Binary file /bin/systemd matches

It is extremely likely that systemd-fsck@.service was hardcoded as a dependency for systemd mount unit files, and the fsck units systemd-fsck@<DISK>.service are NOT BEING CREATED automatically. This is the real problem to be fixed IMO.

This case should be targeted to both: snapd (trusty) AND systemd (trusty)