System fails to start (boot) on battery due to read-only root file-system

Possibly related: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1725458

The systemd/dbus issues are possibly related to the partial fix detailed in

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1591411

However, there are 2 upstream DBus bugs/fixes and I believe one of them isn't in 17.10's DBus at present:

"unix fd in-flight counting broken" https://bugs.freedesktop.org/show_bug.cgi?id=95263
" _dbus_timeout_set_enabled doesn't reset timeout ticking" https://bugs.freedesktop.org/show_bug.cgi?id=95619#c10

systemd-resolved.service and systemd.hostnamed.service show up pretty clearly on the startup console as having problems and their logs show repeated entries of the form:

-- Logs begin at Mon 2017-10-23 13:01:36 EDT, end at Mon 2017-10-23 13:05:52 EDT. --
Oct 23 13:01:38 T450s systemd[1]: systemd-resolved.service: Failed to run 'start' task: Read-only file system
Oct 23 13:01:38 T450s systemd[1]: Failed to start Network Name Resolution.
Oct 23 13:01:38 T450s systemd[1]: systemd-resolved.service: Unit entered failed state.
Oct 23 13:01:38 T450s systemd[1]: systemd-resolved.service: Failed with result 'resources'.
Oct 23 13:01:38 T450s systemd[1]: systemd-resolved.service: Service has no hold-off time, scheduling restart.
Oct 23 13:01:38 T450s systemd[1]: Stopped Network Name Resolution.
Oct 23 13:01:38 T450s systemd[1]: systemd-resolved.service: Failed to run 'start' task: Read-only file system

-- Logs begin at Mon 2017-10-23 13:01:36 EDT, end at Mon 2017-10-23 13:06:20 EDT. --
Oct 23 13:01:38 T450s systemd[1]: systemd-hostnamed.service: Failed to run 'start' task: Read-only file system
Oct 23 13:01:38 T450s systemd[1]: Failed to start Hostname Service.
Oct 23 13:01:38 T450s systemd[1]: systemd-hostnamed.service: Unit entered failed state.
Oct 23 13:01:38 T450s systemd[1]: systemd-hostnamed.service: Failed with result 'resources'.

Status changed to 'Confirmed' because the bug affects multiple users.

Confirmed on a hp Spectre x360 13-4104ng.

On battery power, the system DOESN'T boot up.

With the charger connected, the system boots up just fine.

See https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1725458 for my apport logs.

I can confirm bug on ASUS Zenbook UX305CA. I have laptop-mode and pm-utils installed.

Let me know how can I help you to fix it.

Uninstalling laptop-mode-tools helped me, but it's useful package and it would be nice to get it back.

This must be an Ubuntu specific problem. I haven't encountered the problem
upstream.

https://GitHub.com/rickysarraf/laptop-mode-tools

s3nt fr0m a $martph0ne, excuse typ0s

On 07-Dec-2017 02:36, "timor" <email address hidden> wrote:

> Uninstalling laptop-mode-tools helped me, but it's useful package and it
> would be nice to get it back.
>
> --
> You received this bug notification because you are subscribed to laptop-
> mode-tools in Ubuntu.
> https://bugs.launchpad.net/bugs/1726930
>
> Title:
> System fails to start (boot) on battery due to read-only root file-
> system
>
> Status in laptop-mode-tools package in Ubuntu:
> Confirmed
> Status in systemd package in Ubuntu:
> Confirmed
> Status in systemd package in Debian:
> Fix Released
>
> Bug description:
> This report is to capture the extended diagnostic efforts done on IRC
> #ubuntu for user maszlo, who has a Lenovo T450 with SSD that was
> upgraded from 17.04 to 17.10 and since fails to complete start-up of
> services when powered on battery.
>
> We'd previously applied the "acpi=os=! 'acpi_osi=Windows 2015'"
> workaround in case this was an ACPI DSDT issue.
>
> This appears to be due to a read-only root file-system. What is not
> clear is how the rootfs goes read-only.
>
> The file-system (sda6, ext4) is fsck-ed successfully in the initrd
> according to the /run/initramfs/fsck.log.
>
> From the start-up logs (with "systemd.log_level=debug") we see the
> ext4 driver report a remount operation not once but five times.
>
> $ grep 'EXT4-fs.*sda6' systemd-debug.log
> Oct 23 18:10:44 T450s kernel: EXT4-fs (sda6): mounted filesystem with
> ordered data mode. Opts: (null)
> Oct 23 18:10:44 T450s kernel: EXT4-fs (sda6): re-mounted. Opts:
> errors=remount-ro
> Oct 23 18:10:46 T450s kernel: EXT4-fs (sda6): re-mounted. Opts:
> errors=remount-ro,data=ordered,commit=600
> Oct 23 18:10:47 T450s kernel: EXT4-fs (sda6): re-mounted. Opts:
> errors=remount-ro,data=ordered,commit=600
> Oct 23 18:10:48 T450s kernel: EXT4-fs (sda6): re-mounted. Opts:
> errors=remount-ro,data=ordered,commit=600
> Oct 23 18:10:48 T450s kernel: EXT4-fs (sda6): re-mounted. Opts:
> errors=remount-ro,data=ordered,commit=600
> Oct 23 18:10:49 T450s kernel: EXT4-fs (sda6): re-mounted. Opts:
> errors=remount-ro,data=ordered,commit=600
>
> Those last five appear to be carrying options generated by laptop-
> mode-tools.
>
> User has disabled laptop-mode.service and laptop-mode.timer, and lmt-
> poll.service, but the issue remains.
>
> We see several reports of "read-only file-system":
>
> $ grep 'Read-only' systemd-debug.log
> Oct 23 18:10:46 T450s grub-common[1792]: grub-editenv: error: cannot
> open `/boot/grub/grubenv': Read-only file system.
> Oct 23 18:10:46 T450s systemd[1]: colord.service: Failed to run 'start'
> task: Read-only file system
> Oct 23 18:10:46 T450s gpu-manager[1797]: Warning: writing to
> /var/log/gpu-manager.log failed (Read-only file system)
> Oct 23 18:10:46 T450s systemd[1]: systemd-resolved.service: Failed to
> run 'start' task: Read-only file system
> Oct 23 18:10:46 T450s systemd[1]: systemd-resolved.service: Failed to
> run 's...

Like timor I can confirm the bug on the same laptop (ASUS Zenbook UX305CA), too.

This is more likely to be a kernel bug.

Please attach output of `hdparm -I /dev/sdX` here.

Folks...

Does changing to following in /lib/udev/rules.d/99-laptop-mode.rules
help ?

ACTION=="add", SUBSYSTEM=="usb", RUN+="lmt-udev auto"

This would still be a flaky device or a kernel bug. Because the mentioned laptop-mode-tools version (1.71) brought in forced execution mode, so that it could apply proper power saving values. That's pretty much it. That does not result in read only rootfs because if it logically did, it'd be seen on every machine.

BTW, does Ubuntu have the systemd/udevd mount propagation fix included ?

I want to confirm that changing 'force' to 'auto' for the last entry in /lib/udev/rules.d/99-laptop-mode.rules did allow me to boot successfully.

Thanks for confirming @reckenrode. I'd appreciate if more users confirmed the same.

Works for me too with Ubuntu 17.10 on an Acer Extensa laptop (i5-6200U SkyLake). Same symptoms but changing 'force' to 'auto' as above results in successful booting from battery.

Okay. Thanks for the reports. It could very well be the flakiness in usb
devices that are causing this problem. Anyways, the best fix for this is to
contain usb execution to the runtime module only.

Note to self: per module activation was reported to not activating general
power savings. Check those before any conclusions

s3nt fr0m a $martph0ne, excuse typ0s

On 11-Jan-2018 15:56, "Douw Steyn" <email address hidden> wrote:

> Works for me too with Ubuntu 17.10 on an Acer Extensa laptop (i5-6200U
> SkyLake). Same symptoms but changing 'force' to 'auto' as above results
> in successful booting from battery.
>
> --
> You received this bug notification because you are subscribed to laptop-
> mode-tools in Ubuntu.
> https://bugs.launchpad.net/bugs/1726930
>
> Title:
> System fails to start (boot) on battery due to read-only root file-
> system
>
> Status in laptop-mode-tools package in Ubuntu:
> Confirmed
> Status in systemd package in Ubuntu:
> Confirmed
> Status in systemd package in Debian:
> Fix Released
>
> Bug description:
> This report is to capture the extended diagnostic efforts done on IRC
> #ubuntu for user maszlo, who has a Lenovo T450 with SSD that was
> upgraded from 17.04 to 17.10 and since fails to complete start-up of
> services when powered on battery.
>
> We'd previously applied the "acpi=os=! 'acpi_osi=Windows 2015'"
> workaround in case this was an ACPI DSDT issue.
>
> This appears to be due to a read-only root file-system. What is not
> clear is how the rootfs goes read-only.
>
> The file-system (sda6, ext4) is fsck-ed successfully in the initrd
> according to the /run/initramfs/fsck.log.
>
> From the start-up logs (with "systemd.log_level=debug") we see the
> ext4 driver report a remount operation not once but five times.
>
> $ grep 'EXT4-fs.*sda6' systemd-debug.log
> Oct 23 18:10:44 T450s kernel: EXT4-fs (sda6): mounted filesystem with
> ordered data mode. Opts: (null)
> Oct 23 18:10:44 T450s kernel: EXT4-fs (sda6): re-mounted. Opts:
> errors=remount-ro
> Oct 23 18:10:46 T450s kernel: EXT4-fs (sda6): re-mounted. Opts:
> errors=remount-ro,data=ordered,commit=600
> Oct 23 18:10:47 T450s kernel: EXT4-fs (sda6): re-mounted. Opts:
> errors=remount-ro,data=ordered,commit=600
> Oct 23 18:10:48 T450s kernel: EXT4-fs (sda6): re-mounted. Opts:
> errors=remount-ro,data=ordered,commit=600
> Oct 23 18:10:48 T450s kernel: EXT4-fs (sda6): re-mounted. Opts:
> errors=remount-ro,data=ordered,commit=600
> Oct 23 18:10:49 T450s kernel: EXT4-fs (sda6): re-mounted. Opts:
> errors=remount-ro,data=ordered,commit=600
>
> Those last five appear to be carrying options generated by laptop-
> mode-tools.
>
> User has disabled laptop-mode.service and laptop-mode.timer, and lmt-
> poll.service, but the issue remains.
>
> We see several reports of "read-only file-system":
>
> $ grep 'Read-only' systemd-debug.log
> Oct 23 18:10:46 T450s grub-common[1792]: grub-editenv: error: cannot
> open `/boot/grub/grubenv': Read-only file system.
> Oct 23 18:10:46 T450s systemd[1]: colord.service: Failed to run 'start'
> task: Read-only file system
> Oct 23 18:10:46 T450s gpu-manager[1797]: War...

Hi, I'm using Debian Stretch on a Thinkpad X220 and had the same issue. The advice from Ritesh Raj Sarraf worked for me too and I just registered to confirm this.

To actually solve this, we need to disable min power LPM for faulty device in the Linux kernel. So please attach disk information if possible.

here is my hdpram for both drives

I can confirm that the workaround suggested in #15 works for my Asus UX305CA, too. I attached my hdparm output...

Oh cool. Reminds me of this:
https://askubuntu.com/questions/844459/ubuntu-ssd-was-fast-is-now-extremely-slow/879291#879291

I shall have to find the machine with that Crucial SSD...

I had the same problem since 4.9.0-4-amd64

Recently I updated the kernel to version 4.14.0-3-amd64 and now everything works as it should.

ASUS UX305UA laptop

         Icon name: computer-laptop
           Chassis: laptop
  Operating System: Debian GNU/Linux 9 (stretch)
            Kernel: Linux 4.14.0-3-amd64
      Architecture: x86-64

This kernel has workaround for CT500BX100SSD1 in #23 and Micron_M600_MTFDDAV128MBF in #24.
Use "force" in laptop-mode-tools to test it.

people.canonical.com/~khfeng/lp1726930/

I can confirm this fix worked for me.

ryan,

Can you attach the output of `find /sys/devices -name 'link_power_management_policy' | xargs grep .`?
Thanks!

/sys/devices/pci0000:00/0000:00:1f.2/ata1/host0/scsi_host/host0/link_power_management_policy:med_power_with_dipm
/sys/devices/pci0000:00/0000:00:1f.2/ata2/host1/scsi_host/host1/link_power_management_policy:min_power
/sys/devices/pci0000:00/0000:00:1f.2/ata3/host2/scsi_host/host2/link_power_management_policy:min_power

I don't know how i missed this yesterday. I installed the kernel, then change the auto to force and reboot. I was assuming that reboot was the test. It appears that i have the same issue after another reboot, or shut down when AC is not attached

I did run that link_power_management_policy when from tty2 from the failed boot. The results are the same that I posted.

ryan,
So looks like we can only use max_power for your disk.

What about yours, Stefan?

https://lkml.org/lkml/2018/2/15/1184

It sounds like a workaround might be needed for all Crucial SSDs. More models that are problematic in low power mode are mentioned here:
  https://askubuntu.com/questions/844459/ubuntu-ssd-was-fast-is-now-extremely-slow

Not really.
The BX100 has broken LPM, it's not working under min/med power.

The MX300 in askubuntu.com does have a working LPM, just unimaginably slow.

I'll build a kernel with the workaround I used in comment #30, fallback to med_power_with_dipm when min_power gets selected.

(Sorry for accidentally hitting the wrong button earlier.)

Same problem here on a fresh installation of Xubuntu 17.10 on a Tuxedo Book BU1406 (with Tuxedo-modifications tuxedo.sh ).

Storage:
1 × M.2 SSD: 250 GB Samsung 960 EVO (/dev/nvme0n1).
1 × SATA III HDD: 1 TB Seagate Thin / 5400 rpm / 2.5" (/dev/sda).

Disk encryption with LUKS, dm-crypt and LVM are active.

The workaround from #15 seems to do the trick.

That's because what I instructed in #15 makes laptop-mode-tools ineffective to USB hotplug events. Not something that a user would want.

As I see from the reports, this is what I think is happening.

1) Users have some combination of usb devices, getting hotplugged upon discovery.
2) Step 1 results in early invocation of LMT.
3) Which further results in file systems getting ro mounted. This step is still erratic and I have no clue what the calling parent is, to this ro mount.

For users who've confirmed that #15 in the workaround works.
Can you please apply the workaround, successfully boot, then change back the changes to its original, and then hotplug a USB device ?
This will trigger the same chain of events, just that they'll be triggered upon full boot. Which should give us some insight into what may actually be behaving erratic.

Please attach full dmesg if possible.

As requested, booted using battery power with setting "auto" in /lib/udev/rules.d/99-laptop-mode.rules. Then changed setting back to "force" and plugged in an external USB HDD (Transcend StoreJet).

The USB device seemed to be mounted successfully and I could write to it.

Dmesg output attached.

The "auto" fix works for me too. Having just upgraded an old Dell Vostro v131 to 17.10, it no longer booted from battery but required the mains adaptor to be used. The change noted in #15 worked at it now boots correctly.

@rrs
> BTW, does Ubuntu have the systemd/udevd mount propagation fix included ?

yes no maybe =))))

Added in debian at ed16ca8690eb69c248a0a741def5324eedc9149d
Removed from debian at d529d047752d8492d43e42e8b4223d6d37fa3dd3
Meaning it was for a while, if you mean the Debian bug linked to this bug report.
But not since laptop-mode-tools 1.68 was fixed.

I believe the debian bug attached to this bug report is a red herring.

https://git.launchpad.net/~ubuntu-core-dev/+git/systemd/commit/?h=ubuntu-bionic&id=ed16ca8690eb69c248a0a741def5324eedc9149d

https://git.launchpad.net/~ubuntu-core-dev/+git/systemd/commit/?h=d529d047752d8492d43e42e8b4223d6d37fa3dd3

This bug was fixed in the package linux - 4.15.0-15.16

---------------
linux (4.15.0-15.16) bionic; urgency=medium

  * linux: 4.15.0-15.16 -proposed tracker (LP: #1761177)

  * FFe: Enable configuring resume offset via sysfs (LP: #1760106)
    - PM / hibernate: Make passing hibernate offsets more friendly

  * /dev/bcache/by-uuid links not created after reboot (LP: #1729145)
    - SAUCE: (no-up) bcache: decouple emitting a cached_dev CHANGE uevent

  * Ubuntu18.04:POWER9:DD2.2 - Unable to start a KVM guest with default machine
    type(pseries-bionic) complaining "KVM implementation does not support
    Transactional Memory, try cap-htm=off" (kvm) (LP: #1752026)
    - powerpc: Use feature bit for RTC presence rather than timebase presence
    - powerpc: Book E: Remove unused CPU_FTR_L2CSR bit
    - powerpc: Free up CPU feature bits on 64-bit machines
    - powerpc: Add CPU feature bits for TM bug workarounds on POWER9 v2.2
    - powerpc/powernv: Provide a way to force a core into SMT4 mode
    - KVM: PPC: Book3S HV: Work around transactional memory bugs in POWER9
    - KVM: PPC: Book3S HV: Work around XER[SO] bug in fake suspend mode
    - KVM: PPC: Book3S HV: Work around TEXASR bug in fake suspend state

  * Important Kernel fixes to be backported for Power9 (kvm) (LP: #1758910)
    - powerpc/mm: Fixup tlbie vs store ordering issue on POWER9

  * Ubuntu 18.04 - IO Hang on some namespaces when running HTX with 16
    namespaces (Bolt / NVMe) (LP: #1757497)
    - powerpc/64s: Fix lost pending interrupt due to race causing lost update to
      irq_happened

  * fwts-efi-runtime-dkms 18.03.00-0ubuntu1: fwts-efi-runtime-dkms kernel module
    failed to build (LP: #1760876)
    - [Packaging] include the retpoline extractor in the headers

linux (4.15.0-14.15) bionic; urgency=medium

  * linux: 4.15.0-14.15 -proposed tracker (LP: #1760678)

  * [Bionic] mlx4 ETH - mlnx_qos failed when set some TC to vendor
    (LP: #1758662)
    - net/mlx4_en: Change default QoS settings

  * AT_BASE_PLATFORM in AUXV is absent on kernels available on Ubuntu 17.10
    (LP: #1759312)
    - powerpc/64s: Fix NULL AT_BASE_PLATFORM when using DT CPU features

  * Bionic update to 4.15.15 stable release (LP: #1760585)
    - net: dsa: Fix dsa_is_user_port() test inversion
    - openvswitch: meter: fix the incorrect calculation of max delta_t
    - qed: Fix MPA unalign flow in case header is split across two packets.
    - tcp: purge write queue upon aborting the connection
    - qed: Fix non TCP packets should be dropped on iWARP ll2 connection
    - sysfs: symlink: export sysfs_create_link_nowarn()
    - net: phy: relax error checking when creating sysfs link netdev->phydev
    - devlink: Remove redundant free on error path
    - macvlan: filter out unsupported feature flags
    - net: ipv6: keep sk status consistent after datagram connect failure
    - ipv6: old_dport should be a __be16 in __ip6_datagram_connect()
    - ipv6: sr: fix NULL pointer dereference when setting encap source address
    - ipv6: sr: fix scheduling in RCU when creating seg6 lwtunnel state
    - mlxsw: spectrum_buffers: Set a minimum quota for CPU port traffic
    - net: phy: Tell caller result ...

The kernel fix for this arrived in Xenial/16.04 via upstream stable v4.4.125.

Status changed to 'Confirmed' because the bug affects multiple users.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Removing systemd, as I believe there is nothing to be changed in systemd for this fix to land.

Are the affected users fixed with the proposed kernel? Please test proposed kernel asap! Otherwise the change will be dropped from the kernel again on xenial =(

I can confirm I no longer have issues.

This with the /lib/udev/rules.d/99-laptop-mode.rules changed back to
ACTION=="add", SUBSYSTEM=="usb", RUN+="lmt-udev auto"

This is on 18.04
laptop-mode-tools 1.71-2
kernel 4.15.0-20

Correction, it works with the force argument.
ACTION=="add", SUBSYSTEM=="usb", RUN+="lmt-udev force"

Ryan, thanks for the testing.

The same patch should work for all kernel versions.

This bug was fixed in the package linux - 4.4.0-127.153

---------------
linux (4.4.0-127.153) xenial; urgency=medium

  * CVE-2018-3639 (powerpc)
    - powerpc/pseries: Support firmware disable of RFI flush
    - powerpc/powernv: Support firmware disable of RFI flush
    - powerpc/rfi-flush: Move the logic to avoid a redo into the debugfs code
    - powerpc/rfi-flush: Make it possible to call setup_rfi_flush() again
    - powerpc/rfi-flush: Always enable fallback flush on pseries
    - powerpc/rfi-flush: Differentiate enabled and patched flush types
    - powerpc/rfi-flush: Call setup_rfi_flush() after LPM migration
    - powerpc/pseries: Add new H_GET_CPU_CHARACTERISTICS flags
    - powerpc: Add security feature flags for Spectre/Meltdown
    - powerpc/pseries: Set or clear security feature flags
    - powerpc/powernv: Set or clear security feature flags
    - powerpc/64s: Move cpu_show_meltdown()
    - powerpc/64s: Enhance the information in cpu_show_meltdown()
    - powerpc/powernv: Use the security flags in pnv_setup_rfi_flush()
    - powerpc/pseries: Use the security flags in pseries_setup_rfi_flush()
    - powerpc/64s: Wire up cpu_show_spectre_v1()
    - powerpc/64s: Wire up cpu_show_spectre_v2()
    - powerpc/pseries: Fix clearing of security feature flags
    - powerpc: Move default security feature flags
    - powerpc/pseries: Restore default security feature flags on setup
    - SAUCE: powerpc/64s: Add support for a store forwarding barrier at kernel
      entry/exit

  * CVE-2018-3639 (x86)
    - SAUCE: Clean up IBPB and IBRS control functions and macros
    - SAUCE: Fix up IBPB and IBRS kernel parameters documentation
    - SAUCE: Remove #define X86_FEATURE_PTI
    - x86/cpufeature: Move some of the scattered feature bits to x86_capability
    - x86/cpufeature: Cleanup get_cpu_cap()
    - x86/cpu: Probe CPUID leaf 6 even when cpuid_level == 6
    - x86/cpufeatures: Add CPUID_7_EDX CPUID leaf
    - x86/cpufeatures: Add Intel feature bits for Speculation Control
    - SAUCE: x86/kvm: Expose SPEC_CTRL from the leaf
    - x86/cpufeatures: Add AMD feature bits for Speculation Control
    - x86/msr: Add definitions for new speculation control MSRs
    - SAUCE: x86/msr: Rename MSR spec control feature bits
    - x86/pti: Do not enable PTI on CPUs which are not vulnerable to Meltdown
    - x86/cpufeature: Blacklist SPEC_CTRL/PRED_CMD on early Spectre v2 microcodes
    - x86/speculation: Add basic IBPB (Indirect Branch Prediction Barrier) support
    - x86/speculation: Add <asm/msr-index.h> dependency
    - x86/cpufeatures: Clean up Spectre v2 related CPUID flags
    - x86/cpuid: Fix up "virtual" IBRS/IBPB/STIBP feature bits on Intel
    - SAUCE: x86/speculation: Move vendor specific IBRS/IBPB control code
    - SAUCE: x86: Add alternative_msr_write
    - SAUCE: x86/nospec: Simplify alternative_msr_write()
    - SAUCE: x86/bugs: Concentrate bug detection into a separate function
    - SAUCE: x86/bugs: Concentrate bug reporting into a separate function
    - arch: Introduce post-init read-only memory
    - SAUCE: x86/bugs: Read SPEC_CTRL MSR during boot and re-use reserved bits
    - SAUCE: x86/bugs, KVM: Support the combination of guest a...