Ubuntu 19.10 and focal live cloned create and mount casper-rw partition

Adding some text files

Adding screenshot from 19.10

Adding screenshot from Focal Fossa

(I had to stop and start the graphical desktop environment from a text screen to make it responsive, but that is another problem, I think not related to the bug reported here.)

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1851123

The same problem appears also in other configurations, not only with cloned live systems but with all Ubuntu live systems made with 19.10 and Focal Fossa. When running live-only (without the boot option 'persistent') the casper-rw partition will be mounted and busy.

I tested and verified the problem with

- a system extracted into a FAT32 file system, (that is samll in order to leave drive space behind it for a casper-rw partition. When running live-only (without the boot option 'persistent') the casper-rw partition will be mounted and busy.

- a system created with mkusb (where the casper-rw partition is already created). Also in this case, when selecting the live-only menuentry, the casper-rw partition will be mounted and busy.

Status changed to 'Confirmed' because the bug affects multiple users.

The point of the work I did on this was to automatically save the installer logs to the media. I don't want to change the default behaviour. However, we should make sure other use cases are possible. You can put "nopersistent" on the command line to disable both creation and mounting of casper-rw. We should also make toram (and I guess todisk?) disable that too I guess.

Basically if you're doing anything unusual, I'm fine with not mounting casper-rw. But I want the default to stay the same.

I see and understand now that mounting casper-rw in live-only mode is by intention and not a bug.

I think the important thing is that it is possible to unmount all partitions on the drive when running live-only with 'toram'.

It is also valuable to learn about the boot option 'nopersistent.'

So I am looking forward to you making 'toram' disable mounting the casper-rw partition for logging, but running with the boot option 'persistent' should still make the overlay for persistence work (I mean persistence together with 'toram', releasing '/cdrom' from the 'drive device', which makes the system more responsive, if possible).

Please notice that users can have reasons to switch between 'live-only toram' and 'persistent live'. One reason is maintenance of the casper-rw partition, another reason is to switch between a mode where things can be saved (also malware) and a clean read-only mode.

I think all users of persistent live Ubuntu systems will be happy to learn about all boot options, that are relevant for the persistent live feature and how to use them. So please advice where to find such information (or write about it for us if there is no web site, where it is easy to find). I am willing to help with these efforts. (I had never heard about nopersistent and todisk before you mentioned it.)

Exploring the boot option 'nopersistent':

The bug is there, but there is a workaround: add the boot option 'nopersistent'. This means that with 'nopersistent' and 'toram' all partitions on the boot drive can be unmounted and the drive can be unplugged or the casper-rw partition can be repaired or backed up without problems.

I tried booting a Startup Disk Creator 19.10 flashdrive with F6.
I typed a space and then "persistent".
The drive booted persistent and stayed persistent every time I used this
procedure, otherwise it just booted Live.
I then tried F6 with the word "toram", it booted to RAM.
Then with "nomodeset" and combinations of these and they all worked.
Finally I tried F6 and "nopersistent".
I was able to unmount the casper-rw partition using Disks.
I tried formatting it FAT and NTFS but that did not work.
Finally I was able to wipe the partition using Disks and it just went back
to ISO9660 space.
I lost everything that was in casper-rw, but it was all in the name of
research.

On Fri, Nov 8, 2019 at 9:20 PM sudodus <email address hidden> wrote:

> Exploring the boot option 'nopersistent':
>
> The bug is there, but there is a workaround: add the boot option
> 'nopersistent'. This means that with 'nopersistent' and 'toram' all
> partitions on the boot drive can be unmounted and the drive can be
> unplugged or the casper-rw partition can be repaired or backed up
> without problems.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1851123
>
> Title:
> Ubuntu 19.10 and focal live cloned create and mount casper-rw
> partition
>
> Status in casper package in Ubuntu:
> Confirmed
>
> Bug description:
> It is very good that it is easy to create and use a casper-rw
> partition to run Ubuntu persistent live in the versions 19.10 and
> Focal Fossa, and that it is even created automatically, when there is
> unallocated drive space behind the used part in a live drive.
>
> But I think the system for doing that is 'too eager'. Maybe it is a
> feature that is left from the development and debugging phase.
>
> - It would be enough (best), if the casper-rw partition is created
> only when it is needed, that is when the live system is booted with
> the boot option 'persistent'.
>
> - We (I am talking for users who like persistent live drives) can
> accept that the casper-rw partition is created even when the drive is
> booted live only (without the boot option 'persistent').
>
> - But we think it is a bug, that the live-only system is also mounting
> the casper-rw partition on the mount point /var/log and/or /var/crash
> and keeps it busy so that it cannot be unmounted.
>
> This way the drive is not really live-only, and it is not possible to
> manage the drive space behind the system in an independent way. For
> example, it is not possible to detach the drive by using the boot
> option 'toram', and it is not possible to repair the casper-rw
> partition when running live in the same drive. It will be necessary to
> have two linux systems to do such tasks.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.04
> Package: casper 1.428
> ProcVersionSignature: Ubuntu 5.3.0-18.19-generic 5.3.1
> Uname: Linux 5.3.0-18-generic x86_64
> NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair
> ApportVersion: 2.20.11-0ubuntu9
> Architecture: amd64
> CasperVersion: 1.428
> CurrentDesktop: ubuntu:GNOME
> Date: Sun Nov 3 11:15:29 20...

Been chatting with Sudodus.
Back in the days of 4GB and 8GB flash drives it was not so bad when SDC, mkusb and Rufus used the whole drive to make a Live USB.
Nowadays with 64GB and larger flash drives it seems like a waste.
Making the remaining space Persistent is a great start but even on my desktop, my /home partition is only about 30GB.
Any chance using some of this excess persistence space as NTFS so the drive could be used for data both by Windows and Linux?
Sorta like mkusb does with Persistent drives.

I installed Xubuntu when running a persistent live Ubuntu system

1. With the options 'toram' 'nopersistent'

It was truly live-only (and the installation worked).

2. With the default settings (no extra boot option compared what comes with the iso file)

The casper-rw partition received and stored an install log (and the installation worked).

---
tester@tester-SATELLITE-PRO-C850-19W:~$ sudo lsblk -f
NAME FSTYPE LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1 vfat 9B34-A9E8
├─sda2 ext4 aee3c5e5-e695-49e9-9598-ac8671f30181
├─sda3 ext4 cf57ee56-dba4-4494-9475-07af55933ac3 16,2G 25% /
└─sda4 ext4 e1ce9f48-8214-4cae-ba35-c4a62f7f6d78
sdb
├─sdb1 ntfs usbdata 0B539A8875BBB7D6
├─sdb2
├─sdb3 vfat usbboot 112D-C9F8
├─sdb4 iso9660 Xubuntu 20.04 LTS amd64 2019-11-10-02-05-43-00
└─sdb5 ext4 casper-rw 28567df4-354c-4061-ae21-431c30d01939
sr0
tester@tester-SATELLITE-PRO-C850-19W:~$ sudo mount /dev/sdb5 /mnt
tester@tester-SATELLITE-PRO-C850-19W:~$ sudo du -hs /mnt/*
1,4M /mnt/install-logs-2019-11-10.0
16K /mnt/lost+found
43M /mnt/upper
8,0K /mnt/work
tester@tester-SATELLITE-PRO-C850-19W:~$
---

I realize that this is a useful feature and not a bug.

But as stated earlier, this is new territory, and I think some of us, who use live and persistent live drives may have problems to use drives with this new feature. So it would still be valuable to get a modification such that 'toram' alone will turn off saving log files to the casper-rw partition and make it possible to unmount it.

This bug was fixed in the package casper - 1.432

---------------
casper (1.432) focal; urgency=medium

  * scripts/casper-helpers: pass --no-reread to sfdisk when creating the
    casper-rw partition to reduce the risk of self-races (which probably only
    happen in qemu-system-emulation i.e. in the autopkgtests).

 -- Michael Hudson-Doyle <email address hidden> Wed, 13 Nov 2019 15:07:08 +1300

casper in focal now turns off the automatic log persistence if toram or todisk is used.

Imho, the default behaviour should be overall read-only, as it's more in line with the general perception of a live system as an "I'm just looking" thing. Straying away from this concept could make more than a few people uncomfortable.

Moreover, having a default write behaviour can have adverse consequences in the long term. May a bug arise, there is an increased probability that it would behave destructively, especially considering that we're facing scripted automatic partitioning, which isn't quite the least potentially destructive operation.

On one hand, I understand there are use cases when people want to have installer logs handy on their stick, and having to pass "persistent" each and every time on the command line can be a problem. But On the other hand, there are people who don't want their USB stick written to for several reasons, one being to perform an integrity check before booting (such an unattended write operation would make subsequent integrity checks fail). The only option for these users is to remember adding "nopersistent" each and every time, and with no typing mistake of course, and not forgetting to interrupt the boot process, otherwise there's a write, and it's too late.

It seems to me there should be a solution to make both worlds happy, such as requiring people who want log persistence to pass "persistent" or why not "persistent-log" once on the command line. The system would then create the partition the way it's currently done by default, and during further boots with the default parameters, the system would check if a "casper-rw" volume exists on the live support, and if yes, mount it rw.

Additionnally, to address @cscameron's remark, it could be indeed useful to be able to limit the space automatically allotted to casper-rw on the live medium, such as defining an additional parameter to specify its size, either absolute or relative (as a percentage), but another solution that's already working is to create and format the volume manually (don't forget to label "casper-rw") and specify its size, which is a one time operation, hence should not be too much of a burden.

I tested the bugfix with the Xubuntu Focal daily iso file in BIOS mode using the boot option 'toram', and it works :-)

See the attached screenshot: A casper-rw was not mounted; it was not even created.

After that I rebooted without any extra boot option. The casper-rw partition was created and mounted to /var/crash and /var/log and /var/log was busy, could not be unmounted. 'As usual' with this new feature.

Then I rebooted using the boot option 'toram', and it works, also when there is a casper-rw partition. It is not mounted :-)

I tested also with the boot option 'persistent' works as expected (with and without 'toram') and did some basic checking in UEFI mode too. Everything seems to work as expected.

Thank you Michael :-)

-o-

Olivier Robert raised the issue concerning what should be the default. Maybe it can be discussed here, maybe there should be another forum for that discussion, for example a new bug report or the Ubuntu Forums.

I can raise my concern through another bug for sure, if necessary.

@Olivier,

I think that the developers will stop looking at a bug report after the bug is fixed. So they might not read what we write here. (A discussion here is OK for me, but it may not involve the people who can change the code.)

Until you have collected enough opinion for your issue, there are workarounds to create a truly live-only USB pendrive:

- You can edit your binary iso file and replace the cosmetic boot options 'quiet splash' with the boot option 'nopersistent' (both strings contain 12 characters, so the replacement can be done without any offset in the following part(s) of the iso file. You can do it with 'sed'

sed 's/quiet splash/nopersistent/' standard.iso > nopersistent.iso

and clone 'nopersistent.iso' or flash directly with

sed 's/quiet splash/nopersistent/' standard.iso > /dev/sdx

where x is the device letter for the USB pendrive.

- You do it with a safety belt using 'mkusb-minp' according to the following link,

https://help.ubuntu.com/community/mkusb/minp

I am still reading this (although busy, hence the delay). I quite strongly don't want to change the default but maybe we can make it easier to disable than applying sed to the entire ISO (!). Probably the forums or discourse.ubuntu.com would be a better place to discuss this.

@Michael Hudson-Doyle,

I'm glad that you are reading this :-)

An alternative, that I can see beyond editing the whole iso file:

- Add a menuentry in the iso file. This menuentry should have the boot option 'persistent' alongside the other boot options. So there would be two standard boot options, live and persistent live.

I will try to start a discussion in the Ubuntu Forum for the developing verson-

@nio-wiklund, @mwhudson : Thanks for your replies.

@nio-wiklund : Your solution would probably do the trick, but has the inconvenient of indiscriminately replacing a string with another in the whole image, hence possibly in places where it's not desired.

I think I will rather do one of those two things :

1. Create a partition that occupies the remaining space on the USB stick, hence preventing the live image from creating (and writing to) one without my consent

2. Create a custom Grub config file on the EFI partition on the stick, specifying my own parameters

The first solution is the most straightforward, has the advantage the stick can be used for additional purposes, but the inconvenient that it's hardware-dependent (not all sticks have the same size). To have the integrity check work, I would have to rehash after modifying, and maintain different hashes, one hash per stick.

The second solution requires more changes, but would work for all sticks. It probably won't work if I boot the stick on a non-EFI BIOS though. For integrity-checking purposes I would have to rehash too, but would only require one hash for all sticks.

Concerning your change proposal, adding an additional entry to Grub solves indeed the problem of risking a typo compared to having to type "nopersistent" and/or "toram" at every boot, to avoid undesired writes. However, it still requires to never forget interrupting the boot process.

@mwhudson : Imho, the current situation of the 19.10 live image can be considered a bug, since it changes a default that has been there for more than a decade, and that's in many minds, that a live OS image is volatile.

That's not due to a coding error, like most bugs are, but rather in comparison to usually expected behaviour, or "untold specification".

Moreover, your reasons for wanting it to become a default are unclear, especially considering that the ones who want persistence could achieve that by passing a one-time option at boot, and being OK for subsequent boots, which obviously people who don't want persistence can't do, precisely because they don't want persistence.

For these reasons it can be worth clarifying, outside of a bug process why not. I just started a thread : https://ubuntuforums.org/showthread.php?t=2431843

@ Michael Hudson-Doyle, @ Olivier Robert, and @ everybody else who is interested,

I suggest a simple method to prepare the iso files for binary editing without 'sacrificing' anythng.

Since repeated spaces are collapsed into one space by Launchpad's comment editor, I will provide the man content of this comment as an attached file.

You may want to participate in the following thread,

https://discourse.ubuntu.com/t/make-it-easy-to-create-a-persistent-as-well-as-live-only-usb-live-drive/13671

i just wanted do add than IMHO automatically mounting "anything" in live mode is the worst desgin decision ever....

and btw. it produces bugs beyond your imagination...

for example:

on an customized live image there is an instance of nginx running... in order to start nginx it checks if /var/log/nginx/error.log (and access.log) are available...

now casper mounts "casper-rw" or "writable" into /var/log and those files are gone.. therfore nginx start fails.. this is true for a lot of other situations...

i'm glad to here that i can "opt-out" but the linux way to such "features" would be "opt-in"

so please make this optional !!
thx