fsck not repairing corruption on boot

The "getting stuck in a reboot loop" part is possibly a duplicate of bug 204097.

This looks like an upstart bug. There are certain bugs which e2fsck will not fix automatically, but where it wants a human to look at the filesystem and decide what the right thing is before just going ahead and blindly fixing the problem. (For example, the filesystem corruption might result in an access control list file for sudo or some firewall to go missing, and that might leave the system unprotected.) In those cases, e2fsck will print a message:

 fprintf(stderr, _("\n\n%s: UNEXPECTED INCONSISTENCY; "
  "RUN fsck MANUALLY.\n\t(i.e., without -a or -p options)\n"),
        ctx->device_name);

And then exit with an error code of 4 (filesystem errors left uncorrected) OR'ed into the exit code. See the fsck man page for more details about the error code reporting convention.

This error:

/dev/sda9: Inodes that were not part of a corrupted orphan linked list found. fsck died with exit status 4

is one that will result in e2fsck calling preenhalt(). But it appears that upstart isn't making the message be visible (maybe because it's only paying attention to stdout and not stderr?). And if then reboots, the same thing will happen again, and again....

the fsck man page says
" -y For some filesystem-specific checkers, the -y option will cause
              the fs-specific fsck to always attempt to fix any detected
              filesystem corruption automatically. Sometimes an expert may be
              able to do better driving the fsck manually. Note that not all
              filesystem-specific checkers implement this option. In particu‐
              lar fsck.minix(8) and fsck.cramfs(8) does not support the -y
              option as of this writing."

that makes the benifit of manual checking seem pretty small. maybe the -y option could be add by default.

Could a 'these files may have been damaged' list be given to the user at next log in?

There are two prolems with using -y. First of all, the idea of giving the user "these files may have been dmanaged" at the next login doesn't work if these are access control files for controlling which hosts are allowed to talk to a server, or other forms of security critical files if the machine is running on unattended server configuration. As another example, suppose the filesystem contains a database or some other critical application data which is now corrupt. It may be better to not let the system come back up, since there are many business applications where serving wrong data is far, far worse than serving no data at all. (Think financial applications....)

Secondly, e2fsck -y won't always do the best job if the goal is to recover as much files as possible.

I'm willing to consider adding a paremeter to e2fsck.conf file to enable "reckless mode", which in preen mode blindly tries to fix everything according to hueristics, with no care as to whether a system administrator with human judgement could do a better job. I am concerned about this, because Ubuntu users seem to be more likely to have disk corruption issues more frequently than I've seen from other distro's. Maybe it's because some segment of Ubuntu users are not as careful about the sort of hardware they choose and are using cheaper hardware (as the old joke goes, "whatever falls off the boat from Taiwan, as long as its cheapest"); or maybe because people are encouraged to file Launchpad bugs over hardware issues; or maybe its because of a difference in the maintainance strategy of the distro kernel. So the problem with reckless mode is that they might lose files without even noticing that something bad had happened (i.e., they click away or delete the message of filesystem problems because they don't understand it). Of course these "less-clueful users" are also much less likely to be doing regular backups as well.....

In any case, regardless of whether it is a good idea or not to provide a "reckless mode" for e2fsck, upstart **MUST** display output which is printed by the fsck drivers on standard output and upstart **MUST** respect the fsck driver's wishes if it exits saying that a system administrator should stop and look at the filesystem. At least for a server configuration (and I thought Hardy was going to be tagetted at servers), this is a MUST.

I agree that we should really fix this for Hardy. I'm currently not sure where the problem is (in upstart, or the usplash integration, etc.).

Incidentally, is it possible to safely fake this situation in order to test this? Well, I guess temporarily replacing e2fsck with an "exit 4" shell script should do. :-)

FWIW, I think it would be a bad idea to use something like a "reckless" fsck repair mode by default. Forcefully mounting and booting a broken fs might do more damage than the boot is worth, and it invites users to ignore the error.

I'll do some reproducing and checking where exactly the bug lies.

Sam,

In your original report, you said:

>I did the nosplash thing and got the following
>
>* Checking root file system...
>1254
>fsck 1.40.8 (13-Mar-2008)
>/dev/sda9 contains a file system with error, check forced.
>Checking drive /dev/sda9: 0% (stage 1/5, 1/79)
>Checking drive /dev/sda9: 1% (stage 1/5, 1/79)
>...
>Checking drive /dev/sda9: 11% (stage 1/5, 1/79)
>/dev/sda9: Inodes that were not part of a corrupted orphan linked list found. fsck died with exit status 4

Was this *really* all you saw? There should have been a message saying that you needed to run e2fsck without the -p (preen) option, and it should offered to drop you into single user mode, after demanding a root password.

Did you see any evidence of this, in spash or nospash mode?

And when you say "reboot loop", did you have to hit return or otherwise decline to enter single-user mode?

It should have **not** been necessary to boot a rescue floppy to recover from this. You don't have to on Ubuntu Gutsy in non-splash mode, nor on any other major distribution as far as I know. I leapt to the assumption that this was an upstart problem, since upstart was new to Hardy, but maybe it's something else.....

i snipped out the some lines from the middle with incrementing percentages (and there were normal verbose booting lines above). i did not not see any message telling me to run fsck.

the reboot loops were with no interaction by me. it would just go through BIOS, then grub, then show the splash, then show fsck process, then it would go back to BIOS. i let it do this 3 or 4 times.

is there a way to deliberately cause this sort of disk corruption? i had a google for way to do it, but is seems that most people want to fix corruption ;-)

sam tygier [2008-04-02 1:14 -0000]:
> is there a way to deliberately cause this sort of disk corruption? i had
> a google for way to do it, but is seems that most people want to fix
> corruption ;-)

In my experience, this triggers a fatal error, which is correctable
well:

  sudo tune2fs -s 0 /dev/...; sudo tune2fs -s 1 /dev/...

I. e. flip the 'sparse superblock' bit back and forth.

Actually, that won't replicate the problem you're trying to replicate. In order to do that, you need to induce an error which causes e2fsck to exit with a preenhalt statement when you run it with the -p option.

So for example:

# debugfs -w -R "write /etc/motd test-file" /tmp/foo.img
debugfs 1.40.8 (13-Mar-2008)
Allocated inode: 12
# debugfs -w -R "unlink test-file" /tmp/foo.img
debugfs 1.40.8 (13-Mar-2008)
# debugfs -w -R "set_super_value state 2" /tmp/foo.img
debugfs 1.40.8 (13-Mar-2008)
# e2fsck -p /tmp/foo.img
/tmp/foo.img contains a file system with errors, check forced.
/tmp/foo.img: Unattached inode 12

/tmp/foo.img: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
# echo $?
4

i tried theodore's debugfs commands, at boot the fsck happens during the splash screen and then when it fails i am taken to a prompt (though it is not very clear as there is no # or $) see attachment.

is there any reason that "Inodes that were not part of a corrupted orphan linked list found" errors might be treated differently?

Ted: "since upstart was new to Hardy" - for the record, upstart was introduced in Edgy, i.e. Ubuntu 6.10.

With Ted's example commands I can replicate the reboot loop in splash mode. I'll have a look at this now.

Setting to sysvinit since I suspect the bug is in checkroot.sh (for not displaying the text message properly) and the usplash fsck integration (for not killing usplash on failure).

When I corrupt the disk and boot without 'splash' (either immediately or after a few reboot loops), I get attached screenshot. This looks fine to me, I get the "RUN fsck MANUALLY" notice and a root shell.

However, Sam's screenshot is different: given the check percentage lines on his system, the usplash process is still running (since the usplash integration scripts kick in), but not displayed any more. This suggests that you booted with splash, but usplash crashed somewhere in between?

I fixed the fsck integration to quit usplash on fsck errors > 1. Now I end up with the same screen as Sam. The "RUN fsck MANUALLY" message is not printed. I do get a sulogin shell, but it does not show any prompt. I suspect that the terminal attributes get wrecked. Looking into that now.

I got it now, I think. I uploaded a new sysfsutils which is now sitting in the UNAPPROVED queue, and will be accepted after the Hardy RC release.

FYI, I attach the debdiff.

I tested this with the following cases:

 - clean fs, routine check, cancel
 - clean fs, routine check, no cancel
 - two clean fses with routine check, and mixed cancelling
 - one ext3 and one reiserfsck
 - corrupt ext3 root fs (I get to the console and get the fsck message, as well as usable sulogin)
 - corrupt ext3 non-root fs (dito about sulogin)

This bug was fixed in the package sysvinit - 2.86.ds1-14.1ubuntu45

---------------
sysvinit (2.86.ds1-14.1ubuntu45) hardy; urgency=low

  * Fix handling of fatal fsck errors in the usplash integration. (LP: #209416)
    - usplash-fsck-functions.sh: When fsck exits with an error > 1, this
      signals a non-correctable failure, which will trigger sulogin. Quit
      usplash in this case and restore stdin/out/err, so that the following
      sulogin is actually usable.
    - check{root,fs}.sh: Redirect fsck's stdout/err to /dev/console in usplash
      mode, so that the user will see the "RUN fsck MANUALLY" warning.

 -- Martin Pitt <email address hidden> Wed, 16 Apr 2008 17:34:20 +0200