Ubuntu
nfs-utils package

statd startup races with / becoming writable

Bug #690401 reported by Darin Tay
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
nfs-utils (Ubuntu)
Confirmed
Undecided
Unassigned
Lucid
Confirmed
High
Canonical Server
Maverick
Confirmed
Undecided
Unassigned
Natty
Confirmed
Undecided
Unassigned

Bug Description

This bug is in Lucid.

The init script for statd runs sm-notify in its pre-start, which writes to /var/lib/nfs/state.new. Assuming a default partitioning layout, that's part of the root filesystem. Nothing is guaranteeing that the root filesystem is writable by this point, so there is a race. In particular, if the filesystem needs to be fsck'd, sm-notify will often run before the filesystem is writable and thus fail.

Because of https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/579858, many of our systems end up fsck'ing on every boot, causing this problem to recur frequently.

Since statd isn't starting up, these machines can't mount their nfs mounts.

Repro steps:
1) Get a Lucid machine, default partitioning, setup nfs
2) Make sm-notify actually do work on startup (e.g. "sudo touch /var/lib/nfs/sm.bak/not.a.real.server"; you could also get a real lock, but this is easier, and by using a non-real server, it ensures sm-notify will try again every single boot).
3) Add a (e.g. 20 second) sleep to the start of fsck. Or just replace it with a binary that sleeps for 20 seconds. Or do something to ensure that it will take a long time (messy up your filesystem)
4) Shutdown without unmounting / cleanly (or otherwise force a fsck on startup)
5) Observe that statd fails on next startup. If you have it strace'd, you'll see:
open("/var/lib/nfs/state.new", O_WRONLY|O_CREAT, 0644) = -1 EROFS (Read-only file system)

I've filed this as a separate bug with specific repro instructions, but note that there are two other similar statd bugs:
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/525154 is a different race, and only applies to having /var as a separate partition.
https://bugs.launchpad.net/ubuntu/+source/nfs-utils/+bug/581941 may be the same issue, but the comments seem to be discussing either the /var mount race and/or a different problem where statd can't be started up at all, even after boot.

Tags: glucid

Related branches

lp:~clint-fewbar/ubuntu/natty/nfs-utils/wait-for-local-filesystems
James Hunt (community): Needs Fixing
Steve Langasek: Pending requested
Ubuntu branches: Pending requested
Diff: 116 lines (+58/-6)
5 files modified
debian/changelog (+11/-0)
debian/control (+1/-1)
debian/nfs-common.statd-mounting.upstart (+28/-0)
debian/nfs-common.statd.upstart (+17/-5)
debian/rules (+1/-0)
Revision history for this message
Darin Tay (dtay) wrote :

Also note that if there are locks remaining in /var/lib/nfs/sm/... and this race occurs, sm-notify will fail to notify the nfs server about locks, but exit 0 anyway. backup_hosts() will attempt to move the files into sm.bak and, that having failed (non-fatally) due to a readonly filesystem, get_hosts() will decide there are no hosts to notify and exit 0.

Robbie Williamson (robbiew)
Changed in nfs-utils (Ubuntu):
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Canonical Server Team (canonical-server)
Changed in nfs-utils (Ubuntu Lucid):
status: New → Confirmed
Changed in nfs-utils (Ubuntu Natty):
status: Confirmed → New
Changed in nfs-utils (Ubuntu Lucid):
importance: Undecided → High
Changed in nfs-utils (Ubuntu Natty):
importance: High → Undecided
Changed in nfs-utils (Ubuntu Lucid):
assignee: nobody → Canonical Server Team (canonical-server)
Changed in nfs-utils (Ubuntu Natty):
assignee: Canonical Server Team (canonical-server) → nobody
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

So, the current statd.conf makes sure that its always running when portmap is running with:

start on (started portmap or mounting TYPE=nfs)
stop on stopping portmap

It also makes sure, in case it was started on mounting TYPE=nfs before portmap, that portmap is running, in the pre-start.

portmap is

start on virtual-filesystems and net-device-up IFACE=lo

net-device-up IFACE=lo is special, and is emitted by network-interface, presumably before networking has even been started.

so portmap *may* be running too early.

What we really want to express is

"start statd after portmap, before mounting any nfs filesystems, after gaining write access to local filesystems". We *ALSO* need to express "stop statd before you stop portmap, and after you've started it during normal operation".

Anyway, I tried to have portmap export the current RUNLEVEL and do this:

start on (started portmap RUNLEVEL=[2345]) or (local-filesystems and mounting TYPE=nfs)

But I couldn't get exporting working at all, so I've done that check in the pre-star using UPSTART_EVENTSt, which actually seems to work, but I'm hesitant to present this as the final solution, as it feels rather hacky. Maybe somebody can figure out how to get portmap to export the current RUNLEVEL into its started event.

Also since we're not guaranteed to start with portmap or mounting, but nfs services also may need rpc.statd, I added start/stop on runlevel [2345].

Changed in nfs-utils (Ubuntu Natty):
status: New → Confirmed
Changed in nfs-utils (Ubuntu Maverick):
status: New → Confirmed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.