Merge lp:~jamesodhunt/ubuntu/trusty/sysvinit/log-open-files-on-shutdown into lp:ubuntu/trusty/sysvinit

[James Hunt]

[Steve Langasek (vorlon)]

Hi James,

Apologies again for the delay in reviewing. Some comments:
 - lsof and awk are both in /usr/bin. If /usr is on a separate filesystem, this function is going to fail from /etc/init.d/umountroot. We should really try to make it more robust. Should we be using fuser -vm here (part of psmisc, more likely to be installed), instead of lsof? fuser is located in /bin, and more directly addresses what we're looking to find out.
 - Your /etc/init.d/umountroot change defines the do_log() function but does not invoke it. This looks like a bug to me.
 - If we're going to print anything to the console, then we should also include some sort of pause to give the user a chance to react before the system hard-reboots out from under them. Perhaps either a 'sleep 5' or a 'Press any key to continue' && read in the error case.
 - I think we want to invoke this logging code *only* when there is a failure unmounting the filesystem or mounting the root filesystem read-write. If the root filesystem is successfully mounted ro, and all the other filesystems are successfully unmounted, then there's no reason for us to care what files are open, and spamming this to the console on shutdown (because there will always be a non-zero number of files) is going to be unwelcome noise. And in all the cases where it matters, we know in fact that the root filesystem is *not* remounted ro, so we can reliably log to the root filesystem instead of the console which is better for debugging purposes.

Can you please adjust your patch in light of the above?

review: Needs Fixing

Unmerged revisions

193. By James Hunt on 2013-11-21

* debian/src/initscripts/etc/init.d/umountfs: Log open files on root
  filesystem just prior to unmounting filesystems to aid diagnosis of
  shutdown issues.
* debian/src/initscripts/etc/init.d/umountroot: Check again for open files
  and log to console.

=== modified file 'debian/changelog'
--- debian/changelog	2013-06-04 08:53:47 +0000
+++ debian/changelog	2013-11-21 15:31:05 +0000
@@ -1,3 +1,13 @@
+sysvinit (2.88dsf-41ubuntu4) UNRELEASED; urgency=low
+
+  * debian/src/initscripts/etc/init.d/umountfs: Log open files on root
+    filesystem just prior to unmounting filesystems to aid diagnosis of
+    shutdown issues.
+  * debian/src/initscripts/etc/init.d/umountroot: Check again for open files
+    and log to console.
+
+ -- James Hunt <[e-mail]>  Thu, 21 Nov 2013 15:19:02 +0000
+
 sysvinit (2.88dsf-41ubuntu3) saucy; urgency=low
 
   * Exit the ondemand script silently if we are on a system that has

=== modified file 'debian/src/initscripts/etc/init.d/umountfs'
--- debian/src/initscripts/etc/init.d/umountfs	2013-05-17 06:03:10 +0000
+++ debian/src/initscripts/etc/init.d/umountfs	2013-11-21 15:31:05 +0000
@@ -16,6 +16,25 @@
 
 umask 022
 
+# Before unmounting filesystems, log all files open for writing on the root
+# filesystem for possible analysis on the next boot since such files
+# will cause an unclean shutdown.
+do_log () {
+	if command -v lsof >/dev/null 2>&1; then
+		# capture list of open files on root mountpoint as quickly as possible
+		output=$(lsof -lnPR /)
+
+		# look for first occurence of file open for writing
+		open_files=$(echo "$output"|awk '$5 ~ /^[0-9][0-9]*[uw]/ {print;exit 0}')
+
+		if [ -n "$open_files" ]; then
+			# log full lsof output
+			echo "$output" > /var/log/shutdown-lsof.log 2>/dev/null
+			sync
+		fi
+	fi
+}
+
 do_stop () {
 	PROTECTED_MOUNTS="$(sed -n ':a;/^[^ ]* \/ /!{H;n;ba};{H;s/.*//;x;s/\n//;p}' /proc/mounts)"
 	WEAK_MTPTS="" # be gentle, don't use force
@@ -119,6 +138,7 @@
 	exit 3
 	;;
   stop)
+	do_log
 	do_stop
 	;;
   *)

=== modified file 'debian/src/initscripts/etc/init.d/umountroot'
--- debian/src/initscripts/etc/init.d/umountroot	2013-05-17 06:03:10 +0000
+++ debian/src/initscripts/etc/init.d/umountroot	2013-11-21 15:31:05 +0000
@@ -14,6 +14,26 @@
 
 . /lib/lsb/init-functions
 
+# Before unmounting root, log all files open for writing to the console
+# (since no other locations are available for writing and root may be
+# read-only).
+do_log () {
+	if command -v lsof >/dev/null 2>&1; then
+		# capture list of open files on root mountpoint as quickly as possible
+		output=$(lsof -lnPR /)
+		open_files=$(echo "$output"|awk '$5 ~ /^[0-9][0-9]*[uw]/ {print}')
+
+		if [ -n "$open_files" ]; then
+			echo "INFO: files open at shutdown"
+			# show lsof header
+			echo "$output"|head -n1
+			# just show the open files to avoid completely
+			# flooding the console.
+			echo "$open_files" > /dev/console
+		fi
+	fi
+}
+
 do_stop () {
 	[ "$VERBOSE" = no ] || log_action_begin_msg "Mounting root filesystem read-only"
 	# Ask init to re-exec itself before we go down if it has been