Merge lp:~xnox/upstart/shrink-notifications into lp:upstart

[Dimitri John Ledkov]

[James Hunt (jamesodhunt)]

As discussed, we should not be disabling tests so need to get to the bottom of the failures.

review: Needs Fixing

[lp:~xnox/upstart/shrink-notifications]

1641. By Dimitri John Ledkov on 2014-07-02

Wait for the right pid to exit, reset main-loop state between tests

[Dimitri John Ledkov (xnox)]

Thanks for pointing out that the test-harness is broken and not the test. And hence the test fails because of bad environment.

* added helpers to install handlers, which can be reset with the TEST_RESET_MAIN_LOOP()
* realized that i need to pass the desired pid to the exit handler, instead of simply quiting on the first one
* modified affected tests to use main loop and properly clean-up environment before and after it self

All tests pass now \o/

[James Hunt (jamesodhunt)]

Thanks! However, test_job_process.c is still failing for me:

#2 0x0000000000433a2d in test_start () at tests/test_job_process.c:4160
4160 TEST_FILE_END (output);

Also, re-exec is failing occasionally for me with EBADF thrown by nih_io_reopen().

Also, could you remove that stale comment in job_process_close_handler().

review: Needs Fixing

[lp:~xnox/upstart/shrink-notifications]

1642. By Dimitri John Ledkov on 2014-07-03

Catch up trunk

1643. By Dimitri John Ledkov on 2014-07-03

Reduce races in 'with multiple processes and log' test by waiting for sufficiently large log file to be written.

1644. By Dimitri John Ledkov on 2014-07-03

remove stale comment

[Dimitri John Ledkov (xnox)]

> Thanks! However, test_job_process.c is still failing for me:
>
> #2 0x0000000000433a2d in test_start () at tests/test_job_process.c:4160
> 4160 TEST_FILE_END (output);
>

That test is racy since async branch introduction and has a comment to denote as such. I've not committed a couple of guards to reduce spurious failures of this test, by waiting for sufficiently large enough log file to be written.

> Also, re-exec is failing occasionally for me with EBADF thrown by
> nih_io_reopen().
>

=/

> Also, could you remove that stale comment in job_process_close_handler().

Done.

[James Hunt (jamesodhunt)]

test_job_process.c is frequently hanging for me and there are a few instances spinning at 100% CPU in the background.

review: Needs Fixing

[lp:~xnox/upstart/shrink-notifications]

1645. By Dimitri John Ledkov on 2014-07-08

Merge trunk

1646. By Dimitri John Ledkov on 2014-07-08

100 interations pass

[Dimitri John Ledkov (xnox)]

Backed out jodh's merged tests.
Made tests resilient to tests (pass 100+ iterations).

Fixed races:
- sigchild ariving, before read handler is called

Fixed aborts:
- due to multiple io reads, since process_fd pipe was not cleared
- job_process_error_handler returning EILSEQ
- double freeing io, when read handler is called before close handles

[James Hunt (jamesodhunt)]

Still racy I'm afraid:

not ok 48 - with single-line command running an invalid command, then a 1-line post-stop script
        wrong value for child_exit_status[1], expected 255 got 0
        at tests/test_job_process.c:3193 (test_start).
Aborted (core dumped)
ERROR: failed on iteration 6

#2 0x0000000000428c92 in test_start () at tests/test_job_process.c:3193
3193 TEST_EQ (child_exit_status[1], 255);
(gdb) p child_exit_status[1]
$1 = 0
(gdb)

review: Needs Fixing

[Dimitri John Ledkov (xnox)]

> Still racy I'm afraid:
>
> not ok 48 - with single-line command running an invalid command, then a 1-line
> post-stop script
> wrong value for child_exit_status[1], expected 255 got 0
> at tests/test_job_process.c:3193 (test_start).
> Aborted (core dumped)
> ERROR: failed on iteration 6
>
> #2 0x0000000000428c92 in test_start () at tests/test_job_process.c:3193
> 3193 TEST_EQ (child_exit_status[1], 255);
> (gdb) p child_exit_status[1]
> $1 = 0
> (gdb)

It is better than it was before.

What about review of the actual code changes? Do you approve those?

[James Hunt (jamesodhunt)]

I don't understand the change you've made to job_process_handler():

=== modified file 'init/job_process.c'
--- init/job_process.c 2014-07-08 14:11:53 +0000
+++ init/job_process.c 2014-07-10 07:54:25 +0000
@@ -1474,6 +1474,11 @@
        job = job_process_find (pid, &process);
        if (! job)
                return;
+
+ /* Child setup process died, before finishing the setup.
+ */
+ if (job->process_data[process] && job->process_data[process]->valid)
+ return;

This seems to be what is causing the branch to fail to boot too.

[lp:~xnox/upstart/shrink-notifications]

1647. By Dimitri John Ledkov on 2014-07-10

Backout "fix" that causes boot regression.

This is meant to skip child process sigchild processing if
asynchronous setup notification pipe has not been closed/read-from
yet.

1648. By Dimitri John Ledkov on 2014-07-10

Merge trunk

1649. By Dimitri John Ledkov on 2014-07-10

Assert job_change_state is not blocked

1650. By Dimitri John Ledkov on 2014-07-10

Back out more of -r1647

1651. By Dimitri John Ledkov on 2014-07-10

Revert asserting no blockers, as we can get blockers.

1652. By Dimitri John Ledkov on 2014-07-10

Don't transition and return if blocked.... and don't assert before that check.

1653. By Dimitri John Ledkov on 2014-07-10

Merge

1654. By Dimitri John Ledkov on 2014-07-11

Derace real & mock signal handlers

1655. By Dimitri John Ledkov on 2014-07-11

Merge jodh tests

1656. By Dimitri John Ledkov on 2014-07-11

Make test_initctl pass without inotify support.

=== modified file 'ChangeLog'
--- ChangeLog	2014-07-08 14:11:53 +0000
+++ ChangeLog	2014-07-11 12:53:11 +0000
@@ -1,3 +1,18 @@
[phone]  James Hunt  <[e-mail]>
+
+	* init/job_process.c: Unhide job_process_error_abort() to allow it to be
+	  called by the tests.
+	* init/tests/test_job_process.c: RESILIENT_TEST_CHILD(): New macro
+	  to work around a bug in NIH's TEST_CHILD() where it doesn't handle
+	  EINTR.
+	* init/tests/test_job_process.c: test_handler(): New tests to ensure
+	  async handling can cope with both 'waitid() before select()' and
+	  vice versa:
+	  - "with child exit notification before child setup success notification"
+	  - "with child exit notification before child setup failure notification"
+	  - "with child setup success notification before child exit notification"
+	  - "with child setup failure notification before child exit notification"
+
 2014-07-08  James Hunt  <[e-mail]>
 
 	* init/main.c: main():

=== modified file 'init/job.c'
--- init/job.c	2014-06-12 14:23:47 +0000
+++ init/job.c	2014-07-11 12:53:11 +0000
@@ -396,11 +396,6 @@
 {
 	nih_assert (job != NULL);
 
-	/* We may not be blocked by any events when doing first
-	 * transition.
-	 */
-	nih_assert (job->blocker == NULL);
-
 	while (job->state != state) {
 		JobState old_state;
 		int      unused;

=== modified file 'init/job_process.c'
--- init/job_process.c	2014-07-08 14:11:53 +0000
+++ init/job_process.c	2014-07-11 12:53:11 +0000
@@ -111,9 +111,6 @@
 int disable_respawn = FALSE;
 
 /* Prototypes for static functions */
-static void job_process_error_abort     (int fd, JobProcessErrorType type,
-					 int arg)
-	__attribute__ ((noreturn));
 static void job_process_remap_fd        (int *fd, int reserved_fd, int error_fd);
 
 /**
@@ -983,7 +980,7 @@
  *
  * This function calls the exit() system call, so never returns.
  **/
-static void
+void
 job_process_error_abort (int                 fd,
 			 JobProcessErrorType type,
 			 int                 arg)
@@ -2457,7 +2454,15 @@
 
 	err = nih_error_get ();
 
-	nih_assert (err->number == JOB_PROCESS_ERROR);
+	if (err->number != JOB_PROCESS_ERROR) {
+		nih_warn ("%s: %s", _("Temporary process spawn error"),
+			  err->message);
+		nih_free (err);
+		return;
+	}
+
+	/* Wilco. Out. */
+	nih_io_buffer_shrink (io->recv_buf, len);
 
 	/* Non-temporary error condition */
 	nih_warn (_("Failed to spawn %s %s process: %s"),
@@ -2491,12 +2496,12 @@
 	/* Note that pts_master is closed automatically in the parent
 	 * when the log object is destroyed.
 	 */
+	process_data->valid = FALSE;
 
 	nih_io_shutdown (io);
 
 	/* Invalidate */
 	process_data->job_process_fd = -1;
-	process_data->valid = FALSE;
 }
 
 /**
@@ -2519,6 +2524,8 @@
 	 */
 	if (! process_data)
 		return;
+	if (! process_data->valid)
+		return;
 
 	nih_assert (io);
 

=== modified file 'init/job_process.h'
--- init/job_process.h	2014-06-02 20:29:33 +0000
+++ init/job_process.h	2014-07-11 12:53:11 +0000
@@ -194,6 +194,10 @@
 
 void job_process_error_handler (const char *buf, size_t len);
 
+void job_process_error_abort     (int fd, JobProcessErrorType type,
+					 int arg)
+	__attribute__ ((noreturn));
+
 NIH_END_EXTERN
 
 #endif /* INIT_JOB_PROCESS_H */

=== modified file 'init/tests/test_job_process.c'
--- init/tests/test_job_process.c	2014-07-07 10:15:25 +0000
+++ init/tests/test_job_process.c	2014-07-11 12:53:11 +0000
@@ -122,6 +122,80 @@
 		TEST_EQ (ok, TRUE); \
 	} while (0)
 
+
+/* 
+ * Register regular child handler.
+ * Register another handler to be called after the primary
+ * Upstart handler to allow the test to exit the main loop
+ * quickly on success.
+ * Process the event queue each time through the main loop
+ */
+
+#define TEST_INSTALL_CHILD_HANDLERS()		                \
+	NIH_MUST (nih_child_add_watch (NULL,			\
+				       -1,			\
+				       NIH_CHILD_ALL,		\
+				       test_job_process_handler,\
+				       NULL));			\
+	NIH_MUST (nih_child_add_watch (NULL,			\
+				       -1,			\
+				       NIH_CHILD_ALL,		\
+				       job_process_handler,	\
+				       NULL));			\
+	NIH_MUST (nih_main_loop_add_func (NULL, (NihMainLoopCb)event_poll, \
+					  NULL))
+
+
+#define TEST_CLEAR_CHILD_STATUS()				\
+	do {							\
+		TEST_EQ (waitid (P_ALL, 0, &siginfo, WEXITED | WNOWAIT), -1); \
+		TEST_EQ (errno, ECHILD);			\
+		errno = 0;					\
+		child_exit_after = 1;				\
+		for (int _i = 0; _i < PROCESS_LAST; _i++) {	\
+			child_exit_status[_i] = -1;		\
+		}						\
+	} while (0)
+
+
+/* Modified version of NIH's TEST_CHILD() that is resilient
+ * to temporary errors.
+ */
+#define RESILIENT_TEST_CHILD(_pid) \
+	do { \
+		int _test_fds[2]; \
+		fflush (stdout);	    \
+		fflush (stderr);	    \
+		assert0 (pipe (_test_fds)); \
+		_pid = fork (); \
+		if (_pid > 0) { \
+			ssize_t ret; \
+			char _test_buf[1]; \
+			close (_test_fds[1]); \
+			while (TRUE) { \
+				ret = read (_test_fds[0], _test_buf, 1); \
+				if (ret > 0) { \
+					assert (ret == 1); \
+					break; \
+				} else if (ret < 0 && errno != EINTR) { \
+					nih_assert_not_reached (); \
+				} else { \
+					nih_assert_not_reached (); \
+				} \
+			} \
+			close (_test_fds[0]); \
+		} else if (_pid == 0) { \
+			close (_test_fds[0]); \
+			assert (write (_test_fds[1], "\n", 1) == 1); \
+			close (_test_fds[1]); \
+		} \
+	} while (0); \
+	if (_pid == 0) \
+		for (int _test_child = 0; _test_child < 2; _test_child++) \
+			if (_test_child) { \
+				abort (); \
+			} else
+
 /* Sadly we can't test everything that job_process_spawn() does simply because
  * a lot of it can only be done by root, or in the case of the console stuff,
  * kills whatever had /dev/console (usually X).
@@ -147,6 +221,36 @@
 static int get_available_pty_count (void) __attribute__((unused));
 static void close_all_files (void);
 
+static int child_exit_status[PROCESS_LAST];
+static int child_exit_after;
+
+/**
+ * test_job_process_handler:
+ *
+ * @data: existing NihList that this function will add entries to,
+ * @pid: process that changed,
+ * @event: event that occurred on the child,
+ * @status: exit status, signal raised or ptrace event.
+ *
+ * Handler that just sets some globals and requests the main loop to
+ * exit to allow the test that installs it to check the values passed to
+ * this function as appropriate.
+ **/
+void
+test_job_process_handler (void           *data,
+			  pid_t           pid,
+			  NihChildEvents  event,
+			  int             status)
+{
+	if (event != NIH_CHILD_EXITED)
+		return;
+	child_exit_after--;
+	child_exit_status[child_exit_after] = status;
+	if (child_exit_after < 1)
+		nih_main_loop_exit (0);
+}
+
+
 static void
 child (enum child_tests  test,
        const char       *filename)
@@ -1056,6 +1160,7 @@
 		nih_free (class);
 	}
 
+
 	/* Check that if we try and run a command that doesn't exist,
 	 * job_process_start() raises a ProcessError and the command doesn't
 	 * have any stored process id for it.
@@ -1063,10 +1168,14 @@
 	TEST_FEATURE ("with no such file");
 	TEST_HASH_EMPTY (job_classes);
 
+	TEST_RESET_MAIN_LOOP ();
+	TEST_INSTALL_CHILD_HANDLERS ();
+
 	output = tmpfile ();
 
 	TEST_ALLOC_FAIL {
 		TEST_ALLOC_SAFE {
+			TEST_HASH_EMPTY (job_classes);
 			class = job_class_new (NULL, "test", NULL);
 			class->console = CONSOLE_NONE;
 			class->process[PROCESS_MAIN] = process_new (class);
@@ -1076,12 +1185,16 @@
 			job = job_new (class, "foo");
 			job->goal = JOB_START;
 			job->state = JOB_SPAWNED;
+
+			nih_hash_add (job_classes, &class->entry);
+			TEST_CLEAR_CHILD_STATUS ();
 		}
 
 		TEST_DIVERT_STDERR (output) {
 			job_process_start (job, PROCESS_MAIN);
-			TEST_WATCH_LOOP ();
-			event_poll ();
+			TEST_GT (job->pid[PROCESS_MAIN], 0);
+			TEST_EQ (nih_main_loop (), 0);
+			TEST_EQ (child_exit_status[PROCESS_MAIN], 255);
 		}
 		rewind (output);
 		
@@ -1098,6 +1211,8 @@
 
 	TEST_EQ (rmdir (dirname), 0);
 
+	TEST_RESET_MAIN_LOOP ();
+
 	TEST_FILENAME (dirname);       
 	TEST_EQ (mkdir (dirname, 0755), 0);
 
@@ -3018,6 +3133,9 @@
 	TEST_FEATURE ("with single-line command running an invalid command");
 	TEST_HASH_EMPTY (job_classes);
 
+	TEST_RESET_MAIN_LOOP ();
+	TEST_INSTALL_CHILD_HANDLERS ();
+
 	class = job_class_new (NULL, "buzz", NULL);
 	TEST_NE_P (class, NULL);
 
@@ -3039,14 +3157,21 @@
 	job->goal = JOB_START;
 	job->state = JOB_SPAWNED;
 
+	/* XXX: Manually add the class so job_process_find() works */
+	nih_hash_add (job_classes, &class->entry);
+
 	output = tmpfile ();
 	TEST_NE_P (output, NULL);
+	TEST_CLEAR_CHILD_STATUS ();
 	TEST_DIVERT_STDERR (output) {
 		job_process_start (job, PROCESS_MAIN);
-		TEST_WATCH_UPDATE ();
-		TEST_WATCH_UPDATE ();
-		event_poll ();
+		TEST_GT (job->pid[PROCESS_MAIN], 0);
+		TEST_EQ (nih_main_loop (), 0);
+		TEST_EQ (child_exit_status[PROCESS_MAIN], 255);
 	}
+
+	TEST_FILE_END (output);
+
 	fclose (output);
 
 	/* We don't expect a logfile to be written since there is no
@@ -3056,11 +3181,14 @@
 	TEST_EQ (errno, ENOENT);
 
 	nih_free (class);
+	TEST_RESET_MAIN_LOOP ();
 
 	/************************************************************/
 	TEST_FEATURE ("with single-line command running an invalid command, then a 1-line post-stop script");
 	TEST_HASH_EMPTY (job_classes);
 
+	TEST_INSTALL_CHILD_HANDLERS ();
+
 	class = job_class_new (NULL, "asterix", NULL);
 	TEST_NE_P (class, NULL);
 
@@ -3086,6 +3214,9 @@
 
 	job = job_new (class, "");
 
+	/* XXX: Manually add the class so job_process_find() works */
+	nih_hash_add (job_classes, &class->entry);
+
 	output = tmpfile ();
 	TEST_NE_P (output, NULL);
 	TEST_DIVERT_STDERR (output) {
@@ -3093,37 +3224,18 @@
 		job->goal = JOB_START;
 		job->state = JOB_SPAWNED;
 
+		TEST_CLEAR_CHILD_STATUS ();
+		child_exit_after = 2;
 		job_process_start (job, PROCESS_MAIN);
-		TEST_WATCH_UPDATE ();
-
-		/* We don't expect a logfile to be written since there is no
-		 * accompanying shell to write the error.
-		 */
-		TEST_EQ (stat (filename, &statbuf), -1);
-		TEST_EQ (errno, ENOENT);
-
-		job->goal = JOB_STOP;
-		job->state = JOB_POST_STOP;
-
-		job_process_start (job, PROCESS_POST_STOP);
-
-		TEST_NE (job->pid[PROCESS_POST_STOP], 0);
-		TEST_WATCH_UPDATE ();
-
-		/* Flush the io so that the shell on the client side
-		 * gets the data (the script to execute).
-		 */
-		TEST_WATCH_UPDATE ();
-
-		waitpid (job->pid[PROCESS_POST_STOP], &status, 0);
-		TEST_TRUE (WIFEXITED (status));
-		TEST_EQ (WEXITSTATUS (status), 0);
-
-		TEST_WATCH_UPDATE ();
-
-		/* .. but the post stop should have written data */
+		TEST_GT (job->pid[PROCESS_MAIN], 0);
+		TEST_EQ (nih_main_loop (), 0);
+		/* Check exit status in reverse order */
+		/* Main */
+		TEST_EQ (child_exit_status[1], 255);
+		/* Post-stop */
+		TEST_EQ (child_exit_status[0], 0);
+		/* And the post stop should have written data */
 		TEST_EQ (stat (filename, &statbuf), 0);
-		event_poll ();
 	}
 	fclose (output);
 
@@ -3144,6 +3256,9 @@
 	TEST_FEATURE ("with single-line command running an invalid command, then a 2-line post-stop script");
 	TEST_HASH_EMPTY (job_classes);
 
+	TEST_RESET_MAIN_LOOP ();
+	TEST_INSTALL_CHILD_HANDLERS ();
+
 	class = job_class_new (NULL, "asterix", NULL);
 	TEST_NE_P (class, NULL);
 
@@ -3168,6 +3283,8 @@
 	TEST_EQ (errno, ENOENT);
 
 	job = job_new (class, "");
+	/* XXX: Manually add the class so job_process_find() works */
+	nih_hash_add (job_classes, &class->entry);
 
 	output = tmpfile ();
 	TEST_NE_P (output, NULL);
@@ -3176,39 +3293,17 @@
 		job->goal = JOB_START;
 		job->state = JOB_SPAWNED;
 
+		TEST_CLEAR_CHILD_STATUS ();
+		child_exit_after = 2;
 		job_process_start (job, PROCESS_MAIN);
-		TEST_WATCH_UPDATE ();
-		TEST_WATCH_UPDATE ();
-
-		/* We don't expect a logfile to be written since there is no
-		 * accompanying shell to write the error.
-		 */
-		TEST_EQ (stat (filename, &statbuf), -1);
-		TEST_EQ (errno, ENOENT);
-
-		job->goal = JOB_STOP;
-		job->state = JOB_POST_STOP;
-
-		job_process_start (job, PROCESS_POST_STOP);
-		TEST_WATCH_UPDATE ();
-
-		TEST_NE (job->pid[PROCESS_POST_STOP], 0);
-
-		/* Flush the io so that the shell on the client side
-		 * gets the data (the script to execute).
-		 */
-		TEST_WATCH_UPDATE ();
-
-		waitpid (job->pid[PROCESS_POST_STOP], &status, 0);
-		TEST_TRUE (WIFEXITED (status));
-		TEST_EQ (WEXITSTATUS (status), 0);
-
-		/* Allow the log to be written */
-		TEST_WATCH_UPDATE ();
-
-		/* .. but the post stop should have written data */
+		TEST_NE (job->pid[PROCESS_MAIN], 0);
+
+		TEST_EQ (nih_main_loop (), 0);
+		TEST_EQ (child_exit_status[1], 255);
+		TEST_EQ (child_exit_status[0], 0);
+		
+		/* And the post stop should have written data */
 		TEST_EQ (stat (filename, &statbuf), 0);
-		event_poll ();
 	}
 	fclose (output);
 
@@ -3225,6 +3320,7 @@
 	TEST_EQ (unlink (filename), 0);
 
 	nih_free (class);
+	TEST_RESET_MAIN_LOOP ();
 
 	/************************************************************/
 	TEST_FEATURE ("with single-line command running an invalid command, then a post-stop command");
@@ -3255,6 +3351,9 @@
 
 	job = job_new (class, "");
 
+	TEST_RESET_MAIN_LOOP ();
+	TEST_INSTALL_CHILD_HANDLERS ();
+
 	output = tmpfile ();
 	TEST_NE_P (output, NULL);
 	TEST_DIVERT_STDERR (output) {
@@ -3262,37 +3361,19 @@
 		job->goal = JOB_START;
 		job->state = JOB_SPAWNED;
 
+		TEST_CLEAR_CHILD_STATUS ();
+		child_exit_after = 2;
+
 		job_process_start (job, PROCESS_MAIN);
-		TEST_WATCH_UPDATE ();
-
-		/* We don't expect a logfile to be written since there is no
-		 * accompanying shell to write the error.
-		 */
-		TEST_EQ (stat (filename, &statbuf), -1);
-		TEST_EQ (errno, ENOENT);
-
-		job->goal = JOB_STOP;
-		job->state = JOB_POST_STOP;
-
-		job_process_start (job, PROCESS_POST_STOP);
-		TEST_WATCH_UPDATE ();
-
-		TEST_NE (job->pid[PROCESS_POST_STOP], 0);
-
-		/* Flush the io so that the shell on the client side
-		 * gets the data (the script to execute).
-		 */
-		TEST_WATCH_UPDATE ();
-
-		waitpid (job->pid[PROCESS_POST_STOP], &status, 0);
-		TEST_TRUE (WIFEXITED (status));
-		TEST_EQ (WEXITSTATUS (status), 0);
-
-		TEST_WATCH_UPDATE ();
-
-		/* .. but the post stop should have written data */
+		TEST_GT (job->pid[PROCESS_MAIN], 0);
+		TEST_EQ (nih_main_loop (), 0);
+		TEST_GT (job->pid[PROCESS_POST_STOP], 0);
+		TEST_EQ (child_exit_status[1], 255);
+		TEST_EQ (child_exit_status[0], 0);
+
+		/* The post stop should have written data */
 		TEST_EQ (stat (filename, &statbuf), 0);
-		event_poll ();
+
 	}
 	fclose (output);
 
@@ -3309,6 +3390,8 @@
 
 	nih_free (class);
 
+	TEST_RESET_MAIN_LOOP ();
+
 	/************************************************************/
 	TEST_FEATURE ("with single-line command running an invalid command, then an invalid post-stop command");
 	TEST_HASH_EMPTY (job_classes);
@@ -3338,6 +3421,9 @@
 
 	job = job_new (class, "");
 
+	TEST_RESET_MAIN_LOOP ();
+	TEST_INSTALL_CHILD_HANDLERS ();
+
 	output = tmpfile ();
 	TEST_NE_P (output, NULL);
 	TEST_DIVERT_STDERR (output) {
@@ -3345,29 +3431,24 @@
 		job->goal = JOB_START;
 		job->state = JOB_SPAWNED;
 
+		TEST_CLEAR_CHILD_STATUS ();
+		child_exit_after = 2;
+
 		job_process_start (job, PROCESS_MAIN);
-		TEST_WATCH_UPDATE ();
-
-		/* We don't expect a logfile to be written since there is no
-		 * accompanying shell to write the error.
-		 */
-		TEST_EQ (stat (filename, &statbuf), -1);
-		TEST_EQ (errno, ENOENT);
-
-		job->goal = JOB_STOP;
-		job->state = JOB_POST_STOP;
-
-		job_process_start (job, PROCESS_POST_STOP);
-		TEST_WATCH_UPDATE ();
+		TEST_GT (job->pid[PROCESS_MAIN], 0);
+		TEST_EQ (nih_main_loop (), 0);
+		TEST_EQ (child_exit_status[1], 255);
+		TEST_EQ (child_exit_status[0], 255);
 
 		/* Again, no file expected */
 		TEST_EQ (stat (filename, &statbuf), -1);
 		TEST_EQ (errno, ENOENT);
-		event_poll ();
 	}
 	fclose (output);
 	nih_free (class);
 
+	TEST_RESET_MAIN_LOOP ();
+
 	/************************************************************/
 	TEST_FEATURE ("with single-line command running a valid command, then a 1-line invalid post-stop command");
 	TEST_HASH_EMPTY (job_classes);
@@ -3397,6 +3478,9 @@
 
 	job = job_new (class, "");
 
+	TEST_RESET_MAIN_LOOP ();
+	TEST_INSTALL_CHILD_HANDLERS ();
+
 	output = tmpfile ();
 	TEST_NE_P (output, NULL);
 	TEST_DIVERT_STDERR (output) {
@@ -3404,29 +3488,26 @@
 		job->goal = JOB_START;
 		job->state = JOB_SPAWNED;
 
+		TEST_CLEAR_CHILD_STATUS ();
+
 		job_process_start (job, PROCESS_MAIN);
-
-		TEST_NE (job->pid[PROCESS_MAIN], 0);
-
-		waitpid (job->pid[PROCESS_MAIN], &status, 0);
-		TEST_TRUE (WIFEXITED (status));
-		TEST_EQ (WEXITSTATUS (status), 0);
-
-		/* Flush the io so that the shell on the client side
-		 * gets the data (the script to execute).
-		 */
-		TEST_WATCH_UPDATE ();
+		TEST_GT (job->pid[PROCESS_MAIN], 0);
+		TEST_EQ (nih_main_loop (), 0);
+		TEST_EQ (child_exit_status[0], 0);
+		TEST_EQ (child_exit_status[1], -1);
 
 		/* Expect a log file */
 		TEST_EQ (stat (filename, &statbuf), 0);
 
 		job->goal = JOB_STOP;
 		job->state = JOB_POST_STOP;
+		
+		TEST_CLEAR_CHILD_STATUS ();
 
 		job_process_start (job, PROCESS_POST_STOP);
-		TEST_WATCH_UPDATE ();
-
-		TEST_EQ (job->pid[PROCESS_POST_STOP], 0);
+		TEST_GT (job->pid[PROCESS_POST_STOP], 0);
+		TEST_EQ (nih_main_loop (), 0);
+		TEST_EQ (child_exit_status[0], 255);
 	}
 	fclose (output);
 
@@ -3443,10 +3524,15 @@
 
 	nih_free (class);
 
+	TEST_RESET_MAIN_LOOP ();
+
 	/************************************************************/
 	TEST_FEATURE ("with multi-line script running an invalid command");
 	TEST_HASH_EMPTY (job_classes);
 
+	TEST_RESET_MAIN_LOOP ();
+	TEST_INSTALL_CHILD_HANDLERS ();
+
 	class = job_class_new (NULL, "blah", NULL);
 	TEST_NE_P (class, NULL);
 
@@ -3463,17 +3549,13 @@
 	job = job_new (class, "");
 	job->goal = JOB_START;
 	job->state = JOB_SPAWNED;
+	nih_hash_add (job_classes, &class->entry);
+	TEST_CLEAR_CHILD_STATUS ();
 
 	job_process_start (job, PROCESS_MAIN);
-	TEST_WATCH_UPDATE ();
-
-	TEST_NE (job->pid[PROCESS_MAIN], 0);
-
-	TEST_WATCH_UPDATE ();
-	waitpid (job->pid[PROCESS_MAIN], &status, 0);
-	TEST_TRUE (WIFEXITED (status));
-	TEST_NE (WEXITSTATUS (status), 0);
-	TEST_WATCH_UPDATE ();
+	TEST_GT (job->pid[PROCESS_MAIN], 0);
+	TEST_EQ (nih_main_loop (), 0);
+	TEST_EQ (child_exit_status[0], 127);
 
 	TEST_EQ (stat (filename, &statbuf), 0);
 
@@ -3502,6 +3584,7 @@
 
 	TEST_EQ (unlink (filename), 0);
 	nih_free (class);
+	TEST_RESET_MAIN_LOOP ();
 
 	/************************************************************/
 	TEST_FEATURE ("with multi-line script that writes 1 line to stdout then 1 line to stderr");
@@ -3977,6 +4060,9 @@
 	 */
 	TEST_FEATURE ("with setuid");
 	TEST_HASH_EMPTY (job_classes);
+	TEST_RESET_MAIN_LOOP ();
+	TEST_INSTALL_CHILD_HANDLERS ();
+
 
 	TEST_NE_P (output, NULL);
 	TEST_ALLOC_FAIL {
@@ -3999,30 +4085,30 @@
 			job->goal = JOB_START;
 			job->state = JOB_SPAWNED;
 
+			nih_hash_add (job_classes, &class->entry);
+			TEST_CLEAR_CHILD_STATUS ();
+
 			output = tmpfile ();
 		}
 
 		TEST_DIVERT_STDERR (output) {
 			job_process_start (job, PROCESS_MAIN);
-			TEST_WATCH_UPDATE ();
+			TEST_GT (job->pid[PROCESS_MAIN], 0);
+			TEST_EQ (nih_main_loop (), 0);
 		}
 
 		if (geteuid() == 0 || getuid() == pwd->pw_uid) {
-			TEST_NE (job->pid[PROCESS_MAIN], 0);
-
-			waitpid (job->pid[PROCESS_MAIN], NULL, 0);
 			TEST_EQ (stat (filename, &statbuf), 0);
-		}
-		else {
+		} else {
 			TEST_EQ (stat (filename, &statbuf), -1);
-			event_poll ();
 		}
 
 		unlink (filename);
 		nih_free (class);
 	}
+	
+	TEST_RESET_MAIN_LOOP ();
 
-	/* FIXME with async spawn this test is racy */
 
 	/************************************************************/
 	TEST_FEATURE ("with multiple processes and log");
@@ -4062,10 +4148,10 @@
 
 	job = job_new (class, "");
 	job->goal = JOB_START;
-	job->state = JOB_SPAWNED;
+	job->state = JOB_SPAWNING;
 
 	job_process_start (job, PROCESS_MAIN);
-	while (stat (filename, &statbuf) != 0) {
+	while (stat (filename, &statbuf) != 0 || statbuf.st_size < 9) {
 		TEST_WATCH_UPDATE ();
 	}
 	pid = job->pid[PROCESS_MAIN];
@@ -4077,9 +4163,7 @@
 	TEST_NE_P (output, NULL);
 
 	/* initial output from main process */
-	CHECK_FILE_EQ (output, "started\r\n", TRUE);
-
-	TEST_FILE_END (output);
+	TEST_FILE_EQ (output, "started\r\n");
 
 	TEST_EQ (fclose (output), 0);
 
@@ -4093,7 +4177,10 @@
 	waitpid (pid, &status, 0);
 	TEST_TRUE (WIFEXITED (status));
 	TEST_EQ (WEXITSTATUS (status), 0);
-	TEST_WATCH_UPDATE ();
+
+	while (stat (filename, &statbuf) != 0 || statbuf.st_size < 25) {
+		TEST_WATCH_UPDATE ();
+	}
 
 	output = fopen (filename, "r");
 	TEST_NE_P (output, NULL);
@@ -5488,6 +5575,8 @@
 	char            dirname[PATH_MAX];
 	nih_local char *logfile = NULL;
 	int             fds[2] = { -1, -1};
+	NihIo          *io = NULL;
+	nih_local NihIoBuffer    *buffer= NULL;
 
 	TEST_FILENAME (dirname);       
 	TEST_EQ (mkdir (dirname, 0755), 0);
@@ -8770,13 +8859,288 @@
 
 	fclose (output);
 
+	nih_free (event);
+	event_poll ();
+
+	TEST_RESET_MAIN_LOOP ();
+
+	TEST_NE_P (class->process[PROCESS_MAIN], NULL);
+	TEST_EQ_P (class->process[PROCESS_PRE_START], NULL);
+	TEST_EQ_P (class->process[PROCESS_POST_START], NULL);
+	TEST_EQ_P (class->process[PROCESS_PRE_STOP], NULL);
+	TEST_EQ_P (class->process[PROCESS_POST_STOP], NULL);
+	TEST_EQ_P (class->process[PROCESS_SECURITY], NULL);
+
+	/************************************************************/
+	/* Ensure that if a child is setup successfully and then exits
+	 * before the main loop detects the child fd has closed (due to
+	 * the child calling execvp()), that the state is correct.
+	 */
+	TEST_FEATURE ("with child exit notification before child setup success notification");
+
+	job = job_new (class, "");
+	TEST_NE_P (job, NULL);
+
+	job->goal = JOB_START;
+	job->state = JOB_SPAWNING;
+
+	assert0 (pipe (fds));
+
+	job->process_data[PROCESS_MAIN] = job_process_data_new (job->process_data,
+			job, PROCESS_MAIN, fds[0]);
+	TEST_NE_P (job->process_data[PROCESS_MAIN], NULL);
+
+	TEST_NE_P (class->process[PROCESS_MAIN], NULL);
+	TEST_EQ_P (class->process[PROCESS_PRE_START], NULL);
+	TEST_EQ_P (class->process[PROCESS_POST_START], NULL);
+	TEST_EQ_P (class->process[PROCESS_PRE_STOP], NULL);
+	TEST_EQ_P (class->process[PROCESS_POST_STOP], NULL);
+	TEST_EQ_P (class->process[PROCESS_SECURITY], NULL);
+
+	/* used to check if job_process_terminated() called */
+	job->process_data[PROCESS_MAIN]->status = -1;
+
+	TEST_CHILD (job->pid[PROCESS_MAIN]) {
+		close (fds[0]);
+
+		nih_io_set_cloexec (fds[1]);
+
+		execl ("/bin/true", "/bin/true", NULL);
+	}
+	close (fds[1]);
+
+	TEST_FREE_TAG (job);
+
+	job_process_handler (NULL, job->pid[PROCESS_MAIN],
+			NIH_CHILD_EXITED, 0);
+
+	TEST_NOT_FREE (job);
+
+	TEST_TRUE (job->process_data[PROCESS_MAIN]->valid);
+	TEST_EQ (job->process_data[PROCESS_MAIN]->job_process_fd, fds[0]);
+	TEST_EQ (job->process_data[PROCESS_MAIN]->shell_fd, -1);
+	TEST_EQ_P (job->process_data[PROCESS_MAIN]->script, NULL);
+
+	/* job_process_terminated() should have been called now and the
+	 * status recorded.
+	 */
+	TEST_EQ (job->process_data[PROCESS_MAIN]->status, 0);
+
+	/* goal should not change until the IO handlers have had a
+	 * chance to run.
+	 */
+	TEST_EQ (job->goal, JOB_START);
+	TEST_EQ (job->state, JOB_SPAWNED);
+
+	TEST_NE_P (job->process_data[PROCESS_MAIN], NULL);
+
+	close (fds[0]);
+
+	pid = job->pid[PROCESS_MAIN];
+
+	TEST_EQ (timed_waitpid (pid, 5), pid);
+
+	nih_free (job);
+	TEST_RESET_MAIN_LOOP ();
+
+	/************************************************************/
+	/* Ensure that if a child failed to be setup and then exits
+	 * before the main loop detects the child fd has data to read
+	 * that the state is correct.
+	 */
+	TEST_FEATURE ("with child exit notification before child setup failure notification");
+
+	job = job_new (class, "");
+	TEST_NE_P (job, NULL);
+
+	job->goal = JOB_START;
+	job->state = JOB_SPAWNING;
+
+	assert0 (pipe (fds));
+
+	job->process_data[PROCESS_MAIN] = job_process_data_new (job->process_data,
+			job, PROCESS_MAIN, fds[0]);
+	TEST_NE_P (job->process_data[PROCESS_MAIN], NULL);
+	TEST_EQ (job->process_data[PROCESS_MAIN]->job_process_fd, fds[0]);
+	TEST_EQ (job->process_data[PROCESS_MAIN]->process, PROCESS_MAIN);
+
+	TEST_CHILD (job->pid[PROCESS_MAIN]) {
+		close (fds[0]);
+
+		nih_error_raise_no_memory ();
+		job_process_error_abort (fds[1], JOB_PROCESS_ERROR_CGROUP_SETUP, 0);
+	}
+	close (fds[1]);
+
+	TEST_FREE_TAG (job);
+
+	job_process_handler (NULL, job->pid[PROCESS_MAIN],
+			NIH_CHILD_DUMPED, 255);
+
+	TEST_NOT_FREE (job);
+
+	/* goal should not change until the IO handlers have had a
+	 * chance to run.
+	 */
+	TEST_EQ (job->goal, JOB_START);
+
+	TEST_EQ (job->state, JOB_SPAWNED);
+	TEST_NE_P (job->process_data[PROCESS_MAIN], NULL);
+
+	/* Still valid because the IO handlers haven't fired yet */
+	TEST_TRUE (job->process_data[PROCESS_MAIN]->valid);
+	TEST_EQ (job->process_data[PROCESS_MAIN]->job_process_fd, fds[0]);
+
+	TEST_EQ (job->process_data[PROCESS_MAIN]->shell_fd, -1);
+	TEST_EQ_P (job->process_data[PROCESS_MAIN]->script, NULL);
+
+	/* job_process_terminated() will have been called now */
+	status = job->process_data[PROCESS_MAIN]->status;
+	TEST_GT (status, 0);
+
+	TEST_TRUE (WIFEXITED (status));
+	TEST_EQ (WEXITSTATUS (status), 255);
+
+	/* slurp - don't care about content */
+	buffer = read_from_fd (NULL, fds[0]);
+	close (fds[0]);
+
+	pid = job->pid[PROCESS_MAIN];
+	TEST_EQ (timed_waitpid (pid, 5), pid);
+
+	nih_free (job);
+	TEST_RESET_MAIN_LOOP ();
+
+	/************************************************************/
+	TEST_FEATURE ("with child setup success notification before child exit notification");
+
+	job = job_new (class, "");
+	TEST_NE_P (job, NULL);
+
+	job->goal = JOB_START;
+	job->state = JOB_SPAWNING;
+
+	assert0 (pipe (fds));
+
+	job->process_data[PROCESS_MAIN] = job_process_data_new (job->process_data,
+			job, PROCESS_MAIN, fds[0]);
+	TEST_NE_P (job->process_data[PROCESS_MAIN], NULL);
+	TEST_EQ (job->process_data[PROCESS_MAIN]->job_process_fd, fds[0]);
+	TEST_EQ (job->process_data[PROCESS_MAIN]->process, PROCESS_MAIN);
+
+	io = nih_io_reopen (job->process_data[PROCESS_MAIN],
+			fds[0],
+			NIH_IO_STREAM,
+			(NihIoReader)job_process_child_reader,
+			(NihIoCloseHandler)job_process_close_handler,
+			NULL,
+			job->process_data[PROCESS_MAIN]);
+	TEST_NE_P (io, NULL);
+
+	TEST_CHILD (job->pid[PROCESS_MAIN]) {
+		close (fds[0]);
+
+		nih_io_set_cloexec (fds[1]);
+
+		execl ("/bin/true", "/bin/true", NULL);
+	}
+	close (fds[1]);
+
+	job_process_close_handler (job->process_data[PROCESS_MAIN], io);
+
+	TEST_EQ (job->goal, JOB_START);
+	TEST_EQ (job->state, JOB_SPAWNED);
+	TEST_NE_P (job->process_data[PROCESS_MAIN], NULL);
+
+	/* Invalid because the IO handlers have now run */
+	TEST_FALSE (job->process_data[PROCESS_MAIN]->valid);
+	TEST_EQ (job->process_data[PROCESS_MAIN]->job_process_fd, -1);
+
+	TEST_EQ (job->process_data[PROCESS_MAIN]->shell_fd, -1);
+	TEST_EQ_P (job->process_data[PROCESS_MAIN]->script, NULL);
+	TEST_EQ (job->process_data[PROCESS_MAIN]->status, 0);
+
+	close (fds[0]);
+
+	pid = job->pid[PROCESS_MAIN];
+	TEST_EQ (timed_waitpid (pid, 5), pid);
+
+	nih_free (job);
+	TEST_RESET_MAIN_LOOP ();
+
+	/************************************************************/
+	TEST_FEATURE ("with child setup failure notification before child exit notification");
+
+	job = job_new (class, "");
+	TEST_NE_P (job, NULL);
+
+	job->goal = JOB_START;
+	job->state = JOB_SPAWNED;
+
+	assert0 (pipe (fds));
+
+	job->process_data[PROCESS_MAIN] = job_process_data_new (job->process_data,
+			job, PROCESS_MAIN, fds[0]);
+	TEST_NE_P (job->process_data[PROCESS_MAIN], NULL);
+	TEST_EQ (job->process_data[PROCESS_MAIN]->job_process_fd, fds[0]);
+	TEST_EQ (job->process_data[PROCESS_MAIN]->process, PROCESS_MAIN);
+
+	/* Can't use job_register_child_handler() as we want the return
+	 * value.
+	 */
+	io = nih_io_reopen (job->process_data[PROCESS_MAIN],
+			fds[0],
+			NIH_IO_STREAM,
+			(NihIoReader)job_process_child_reader,
+			(NihIoCloseHandler)job_process_close_handler,
+			NULL,
+			job->process_data[PROCESS_MAIN]);
+	TEST_NE_P (io, NULL);
+
+	RESILIENT_TEST_CHILD (job->pid[PROCESS_MAIN]) {
+		close (fds[0]);
+		nih_error_raise_no_memory ();
+		job_process_error_abort (fds[1], JOB_PROCESS_ERROR_CGROUP_SETUP, 0);
+	}
+	close (fds[1]);
+	pid = job->pid[PROCESS_MAIN];
+
+	buffer = read_from_fd (NULL, fds[0]);
+
+	job_process_child_reader (job->process_data[PROCESS_MAIN],
+			io, buffer->buf, buffer->len);
+
+	/* Setup failed, so goal should have changed to stop */
+	TEST_EQ (job->goal, JOB_STOP);
+	TEST_EQ (job->state, JOB_STOPPING);
+
+	TEST_NE_P (job->process_data[PROCESS_MAIN], NULL);
+
+	/* Invalid because the IO handlers have now run */
+	TEST_FALSE (job->process_data[PROCESS_MAIN]->valid);
+	TEST_EQ (job->process_data[PROCESS_MAIN]->job_process_fd, -1);
+
+	TEST_EQ (job->process_data[PROCESS_MAIN]->shell_fd, -1);
+	TEST_EQ_P (job->process_data[PROCESS_MAIN]->script, NULL);
+	TEST_EQ (job->process_data[PROCESS_MAIN]->shell_fd, -1);
+
+	/* Still zero because the process handler hasn't run yet */
+	TEST_EQ (job->process_data[PROCESS_MAIN]->status, 0);
+
+	close (fds[0]);
+
+	TEST_EQ (timed_waitpid (pid, 5), pid);
+
+	nih_free (job);
+
+	TEST_RESET_MAIN_LOOP ();
+
+	/************************************************************/
+
 	nih_free (class);
 	file->job = NULL;
 	nih_free (source);
 
-	nih_free (event);
-	event_poll ();
-
 	TEST_EQ (rmdir (dirname), 0);
 	TEST_EQ (unsetenv ("UPSTART_LOGDIR"), 0);
 }

=== modified file 'test/test_util_common.c'
--- test/test_util_common.c	2014-06-02 23:32:49 +0000
+++ test/test_util_common.c	2014-07-11 12:53:11 +0000
@@ -375,6 +375,8 @@
  *
  * Simplified waitpid(2) with timeout using a pipe to allow select(2)
  * with timeout to be used to wait for process state change.
+ *
+ * Returns: as waitpid(2).
  **/
 pid_t
 timed_waitpid (pid_t pid, time_t timeout)
@@ -1000,7 +1002,7 @@
 			nih_local char *cmd = NULL;
 
 			/* Clean up if tests forgot to */
-			cmd = NIH_MUST (nih_sprintf (NULL, "rm %s/*.session 2>/dev/null", path));
+			cmd = NIH_MUST (nih_sprintf (NULL, "rm -f %s/*.session 2>/dev/null", path));
 			assert0 (system (cmd));
 
 			/* Remove the directory tree the first Session Init created */
@@ -1094,10 +1096,13 @@
 				buffer->buf + buffer->len,
 				buffer->size - buffer->len);
 
-		if (len <= 0)
-			break;
-		else if (len > 0)
+		if (len < 0 && errno != EAGAIN && errno != EINTR) {
+			break;
+		} else if (! len) {
+			break;
+		} else if (len > 0) {
 			buffer->len += len;
+		}
 	}
 
 	close (fd);

=== modified file 'util/tests/test_initctl.c'
--- util/tests/test_initctl.c	2014-07-01 15:34:42 +0000
+++ util/tests/test_initctl.c	2014-07-11 12:53:11 +0000
@@ -10968,9 +10968,9 @@
 	/*******************************************************************/
 	TEST_FEATURE ("single job");
 
-	START_UPSTART (upstart_pid, FALSE);
 	CREATE_FILE (dirname, "foo.conf",
 			"exec echo hello");
+	START_UPSTART (upstart_pid, FALSE);
 
 	cmd = nih_sprintf (NULL, "%s list 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
@@ -10984,10 +10984,10 @@
 	/*******************************************************************/
 	TEST_FEATURE ("3 jobs and re-exec");
 
-	START_UPSTART (upstart_pid, FALSE);
-	CREATE_FILE (dirname, "foo.conf", "exec echo foo");
+        CREATE_FILE (dirname, "foo.conf", "exec echo foo");
 	CREATE_FILE (dirname, "bar.conf", "exec echo bar");
 	CREATE_FILE (dirname, "baz.conf", "exec echo bar");
+	START_UPSTART (upstart_pid, FALSE);
 
 	cmd = nih_sprintf (NULL, "%s list 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
@@ -11110,6 +11110,12 @@
 
 	CREATE_FILE (confdir, "foo.conf", contents);
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s start foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -12482,6 +12488,12 @@
 			"author \"foo\"\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s show-config foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -12507,6 +12519,12 @@
 			"emits \"thing\"\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s show-config foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -12535,6 +12553,12 @@
 			"emits \"thong\"\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s show-config foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -12564,6 +12588,12 @@
 			"start on (A and B)\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s show-config foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -12593,6 +12623,12 @@
 			"start on (A and B)\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s show-config foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -12625,6 +12661,12 @@
 			"emits \"stime\"\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s show-config foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -12657,6 +12699,12 @@
 			"stop on (A or B)\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s show-config foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -12686,6 +12734,12 @@
 			"stop on (A or B)\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s show-config foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -12718,6 +12772,12 @@
 			"emits \"stime\"\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s show-config foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -12753,6 +12813,12 @@
 			"start on (starting JOB=\"boo\" or B x=y)\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s show-config foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -12791,6 +12857,12 @@
 			"start on (starting JOB=\"boo\" P=Q c=sea or B x=y)\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s show-config foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -12830,6 +12902,12 @@
 			"start on A and (B FOO=BAR or starting C x=y)\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s show-config foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -12876,6 +12954,12 @@
 			"(stopped gdm or stopped kdm or stopped xdm A=B or stopping lxdm)))\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s show-config foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -13011,6 +13095,12 @@
 			"task\n"
 			"exec true");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s check-config 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -13031,6 +13121,12 @@
 	CREATE_FILE (dirname, "baz.conf",
 			"emits wibble");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s check-config 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -13052,6 +13148,12 @@
 			"task\n"
 			"exec true");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s check-config --ignore-events=wibble 2>&1",
 			get_initctl ());
 	TEST_NE_P (cmd, NULL);
@@ -13067,6 +13169,12 @@
 	CREATE_FILE (dirname, "foo.conf",
 			"start on (fred and wilma)");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s check-config --ignore-events=wilma,foo,fred 2>&1",
 			get_initctl ());
 	TEST_NE_P (cmd, NULL);
@@ -13091,6 +13199,12 @@
 	CREATE_FILE (dirname, "mountall.conf", "exec true");
 	CREATE_FILE (dirname, "gdm.conf"     , "exec true");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s check-config --ignore-events=runlevel 2>&1",
 			get_initctl ());
 	TEST_NE_P (cmd, NULL);
@@ -13115,6 +13229,12 @@
 
 	CREATE_FILE (dirname, "mountall.conf", "exec true");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s check-config --ignore-events=runlevel 2>&1",
 			get_initctl ());
 	TEST_NE_P (cmd, NULL);
@@ -13146,6 +13266,12 @@
 	CREATE_FILE (dirname, "mountall.conf", "exec true");
 	CREATE_FILE (dirname, "gdm.conf"     , "exec true");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s check-config --ignore-events=runlevel 2>&1",
 			get_initctl ());
 	TEST_NE_P (cmd, NULL);
@@ -13170,6 +13296,12 @@
 
 	CREATE_FILE (dirname, "mountall.conf", "exec true");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s check-config --ignore-events=runlevel 2>&1",
 			get_initctl ());
 	TEST_NE_P (cmd, NULL);
@@ -13203,6 +13335,12 @@
 	CREATE_FILE (dirname, "portmap.conf", "exec true");
 	CREATE_FILE (dirname, "beano.conf", "exec true");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s check-config --ignore-events=runlevel 2>&1",
 			get_initctl ());
 	TEST_NE_P (cmd, NULL);
@@ -13240,6 +13378,12 @@
 			"emits hello");
 	CREATE_FILE (dirname, "gdm.conf", "exec true");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s check-config >&1",
 			get_initctl ());
 	TEST_NE_P (cmd, NULL);
@@ -13272,6 +13416,12 @@
 	CREATE_FILE (dirname, "mountall.conf", "exec true");
 	CREATE_FILE (dirname, "portmap.conf", "exec true");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s check-config 2>&1",
 			get_initctl ());
 	TEST_NE_P (cmd, NULL);
@@ -13313,6 +13463,12 @@
 	CREATE_FILE (dirname, "wibble.conf", "emits wibble");
 	CREATE_FILE (dirname, "beano.conf", "exec true");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s check-config 2>&1",
 			get_initctl ());
 	TEST_NE_P (cmd, NULL);
@@ -13350,6 +13506,12 @@
 	CREATE_FILE (dirname, "wibble.conf", "emits wibble");
 	CREATE_FILE (dirname, "beano.conf", "exec true");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s check-config --warn 2>&1",
 			get_initctl ());
 	TEST_NE_P (cmd, NULL);
@@ -16080,6 +16242,12 @@
 			"author \"foo\"\n"
 			"description \"wibble\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s usage foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -16096,6 +16264,12 @@
 	CREATE_FILE (dirname, "foo.conf",
 			"usage \"this is usage\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s usage foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);
@@ -16113,6 +16287,12 @@
 			"instance $FOO\n"
 			"usage \"this is usage\"");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	memset (&command, 0, sizeof command);
 	args[0] = "foo";
 	args[1] = NULL;
@@ -16246,6 +16426,12 @@
 
 	CREATE_FILE (confdir, "foo.conf", "exec env");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &line_count);
+	TEST_EQ (line_count, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s start foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &line_count);
@@ -16393,6 +16579,12 @@
 
 	CREATE_FILE (confdir, "empty-env.conf", contents);
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &line_count);
+	TEST_EQ (line_count, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s start empty-env 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &line_count);
@@ -17504,6 +17696,12 @@
 
 	CREATE_FILE (confdir, "modified-env.conf", "exec env");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &line_count);
+	TEST_EQ (line_count, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s start modified-env 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &line_count);
@@ -17629,6 +17827,12 @@
 
 	CREATE_FILE (confdir, "foo.conf", contents);
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &line_count);
+	TEST_EQ (line_count, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s start foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 
@@ -17674,6 +17878,12 @@
 
 	CREATE_FILE (confdir, "foo.conf", contents);
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &line_count);
+	TEST_EQ (line_count, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s start foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &line_count);
@@ -17705,6 +17915,12 @@
 
 	CREATE_FILE (confdir, "bar.conf", contents);
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &line_count);
+	TEST_EQ (line_count, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s start bar 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &line_count);
@@ -17788,6 +18004,12 @@
 
 	CREATE_FILE (confdir, "foo.conf", "exec env");
 
+	cmd = nih_sprintf (NULL, "%s reload-configuration 2>&1", get_initctl ());
+	TEST_NE_P (cmd, NULL);
+	RUN_COMMAND (NULL, cmd, &output, &lines);
+	TEST_EQ (lines, 0);
+	nih_free (output);
+
 	cmd = nih_sprintf (NULL, "%s start foo 2>&1", get_initctl ());
 	TEST_NE_P (cmd, NULL);
 	RUN_COMMAND (NULL, cmd, &output, &lines);