Merge lp:~clay-gerrard/swift/swift_init_ng into lp:~hudson-openstack/swift/trunk

[clayg]

I rewrote swift-init cause I wanted it to do somethings it didn't, and I
tried to make some of the error conditions a little better too. It has
some new features, but overall it's behavior is mostly compatible with
the old swift-init (input, return codes) - but if someone had some weird
greps on the output they're hozed.

swift-init --help is a good place to start

some highlights might be:
 $swift-init start main --wait # wait for all "main" services to start,
printing any errors to console
 $swift-init rest wait # same as above but for replicators, updaters & auditors
 $swift-init stop object -c1 # kill the first object-server "node"
 $swift-init object-server status # make sure it's dead - yay status!
 $swift-init proxy auth reload # devauth? srsly?
 $swift-init *-replicator once -n # run the object, container,
and account replicators in "once mode" and watch them log to console
until they're finished

... probably some other stuff too. Looking forward to feedback...

[gholt (gholt)]

First, thanks! This should greatly increase quality of life. :)

Here are my comments so far (just a once over, haven't read the tests):

General:
Internationalization?
I think you have to update the doc/source stuff some more so that the new swift.common.manager appears.
With your "wait" mode stuff, I think we can maybe drop the "sleep 5" in resetswift? [doc/source/development_saio.rst line 534]

swift/common/manager.py:
24: Commonly imports are: stdlib imports, blank line, local swift imports
60: Is there a reason for this explicit return?
90: Why "while interval" why not just "while True" or maybe better "while time.time() < end" and clean up the bottom of the loop?
106: Is there a possibility of a problem modifying the dict you're iterating over?
114: Is there a reason for this explicit return?
213: E501 line too long (81 characters)
214: Typo "yeiled"
228: Hardcoded "15" instead of KILL_WAIT
315: Maybe use self.type = server.rsplit('-', 1)[0] instead?
361: Reference to "ini files"
365: Reference to "ini files"
401: E501 line too long (80 characters)
401: typo, grammar: "limt" and sentence is rambling
430: 3 should probably be errno.ESRCH instead
552: Reference to "ini files"
582: Isn't this TODO done?

swift/common/utils.py:
80: Not sure why you changed this since [from pydoc sys.exit] "If it is another kind of object, it will be printed and the system exit status will be one (i.e., failure)."
825: I heard that "dir" as a variable name is frowned upon by somebody somewhere.

swift/common/wsgi.py
124: Honestly? This was your change here? :P
127: Well, I mean, doesn't this one deserve some space? :)

I haven't yet begun the unit tests. Figured I'd post this and do those later.

[clayg (clay-gerrard)]

yes, that all seems reasonable, thanks.

[gholt (gholt)]

review: Approve

[lp:~clay-gerrard/swift/swift_init_ng]

222. By clayg on 2011-02-16

added [options] to command usage

223. By clayg on 2011-02-16

mermged upstream changes

[Chuck Thier (cthier)]

Looks great clay. Thanks!

review: Approve

=== modified file 'bin/swift-init' (properties changed: +x to -x)
--- bin/swift-init	2011-01-19 23:21:57 +0000
+++ bin/swift-init	2011-02-16 21:01:55 +0000
@@ -14,180 +14,60 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from __future__ import with_statement
-import errno
-import glob
-import os
-import resource
-import signal
 import sys
-import time
-
-ALL_SERVERS = ['account-auditor', 'account-server', 'container-auditor',
-    'container-replicator', 'container-server', 'container-updater',
-    'object-auditor', 'object-server', 'object-replicator', 'object-updater',
-    'proxy-server', 'account-replicator', 'auth-server', 'account-reaper']
-GRACEFUL_SHUTDOWN_SERVERS = ['account-server', 'container-server',
-    'object-server', 'proxy-server', 'auth-server']
-MAX_DESCRIPTORS = 32768
-MAX_MEMORY = (1024 * 1024 * 1024) * 2 # 2 GB
-
-_junk, server, command = sys.argv
-if server == 'all':
-    servers = ALL_SERVERS
-else:
-    if '-' not in server:
-        server = '%s-server' % server
-    servers = [server]
-command = command.lower()
-
-def pid_files(server):
-    if os.path.exists('/var/run/swift/%s.pid' % server):
-        pid_files = ['/var/run/swift/%s.pid' % server]
-    else:
-        pid_files = glob.glob('/var/run/swift/%s/*.pid' % server)
-    for pid_file in pid_files:
-        pid = int(open(pid_file).read().strip())
-        yield pid_file, pid
-
-def do_start(server, once=False):
-    server_type = '-'.join(server.split('-')[:-1])
-
-    for pid_file, pid in pid_files(server):
-        if os.path.exists('/proc/%s' % pid):
-            print "%s appears to already be running: %s" % (server, pid_file)
-            return
-        else:
-            print "Removing stale pid file %s" % pid_file
-            os.unlink(pid_file)
-
+from optparse import OptionParser
+
+from swift.common.manager import Server, Manager, UnknownCommandError
+
+USAGE = """%prog <server> [<server> ...] <command> [options]
+
+Commands:
+""" + '\n'.join(["%16s: %s" % x for x in Manager.list_commands()])
+
+
+def main():
+    parser = OptionParser(USAGE)
+    parser.add_option('-v', '--verbose', action="store_true",
+                      default=False, help="display verbose output")
+    parser.add_option('-w', '--no-wait', action="store_false", dest="wait",
+                      default=True, help="won't wait for server to start "
+                      "before returning")
+    parser.add_option('-o', '--once', action="store_true",
+                      default=False, help="only run one pass of daemon")
+    # this is a negative option, default is options.daemon = True
+    parser.add_option('-n', '--no-daemon', action="store_false", dest="daemon",
+                      default=True, help="start server interactively")
+    parser.add_option('-g', '--graceful', action="store_true",
+                      default=False, help="send SIGHUP to supporting servers")
+    parser.add_option('-c', '--config-num', metavar="N", type="int",
+                      dest="number", default=0,
+                      help="send command to the Nth server only")
+    options, args = parser.parse_args()
+
+    if len(args) < 2:
+        parser.print_help()
+        print 'ERROR: specify server(s) and command'
+        return 1
+
+    command = args[-1]
+    servers = args[:-1]
+
+    # this is just a silly swap for me cause I always try to "start main"
+    commands = dict(Manager.list_commands()).keys()
+    if command not in commands and servers[0] in commands:
+        servers.append(command)
+        command = servers.pop(0)
+
+    manager = Manager(servers)
     try:
-        resource.setrlimit(resource.RLIMIT_NOFILE,
-                (MAX_DESCRIPTORS, MAX_DESCRIPTORS))
-        resource.setrlimit(resource.RLIMIT_DATA,
-                (MAX_MEMORY, MAX_MEMORY))
-    except ValueError:
-        print "Unable to increase file descriptor limit.  Running as non-root?"
-    os.environ['PYTHON_EGG_CACHE'] = '/tmp'
-
-    def write_pid_file(pid_file, pid):
-        dir, file = os.path.split(pid_file)
-        if not os.path.exists(dir):
-            try:
-                os.makedirs(dir)
-            except OSError, err:
-                if err.errno == errno.EACCES:
-                    sys.exit('Unable to create %s.  Running as non-root?' % dir)
-        fp = open(pid_file, 'w')
-        fp.write('%d\n' % pid)
-        fp.close()
-
-    def launch(ini_file, pid_file):
-        cmd = 'swift-%s' % server
-        args = [server, ini_file]
-        if once:
-            print 'Running %s once' % server
-            args.append('once')
-        else:
-            print 'Starting %s' % server
-
-        pid = os.fork()
-        if pid == 0:
-            os.setsid()
-            with open(os.devnull, 'r+b') as nullfile:
-                for desc in (0, 1, 2): # close stdio
-                    try:
-                        os.dup2(nullfile.fileno(), desc)
-                    except OSError:
-                        pass
-            try:
-                if once:
-                    os.execlp('swift-%s' % server, server,
-                        ini_file, 'once')
-                else:
-                    os.execlp('swift-%s' % server, server, ini_file)
-            except OSError:
-                print 'unable to launch %s' % server
-            sys.exit(0)
-        else:
-            write_pid_file(pid_file, pid)
-
-    ini_file = '/etc/swift/%s-server.conf' % server_type
-    if os.path.exists(ini_file):
-        # single config file over-rides config dirs
-        pid_file = '/var/run/swift/%s.pid' % server
-        launch_args = [(ini_file, pid_file)]
-    elif os.path.exists('/etc/swift/%s-server/' % server_type):
-        # found config directory, searching for config file(s)
-        launch_args = []
-        for num, ini_file in enumerate(glob.glob('/etc/swift/%s-server/*.conf' \
-                        % server_type)):
-            pid_file = '/var/run/swift/%s/%d.pid' % (server, num)
-            # start a server for each ini_file found
-            launch_args.append((ini_file, pid_file))
-    else:
-        # maybe there's a config file(s) out there, but I couldn't find it!
-        print 'Unable to locate config file for %s. %s does not exist?' % \
-                    (server, ini_file)
-        return
-
-    # start all servers
-    for ini_file, pid_file in launch_args:
-        launch(ini_file, pid_file)
-
-def do_stop(server, graceful=False):
-    if graceful and server in GRACEFUL_SHUTDOWN_SERVERS:
-        sig = signal.SIGHUP
-    else:
-        sig = signal.SIGTERM
-
-    did_anything = False
-    pfiles = pid_files(server)
-    for pid_file, pid in pfiles:
-        did_anything = True
-        try:
-            print 'Stopping %s  pid: %s  signal: %s' % (server, pid, sig)
-            os.kill(pid, sig)
-        except OSError:
-            print "Process %d not running" % pid
-        try:
-            os.unlink(pid_file)
-        except OSError:
-            pass
-    for pid_file, pid in pfiles:
-        for _junk in xrange(150):  # 15 seconds
-            if not os.path.exists('/proc/%s' % pid):
-                break
-            time.sleep(0.1)
-        else:
-            print 'Waited 15 seconds for pid %s (%s) to die; giving up' % \
-                  (pid, pid_file)
-    if not did_anything:
-        print 'No %s running' % server
-
-if command == 'start':
-    for server in servers:
-        do_start(server)
-
-if command == 'stop':
-    for server in servers:
-        do_stop(server)
-
-if command == 'shutdown':
-    for server in servers:
-        do_stop(server, graceful=True)
-
-if command == 'restart':
-    for server in servers:
-        do_stop(server)
-    for server in servers:
-        do_start(server)
-
-if command == 'reload' or command == 'force-reload':
-    for server in servers:
-        do_stop(server, graceful=True)
-        do_start(server)
-
-if command == 'once':
-    for server in servers:
-        do_start(server, once=True)
+        status = manager.run_command(command, **options.__dict__)
+    except UnknownCommandError:
+        parser.print_help()
+        print 'ERROR: unknown command, %s' % command
+        status = 1
+
+    return 1 if status else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

=== modified file 'doc/source/development_saio.rst'
--- doc/source/development_saio.rst	2011-01-19 22:05:22 +0000
+++ doc/source/development_saio.rst	2011-02-16 21:01:55 +0000
@@ -531,7 +531,6 @@
         #!/bin/bash
 
         swift-init all stop
-        sleep 5
         sudo umount /mnt/sdb1
         sudo mkfs.xfs -f -i size=1024 /dev/sdb1
         sudo mount /mnt/sdb1
@@ -573,12 +572,9 @@
 
         #!/bin/bash
 
+        swift-init main start
         # The auth-server line is only needed for DevAuth:
         swift-init auth-server start
-        swift-init proxy-server start
-        swift-init account-server start
-        swift-init container-server start
-        swift-init object-server start
 
   #. For Swauth (not needed for DevAuth), create `~/bin/recreateaccounts`::
   
@@ -600,15 +596,7 @@
         # /etc/swift/auth-server.conf). This swift-auth-recreate-accounts line
         # is only needed for DevAuth:
         swift-auth-recreate-accounts -K devauth
-        swift-init object-updater start
-        swift-init container-updater start
-        swift-init object-replicator start
-        swift-init container-replicator start
-        swift-init account-replicator start
-        swift-init object-auditor start
-        swift-init container-auditor start
-        swift-init account-auditor start
-        swift-init account-reaper start
+        swift-init rest start
 
   #. `chmod +x ~/bin/*`
   #. `remakerings`

=== modified file 'doc/source/misc.rst'
--- doc/source/misc.rst	2011-01-14 19:49:05 +0000
+++ doc/source/misc.rst	2011-02-16 21:01:55 +0000
@@ -116,6 +116,13 @@
     :members:
     :show-inheritance:
 
+Manager
+=========
+
+.. automodule:: swift.common.manager
+    :members:
+    :show-inheritance:
+
 Ratelimit
 =========
 

=== modified file 'swift/common/daemon.py'
--- swift/common/daemon.py	2011-02-02 21:39:08 +0000
+++ swift/common/daemon.py	2011-02-16 21:01:55 +0000
@@ -39,8 +39,8 @@
     def run(self, once=False, **kwargs):
         """Run the daemon"""
         utils.validate_configuration()
+        utils.drop_privileges(self.conf.get('user', 'swift'))
         utils.capture_stdio(self.logger, **kwargs)
-        utils.drop_privileges(self.conf.get('user', 'swift'))
 
         def kill_children(*args):
             signal.signal(signal.SIGTERM, signal.SIG_IGN)

=== added file 'swift/common/manager.py'
--- swift/common/manager.py	1970-01-01 00:00:00 +0000
+++ swift/common/manager.py	2011-02-16 21:01:55 +0000
@@ -0,0 +1,605 @@
+# Copyright (c) 2010-2011 OpenStack, LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import with_statement
+import functools
+import errno
+import os
+import resource
+import signal
+import sys
+import time
+import subprocess
+import re
+
+from swift.common.utils import search_tree, remove_file, write_file
+
+SWIFT_DIR = '/etc/swift'
+RUN_DIR = '/var/run/swift'
+
+# auth-server has been removed from ALL_SERVERS, start it explicitly
+ALL_SERVERS = ['account-auditor', 'account-server', 'container-auditor',
+    'container-replicator', 'container-server', 'container-updater',
+    'object-auditor', 'object-server', 'object-replicator', 'object-updater',
+    'proxy-server', 'account-replicator', 'account-reaper']
+MAIN_SERVERS = ['proxy-server', 'account-server', 'container-server',
+                'object-server']
+REST_SERVERS = [s for s in ALL_SERVERS if s not in MAIN_SERVERS]
+GRACEFUL_SHUTDOWN_SERVERS = MAIN_SERVERS + ['auth-server']
+START_ONCE_SERVERS = REST_SERVERS
+
+KILL_WAIT = 15  # seconds to wait for servers to die
+
+MAX_DESCRIPTORS = 32768
+MAX_MEMORY = (1024 * 1024 * 1024) * 2  # 2 GB
+
+
+def setup_env():
+    """Try to increase resource limits of the OS. Move PYTHON_EGG_CACHE to /tmp
+    """
+    try:
+        resource.setrlimit(resource.RLIMIT_NOFILE,
+                (MAX_DESCRIPTORS, MAX_DESCRIPTORS))
+        resource.setrlimit(resource.RLIMIT_DATA,
+                (MAX_MEMORY, MAX_MEMORY))
+    except ValueError:
+        print _("WARNING: Unable to increase file descriptor limit.  "
+                "Running as non-root?")
+
+    os.environ['PYTHON_EGG_CACHE'] = '/tmp'
+
+
+def command(func):
+    """
+    Decorator to declare which methods are accessible as commands, commands
+    always return 1 or 0, where 0 should indicate success.
+
+    :param func: function to make public
+    """
+    func.publicly_accessible = True
+
+    @functools.wraps(func)
+    def wrapped(*a, **kw):
+        rv = func(*a, **kw)
+        return 1 if rv else 0
+    return wrapped
+
+
+def watch_server_pids(server_pids, interval=1, **kwargs):
+    """Monitor a collection of server pids yeilding back those pids that
+    aren't responding to signals.
+
+    :param server_pids: a dict, lists of pids [int,...] keyed on
+                        Server objects
+    """
+    status = {}
+    start = time.time()
+    end = start + interval
+    server_pids = dict(server_pids)  # make a copy
+    while True:
+        for server, pids in server_pids.items():
+            for pid in pids:
+                try:
+                    # let pid stop if it wants to
+                    os.waitpid(pid, os.WNOHANG)
+                except OSError, e:
+                    if e.errno not in (errno.ECHILD, errno.ESRCH):
+                        raise  # else no such child/process
+            # check running pids for server
+            status[server] = server.get_running_pids(**kwargs)
+            for pid in pids:
+                # original pids no longer in running pids!
+                if pid not in status[server]:
+                    yield server, pid
+            # update active pids list using running_pids
+            server_pids[server] = status[server]
+        if not [p for server, pids in status.items() for p in pids]:
+            # no more running pids
+            break
+        if time.time() > end:
+            break
+        else:
+            time.sleep(0.1)
+
+
+class UnknownCommandError(Exception):
+    pass
+
+
+class Manager():
+    """Main class for performing commands on groups of servers.
+
+    :param servers: list of server names as strings
+
+    """
+
+    def __init__(self, servers):
+        server_names = set()
+        for server in servers:
+            if server == 'all':
+                server_names.update(ALL_SERVERS)
+            elif server == 'main':
+                server_names.update(MAIN_SERVERS)
+            elif server == 'rest':
+                server_names.update(REST_SERVERS)
+            elif '*' in server:
+                # convert glob to regex
+                server_names.update([s for s in ALL_SERVERS if
+                                     re.match(server.replace('*', '.*'), s)])
+            else:
+                server_names.add(server)
+
+        self.servers = set()
+        for name in server_names:
+            self.servers.add(Server(name))
+
+    @command
+    def status(self, **kwargs):
+        """display status of tracked pids for server
+        """
+        status = 0
+        for server in self.servers:
+            status += server.status(**kwargs)
+        return status
+
+    @command
+    def start(self, **kwargs):
+        """starts a server
+        """
+        setup_env()
+        status = 0
+
+        for server in self.servers:
+            server.launch(**kwargs)
+        if not kwargs.get('daemon', True):
+            for server in self.servers:
+                try:
+                    status += server.interact(**kwargs)
+                except KeyboardInterrupt:
+                    print _('\nuser quit')
+                    self.stop(**kwargs)
+                    break
+        elif kwargs.get('wait', True):
+            for server in self.servers:
+                status += server.wait(**kwargs)
+        return status
+
+    @command
+    def no_wait(self, **kwargs):
+        """spawn server and return immediately
+        """
+        kwargs['wait'] = False
+        return self.start(**kwargs)
+
+    @command
+    def no_daemon(self, **kwargs):
+        """start a server interactively
+        """
+        kwargs['daemon'] = False
+        return self.start(**kwargs)
+
+    @command
+    def once(self, **kwargs):
+        """start server and run one pass on supporting daemons
+        """
+        kwargs['once'] = True
+        return self.start(**kwargs)
+
+    @command
+    def stop(self, **kwargs):
+        """stops a server
+        """
+        server_pids = {}
+        for server in self.servers:
+            signaled_pids = server.stop(**kwargs)
+            if not signaled_pids:
+                print _('No %s running') % server
+            else:
+                server_pids[server] = signaled_pids
+
+        # all signaled_pids, i.e. list(itertools.chain(*server_pids.values()))
+        signaled_pids = [p for server, pids in server_pids.items()
+                         for p in pids]
+        # keep track of the pids yeiled back as killed for all servers
+        killed_pids = set()
+        for server, killed_pid in watch_server_pids(server_pids,
+                                                interval=KILL_WAIT, **kwargs):
+            print _("%s (%s) appears to have stopped") % (server, killed_pid)
+            killed_pids.add(killed_pid)
+            if not killed_pids.symmetric_difference(signaled_pids):
+                # all proccesses have been stopped
+                return 0
+
+        # reached interval n watch_pids w/o killing all servers
+        for server, pids in server_pids.items():
+            if not killed_pids.issuperset(pids):
+                # some pids of this server were not killed
+                print _('Waited %s seconds for %s to die; giving up') % (
+                    KILL_WAIT, server)
+        return 1
+
+    @command
+    def shutdown(self, **kwargs):
+        """allow current requests to finish on supporting servers
+        """
+        kwargs['graceful'] = True
+        status = 0
+        status += self.stop(**kwargs)
+        return status
+
+    @command
+    def restart(self, **kwargs):
+        """stops then restarts server
+        """
+        status = 0
+        status += self.stop(**kwargs)
+        status += self.start(**kwargs)
+        return status
+
+    @command
+    def reload(self, **kwargs):
+        """graceful shutdown then restart on supporting servers
+        """
+        kwargs['graceful'] = True
+        status = 0
+        for server in self.servers:
+            m = Manager([server.server])
+            status += m.stop(**kwargs)
+            status += m.start(**kwargs)
+        return status
+
+    @command
+    def force_reload(self, **kwargs):
+        """alias for reload
+        """
+        return self.reload(**kwargs)
+
+    def get_command(self, cmd):
+        """Find and return the decorated method named like cmd
+
+        :param cmd: the command to get, a string, if not found raises
+                    UnknownCommandError
+
+        """
+        cmd = cmd.lower().replace('-', '_')
+        try:
+            f = getattr(self, cmd)
+        except AttributeError:
+            raise UnknownCommandError(cmd)
+        if not hasattr(f, 'publicly_accessible'):
+            raise UnknownCommandError(cmd)
+        return f
+
+    @classmethod
+    def list_commands(cls):
+        """Get all publicly accessible commands
+
+        :returns: a list of string tuples (cmd, help), the method names who are
+                  decorated as commands
+        """
+        get_method = lambda cmd: getattr(cls, cmd)
+        return sorted([(x.replace('_', '-'), get_method(x).__doc__.strip())
+                       for x in dir(cls) if
+                       getattr(get_method(x), 'publicly_accessible', False)])
+
+    def run_command(self, cmd, **kwargs):
+        """Find the named command and run it
+
+        :param cmd: the command name to run
+
+        """
+        f = self.get_command(cmd)
+        return f(**kwargs)
+
+
+class Server():
+    """Manage operations on a server or group of servers of similar type
+
+    :param server: name of server
+    """
+
+    def __init__(self, server):
+        if '-' not in server:
+            server = '%s-server' % server
+        self.server = server.lower()
+        self.type = server.rsplit('-', 1)[0]
+        self.cmd = 'swift-%s' % server
+        self.procs = []
+
+    def __str__(self):
+        return self.server
+
+    def __repr__(self):
+        return "%s(%s)" % (self.__class__.__name__, repr(str(self)))
+
+    def __hash__(self):
+        return hash(str(self))
+
+    def __eq__(self, other):
+        try:
+            return self.server == other.server
+        except AttributeError:
+            return False
+
+    def get_pid_file_name(self, conf_file):
+        """Translate conf_file to a corresponding pid_file
+
+        :param conf_file: an conf_file for this server, a string
+
+        :returns: the pid_file for this conf_file
+
+        """
+        return conf_file.replace(
+            os.path.normpath(SWIFT_DIR), RUN_DIR, 1).replace(
+                '%s-server' % self.type, self.server, 1).rsplit(
+                    '.conf', 1)[0] + '.pid'
+
+    def get_conf_file_name(self, pid_file):
+        """Translate pid_file to a corresponding conf_file
+
+        :param pid_file: a pid_file for this server, a string
+
+        :returns: the conf_file for this pid_file
+
+        """
+        return pid_file.replace(
+            os.path.normpath(RUN_DIR), SWIFT_DIR, 1).replace(
+                self.server, '%s-server' % self.type, 1).rsplit(
+                    '.pid', 1)[0] + '.conf'
+
+    def conf_files(self, **kwargs):
+        """Get conf files for this server
+
+        :param: number, if supplied will only lookup the nth server
+
+        :returns: list of conf files
+        """
+        found_conf_files = search_tree(SWIFT_DIR, '%s-server*' % self.type,
+                                      '.conf')
+        number = kwargs.get('number')
+        if number:
+            try:
+                conf_files = [found_conf_files[number - 1]]
+            except IndexError:
+                conf_files = []
+        else:
+            conf_files = found_conf_files
+        if not conf_files:
+            # maybe there's a config file(s) out there, but I couldn't find it!
+            if not kwargs.get('quiet'):
+                print _('Unable to locate config %sfor %s') % (
+                    ('number %s ' % number if number else ''), self.server)
+            if kwargs.get('verbose') and not kwargs.get('quiet'):
+                if found_conf_files:
+                    print _('Found configs:')
+                for i, conf_file in enumerate(found_conf_files):
+                    print '  %d) %s' % (i + 1, conf_file)
+
+        return conf_files
+
+    def pid_files(self, **kwargs):
+        """Get pid files for this server
+
+        :param: number, if supplied will only lookup the nth server
+
+        :returns: list of pid files
+        """
+        pid_files = search_tree(RUN_DIR, '%s*' % self.server, '.pid')
+        if kwargs.get('number', 0):
+            conf_files = self.conf_files(**kwargs)
+            # filter pid_files to match the index of numbered conf_file
+            pid_files = [pid_file for pid_file in pid_files if
+                         self.get_conf_file_name(pid_file) in conf_files]
+        return pid_files
+
+    def iter_pid_files(self, **kwargs):
+        """Generator, yields (pid_file, pids)
+        """
+        for pid_file in self.pid_files(**kwargs):
+            yield pid_file, int(open(pid_file).read().strip())
+
+    def signal_pids(self, sig, **kwargs):
+        """Send a signal to pids for this server
+
+        :param sig: signal to send
+
+        :returns: a dict mapping pids (ints) to pid_files (paths)
+
+        """
+        pids = {}
+        for pid_file, pid in self.iter_pid_files(**kwargs):
+            try:
+                if sig != signal.SIG_DFL:
+                    print _('Signal %s  pid: %s  signal: %s') % (self.server,
+                                                                 pid, sig)
+                os.kill(pid, sig)
+            except OSError, e:
+                if e.errno == errno.ESRCH:
+                    # pid does not exist
+                    if kwargs.get('verbose'):
+                        print _("Removing stale pid file %s") % pid_file
+                    remove_file(pid_file)
+            else:
+                # process exists
+                pids[pid] = pid_file
+        return pids
+
+    def get_running_pids(self, **kwargs):
+        """Get running pids
+
+        :returns: a dict mapping pids (ints) to pid_files (paths)
+
+        """
+        return self.signal_pids(signal.SIG_DFL, **kwargs)  # send noop
+
+    def kill_running_pids(self, **kwargs):
+        """Kill running pids
+
+        :param graceful: if True, attempt SIGHUP on supporting servers
+
+        :returns: a dict mapping pids (ints) to pid_files (paths)
+
+        """
+        graceful = kwargs.get('graceful')
+        if graceful and self.server in GRACEFUL_SHUTDOWN_SERVERS:
+            sig = signal.SIGHUP
+        else:
+            sig = signal.SIGTERM
+        return self.signal_pids(sig, **kwargs)
+
+    def status(self, pids=None, **kwargs):
+        """Display status of server
+
+        :param: pids, if not supplied pids will be populated automatically
+        :param: number, if supplied will only lookup the nth server
+
+        :returns: 1 if server is not running, 0 otherwise
+        """
+        if pids is None:
+            pids = self.get_running_pids(**kwargs)
+        if not pids:
+            number = kwargs.get('number', 0)
+            if number:
+                kwargs['quiet'] = True
+                conf_files = self.conf_files(**kwargs)
+                if conf_files:
+                    print _("%s #%d not running (%s)") % (self.server, number,
+                                                          conf_files[0])
+            else:
+                print _("No %s running") % self.server
+            return 1
+        for pid, pid_file in pids.items():
+            conf_file = self.get_conf_file_name(pid_file)
+            print _("%s running (%s - %s)") % (self.server, pid, conf_file)
+        return 0
+
+    def spawn(self, conf_file, once=False, wait=True, daemon=True, **kwargs):
+        """Launch a subprocess for this server.
+
+        :param conf_file: path to conf_file to use as first arg
+        :param once: boolean, add once argument to command
+        :param wait: boolean, if true capture stdout with a pipe
+        :param daemon: boolean, if true ask server to log to console
+
+        :returns : the pid of the spawned process
+        """
+        args = [self.cmd, conf_file]
+        if once:
+            args.append('once')
+        if not daemon:
+            # ask the server to log to console
+            args.append('verbose')
+
+        # figure out what we're going to do with stdio
+        if not daemon:
+            # do nothing, this process is open until the spawns close anyway
+            re_out = None
+            re_err = None
+        else:
+            re_err = subprocess.STDOUT
+            if wait:
+                # we're going to need to block on this...
+                re_out = subprocess.PIPE
+            else:
+                re_out = open(os.devnull, 'w+b')
+        proc = subprocess.Popen(args, stdout=re_out, stderr=re_err)
+        pid_file = self.get_pid_file_name(conf_file)
+        write_file(pid_file, proc.pid)
+        self.procs.append(proc)
+        return proc.pid
+
+    def wait(self, **kwargs):
+        """
+        wait on spawned procs to start
+        """
+        status = 0
+        for proc in self.procs:
+            # wait for process to close it's stdout
+            output = proc.stdout.read()
+            if output:
+                print output
+                proc.communicate()
+            if proc.returncode:
+                status += 1
+        return status
+
+    def interact(self, **kwargs):
+        """
+        wait on spawned procs to terminate
+        """
+        status = 0
+        for proc in self.procs:
+            # wait for process to terminate
+            proc.communicate()
+            if proc.returncode:
+                status += 1
+        return status
+
+    def launch(self, **kwargs):
+        """
+        Collect conf files and attempt to spawn the processes for this server
+        """
+        conf_files = self.conf_files(**kwargs)
+        if not conf_files:
+            return []
+
+        pids = self.get_running_pids(**kwargs)
+
+        already_started = False
+        for pid, pid_file in pids.items():
+            conf_file = self.get_conf_file_name(pid_file)
+            # for legacy compat you can't start other servers if one server is
+            # already running (unless -n specifies which one you want), this
+            # restriction could potentially be lifted, and launch could start
+            # any unstarted instances
+            if conf_file in conf_files:
+                already_started = True
+                print _("%s running (%s - %s)") % (self.server, pid, conf_file)
+            elif not kwargs.get('number', 0):
+                already_started = True
+                print _("%s running (%s - %s)") % (self.server, pid, pid_file)
+
+        if already_started:
+            print _("%s already started...") % self.server
+            return []
+
+        if self.server not in START_ONCE_SERVERS:
+            kwargs['once'] = False
+
+        pids = {}
+        for conf_file in conf_files:
+            if kwargs.get('once'):
+                msg = _('Running %s once') % self.server
+            else:
+                msg = _('Starting %s') % self.server
+            print '%s...(%s)' % (msg, conf_file)
+            try:
+                pid = self.spawn(conf_file, **kwargs)
+            except OSError, e:
+                if e.errno == errno.ENOENT:
+                    # TODO: should I check if self.cmd exists earlier?
+                    print _("%s does not exist") % self.cmd
+                    break
+            pids[pid] = conf_file
+
+        return pids
+
+    def stop(self, **kwargs):
+        """Send stop signals to pids for this server
+
+        :returns: a dict mapping pids (ints) to pid_files (paths)
+
+        """
+        return self.kill_running_pids(**kwargs)

=== modified file 'swift/common/utils.py'
--- swift/common/utils.py	2011-02-11 17:27:05 +0000
+++ swift/common/utils.py	2011-02-16 21:01:55 +0000
@@ -34,6 +34,7 @@
 from optparse import OptionParser
 from tempfile import mkstemp
 import cPickle as pickle
+import glob
 from urlparse import urlparse as stdlib_urlparse, ParseResult
 
 import eventlet
@@ -471,7 +472,10 @@
     stdio_fds = [0, 1, 2]
     for _junk, handler in getattr(get_logger,
                                   'console_handler4logger', {}).items():
-        stdio_fds.remove(handler.stream.fileno())
+        try:
+            stdio_fds.remove(handler.stream.fileno())
+        except ValueError:
+            pass  # fd not in list
 
     with open(os.devnull, 'r+b') as nullfile:
         # close stdio (excludes fds open for logging)
@@ -786,6 +790,60 @@
         renamer(tmppath, dest)
 
 
+def search_tree(root, glob_match, ext):
+    """Look in root, for any files/dirs matching glob, recurively traversing
+    any found directories looking for files ending with ext
+
+    :param root: start of search path
+    :param glob_match: glob to match in root, matching dirs are traversed with
+                       os.walk
+    :param ext: only files that end in ext will be returned
+
+    :returns: list of full paths to matching files, sorted
+
+    """
+    found_files = []
+    for path in glob.glob(os.path.join(root, glob_match)):
+        if path.endswith(ext):
+            found_files.append(path)
+        else:
+            for root, dirs, files in os.walk(path):
+                for file in files:
+                    if file.endswith(ext):
+                        found_files.append(os.path.join(root, file))
+    return sorted(found_files)
+
+
+def write_file(path, contents):
+    """Write contents to file at path
+
+    :param path: any path, subdirs will be created as needed
+    :param contents: data to write to file, will be converted to string
+
+    """
+    dirname, name = os.path.split(path)
+    if not os.path.exists(dirname):
+        try:
+            os.makedirs(dirname)
+        except OSError, err:
+            if err.errno == errno.EACCES:
+                sys.exit('Unable to create %s.  Running as '
+                         'non-root?' % dirname)
+    with open(path, 'w') as f:
+        f.write('%s' % contents)
+
+
+def remove_file(path):
+    """Quiet wrapper for os.unlink, OSErrors are suppressed
+
+    :param path: first and only argument passed to os.unlink
+    """
+    try:
+        os.unlink(path)
+    except OSError:
+        pass
+
+
 def audit_location_generator(devices, datadir, mount_check=True, logger=None):
     '''
     Given a devices path and a data directory, yield (path, device,

=== modified file 'swift/common/wsgi.py'
--- swift/common/wsgi.py	2011-02-11 17:27:05 +0000
+++ swift/common/wsgi.py	2011-02-16 21:01:55 +0000
@@ -119,8 +119,6 @@
         logger = get_logger(conf, log_name,
             log_to_console=kwargs.pop('verbose', False), log_route='wsgi')
 
-    # redirect errors to logger and close stdio
-    capture_stdio(logger)
     # bind to address and port
     sock = get_socket(conf, default_port=kwargs.get('default_port', 8080))
     # remaining tasks should not require elevated privileges
@@ -129,6 +127,9 @@
     # finally after binding to ports and privilege drop, run app __init__ code
     app = loadapp('config:%s' % conf_file, global_conf={'log_name': log_name})
 
+    # redirect errors to logger and close stdio
+    capture_stdio(logger)
+
     def run_server():
         wsgi.HttpProtocol.default_request_version = "HTTP/1.0"
         eventlet.hubs.use_hub('poll')

=== modified file 'test/unit/__init__.py'
--- test/unit/__init__.py	2011-01-19 22:19:43 +0000
+++ test/unit/__init__.py	2011-02-16 21:01:55 +0000
@@ -4,6 +4,8 @@
 from contextlib import contextmanager
 from tempfile import NamedTemporaryFile
 from eventlet.green import socket
+from tempfile import mkdtemp
+from shutil import rmtree
 
 
 def readuntil2crlfs(fd):
@@ -68,6 +70,27 @@
 xattr.getxattr = _getxattr
 
 
+@contextmanager
+def temptree(files, contents=''):
+    # generate enough contents to fill the files
+    c = len(files)
+    contents = (list(contents) + [''] * c)[:c]
+    tempdir = mkdtemp()
+    for path, content in zip(files, contents):
+        if os.path.isabs(path):
+            path = '.' + path
+        new_path = os.path.join(tempdir, path)
+        subdir = os.path.dirname(new_path)
+        if not os.path.exists(subdir):
+            os.makedirs(subdir)
+        with open(new_path, 'w') as f:
+            f.write(str(content))
+    try:
+        yield tempdir
+    finally:
+        rmtree(tempdir)
+
+
 class MockTrue(object):
     """
     Instances of MockTrue evaluate like True

=== added file 'test/unit/common/test_manager.py'
--- test/unit/common/test_manager.py	1970-01-01 00:00:00 +0000
+++ test/unit/common/test_manager.py	2011-02-16 21:01:55 +0000
@@ -0,0 +1,1636 @@
+# Copyright (c) 2010-2011 OpenStack, LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+# implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+from nose import SkipTest
+from test.unit import temptree
+
+import os
+import sys
+import resource
+import signal
+import errno
+from contextlib import contextmanager
+from collections import defaultdict
+from threading import Thread
+from time import sleep, time
+
+from swift.common import manager
+
+DUMMY_SIG = 1
+
+
+class MockOs():
+
+    def __init__(self, pids):
+        self.running_pids = pids
+        self.pid_sigs = defaultdict(list)
+        self.closed_fds = []
+        self.child_pid = 9999  # fork defaults to test parent process path
+        self.execlp_called = False
+
+    def kill(self, pid, sig):
+        if pid not in self.running_pids:
+            raise OSError(3, 'No such process')
+        self.pid_sigs[pid].append(sig)
+
+    def __getattr__(self, name):
+        # I only over-ride portions of the os module
+        try:
+            return object.__getattr__(self, name)
+        except AttributeError:
+            return getattr(os, name)
+
+
+def pop_stream(f):
+    """read everything out of file from the top and clear it out
+    """
+    f.flush()
+    f.seek(0)
+    output = f.read()
+    f.seek(0)
+    f.truncate()
+    #print >> sys.stderr, output
+    return output
+
+
+class TestManagerModule(unittest.TestCase):
+
+    def test_servers(self):
+        main_plus_rest = set(manager.MAIN_SERVERS + manager.REST_SERVERS)
+        self.assertEquals(set(manager.ALL_SERVERS), main_plus_rest)
+        # make sure there's no server listed in both
+        self.assertEquals(len(main_plus_rest), len(manager.MAIN_SERVERS) +
+                          len(manager.REST_SERVERS))
+
+    def test_setup_env(self):
+        class MockResource():
+            def __init__(self, error=None):
+                self.error = error
+                self.called_with_args = []
+
+            def setrlimit(self, resource, limits):
+                if self.error:
+                    raise self.error
+                self.called_with_args.append((resource, limits))
+
+            def __getattr__(self, name):
+                # I only over-ride portions of the resource module
+                try:
+                    return object.__getattr__(self, name)
+                except AttributeError:
+                    return getattr(resource, name)
+
+        _orig_resource = manager.resource
+        _orig_environ = os.environ
+        try:
+            manager.resource = MockResource()
+            manager.os.environ = {}
+            manager.setup_env()
+            expected = [
+                (resource.RLIMIT_NOFILE, (manager.MAX_DESCRIPTORS,
+                                          manager.MAX_DESCRIPTORS)),
+                (resource.RLIMIT_DATA, (manager.MAX_MEMORY,
+                                        manager.MAX_MEMORY)),
+            ]
+            self.assertEquals(manager.resource.called_with_args, expected)
+            self.assertEquals(manager.os.environ['PYTHON_EGG_CACHE'], '/tmp')
+
+            # test error condition
+            manager.resource = MockResource(error=ValueError())
+            manager.os.environ = {}
+            manager.setup_env()
+            self.assertEquals(manager.resource.called_with_args, [])
+            self.assertEquals(manager.os.environ['PYTHON_EGG_CACHE'], '/tmp')
+
+            manager.resource = MockResource(error=OSError())
+            manager.os.environ = {}
+            self.assertRaises(OSError, manager.setup_env)
+            self.assertEquals(manager.os.environ.get('PYTHON_EGG_CACHE'), None)
+        finally:
+            manager.resource = _orig_resource
+            os.environ = _orig_environ
+
+    def test_command_wrapper(self):
+        @manager.command
+        def myfunc(arg1):
+            """test doc
+            """
+            return arg1
+
+        self.assertEquals(myfunc.__doc__.strip(), 'test doc')
+        self.assertEquals(myfunc(1), 1)
+        self.assertEquals(myfunc(0), 0)
+        self.assertEquals(myfunc(True), 1)
+        self.assertEquals(myfunc(False), 0)
+        self.assert_(hasattr(myfunc, 'publicly_accessible'))
+        self.assert_(myfunc.publicly_accessible)
+
+    def test_watch_server_pids(self):
+        class MockOs():
+            WNOHANG = os.WNOHANG
+
+            def __init__(self, pid_map={}):
+                self.pid_map = {}
+                for pid, v in pid_map.items():
+                    self.pid_map[pid] = (x for x in v)
+
+            def waitpid(self, pid, options):
+                try:
+                    rv = self.pid_map[pid].next()
+                except StopIteration:
+                    raise OSError(errno.ECHILD, os.strerror(errno.ECHILD))
+                except KeyError:
+                    raise OSError(errno.ESRCH, os.strerror(errno.ESRCH))
+                if isinstance(rv, Exception):
+                    raise rv
+                else:
+                    return rv
+
+        class MockTime():
+            def __init__(self, ticks=None):
+                self.tock = time()
+                if not ticks:
+                    ticks = []
+
+                self.ticks = (t for t in ticks)
+
+            def time(self):
+                try:
+                    self.tock += self.ticks.next()
+                except StopIteration:
+                    self.tock += 1
+                return self.tock
+
+            def sleep(*args):
+                return
+
+        class MockServer():
+
+            def __init__(self, pids, zombie=0):
+                self.heartbeat = (pids for _ in range(zombie))
+
+            def get_running_pids(self):
+                try:
+                    rv = self.heartbeat.next()
+                    return rv
+                except StopIteration:
+                    return {}
+
+        _orig_os = manager.os
+        _orig_time = manager.time
+        _orig_server = manager.Server
+        try:
+            manager.time = MockTime()
+            manager.os = MockOs()
+            # this server always says it's dead when you ask for running pids
+            server = MockServer([1])
+            # list of pids keyed on servers to watch
+            server_pids = {
+                server: [1],
+            }
+            # basic test, server dies
+            gen = manager.watch_server_pids(server_pids)
+            expected = [(server, 1)]
+            self.assertEquals([x for x in gen], expected)
+            # start long running server and short interval
+            server = MockServer([1], zombie=15)
+            server_pids = {
+                server: [1],
+            }
+            gen = manager.watch_server_pids(server_pids)
+            self.assertEquals([x for x in gen], [])
+            # wait a little longer
+            gen = manager.watch_server_pids(server_pids, interval=15)
+            self.assertEquals([x for x in gen], [(server, 1)])
+            # zombie process
+            server = MockServer([1], zombie=200)
+            server_pids = {
+                server: [1],
+            }
+            # test weird os error
+            manager.os = MockOs({1: [OSError()]})
+            gen = manager.watch_server_pids(server_pids)
+            self.assertRaises(OSError, lambda: [x for x in gen])
+            # test multi-server
+            server1 = MockServer([1, 10], zombie=200)
+            server2 = MockServer([2, 20], zombie=8)
+            server_pids = {
+                server1: [1, 10],
+                server2: [2, 20],
+            }
+            pid_map = {
+                1: [None for _ in range(10)],
+                2: [None for _ in range(8)],
+                20: [None for _ in range(4)],
+            }
+            manager.os = MockOs(pid_map)
+            gen = manager.watch_server_pids(server_pids,
+                                            interval=manager.KILL_WAIT)
+            expected = [
+                (server2, 2),
+                (server2, 20),
+            ]
+            self.assertEquals([x for x in gen], expected)
+
+        finally:
+            manager.os = _orig_os
+            manager.time = _orig_time
+            manager.Server = _orig_server
+
+    def test_exc(self):
+        self.assert_(issubclass(manager.UnknownCommandError, Exception))
+
+
+class TestServer(unittest.TestCase):
+
+    def tearDown(self):
+        reload(manager)
+
+    def join_swift_dir(self, path):
+        return os.path.join(manager.SWIFT_DIR, path)
+
+    def join_run_dir(self, path):
+        return os.path.join(manager.RUN_DIR, path)
+
+    def test_create_server(self):
+        server = manager.Server('proxy')
+        self.assertEquals(server.server, 'proxy-server')
+        self.assertEquals(server.type, 'proxy')
+        self.assertEquals(server.cmd, 'swift-proxy-server')
+        server = manager.Server('object-replicator')
+        self.assertEquals(server.server, 'object-replicator')
+        self.assertEquals(server.type, 'object')
+        self.assertEquals(server.cmd, 'swift-object-replicator')
+
+    def test_server_to_string(self):
+        server = manager.Server('Proxy')
+        self.assertEquals(str(server), 'proxy-server')
+        server = manager.Server('object-replicator')
+        self.assertEquals(str(server), 'object-replicator')
+
+    def test_server_repr(self):
+        server = manager.Server('proxy')
+        self.assert_(server.__class__.__name__ in repr(server))
+        self.assert_(str(server) in repr(server))
+
+    def test_server_equality(self):
+        server1 = manager.Server('Proxy')
+        server2 = manager.Server('proxy-server')
+        self.assertEquals(server1, server2)
+        # it is NOT a string
+        self.assertNotEquals(server1, 'proxy-server')
+
+    def test_get_pid_file_name(self):
+        server = manager.Server('proxy')
+        conf_file = self.join_swift_dir('proxy-server.conf')
+        pid_file = self.join_run_dir('proxy-server.pid')
+        self.assertEquals(pid_file, server.get_pid_file_name(conf_file))
+        server = manager.Server('object-replicator')
+        conf_file = self.join_swift_dir('object-server/1.conf')
+        pid_file = self.join_run_dir('object-replicator/1.pid')
+        self.assertEquals(pid_file, server.get_pid_file_name(conf_file))
+        server = manager.Server('container-auditor')
+        conf_file = self.join_swift_dir(
+            'container-server/1/container-auditor.conf')
+        pid_file = self.join_run_dir(
+            'container-auditor/1/container-auditor.pid')
+        self.assertEquals(pid_file, server.get_pid_file_name(conf_file))
+
+    def test_get_conf_file_name(self):
+        server = manager.Server('proxy')
+        conf_file = self.join_swift_dir('proxy-server.conf')
+        pid_file = self.join_run_dir('proxy-server.pid')
+        self.assertEquals(conf_file, server.get_conf_file_name(pid_file))
+        server = manager.Server('object-replicator')
+        conf_file = self.join_swift_dir('object-server/1.conf')
+        pid_file = self.join_run_dir('object-replicator/1.pid')
+        self.assertEquals(conf_file, server.get_conf_file_name(pid_file))
+        server = manager.Server('container-auditor')
+        conf_file = self.join_swift_dir(
+            'container-server/1/container-auditor.conf')
+        pid_file = self.join_run_dir(
+            'container-auditor/1/container-auditor.pid')
+        self.assertEquals(conf_file, server.get_conf_file_name(pid_file))
+
+    def test_conf_files(self):
+        # test get single conf file
+        conf_files = (
+            'proxy-server.conf',
+            'proxy-server.ini',
+            'auth-server.conf',
+        )
+        with temptree(conf_files) as t:
+            manager.SWIFT_DIR = t
+            server = manager.Server('proxy')
+            conf_files = server.conf_files()
+            self.assertEquals(len(conf_files), 1)
+            conf_file = conf_files[0]
+            proxy_conf = self.join_swift_dir('proxy-server.conf')
+            self.assertEquals(conf_file, proxy_conf)
+
+        # test multi server conf files & grouping of server-type config
+        conf_files = (
+            'object-server1.conf',
+            'object-server/2.conf',
+            'object-server/object3.conf',
+            'object-server/conf/server4.conf',
+            'object-server.txt',
+            'proxy-server.conf',
+        )
+        with temptree(conf_files) as t:
+            manager.SWIFT_DIR = t
+            server = manager.Server('object-replicator')
+            conf_files = server.conf_files()
+            self.assertEquals(len(conf_files), 4)
+            c1 = self.join_swift_dir('object-server1.conf')
+            c2 = self.join_swift_dir('object-server/2.conf')
+            c3 = self.join_swift_dir('object-server/object3.conf')
+            c4 = self.join_swift_dir('object-server/conf/server4.conf')
+            for c in [c1, c2, c3, c4]:
+                self.assert_(c in conf_files)
+            # test configs returned sorted
+            sorted_confs = sorted([c1, c2, c3, c4])
+            self.assertEquals(conf_files, sorted_confs)
+
+        # test get single numbered conf
+        conf_files = (
+            'account-server/1.conf',
+            'account-server/2.conf',
+            'account-server/3.conf',
+            'account-server/4.conf',
+        )
+        with temptree(conf_files) as t:
+            manager.SWIFT_DIR = t
+            server = manager.Server('account')
+            conf_files = server.conf_files(number=2)
+            self.assertEquals(len(conf_files), 1)
+            conf_file = conf_files[0]
+            self.assertEquals(conf_file,
+                              self.join_swift_dir('account-server/2.conf'))
+            # test missing config number
+            conf_files = server.conf_files(number=5)
+            self.assertFalse(conf_files)
+
+        # test verbose & quiet
+        conf_files = (
+            'auth-server.ini',
+            'container-server/1.conf',
+        )
+        with temptree(conf_files) as t:
+            manager.SWIFT_DIR = t
+            old_stdout = sys.stdout
+            try:
+                with open(os.path.join(t, 'output'), 'w+') as f:
+                    sys.stdout = f
+                    server = manager.Server('auth')
+                    # check warn "unable to locate"
+                    conf_files = server.conf_files()
+                    self.assertFalse(conf_files)
+                    self.assert_('unable to locate' in pop_stream(f).lower())
+                    # check quiet will silence warning
+                    conf_files = server.conf_files(verbose=True, quiet=True)
+                    self.assertEquals(pop_stream(f), '')
+                    # check found config no warning
+                    server = manager.Server('container-auditor')
+                    conf_files = server.conf_files()
+                    self.assertEquals(pop_stream(f), '')
+                    # check missing config number warn "unable to locate"
+                    conf_files = server.conf_files(number=2)
+                    self.assert_('unable to locate' in pop_stream(f).lower())
+                    # check verbose lists configs
+                    conf_files = server.conf_files(number=2, verbose=True)
+                    c1 = self.join_swift_dir('container-server/1.conf')
+                    self.assert_(c1 in pop_stream(f))
+            finally:
+                sys.stdout = old_stdout
+
+    def test_iter_pid_files(self):
+        """
+        Server.iter_pid_files is kinda boring, test the
+        Server.pid_files stuff here as well
+        """
+        pid_files = (
+            ('proxy-server.pid', 1),
+            ('auth-server.pid', 'blah'),
+            ('object-replicator/1.pid', 11),
+            ('object-replicator/2.pid', 12),
+        )
+        files, contents = zip(*pid_files)
+        with temptree(files, contents) as t:
+            manager.RUN_DIR = t
+            server = manager.Server('proxy')
+            # test get one file
+            iter = server.iter_pid_files()
+            pid_file, pid = iter.next()
+            self.assertEquals(pid_file, self.join_run_dir('proxy-server.pid'))
+            self.assertEquals(pid, 1)
+            # ... and only one file
+            self.assertRaises(StopIteration, iter.next)
+            # test invalid value in pid file
+            server = manager.Server('auth')
+            self.assertRaises(ValueError, server.iter_pid_files().next)
+            # test object-server doesn't steal pids from object-replicator
+            server = manager.Server('object')
+            self.assertRaises(StopIteration, server.iter_pid_files().next)
+            # test multi-pid iter
+            server = manager.Server('object-replicator')
+            real_map = {
+                11: self.join_run_dir('object-replicator/1.pid'),
+                12: self.join_run_dir('object-replicator/2.pid'),
+            }
+            pid_map = {}
+            for pid_file, pid in server.iter_pid_files():
+                pid_map[pid] = pid_file
+            self.assertEquals(pid_map, real_map)
+
+        # test get pid_files by number
+        conf_files = (
+            'object-server/1.conf',
+            'object-server/2.conf',
+            'object-server/3.conf',
+            'object-server/4.conf',
+        )
+
+        pid_files = (
+            ('object-server/1.pid', 1),
+            ('object-server/2.pid', 2),
+            ('object-server/5.pid', 5),
+        )
+
+        with temptree(conf_files) as swift_dir:
+            manager.SWIFT_DIR = swift_dir
+            files, pids = zip(*pid_files)
+            with temptree(files, pids) as t:
+                manager.RUN_DIR = t
+                server = manager.Server('object')
+                # test get all pid files
+                real_map = {
+                    1: self.join_run_dir('object-server/1.pid'),
+                    2: self.join_run_dir('object-server/2.pid'),
+                    5: self.join_run_dir('object-server/5.pid'),
+                }
+                pid_map = {}
+                for pid_file, pid in server.iter_pid_files():
+                    pid_map[pid] = pid_file
+                self.assertEquals(pid_map, real_map)
+                # test get pid with matching conf
+                pids = list(server.iter_pid_files(number=2))
+                self.assertEquals(len(pids), 1)
+                pid_file, pid = pids[0]
+                self.assertEquals(pid, 2)
+                pid_two = self.join_run_dir('object-server/2.pid')
+                self.assertEquals(pid_file, pid_two)
+                # try to iter on a pid number with a matching conf but no pid
+                pids = list(server.iter_pid_files(number=3))
+                self.assertFalse(pids)
+                # test get pids w/o matching conf
+                pids = list(server.iter_pid_files(number=5))
+                self.assertFalse(pids)
+
+    def test_signal_pids(self):
+        pid_files = (
+            ('proxy-server.pid', 1),
+            ('auth-server.pid', 2),
+        )
+        files, pids = zip(*pid_files)
+        with temptree(files, pids) as t:
+            manager.RUN_DIR = t
+            # mock os with both pids running
+            manager.os = MockOs([1, 2])
+            server = manager.Server('proxy')
+            pids = server.signal_pids(DUMMY_SIG)
+            self.assertEquals(len(pids), 1)
+            self.assert_(1 in pids)
+            self.assertEquals(manager.os.pid_sigs[1], [DUMMY_SIG])
+            # make sure other process not signaled
+            self.assertFalse(2 in pids)
+            self.assertFalse(2 in manager.os.pid_sigs)
+            # capture stdio
+            old_stdout = sys.stdout
+            try:
+                with open(os.path.join(t, 'output'), 'w+') as f:
+                    sys.stdout = f
+                    #test print details
+                    pids = server.signal_pids(DUMMY_SIG)
+                    output = pop_stream(f)
+                    self.assert_('pid: %s' % 1 in output)
+                    self.assert_('signal: %s' % DUMMY_SIG in output)
+                    # test no details on signal.SIG_DFL
+                    pids = server.signal_pids(signal.SIG_DFL)
+                    self.assertEquals(pop_stream(f), '')
+                    # reset mock os so only the other server is running
+                    manager.os = MockOs([2])
+                    # test pid not running
+                    pids = server.signal_pids(signal.SIG_DFL)
+                    self.assert_(1 not in pids)
+                    self.assert_(1 not in manager.os.pid_sigs)
+                    # test remove stale pid file
+                    self.assertFalse(os.path.exists(
+                        self.join_run_dir('proxy-server.pid')))
+                    # reset mock os with no running pids
+                    manager.os = MockOs([])
+                    server = manager.Server('auth')
+                    # test verbose warns on removing pid file
+                    pids = server.signal_pids(signal.SIG_DFL, verbose=True)
+                    output = pop_stream(f)
+                    self.assert_('stale pid' in output.lower())
+                    auth_pid = self.join_run_dir('auth-server.pid')
+                    self.assert_(auth_pid in output)
+            finally:
+                sys.stdout = old_stdout
+
+    def test_get_running_pids(self):
+        # test only gets running pids
+        pid_files = (
+            ('test-server1.pid', 1),
+            ('test-server2.pid', 2),
+        )
+        with temptree(*zip(*pid_files)) as t:
+            manager.RUN_DIR = t
+            server = manager.Server('test-server')
+            # mock os, only pid '1' is running
+            manager.os = MockOs([1])
+            running_pids = server.get_running_pids()
+            self.assertEquals(len(running_pids), 1)
+            self.assert_(1 in running_pids)
+            self.assert_(2 not in running_pids)
+            # test persistant running pid files
+            self.assert_(os.path.exists(os.path.join(t, 'test-server1.pid')))
+            # test clean up stale pids
+            pid_two = self.join_swift_dir('test-server2.pid')
+            self.assertFalse(os.path.exists(pid_two))
+            # reset mock os, no pids running
+            manager.os = MockOs([])
+            running_pids = server.get_running_pids()
+            self.assertFalse(running_pids)
+            # and now all pid files are cleaned out
+            pid_one = self.join_run_dir('test-server1.pid')
+            self.assertFalse(os.path.exists(pid_one))
+            all_pids = os.listdir(t)
+            self.assertEquals(len(all_pids), 0)
+
+        # test only get pids for right server
+        pid_files = (
+            ('thing-doer.pid', 1),
+            ('thing-sayer.pid', 2),
+            ('other-doer.pid', 3),
+            ('other-sayer.pid', 4),
+        )
+        files, pids = zip(*pid_files)
+        with temptree(files, pids) as t:
+            manager.RUN_DIR = t
+            # all pids are running
+            manager.os = MockOs(pids)
+            server = manager.Server('thing-doer')
+            running_pids = server.get_running_pids()
+            # only thing-doer.pid, 1
+            self.assertEquals(len(running_pids), 1)
+            self.assert_(1 in running_pids)
+            # no other pids returned
+            for n in (2, 3, 4):
+                self.assert_(n not in running_pids)
+            # assert stale pids for other servers ignored
+            manager.os = MockOs([1])  # only thing-doer is running
+            running_pids = server.get_running_pids()
+            for f in ('thing-sayer.pid', 'other-doer.pid', 'other-sayer.pid'):
+                # other server pid files persist
+                self.assert_(os.path.exists, os.path.join(t, f))
+            # verify that servers are in fact not running
+            for server_name in ('thing-sayer', 'other-doer', 'other-sayer'):
+                server = manager.Server(server_name)
+                running_pids = server.get_running_pids()
+                self.assertFalse(running_pids)
+            # and now all OTHER pid files are cleaned out
+            all_pids = os.listdir(t)
+            self.assertEquals(len(all_pids), 1)
+            self.assert_(os.path.exists(os.path.join(t, 'thing-doer.pid')))
+
+    def test_kill_running_pids(self):
+        pid_files = (
+            ('object-server.pid', 1),
+            ('object-replicator1.pid', 11),
+            ('object-replicator2.pid', 12),
+        )
+        files, running_pids = zip(*pid_files)
+        with temptree(files, running_pids) as t:
+            manager.RUN_DIR = t
+            server = manager.Server('object')
+            # test no servers running
+            manager.os = MockOs([])
+            pids = server.kill_running_pids()
+            self.assertFalse(pids, pids)
+        files, running_pids = zip(*pid_files)
+        with temptree(files, running_pids) as t:
+            manager.RUN_DIR = t
+            # start up pid
+            manager.os = MockOs([1])
+            # test kill one pid
+            pids = server.kill_running_pids()
+            self.assertEquals(len(pids), 1)
+            self.assert_(1 in pids)
+            self.assertEquals(manager.os.pid_sigs[1], [signal.SIGTERM])
+            # reset os mock
+            manager.os = MockOs([1])
+            # test shutdown
+            self.assert_('object-server' in
+                         manager.GRACEFUL_SHUTDOWN_SERVERS)
+            pids = server.kill_running_pids(graceful=True)
+            self.assertEquals(len(pids), 1)
+            self.assert_(1 in pids)
+            self.assertEquals(manager.os.pid_sigs[1], [signal.SIGHUP])
+            # start up other servers
+            manager.os = MockOs([11, 12])
+            # test multi server kill & ignore graceful on unsupported server
+            self.assertFalse('object-replicator' in
+                             manager.GRACEFUL_SHUTDOWN_SERVERS)
+            server = manager.Server('object-replicator')
+            pids = server.kill_running_pids(graceful=True)
+            self.assertEquals(len(pids), 2)
+            for pid in (11, 12):
+                self.assert_(pid in pids)
+                self.assertEquals(manager.os.pid_sigs[pid],
+                                  [signal.SIGTERM])
+            # and the other pid is of course not signaled
+            self.assert_(1 not in manager.os.pid_sigs)
+
+    def test_status(self):
+        conf_files = (
+            'test-server/1.conf',
+            'test-server/2.conf',
+            'test-server/3.conf',
+            'test-server/4.conf',
+        )
+
+        pid_files = (
+            ('test-server/1.pid', 1),
+            ('test-server/2.pid', 2),
+            ('test-server/3.pid', 3),
+            ('test-server/4.pid', 4),
+        )
+
+        with temptree(conf_files) as swift_dir:
+            manager.SWIFT_DIR = swift_dir
+            files, pids = zip(*pid_files)
+            with temptree(files, pids) as t:
+                manager.RUN_DIR = t
+                # setup running servers
+                server = manager.Server('test')
+                # capture stdio
+                old_stdout = sys.stdout
+                try:
+                    with open(os.path.join(t, 'output'), 'w+') as f:
+                        sys.stdout = f
+                        # test status for all running
+                        manager.os = MockOs(pids)
+                        self.assertEquals(server.status(), 0)
+                        output = pop_stream(f).strip().splitlines()
+                        self.assertEquals(len(output), 4)
+                        for line in output:
+                            self.assert_('test-server running' in line)
+                        # test get single server by number
+                        self.assertEquals(server.status(number=4), 0)
+                        output = pop_stream(f).strip().splitlines()
+                        self.assertEquals(len(output), 1)
+                        line = output[0]
+                        self.assert_('test-server running' in line)
+                        conf_four = self.join_swift_dir(conf_files[3])
+                        self.assert_('4 - %s' % conf_four in line)
+                        # test some servers not running
+                        manager.os = MockOs([1, 2, 3])
+                        self.assertEquals(server.status(), 0)
+                        output = pop_stream(f).strip().splitlines()
+                        self.assertEquals(len(output), 3)
+                        for line in output:
+                            self.assert_('test-server running' in line)
+                        # test single server not running
+                        manager.os = MockOs([1, 2])
+                        self.assertEquals(server.status(number=3), 1)
+                        output = pop_stream(f).strip().splitlines()
+                        self.assertEquals(len(output), 1)
+                        line = output[0]
+                        self.assert_('not running' in line)
+                        conf_three = self.join_swift_dir(conf_files[2])
+                        self.assert_(conf_three in line)
+                        # test no running pids
+                        manager.os = MockOs([])
+                        self.assertEquals(server.status(), 1)
+                        output = pop_stream(f).lower()
+                        self.assert_('no test-server running' in output)
+                        # test use provided pids
+                        pids = {
+                            1: '1.pid',
+                            2: '2.pid',
+                        }
+                        # shouldn't call get_running_pids
+                        called = []
+
+                        def mock(*args, **kwargs):
+                            called.append(True)
+                        server.get_running_pids = mock
+                        status = server.status(pids=pids)
+                        self.assertEquals(status, 0)
+                        self.assertFalse(called)
+                        output = pop_stream(f).strip().splitlines()
+                        self.assertEquals(len(output), 2)
+                        for line in output:
+                            self.assert_('test-server running' in line)
+                finally:
+                    sys.stdout = old_stdout
+
+    def test_spawn(self):
+
+        # mocks
+        class MockProcess():
+
+            NOTHING = 'default besides None'
+            STDOUT = 'stdout'
+            PIPE = 'pipe'
+
+            def __init__(self, pids=None):
+                if pids is None:
+                    pids = []
+                self.pids = (p for p in pids)
+
+            def Popen(self, args, **kwargs):
+                return MockProc(self.pids.next(), args, **kwargs)
+
+        class MockProc():
+
+            def __init__(self, pid, args, stdout=MockProcess.NOTHING,
+                         stderr=MockProcess.NOTHING):
+                self.pid = pid
+                self.args = args
+                self.stdout = stdout
+                if stderr == MockProcess.STDOUT:
+                    self.stderr = self.stdout
+                else:
+                    self.stderr = stderr
+
+        # setup running servers
+        server = manager.Server('test')
+
+        with temptree(['test-server.conf']) as swift_dir:
+            manager.SWIFT_DIR = swift_dir
+            with temptree([]) as t:
+                manager.RUN_DIR = t
+                old_subprocess = manager.subprocess
+                try:
+                    # test single server process calls spawn once
+                    manager.subprocess = MockProcess([1])
+                    conf_file = self.join_swift_dir('test-server.conf')
+                    # spawn server no kwargs
+                    server.spawn(conf_file)
+                    # test pid file
+                    pid_file = self.join_run_dir('test-server.pid')
+                    self.assert_(os.path.exists(pid_file))
+                    pid_on_disk = int(open(pid_file).read().strip())
+                    self.assertEquals(pid_on_disk, 1)
+                    # assert procs args
+                    self.assert_(server.procs)
+                    self.assertEquals(len(server.procs), 1)
+                    proc = server.procs[0]
+                    expected_args = [
+                        'swift-test-server',
+                        conf_file,
+                    ]
+                    self.assertEquals(proc.args, expected_args)
+                    # assert stdout is piped
+                    self.assertEquals(proc.stdout, MockProcess.PIPE)
+                    self.assertEquals(proc.stderr, proc.stdout)
+                    # test multi server process calls spawn multiple times
+                    manager.subprocess = MockProcess([11, 12, 13, 14])
+                    conf1 = self.join_swift_dir('test-server/1.conf')
+                    conf2 = self.join_swift_dir('test-server/2.conf')
+                    conf3 = self.join_swift_dir('test-server/3.conf')
+                    conf4 = self.join_swift_dir('test-server/4.conf')
+                    server = manager.Server('test')
+                    # test server run once
+                    server.spawn(conf1, once=True)
+                    self.assert_(server.procs)
+                    self.assertEquals(len(server.procs), 1)
+                    proc = server.procs[0]
+                    expected_args = ['swift-test-server', conf1, 'once']
+                    # assert stdout is piped
+                    self.assertEquals(proc.stdout, MockProcess.PIPE)
+                    self.assertEquals(proc.stderr, proc.stdout)
+                    # test server not daemon
+                    server.spawn(conf2, daemon=False)
+                    self.assert_(server.procs)
+                    self.assertEquals(len(server.procs), 2)
+                    proc = server.procs[1]
+                    expected_args = ['swift-test-server', conf2, 'verbose']
+                    self.assertEquals(proc.args, expected_args)
+                    # assert stdout is not changed
+                    self.assertEquals(proc.stdout, None)
+                    self.assertEquals(proc.stderr, None)
+                    # test server wait
+                    server.spawn(conf3, wait=False)
+                    self.assert_(server.procs)
+                    self.assertEquals(len(server.procs), 3)
+                    proc = server.procs[2]
+                    # assert stdout is /dev/null
+                    self.assert_(isinstance(proc.stdout, file))
+                    self.assertEquals(proc.stdout.name, os.devnull)
+                    self.assertEquals(proc.stdout.mode, 'w+b')
+                    self.assertEquals(proc.stderr, proc.stdout)
+                    # test not daemon over-rides wait
+                    server.spawn(conf4, wait=False, daemon=False, once=True)
+                    self.assert_(server.procs)
+                    self.assertEquals(len(server.procs), 4)
+                    proc = server.procs[3]
+                    expected_args = ['swift-test-server', conf4, 'once',
+                                     'verbose']
+                    self.assertEquals(proc.args, expected_args)
+                    # daemon behavior should trump wait, once shouldn't matter
+                    self.assertEquals(proc.stdout, None)
+                    self.assertEquals(proc.stderr, None)
+                    # assert pids
+                    for i, proc in enumerate(server.procs):
+                        pid_file = self.join_run_dir('test-server/%d.pid' %
+                                                     (i + 1))
+                        pid_on_disk = int(open(pid_file).read().strip())
+                        self.assertEquals(pid_on_disk, proc.pid)
+                finally:
+                    manager.subprocess = old_subprocess
+
+    def test_wait(self):
+        server = manager.Server('test')
+        self.assertEquals(server.wait(), 0)
+
+        class MockProcess(Thread):
+            def __init__(self, delay=0.1, fail_to_start=False):
+                Thread.__init__(self)
+                # setup pipe
+                rfd, wfd = os.pipe()
+                # subprocess connection to read stdout
+                self.stdout = os.fdopen(rfd)
+                # real process connection to write stdout
+                self._stdout = os.fdopen(wfd, 'w')
+                self.delay = delay
+                self.finished = False
+                self.returncode = None
+                if fail_to_start:
+                    self._returncode = 1
+                    self.run = self.fail
+                else:
+                    self._returncode = 0
+
+            def __enter__(self):
+                self.start()
+                return self
+
+            def __exit__(self, *args):
+                if self.isAlive():
+                    self.join()
+
+            def close_stdout(self):
+                self._stdout.flush()
+                with open(os.devnull, 'wb') as nullfile:
+                    try:
+                        os.dup2(nullfile.fileno(), self._stdout.fileno())
+                    except OSError:
+                        pass
+
+            def fail(self):
+                print >>self._stdout, 'mock process started'
+                sleep(self.delay)  # perform setup processing
+                print >>self._stdout, 'mock process failed to start'
+                self.close_stdout()
+
+            def communicate(self):
+                self.returncode = self._returncode
+
+            def run(self):
+                print >>self._stdout, 'mock process started'
+                sleep(self.delay)  # perform setup processing
+                print >>self._stdout, 'setup complete!'
+                self.close_stdout()
+                sleep(self.delay)  # do some more processing
+                print >>self._stdout, 'mock process finished'
+                self.finished = True
+
+        with temptree([]) as t:
+            old_stdout = sys.stdout
+            try:
+                with open(os.path.join(t, 'output'), 'w+') as f:
+                    # acctually capture the read stdout (for prints)
+                    sys.stdout = f
+                    # test closing pipe in subprocess unblocks read
+                    with MockProcess() as proc:
+                        server.procs = [proc]
+                        status = server.wait()
+                        self.assertEquals(status, 0)
+                        # wait should return as soon as stdout is closed
+                        self.assert_(proc.isAlive())
+                        self.assertFalse(proc.finished)
+                    self.assert_(proc.finished)  # make sure it did finish...
+                    # test output kwarg prints subprocess output
+                    with MockProcess() as proc:
+                        server.procs = [proc]
+                        status = server.wait(output=True)
+                    output = pop_stream(f)
+                    self.assert_('mock process started' in output)
+                    self.assert_('setup complete' in output)
+                    # make sure we don't get prints after stdout was closed
+                    self.assert_('mock process finished' not in output)
+                    # test process which fails to start
+                    with MockProcess(fail_to_start=True) as proc:
+                        server.procs = [proc]
+                        status = server.wait()
+                        self.assertEquals(status, 1)
+                    self.assert_('failed' in pop_stream(f))
+                    # test multiple procs
+                    procs = [MockProcess() for i in range(3)]
+                    for proc in procs:
+                        proc.start()
+                    server.procs = procs
+                    status = server.wait()
+                    self.assertEquals(status, 0)
+                    for proc in procs:
+                        self.assert_(proc.isAlive())
+                    for proc in procs:
+                        proc.join()
+            finally:
+                sys.stdout = old_stdout
+
+    def test_interact(self):
+        class MockProcess():
+
+            def __init__(self, fail=False):
+                self.returncode = None
+                if fail:
+                    self._returncode = 1
+                else:
+                    self._returncode = 0
+
+            def communicate(self):
+                self.returncode = self._returncode
+                return '', ''
+
+        server = manager.Server('test')
+        server.procs = [MockProcess()]
+        self.assertEquals(server.interact(), 0)
+        server.procs = [MockProcess(fail=True)]
+        self.assertEquals(server.interact(), 1)
+        procs = []
+        for fail in (False, True, True):
+            procs.append(MockProcess(fail=fail))
+        server.procs = procs
+        self.assert_(server.interact() > 0)
+
+    def test_launch(self):
+        # stubs
+        conf_files = (
+            'proxy-server.conf',
+            'auth-server.conf',
+            'object-server/1.conf',
+            'object-server/2.conf',
+            'object-server/3.conf',
+            'object-server/4.conf',
+        )
+        pid_files = (
+            ('proxy-server.pid', 1),
+            ('proxy-server/2.pid', 2),
+        )
+
+        #mocks
+        class MockSpawn():
+
+            def __init__(self, pids=None):
+                self.conf_files = []
+                self.kwargs = []
+                if not pids:
+                    def one_forever():
+                        while True:
+                            yield 1
+                    self.pids = one_forever()
+                else:
+                    self.pids = (x for x in pids)
+
+            def __call__(self, conf_file, **kwargs):
+                self.conf_files.append(conf_file)
+                self.kwargs.append(kwargs)
+                rv = self.pids.next()
+                if isinstance(rv, Exception):
+                    raise rv
+                else:
+                    return rv
+
+        with temptree(conf_files) as swift_dir:
+            manager.SWIFT_DIR = swift_dir
+            files, pids = zip(*pid_files)
+            with temptree(files, pids) as t:
+                manager.RUN_DIR = t
+                old_stdout = sys.stdout
+                try:
+                    with open(os.path.join(t, 'output'), 'w+') as f:
+                        sys.stdout = f
+                        # can't start server w/o an conf
+                        server = manager.Server('test')
+                        self.assertFalse(server.launch())
+                        # start mock os running all pids
+                        manager.os = MockOs(pids)
+                        server = manager.Server('proxy')
+                        # can't start server if it's already running
+                        self.assertFalse(server.launch())
+                        output = pop_stream(f)
+                        self.assert_('running' in output)
+                        conf_file = self.join_swift_dir('proxy-server.conf')
+                        self.assert_(conf_file in output)
+                        pid_file = self.join_run_dir('proxy-server/2.pid')
+                        self.assert_(pid_file in output)
+                        self.assert_('already started' in output)
+                        # no running pids
+                        manager.os = MockOs([])
+                        # test ignore once for non-start-once server
+                        mock_spawn = MockSpawn([1])
+                        server.spawn = mock_spawn
+                        conf_file = self.join_swift_dir('proxy-server.conf')
+                        expected = {
+                            1: conf_file,
+                        }
+                        self.assertEquals(server.launch(once=True), expected)
+                        self.assertEquals(mock_spawn.conf_files, [conf_file])
+                        expected = {
+                            'once': False,
+                        }
+                        self.assertEquals(mock_spawn.kwargs, [expected])
+                        output = pop_stream(f)
+                        self.assert_('Starting' in output)
+                        self.assert_('once' not in output)
+                        # test multi-server kwarg once
+                        server = manager.Server('object-replicator')
+                        mock_spawn = MockSpawn([1, 2, 3, 4])
+                        server.spawn = mock_spawn
+                        conf1 = self.join_swift_dir('object-server/1.conf')
+                        conf2 = self.join_swift_dir('object-server/2.conf')
+                        conf3 = self.join_swift_dir('object-server/3.conf')
+                        conf4 = self.join_swift_dir('object-server/4.conf')
+                        expected = {
+                            1: conf1,
+                            2: conf2,
+                            3: conf3,
+                            4: conf4,
+                        }
+                        self.assertEquals(server.launch(once=True), expected)
+                        self.assertEquals(mock_spawn.conf_files, [conf1, conf2,
+                                                                 conf3, conf4])
+                        expected = {
+                            'once': True,
+                        }
+                        self.assertEquals(len(mock_spawn.kwargs), 4)
+                        for kwargs in mock_spawn.kwargs:
+                            self.assertEquals(kwargs, expected)
+                        # test number kwarg
+                        mock_spawn = MockSpawn([4])
+                        server.spawn = mock_spawn
+                        expected = {
+                            4: conf4,
+                        }
+                        self.assertEquals(server.launch(number=4), expected)
+                        self.assertEquals(mock_spawn.conf_files, [conf4])
+                        expected = {
+                            'number': 4
+                        }
+                        self.assertEquals(mock_spawn.kwargs, [expected])
+                        # test cmd does not exist
+                        server = manager.Server('auth')
+                        mock_spawn = MockSpawn([OSError(errno.ENOENT, 'blah')])
+                        server.spawn = mock_spawn
+                        self.assertEquals(server.launch(), {})
+                        self.assert_('swift-auth-server does not exist' in
+                                     pop_stream(f))
+                finally:
+                    sys.stdout = old_stdout
+
+    def test_stop(self):
+        conf_files = (
+            'account-server/1.conf',
+            'account-server/2.conf',
+            'account-server/3.conf',
+            'account-server/4.conf',
+        )
+        pid_files = (
+            ('account-reaper/1.pid', 1),
+            ('account-reaper/2.pid', 2),
+            ('account-reaper/3.pid', 3),
+            ('account-reaper/4.pid', 4),
+        )
+
+        with temptree(conf_files) as swift_dir:
+            manager.SWIFT_DIR = swift_dir
+            files, pids = zip(*pid_files)
+            with temptree(files, pids) as t:
+                manager.RUN_DIR = t
+                # start all pids in mock os
+                manager.os = MockOs(pids)
+                server = manager.Server('account-reaper')
+                # test kill all running pids
+                pids = server.stop()
+                self.assertEquals(len(pids), 4)
+                for pid in (1, 2, 3, 4):
+                    self.assert_(pid in pids)
+                    self.assertEquals(manager.os.pid_sigs[pid],
+                                      [signal.SIGTERM])
+                conf1 = self.join_swift_dir('account-reaper/1.conf')
+                conf2 = self.join_swift_dir('account-reaper/2.conf')
+                conf3 = self.join_swift_dir('account-reaper/3.conf')
+                conf4 = self.join_swift_dir('account-reaper/4.conf')
+                # reset mock os with only 2 running pids
+                manager.os = MockOs([3, 4])
+                pids = server.stop()
+                self.assertEquals(len(pids), 2)
+                for pid in (3, 4):
+                    self.assert_(pid in pids)
+                    self.assertEquals(manager.os.pid_sigs[pid],
+                                      [signal.SIGTERM])
+                self.assertFalse(os.path.exists(conf1))
+                self.assertFalse(os.path.exists(conf2))
+                # test number kwarg
+                manager.os = MockOs([3, 4])
+                pids = server.stop(number=3)
+                self.assertEquals(len(pids), 1)
+                expected = {
+                    3: conf3,
+                }
+                self.assert_(pids, expected)
+                self.assertEquals(manager.os.pid_sigs[3], [signal.SIGTERM])
+                self.assertFalse(os.path.exists(conf4))
+                self.assertFalse(os.path.exists(conf3))
+
+
+class TestManager(unittest.TestCase):
+
+    def test_create(self):
+        m = manager.Manager(['test'])
+        self.assertEquals(len(m.servers), 1)
+        server = m.servers.pop()
+        self.assert_(isinstance(server, manager.Server))
+        self.assertEquals(server.server, 'test-server')
+        # test multi-server and simple dedupe
+        servers = ['object-replicator', 'object-auditor', 'object-replicator']
+        m = manager.Manager(servers)
+        self.assertEquals(len(m.servers), 2)
+        for server in m.servers:
+            self.assert_(server.server in servers)
+        # test all
+        m = manager.Manager(['all'])
+        self.assertEquals(len(m.servers), len(manager.ALL_SERVERS))
+        for server in m.servers:
+            self.assert_(server.server in manager.ALL_SERVERS)
+        # test main
+        m = manager.Manager(['main'])
+        self.assertEquals(len(m.servers), len(manager.MAIN_SERVERS))
+        for server in m.servers:
+            self.assert_(server.server in manager.MAIN_SERVERS)
+        # test rest
+        m = manager.Manager(['rest'])
+        self.assertEquals(len(m.servers), len(manager.REST_SERVERS))
+        for server in m.servers:
+            self.assert_(server.server in manager.REST_SERVERS)
+        # test main + rest == all
+        m = manager.Manager(['main', 'rest'])
+        self.assertEquals(len(m.servers), len(manager.ALL_SERVERS))
+        for server in m.servers:
+            self.assert_(server.server in manager.ALL_SERVERS)
+        # test dedupe
+        m = manager.Manager(['main', 'rest', 'proxy', 'object',
+                                           'container', 'account'])
+        self.assertEquals(len(m.servers), len(manager.ALL_SERVERS))
+        for server in m.servers:
+            self.assert_(server.server in manager.ALL_SERVERS)
+        # test glob
+        m = manager.Manager(['object-*'])
+        object_servers = [s for s in manager.ALL_SERVERS if
+                          s.startswith('object')]
+        self.assertEquals(len(m.servers), len(object_servers))
+        for s in m.servers:
+            self.assert_(str(s) in object_servers)
+        m = manager.Manager(['*-replicator'])
+        replicators = [s for s in manager.ALL_SERVERS if
+                       s.endswith('replicator')]
+        for s in m.servers:
+            self.assert_(str(s) in replicators)
+
+    def test_status(self):
+        class MockServer():
+
+            def __init__(self, server):
+                self.server = server
+                self.called_kwargs = []
+
+            def status(self, **kwargs):
+                self.called_kwargs.append(kwargs)
+                if 'error' in self.server:
+                    return 1
+                else:
+                    return 0
+
+        old_server_class = manager.Server
+        try:
+            manager.Server = MockServer
+            m = manager.Manager(['test'])
+            status = m.status()
+            self.assertEquals(status, 0)
+            m = manager.Manager(['error'])
+            status = m.status()
+            self.assertEquals(status, 1)
+            # test multi-server
+            m = manager.Manager(['test', 'error'])
+            kwargs = {'key': 'value'}
+            status = m.status(**kwargs)
+            self.assertEquals(status, 1)
+            for server in m.servers:
+                self.assertEquals(server.called_kwargs, [kwargs])
+        finally:
+            manager.Server = old_server_class
+
+    def test_start(self):
+        def mock_setup_env():
+            getattr(mock_setup_env, 'called', []).append(True)
+
+        class MockServer():
+            def __init__(self, server):
+                self.server = server
+                self.called = defaultdict(list)
+
+            def launch(self, **kwargs):
+                self.called['launch'].append(kwargs)
+
+            def wait(self, **kwargs):
+                self.called['wait'].append(kwargs)
+                return int('error' in self.server)
+
+            def stop(self, **kwargs):
+                self.called['stop'].append(kwargs)
+
+            def interact(self, **kwargs):
+                self.called['interact'].append(kwargs)
+                if 'raise' in self.server:
+                    raise KeyboardInterrupt
+                elif 'error' in self.server:
+                    return 1
+                else:
+                    return 0
+
+        old_setup_env = manager.setup_env
+        old_swift_server = manager.Server
+        try:
+            manager.setup_env = mock_setup_env
+            manager.Server = MockServer
+
+            # test no errors on launch
+            m = manager.Manager(['proxy'])
+            status = m.start()
+            self.assertEquals(status, 0)
+            for server in m.servers:
+                self.assertEquals(server.called['launch'], [{}])
+
+            # test error on launch
+            m = manager.Manager(['proxy', 'error'])
+            status = m.start()
+            self.assertEquals(status, 1)
+            for server in m.servers:
+                self.assertEquals(server.called['launch'], [{}])
+                self.assertEquals(server.called['wait'], [{}])
+
+            # test interact
+            m = manager.Manager(['proxy', 'error'])
+            kwargs = {'daemon': False}
+            status = m.start(**kwargs)
+            self.assertEquals(status, 1)
+            for server in m.servers:
+                self.assertEquals(server.called['launch'], [kwargs])
+                self.assertEquals(server.called['interact'], [kwargs])
+            m = manager.Manager(['raise'])
+            kwargs = {'daemon': False}
+            status = m.start(**kwargs)
+
+        finally:
+            manager.setup_env = old_setup_env
+            manager.Server = old_swift_server
+
+    def test_no_wait(self):
+        class MockServer():
+            def __init__(self, server):
+                self.server = server
+                self.called = defaultdict(list)
+
+            def launch(self, **kwargs):
+                self.called['launch'].append(kwargs)
+
+            def wait(self, **kwargs):
+                self.called['wait'].append(kwargs)
+                return int('error' in self.server)
+
+        orig_swift_server = manager.Server
+        try:
+            manager.Server = MockServer
+            # test success
+            init = manager.Manager(['proxy'])
+            status = init.no_wait()
+            self.assertEquals(status, 0)
+            for server in init.servers:
+                self.assertEquals(len(server.called['launch']), 1)
+                called_kwargs = server.called['launch'][0]
+                self.assertFalse(called_kwargs['wait'])
+                self.assertFalse(server.called['wait'])
+            # test no errocode status even on error
+            init = manager.Manager(['error'])
+            status = init.no_wait()
+            self.assertEquals(status, 0)
+            for server in init.servers:
+                self.assertEquals(len(server.called['launch']), 1)
+                called_kwargs = server.called['launch'][0]
+                self.assert_('wait' in called_kwargs)
+                self.assertFalse(called_kwargs['wait'])
+                self.assertFalse(server.called['wait'])
+            # test wait with once option
+            init = manager.Manager(['updater', 'replicator-error'])
+            status = init.no_wait(once=True)
+            self.assertEquals(status, 0)
+            for server in init.servers:
+                self.assertEquals(len(server.called['launch']), 1)
+                called_kwargs = server.called['launch'][0]
+                self.assert_('wait' in called_kwargs)
+                self.assertFalse(called_kwargs['wait'])
+                self.assert_('once' in called_kwargs)
+                self.assert_(called_kwargs['once'])
+                self.assertFalse(server.called['wait'])
+        finally:
+            manager.Server = orig_swift_server
+
+    def test_no_daemon(self):
+        class MockServer():
+
+            def __init__(self, server):
+                self.server = server
+                self.called = defaultdict(list)
+
+            def launch(self, **kwargs):
+                self.called['launch'].append(kwargs)
+
+            def interact(self, **kwargs):
+                self.called['interact'].append(kwargs)
+                return int('error' in self.server)
+
+        orig_swift_server = manager.Server
+        try:
+            manager.Server = MockServer
+            # test success
+            init = manager.Manager(['proxy'])
+            stats = init.no_daemon()
+            self.assertEquals(stats, 0)
+            # test error
+            init = manager.Manager(['proxy', 'object-error'])
+            stats = init.no_daemon()
+            self.assertEquals(stats, 1)
+            # test once
+            init = manager.Manager(['proxy', 'object-error'])
+            stats = init.no_daemon()
+            for server in init.servers:
+                self.assertEquals(len(server.called['launch']), 1)
+                self.assertEquals(len(server.called['wait']), 0)
+                self.assertEquals(len(server.called['interact']), 1)
+        finally:
+            manager.Server = orig_swift_server
+
+    def test_once(self):
+        class MockServer():
+
+            def __init__(self, server):
+                self.server = server
+                self.called = defaultdict(list)
+
+            def wait(self, **kwargs):
+                self.called['wait'].append(kwargs)
+                if 'error' in self.server:
+                    return 1
+                else:
+                    return 0
+
+            def launch(self, **kwargs):
+                return self.called['launch'].append(kwargs)
+
+        orig_swift_server = manager.Server
+        try:
+            manager.Server = MockServer
+            # test no errors
+            init = manager.Manager(['account-reaper'])
+            status = init.once()
+            self.assertEquals(status, 0)
+            # test error code on error
+            init = manager.Manager(['error-reaper'])
+            status = init.once()
+            self.assertEquals(status, 1)
+            for server in init.servers:
+                self.assertEquals(len(server.called['launch']), 1)
+                called_kwargs = server.called['launch'][0]
+                self.assertEquals(called_kwargs, {'once': True})
+                self.assertEquals(len(server.called['wait']), 1)
+                self.assertEquals(len(server.called['interact']), 0)
+        finally:
+            manager.Server = orig_swift_server
+
+    def test_stop(self):
+        class MockServerFactory():
+            class MockServer():
+                def __init__(self, pids):
+                    self.pids = pids
+
+                def stop(self, **kwargs):
+                    return self.pids
+
+            def __init__(self, server_pids):
+                self.server_pids = server_pids
+
+            def __call__(self, server):
+                return MockServerFactory.MockServer(self.server_pids[server])
+
+        def mock_watch_server_pids(server_pids, **kwargs):
+            for server, pids in server_pids.items():
+                for pid in pids:
+                    if pid is None:
+                        continue
+                    yield server, pid
+
+        _orig_server = manager.Server
+        _orig_watch_server_pids = manager.watch_server_pids
+        try:
+            manager.watch_server_pids = mock_watch_server_pids
+            # test stop one server
+            server_pids = {
+                'test': [1]
+            }
+            manager.Server = MockServerFactory(server_pids)
+            m = manager.Manager(['test'])
+            status = m.stop()
+            self.assertEquals(status, 0)
+            # test not running
+            server_pids = {
+                'test': []
+            }
+            manager.Server = MockServerFactory(server_pids)
+            m = manager.Manager(['test'])
+            status = m.stop()
+            self.assertEquals(status, 1)
+            # test won't die
+            server_pids = {
+                'test': [None]
+            }
+            manager.Server = MockServerFactory(server_pids)
+            m = manager.Manager(['test'])
+            status = m.stop()
+            self.assertEquals(status, 1)
+
+        finally:
+            manager.Server = _orig_server
+            manager.watch_server_pids = _orig_watch_server_pids
+
+    # TODO: more tests
+    def test_shutdown(self):
+        m = manager.Manager(['test'])
+        m.stop_was_called = False
+
+        def mock_stop(*args, **kwargs):
+            m.stop_was_called = True
+            expected = {'graceful': True}
+            self.assertEquals(kwargs, expected)
+            return 0
+        m.stop = mock_stop
+        status = m.shutdown()
+        self.assertEquals(status, 0)
+        self.assertEquals(m.stop_was_called, True)
+
+    def test_restart(self):
+        m = manager.Manager(['test'])
+        m.stop_was_called = False
+
+        def mock_stop(*args, **kwargs):
+            m.stop_was_called = True
+            return 0
+        m.start_was_called = False
+
+        def mock_start(*args, **kwargs):
+            m.start_was_called = True
+            return 0
+        m.stop = mock_stop
+        m.start = mock_start
+        status = m.restart()
+        self.assertEquals(status, 0)
+        self.assertEquals(m.stop_was_called, True)
+        self.assertEquals(m.start_was_called, True)
+
+    def test_reload(self):
+        class MockManager():
+            called = defaultdict(list)
+
+            def __init__(self, servers):
+                pass
+
+            @classmethod
+            def reset_called(cls):
+                cls.called = defaultdict(list)
+
+            def stop(self, **kwargs):
+                MockManager.called['stop'].append(kwargs)
+                return 0
+
+            def start(self, **kwargs):
+                MockManager.called['start'].append(kwargs)
+                return 0
+
+        _orig_manager = manager.Manager
+        try:
+            m = _orig_manager(['auth'])
+            for server in m.servers:
+                self.assert_(server.server in
+                             manager.GRACEFUL_SHUTDOWN_SERVERS)
+            manager.Manager = MockManager
+            status = m.reload()
+            self.assertEquals(status, 0)
+            expected = {
+                'start': [{'graceful': True}],
+                'stop': [{'graceful': True}],
+            }
+            self.assertEquals(MockManager.called, expected)
+            # test force graceful
+            MockManager.reset_called()
+            m = _orig_manager(['*-server'])
+            self.assert_(len(m.servers), 4)
+            for server in m.servers:
+                self.assert_(server.server in
+                             manager.GRACEFUL_SHUTDOWN_SERVERS)
+            manager.Manager = MockManager
+            status = m.reload(graceful=False)
+            self.assertEquals(status, 0)
+            expected = {
+                'start': [{'graceful': True}] * 4,
+                'stop': [{'graceful': True}] * 4,
+            }
+            self.assertEquals(MockManager.called, expected)
+
+        finally:
+            manager.Manager = _orig_manager
+
+    def test_force_reload(self):
+        m = manager.Manager(['test'])
+        m.reload_was_called = False
+
+        def mock_reload(*args, **kwargs):
+            m.reload_was_called = True
+            return 0
+        m.reload = mock_reload
+        status = m.force_reload()
+        self.assertEquals(status, 0)
+        self.assertEquals(m.reload_was_called, True)
+
+    def test_get_command(self):
+        m = manager.Manager(['test'])
+        self.assertEquals(m.start, m.get_command('start'))
+        self.assertEquals(m.force_reload, m.get_command('force-reload'))
+        self.assertEquals(m.get_command('force-reload'),
+                          m.get_command('force_reload'))
+        self.assertRaises(manager.UnknownCommandError, m.get_command,
+                          'no_command')
+        self.assertRaises(manager.UnknownCommandError, m.get_command,
+                          '__init__')
+
+    def test_list_commands(self):
+        for cmd, help in manager.Manager.list_commands():
+            method = getattr(manager.Manager, cmd.replace('-', '_'), None)
+            self.assert_(method, '%s is not a command' % cmd)
+            self.assert_(getattr(method, 'publicly_accessible', False))
+            self.assertEquals(method.__doc__.strip(), help)
+
+    def test_run_command(self):
+        m = manager.Manager(['test'])
+        m.cmd_was_called = False
+
+        def mock_cmd(*args, **kwargs):
+            m.cmd_was_called = True
+            expected = {'kw1': True, 'kw2': False}
+            self.assertEquals(kwargs, expected)
+            return 0
+        mock_cmd.publicly_accessible = True
+        m.mock_cmd = mock_cmd
+        kwargs = {'kw1': True, 'kw2': False}
+        status = m.run_command('mock_cmd', **kwargs)
+        self.assertEquals(status, 0)
+        self.assertEquals(m.cmd_was_called, True)
+
+if __name__ == '__main__':
+    unittest.main()

=== modified file 'test/unit/common/test_utils.py'
--- test/unit/common/test_utils.py	2011-02-11 17:27:05 +0000
+++ test/unit/common/test_utils.py	2011-02-16 21:01:55 +0000
@@ -16,6 +16,7 @@
 """ Tests for swift.common.utils """
 
 from __future__ import with_statement
+from test.unit import temptree
 import logging
 import mimetools
 import os
@@ -660,5 +661,86 @@
         self.assertTrue(abs(100 - (time.time() - start) * 100) < 10)
 
 
+    def test_search_tree(self):
+        # file match & ext miss
+        with temptree(['asdf.conf', 'blarg.conf', 'asdf.cfg']) as t:
+            asdf = utils.search_tree(t, 'a*', '.conf')
+            self.assertEquals(len(asdf), 1)
+            self.assertEquals(asdf[0],
+                              os.path.join(t, 'asdf.conf'))
+
+        # multi-file match & glob miss & sort
+        with temptree(['application.bin', 'apple.bin', 'apropos.bin']) as t:
+            app_bins = utils.search_tree(t, 'app*', 'bin')
+            self.assertEquals(len(app_bins), 2)
+            self.assertEquals(app_bins[0],
+                              os.path.join(t, 'apple.bin'))
+            self.assertEquals(app_bins[1],
+                              os.path.join(t, 'application.bin'))
+
+        # test file in folder & ext miss & glob miss
+        files = (
+            'sub/file1.ini',
+            'sub/file2.conf',
+            'sub.bin',
+            'bus.ini',
+            'bus/file3.ini',
+        )
+        with temptree(files) as t:
+            sub_ini = utils.search_tree(t, 'sub*', '.ini')
+            self.assertEquals(len(sub_ini), 1)
+            self.assertEquals(sub_ini[0],
+                              os.path.join(t, 'sub/file1.ini'))
+
+        # test multi-file in folder & sub-folder & ext miss & glob miss
+        files = (
+            'folder_file.txt',
+            'folder/1.txt',
+            'folder/sub/2.txt',
+            'folder2/3.txt',
+            'Folder3/4.txt'
+            'folder.rc',
+        )
+        with temptree(files) as t:
+            folder_texts = utils.search_tree(t, 'folder*', '.txt')
+            self.assertEquals(len(folder_texts), 4)
+            f1 = os.path.join(t, 'folder_file.txt')
+            f2 = os.path.join(t, 'folder/1.txt')
+            f3 = os.path.join(t, 'folder/sub/2.txt')
+            f4 = os.path.join(t, 'folder2/3.txt')
+            for f in [f1, f2, f3, f4]:
+                self.assert_(f in folder_texts)
+
+    def test_write_file(self):
+        with temptree([]) as t:
+            file_name = os.path.join(t, 'test')
+            utils.write_file(file_name, 'test')
+            with open(file_name, 'r') as f:
+                contents = f.read()
+            self.assertEquals(contents, 'test')
+            # and also subdirs
+            file_name = os.path.join(t, 'subdir/test2')
+            utils.write_file(file_name, 'test2')
+            with open(file_name, 'r') as f:
+                contents = f.read()
+            self.assertEquals(contents, 'test2')
+            # but can't over-write files
+            file_name = os.path.join(t, 'subdir/test2/test3')
+            self.assertRaises(IOError, utils.write_file, file_name,
+                              'test3')
+
+    def test_remove_file(self):
+        with temptree([]) as t:
+            file_name = os.path.join(t, 'blah.pid')
+            # assert no raise
+            self.assertEquals(os.path.exists(file_name), False)
+            self.assertEquals(utils.remove_file(file_name), None)
+            with open(file_name, 'w') as f:
+                f.write('1')
+            self.assert_(os.path.exists(file_name))
+            self.assertEquals(utils.remove_file(file_name), None)
+            self.assertFalse(os.path.exists(file_name))
+
+
 if __name__ == '__main__':
     unittest.main()