Merge lp:~ipython-dev/ipython/kernel-config into lp:ipython/0.11

[Brian Granger]

[Brian Granger (ellisonbg)]

This is the result of a massive effort to refactor IPython.kernel to use the new config system. This also includes an improved ipcluster/ipcontroller/ipengine command line programs. Here are the main files that should be reviewed:

IPython.kernel:

* clusterdir.py
* ipcontrollerapp.py
* ipengineapp.py
* launcher.py
* ipclusterapp.py

IPython.config.default:

* ipcontroller_config.py
* ipengine_config.py
* ipcluster_config.py

I have also fixed a number of bugs and problems in IPython itself:

* The command line options use a new style.
* %debug is fixed.
* New magics %load_ext/%unload_ext/%reload_ext/%install_profiles/%install_default_config.

Enjoy!

[lp:~ipython-dev/ipython/kernel-config]

1269. By Brian Granger on 2009-11-15

Added better documentation to command line programs.

1270. By Brian Granger on 2009-11-16

Initial draft of Windows HPC documentation.

1271. By Brian Granger on 2009-11-20

Work in the documentation.

1272. By bgranger <bgranger@red> on 2009-11-20

Adding figures for options pricing example.

1273. By Brian Granger on 2009-11-21

Final work on the Win HPC whitepaper.

[Fernando Perez (fdo.perez)]

Summary: Excellent work! Many thanks for all of this.

I'm not merging it now just in case you want to do it yourself tomorrow (12/30) first thing or have some other little change you'd like to put in. But if you don't merge it I'll go ahead and do it, and will begin adding code I have (like testing stuff) so we can push towards 0.11. We still have cleanups and things to do, but all of that is best done on top of a more functional trunk, which this branch helps LOT with.

Per-commit comments:

-r1264: ok

-r1265: in kernel/ipclusterapp.py, should we output something visible to the user indicating the 4s pause at shutdown? I won't add it myself, as I'm not sure this should go in; most of our output is now via logging, right? Putting this in the log wouldn't be too useful, it would be strictly a convenience for the user.

-r1266: ok, just upstream

-r1267: great, more examples!

-r1268: good, these are changes we'd discussed over the phone.

-r1269: better docs, excellent...

-r1270, 71, 72, 73: this document is stellar, we'd reviewed it already.

review: Approve

[Brian Granger (ellisonbg)]

> I'm not merging it now just in case you want to do it yourself tomorrow (12/30) first thing or have some other little change you'd like to put in.  But if you don't merge it I'll go ahead and do it, and will begin adding code I have (like testing stuff) so we can push towards 0.11.  We still have cleanups and things to do, but all of that is best done on top of a more functional trunk, which this branch helps LOT with.

Thanks for the review. Go ahead and do the merge, I am working on
other things right now.

> Per-commit comments:
>
> -r1264: ok
>
> -r1265: in kernel/ipclusterapp.py, should we output something visible to the user indicating the 4s pause at shutdown?  I won't add it myself, as I'm not sure this should go in; most of our output is now via logging, right?  Putting this in the log wouldn't be too useful, it would be strictly a convenience for the user.

Yes, I can add a message about this to the logs.

> -r1266: ok, just upstream
>
> -r1267: great, more examples!
>
> -r1268: good, these are changes we'd discussed over the phone.
>
> -r1269: better docs, excellent...
>
> -r1270, 71, 72, 73: this document is stellar, we'd reviewed it already.
>
> --
> https://code.launchpad.net/~ipython-dev/ipython/kernel-config/+merge/14900
> You proposed lp:~ipython-dev/ipython/kernel-config for merging.
>

--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>

[Fernando Perez (fdo.perez)]

On Wed, Dec 30, 2009 at 8:39 AM, Brian Granger <email address hidden> wrote:
>
> Thanks for the review.  Go ahead and do the merge, I am working on
> other things right now

Done, merged and pushed with an extra tiny fix. Thanks again, great job!

=== removed file 'IPython/config/api.py'
--- IPython/config/api.py	2009-07-02 02:49:32 +0000
+++ IPython/config/api.py	1970-01-01 00:00:00 +0000
@@ -1,102 +0,0 @@
-# encoding: utf-8
-
-"""This is the official entry point to IPython's configuration system.  """
-
-__docformat__ = "restructuredtext en"
-
-#-------------------------------------------------------------------------------
-#  Copyright (C) 2008  The IPython Development Team
-#
-#  Distributed under the terms of the BSD License.  The full license is in
-#  the file COPYING, distributed as part of this software.
-#-------------------------------------------------------------------------------
-
-#-------------------------------------------------------------------------------
-# Imports
-#-------------------------------------------------------------------------------
-
-import os
-from os.path import join as pjoin
-
-from IPython.utils.genutils import get_home_dir, get_ipython_dir
-from IPython.external.configobj import ConfigObj
-
-
-class ConfigObjManager(object):
-    
-    def __init__(self, configObj, filename):
-        self.current = configObj
-        self.current.indent_type = '    '
-        self.filename = filename
-        # self.write_default_config_file()
-        
-    def get_config_obj(self):
-        return self.current
-    
-    def update_config_obj(self, newConfig):
-        self.current.merge(newConfig)
-        
-    def update_config_obj_from_file(self, filename):
-        newConfig = ConfigObj(filename, file_error=False)
-        self.current.merge(newConfig)
-        
-    def update_config_obj_from_default_file(self, ipythondir=None):
-        fname = self.resolve_file_path(self.filename, ipythondir)
-        self.update_config_obj_from_file(fname)
-
-    def write_config_obj_to_file(self, filename):
-        f = open(filename, 'w')
-        self.current.write(f)
-        f.close()
-
-    def write_default_config_file(self):
-        ipdir = get_ipython_dir()
-        fname = pjoin(ipdir, self.filename)
-        if not os.path.isfile(fname):
-            print "Writing the configuration file to: " + fname
-            self.write_config_obj_to_file(fname)
-    
-    def _import(self, key):
-        package = '.'.join(key.split('.')[0:-1])
-        obj = key.split('.')[-1]
-        execString = 'from %s import %s' % (package, obj)
-        exec execString
-        exec 'temp = %s' % obj 
-        return temp
-
-    def resolve_file_path(self, filename, ipythondir = None):
-        """Resolve filenames into absolute paths.
-
-        This function looks in the following directories in order:
-
-        1.  In the current working directory or by absolute path with ~ expanded
-        2.  In ipythondir if that is set
-        3.  In the IPYTHONDIR environment variable if it exists
-        4.  In the ~/.ipython directory
-
-        Note: The IPYTHONDIR is also used by the trunk version of IPython so
-               changing it will also affect it was well.
-        """
-
-        # In cwd or by absolute path with ~ expanded
-        trythis = os.path.expanduser(filename)
-        if os.path.isfile(trythis):
-            return trythis
-
-        # In ipythondir if it is set
-        if ipythondir is not None:
-            trythis = pjoin(ipythondir, filename)
-            if os.path.isfile(trythis):
-                return trythis        
-
-        trythis = pjoin(get_ipython_dir(), filename)
-        if os.path.isfile(trythis):
-            return trythis
-
-        return None
-
-
-    
-    
-
-    

=== added file 'IPython/config/default/ipcluster_config.py'
--- IPython/config/default/ipcluster_config.py	1970-01-01 00:00:00 +0000
+++ IPython/config/default/ipcluster_config.py	2009-11-21 00:11:10 +0000
@@ -0,0 +1,184 @@
+import os
+
+c = get_config()
+
+#-----------------------------------------------------------------------------
+# Select which launchers to use
+#-----------------------------------------------------------------------------
+
+# This allows you to control what method is used to start the controller
+# and engines.  The following methods are currently supported:
+# - Start as a regular process on localhost.
+# - Start using mpiexec.
+# - Start using the Windows HPC Server 2008 scheduler
+# - Start using PBS
+# - Start using SSH (currently broken)
+
+
+# The selected launchers can be configured below.
+
+# Options are:
+# - LocalControllerLauncher
+# - MPIExecControllerLauncher
+# - PBSControllerLauncher
+# - WindowsHPCControllerLauncher
+# c.Global.controller_launcher = 'IPython.kernel.launcher.LocalControllerLauncher'
+
+# Options are:
+# - LocalEngineSetLauncher
+# - MPIExecEngineSetLauncher
+# - PBSEngineSetLauncher
+# - WindowsHPCEngineSetLauncher
+# c.Global.engine_launcher = 'IPython.kernel.launcher.LocalEngineSetLauncher'
+
+#-----------------------------------------------------------------------------
+# Global configuration
+#-----------------------------------------------------------------------------
+
+# The default number of engines that will be started. This is overridden by
+# the -n command line option: "ipcluster start -n 4"
+# c.Global.n = 2
+
+# Log to a file in cluster_dir/log, otherwise just log to sys.stdout.
+# c.Global.log_to_file = False
+
+# Remove old logs from cluster_dir/log before starting.
+# c.Global.clean_logs = True
+
+# The working directory for the process. The application will use os.chdir
+# to change to this directory before starting.
+# c.Global.work_dir = os.getcwd()
+
+
+#-----------------------------------------------------------------------------
+# Local process launchers
+#-----------------------------------------------------------------------------
+
+# The command line arguments to call the controller with.
+# c.LocalControllerLauncher.controller_args = \
+#    ['--log-to-file','--log-level', '40']
+
+# The working directory for the controller
+# c.LocalEngineSetLauncher.work_dir = u''
+
+# Command line argument passed to the engines.
+# c.LocalEngineSetLauncher.engine_args = ['--log-to-file','--log-level', '40']
+
+#-----------------------------------------------------------------------------
+# MPIExec launchers
+#-----------------------------------------------------------------------------
+
+# The mpiexec/mpirun command to use in started the controller.
+# c.MPIExecControllerLauncher.mpi_cmd = ['mpiexec']
+
+# Additional arguments to pass to the actual mpiexec command.
+# c.MPIExecControllerLauncher.mpi_args = []
+
+# The command line argument to call the controller with.
+# c.MPIExecControllerLauncher.controller_args = \
+#     ['--log-to-file','--log-level', '40']
+
+
+# The mpiexec/mpirun command to use in started the controller.
+# c.MPIExecEngineSetLauncher.mpi_cmd = ['mpiexec']
+
+# Additional arguments to pass to the actual mpiexec command.
+# c.MPIExecEngineSetLauncher.mpi_args = []
+
+# Command line argument passed to the engines.
+# c.MPIExecEngineSetLauncher.engine_args = ['--log-to-file','--log-level', '40']
+
+# The default number of engines to start if not given elsewhere.
+# c.MPIExecEngineSetLauncher.n = 1
+
+#-----------------------------------------------------------------------------
+# SSH launchers
+#-----------------------------------------------------------------------------
+
+# Todo
+
+
+#-----------------------------------------------------------------------------
+# Unix batch (PBS) schedulers launchers
+#-----------------------------------------------------------------------------
+
+# The command line program to use to submit a PBS job.
+# c.PBSControllerLauncher.submit_command = 'qsub'
+
+# The command line program to use to delete a PBS job.
+# c.PBSControllerLauncher.delete_command = 'qdel'
+
+# A regular expression that takes the output of qsub and find the job id.
+# c.PBSControllerLauncher.job_id_regexp = r'\d+'
+
+# The batch submission script used to start the controller. This is where
+# environment variables would be setup, etc. This string is interpolated using
+# the Itpl module in IPython.external. Basically, you can use ${n} for the
+# number of engine and ${cluster_dir} for the cluster_dir.
+# c.PBSControllerLauncher.batch_template = """"""
+
+# The name of the instantiated batch script that will actually be used to
+# submit the job. This will be written to the cluster directory.
+# c.PBSControllerLauncher.batch_file_name = u'pbs_batch_script_controller'
+
+
+# The command line program to use to submit a PBS job.
+# c.PBSEngineSetLauncher.submit_command = 'qsub'
+
+# The command line program to use to delete a PBS job.
+# c.PBSEngineSetLauncher.delete_command = 'qdel'
+
+# A regular expression that takes the output of qsub and find the job id.
+# c.PBSEngineSetLauncher.job_id_regexp = r'\d+'
+
+# The batch submission script used to start the engines. This is where
+# environment variables would be setup, etc. This string is interpolated using
+# the Itpl module in IPython.external. Basically, you can use ${n} for the
+# number of engine and ${cluster_dir} for the cluster_dir.
+# c.PBSEngineSetLauncher.batch_template = """"""
+
+# The name of the instantiated batch script that will actually be used to
+# submit the job. This will be written to the cluster directory.
+# c.PBSEngineSetLauncher.batch_file_name = u'pbs_batch_script_engines'
+
+#-----------------------------------------------------------------------------
+# Windows HPC Server 2008 launcher configuration
+#-----------------------------------------------------------------------------
+
+# c.IPControllerJob.job_name = 'IPController'
+# c.IPControllerJob.is_exclusive = False
+# c.IPControllerJob.username = r'USERDOMAIN\USERNAME'
+# c.IPControllerJob.priority = 'Highest'
+# c.IPControllerJob.requested_nodes = ''
+# c.IPControllerJob.project = 'MyProject'
+
+# c.IPControllerTask.task_name = 'IPController'
+# c.IPControllerTask.controller_cmd = [u'ipcontroller.exe']
+# c.IPControllerTask.controller_args = ['--log-to-file', '--log-level', '40']
+# c.IPControllerTask.environment_variables = {}
+
+# c.WindowsHPCControllerLauncher.scheduler = 'HEADNODE'
+# c.WindowsHPCControllerLauncher.job_file_name = u'ipcontroller_job.xml'
+
+
+# c.IPEngineSetJob.job_name = 'IPEngineSet'
+# c.IPEngineSetJob.is_exclusive = False
+# c.IPEngineSetJob.username = r'USERDOMAIN\USERNAME'
+# c.IPEngineSetJob.priority = 'Highest'
+# c.IPEngineSetJob.requested_nodes = ''
+# c.IPEngineSetJob.project = 'MyProject'
+
+# c.IPEngineTask.task_name = 'IPEngine'
+# c.IPEngineTask.engine_cmd = [u'ipengine.exe']
+# c.IPEngineTask.engine_args = ['--log-to-file', '--log-level', '40']
+# c.IPEngineTask.environment_variables = {}
+
+# c.WindowsHPCEngineSetLauncher.scheduler = 'HEADNODE'
+# c.WindowsHPCEngineSetLauncher.job_file_name = u'ipengineset_job.xml'
+
+
+
+
+
+
+

=== added file 'IPython/config/default/ipcontroller_config.py'
--- IPython/config/default/ipcontroller_config.py	1970-01-01 00:00:00 +0000
+++ IPython/config/default/ipcontroller_config.py	2009-11-21 00:11:10 +0000
@@ -0,0 +1,136 @@
+from IPython.config.loader import Config
+
+c = get_config()
+
+#-----------------------------------------------------------------------------
+# Global configuration
+#-----------------------------------------------------------------------------
+
+# Basic Global config attributes
+
+# Start up messages are logged to stdout using the logging module.
+# These all happen before the twisted reactor is started and are
+# useful for debugging purposes. Can be (10=DEBUG,20=INFO,30=WARN,40=CRITICAL) 
+# and smaller is more verbose.
+# c.Global.log_level = 20
+
+# Log to a file in cluster_dir/log, otherwise just log to sys.stdout.
+# c.Global.log_to_file = False
+
+# Remove old logs from cluster_dir/log before starting.
+# c.Global.clean_logs = True
+
+# A list of Python statements that will be run before starting the 
+# controller. This is provided because occasionally certain things need to 
+# be imported in the controller for pickling to work.
+# c.Global.import_statements = ['import math']
+
+# Reuse the controller's FURL files. If False, FURL files are regenerated
+# each time the controller is run. If True, they will be reused, *but*, you
+# also must set the network ports by hand. If set, this will override the
+# values set for the client and engine connections below.
+# c.Global.reuse_furls = True
+
+# Enable SSL encryption on all connections to the controller. If set, this
+# will override the values set for the client and engine connections below.
+# c.Global.secure = True
+
+# The working directory for the process. The application will use os.chdir
+# to change to this directory before starting.
+# c.Global.work_dir = os.getcwd()
+
+#-----------------------------------------------------------------------------
+# Configure the client services
+#-----------------------------------------------------------------------------
+
+# Basic client service config attributes
+
+# The network interface the controller will listen on for client connections.
+# This should be an IP address or hostname of the controller's host. The empty
+# string means listen on all interfaces.
+# c.FCClientServiceFactory.ip = ''
+
+# The TCP/IP port the controller will listen on for client connections. If 0
+# a random port will be used. If the controller's host has a firewall running
+# it must allow incoming traffic on this port.
+# c.FCClientServiceFactory.port = 0
+
+# The client learns how to connect to the controller by looking at the
+# location field embedded in the FURL. If this field is empty, all network
+# interfaces that the controller is listening on will be listed. To have the
+# client connect on a particular interface, list it here.
+# c.FCClientServiceFactory.location = ''
+
+# Use SSL encryption for the client connection.
+# c.FCClientServiceFactory.secure =  True
+
+# Reuse the client FURL each time the controller is started. If set, you must
+# also pick a specific network port above (FCClientServiceFactory.port).
+# c.FCClientServiceFactory.reuse_furls = False
+
+#-----------------------------------------------------------------------------
+# Configure the engine services
+#-----------------------------------------------------------------------------
+
+# Basic config attributes for the engine services.
+
+# The network interface the controller will listen on for engine connections.
+# This should be an IP address or hostname of the controller's host. The empty
+# string means listen on all interfaces.
+# c.FCEngineServiceFactory.ip = ''
+
+# The TCP/IP port the controller will listen on for engine connections. If 0
+# a random port will be used. If the controller's host has a firewall running
+# it must allow incoming traffic on this port.
+# c.FCEngineServiceFactory.port = 0
+
+# The engine learns how to connect to the controller by looking at the
+# location field embedded in the FURL. If this field is empty, all network
+# interfaces that the controller is listening on will be listed. To have the
+# client connect on a particular interface, list it here.
+# c.FCEngineServiceFactory.location = ''
+
+# Use SSL encryption for the engine connection.
+# c.FCEngineServiceFactory.secure = True
+
+# Reuse the client FURL each time the controller is started. If set, you must
+# also pick a specific network port above (FCClientServiceFactory.port).
+# c.FCEngineServiceFactory.reuse_furls = False
+
+#-----------------------------------------------------------------------------
+# Developer level configuration attributes
+#-----------------------------------------------------------------------------
+
+# You shouldn't have to modify anything in this section. These attributes
+# are more for developers who want to change the behavior of the controller
+# at a fundamental level.
+
+# c.FCClientServiceFactory.cert_file = u'ipcontroller-client.pem'
+
+# default_client_interfaces = Config()
+# default_client_interfaces.Task.interface_chain = [
+#     'IPython.kernel.task.ITaskController',
+#     'IPython.kernel.taskfc.IFCTaskController'
+# ]
+# 
+# default_client_interfaces.Task.furl_file = u'ipcontroller-tc.furl'
+# 
+# default_client_interfaces.MultiEngine.interface_chain = [
+#     'IPython.kernel.multiengine.IMultiEngine',
+#     'IPython.kernel.multienginefc.IFCSynchronousMultiEngine'
+# ]
+# 
+# default_client_interfaces.MultiEngine.furl_file = u'ipcontroller-mec.furl'
+# 
+# c.FCEngineServiceFactory.interfaces = default_client_interfaces
+
+# c.FCEngineServiceFactory.cert_file = u'ipcontroller-engine.pem'
+
+# default_engine_interfaces = Config()
+# default_engine_interfaces.Default.interface_chain = [
+#     'IPython.kernel.enginefc.IFCControllerBase'
+# ]
+# 
+# default_engine_interfaces.Default.furl_file = u'ipcontroller-engine.furl'
+# 
+# c.FCEngineServiceFactory.interfaces = default_engine_interfaces

=== added file 'IPython/config/default/ipengine_config.py'
--- IPython/config/default/ipengine_config.py	1970-01-01 00:00:00 +0000
+++ IPython/config/default/ipengine_config.py	2009-11-21 00:11:10 +0000
@@ -0,0 +1,90 @@
+c = get_config()
+
+#-----------------------------------------------------------------------------
+# Global configuration
+#-----------------------------------------------------------------------------
+
+# Start up messages are logged to stdout using the logging module.
+# These all happen before the twisted reactor is started and are
+# useful for debugging purposes. Can be (10=DEBUG,20=INFO,30=WARN,40=CRITICAL) 
+# and smaller is more verbose.
+# c.Global.log_level = 20
+
+# Log to a file in cluster_dir/log, otherwise just log to sys.stdout.
+# c.Global.log_to_file = False
+
+# Remove old logs from cluster_dir/log before starting.
+# c.Global.clean_logs = True
+
+# A list of strings that will be executed in the users namespace on the engine
+# before it connects to the controller.
+# c.Global.exec_lines = ['import numpy']
+
+# The engine will try to connect to the controller multiple times, to allow
+# the controller time to startup and write its FURL file. These parameters 
+# control the number of retries (connect_max_tries) and the initial delay
+# (connect_delay) between attemps. The actual delay between attempts gets
+# longer each time by a factor of 1.5 (delay[i] = 1.5*delay[i-1])
+# those attemps.
+# c.Global.connect_delay = 0.1
+# c.Global.connect_max_tries = 15
+
+# By default, the engine will look for the controller's FURL file in its own
+# cluster directory. Sometimes, the FURL file will be elsewhere and this 
+# attribute can be set to the full path of the FURL file.
+# c.Global.furl_file = u''
+
+# The working directory for the process. The application will use os.chdir
+# to change to this directory before starting.
+# c.Global.work_dir = os.getcwd()
+
+#-----------------------------------------------------------------------------
+# MPI configuration
+#-----------------------------------------------------------------------------
+
+# Upon starting the engine can be configured to call MPI_Init. This section
+# configures that.
+
+# Select which MPI section to execute to setup MPI. The value of this 
+# attribute must match the name of another attribute in the MPI config 
+# section (mpi4py, pytrilinos, etc.). This can also be set by the --mpi
+# command line option.
+# c.MPI.use = ''
+
+# Initialize MPI using mpi4py. To use this, set c.MPI.use = 'mpi4py' to use
+# --mpi=mpi4py at the command line.
+# c.MPI.mpi4py = """from mpi4py import MPI as mpi
+# mpi.size = mpi.COMM_WORLD.Get_size()
+# mpi.rank = mpi.COMM_WORLD.Get_rank()
+# """
+
+# Initialize MPI using pytrilinos. To use this, set c.MPI.use = 'pytrilinos' 
+# to use --mpi=pytrilinos at the command line.
+# c.MPI.pytrilinos = """from PyTrilinos import Epetra
+# class SimpleStruct:
+# pass
+# mpi = SimpleStruct()
+# mpi.rank = 0
+# mpi.size = 0
+# """
+
+#-----------------------------------------------------------------------------
+# Developer level configuration attributes
+#-----------------------------------------------------------------------------
+
+# You shouldn't have to modify anything in this section. These attributes
+# are more for developers who want to change the behavior of the controller
+# at a fundamental level.
+
+# You should not have to change these attributes.
+
+# c.Global.shell_class = 'IPython.kernel.core.interpreter.Interpreter'
+
+# c.Global.furl_file_name = u'ipcontroller-engine.furl'
+
+
+
+
+
+
+

=== modified file 'IPython/config/default/ipython_config.py'
--- IPython/config/default/ipython_config.py	2009-09-28 05:57:44 +0000
+++ IPython/config/default/ipython_config.py	2009-11-21 00:11:10 +0000
@@ -13,7 +13,7 @@
 
 # Set this to determine the detail of what is logged at startup.
 # The default is 30 and possible values are 0,10,20,30,40,50.
-c.Global.log_level = 20
+# c.Global.log_level = 20
 
 # This should be a list of importable Python modules that have an
 # load_in_ipython(ip) method.  This method gets called when the extension
@@ -35,7 +35,7 @@
 # These files are run in IPython in the user's namespace.  Files with a .py
 # extension need to be pure Python.  Files with a .ipy extension can have
 # custom IPython syntax (like magics, etc.).  
-# These files need to be in the cwd, the ipythondir or be absolute paths.
+# These files need to be in the cwd, the ipython_dir or be absolute paths.
 # c.Global.exec_files = [
 #     'mycode.py',
 #     'fancy.ipy'
@@ -71,9 +71,9 @@
 
 # c.InteractiveShell.logstart = True
 
-# c.InteractiveShell.logfile = 'ipython_log.py'
+# c.InteractiveShell.logfile = u'ipython_log.py'
 
-# c.InteractiveShell.logappend = 'mylog.py'
+# c.InteractiveShell.logappend = u'mylog.py'
 
 # c.InteractiveShell.object_info_string_level = 0
 

=== modified file 'IPython/config/loader.py'
--- IPython/config/loader.py	2009-09-15 05:54:31 +0000
+++ IPython/config/loader.py	2009-11-21 00:11:10 +0000
@@ -244,8 +244,14 @@
         # with the parents.
         def load_subconfig(fname):
             loader = PyFileConfigLoader(fname, self.path)
-            sub_config = loader.load_config()
-            self.config._merge(sub_config)
+            try:
+                sub_config = loader.load_config()
+            except IOError:
+                # Pass silently if the sub config is not there. This happens
+                # when a user us using a profile, but not the default config.
+                pass
+            else:
+                self.config._merge(sub_config)
 
         # Again, this needs to be a closure and should be used in config
         # files to get the config being loaded.
@@ -271,6 +277,7 @@
 class NoConfigDefault(object): pass
 NoConfigDefault = NoConfigDefault()
 
+
 class ArgParseConfigLoader(CommandLineConfigLoader):
     
     # arguments = [(('-f','--file'),dict(type=str,dest='file'))]

=== renamed file 'IPython/config/profile/__init_.py' => 'IPython/config/profile/__init__.py'
=== added file 'IPython/config/profile/ipython_config_cluster.py'
--- IPython/config/profile/ipython_config_cluster.py	1970-01-01 00:00:00 +0000
+++ IPython/config/profile/ipython_config_cluster.py	2009-11-21 00:11:10 +0000
@@ -0,0 +1,24 @@
+c = get_config()
+
+# This can be used at any point in a config file to load a sub config
+# and merge it into the current one.
+load_subconfig('ipython_config.py')
+
+lines = """
+from IPython.kernel.client import *
+"""
+
+# You have to make sure that attributes that are containers already
+# exist before using them.  Simple assigning a new list will override
+# all previous values.
+if hasattr(c.Global, 'exec_lines'):
+    c.Global.exec_lines.append(lines)
+else:
+    c.Global.exec_lines = [lines]
+
+# Load the parallelmagic extension to enable %result, %px, %autopx magics.
+if hasattr(c.Global, 'extensions'):
+    c.Global.extensions.append('parallelmagic')
+else:
+    c.Global.extensions = ['parallelmagic']
+

=== modified file 'IPython/core/application.py'
--- IPython/core/application.py	2009-09-18 03:26:43 +0000
+++ IPython/core/application.py	2009-11-21 00:11:10 +0000
@@ -1,7 +1,13 @@
 #!/usr/bin/env python
 # encoding: utf-8
 """
-An application for IPython
+An application for IPython.
+
+All top-level applications should use the classes in this module for
+handling configuration and creating componenets.
+
+The job of an :class:`Application` is to create the master configuration 
+object and then create the components, passing the config to them.
 
 Authors:
 
@@ -26,10 +32,9 @@
 import logging
 import os
 import sys
-import traceback
-from copy import deepcopy
 
-from IPython.utils.genutils import get_ipython_dir, filefind
+from IPython.core import release
+from IPython.utils.genutils import get_ipython_dir
 from IPython.config.loader import (
     PyFileConfigLoader,
     ArgParseConfigLoader,
@@ -42,22 +47,27 @@
 #-----------------------------------------------------------------------------
 
 
-class IPythonArgParseConfigLoader(ArgParseConfigLoader):
+class BaseAppArgParseConfigLoader(ArgParseConfigLoader):
     """Default command line options for IPython based applications."""
 
     def _add_other_arguments(self):
-        self.parser.add_argument('-ipythondir',dest='Global.ipythondir',type=str,
-            help='Set to override default location of Global.ipythondir.',
+        self.parser.add_argument('--ipython-dir', 
+            dest='Global.ipython_dir',type=unicode,
+            help='Set to override default location of Global.ipython_dir.',
             default=NoConfigDefault,
-            metavar='Global.ipythondir')
-        self.parser.add_argument('-p','-profile',dest='Global.profile',type=str,
+            metavar='Global.ipython_dir')
+        self.parser.add_argument('-p', '--profile',
+            dest='Global.profile',type=unicode,
             help='The string name of the ipython profile to be used.',
             default=NoConfigDefault,
             metavar='Global.profile')
-        self.parser.add_argument('-log_level',dest="Global.log_level",type=int,
+        self.parser.add_argument('--log-level',
+            dest="Global.log_level",type=int,
             help='Set the log level (0,10,20,30,40,50).  Default is 30.',
-            default=NoConfigDefault)
-        self.parser.add_argument('-config_file',dest='Global.config_file',type=str,
+            default=NoConfigDefault,
+            metavar='Global.log_level')
+        self.parser.add_argument('--config-file',
+            dest='Global.config_file',type=unicode,
             help='Set the config file name to override default.',
             default=NoConfigDefault,
             metavar='Global.config_file')
@@ -68,20 +78,24 @@
 
 
 class Application(object):
-    """Load a config, construct an app and run it.
-    """
+    """Load a config, construct components and set them running."""
 
-    config_file_name = 'ipython_config.py'
-    name = 'ipython'
+    name = u'ipython'
+    description = 'IPython: an enhanced interactive Python shell.'
+    config_file_name = u'ipython_config.py'
+    default_log_level = logging.WARN
 
     def __init__(self):
+        self._exiting = False
         self.init_logger()
+        # Track the default and actual separately because some messages are
+        # only printed if we aren't using the default.
         self.default_config_file_name = self.config_file_name
 
     def init_logger(self):
         self.log = logging.getLogger(self.__class__.__name__)
         # This is used as the default until the command line arguments are read.
-        self.log.setLevel(logging.WARN)
+        self.log.setLevel(self.default_log_level)
         self._log_handler = logging.StreamHandler()
         self._log_formatter = logging.Formatter("[%(name)s] %(message)s")
         self._log_handler.setFormatter(self._log_formatter)
@@ -98,16 +112,24 @@
     def start(self):
         """Start the application."""
         self.attempt(self.create_default_config)
+        self.log_default_config()
+        self.set_default_config_log_level()
         self.attempt(self.pre_load_command_line_config)
         self.attempt(self.load_command_line_config, action='abort')
+        self.set_command_line_config_log_level()
         self.attempt(self.post_load_command_line_config)
-        self.attempt(self.find_ipythondir)
+        self.log_command_line_config()
+        self.attempt(self.find_ipython_dir)
+        self.attempt(self.find_resources)
         self.attempt(self.find_config_file_name)
         self.attempt(self.find_config_file_paths)
         self.attempt(self.pre_load_file_config)
         self.attempt(self.load_file_config)
+        self.set_file_config_log_level()
         self.attempt(self.post_load_file_config)
+        self.log_file_config()
         self.attempt(self.merge_configs)
+        self.log_master_config()
         self.attempt(self.pre_construct)
         self.attempt(self.construct)
         self.attempt(self.post_construct)
@@ -127,65 +149,89 @@
         don't belong to a particular component.
         """
         self.default_config = Config()
-        self.default_config.Global.ipythondir = get_ipython_dir()
+        self.default_config.Global.ipython_dir = get_ipython_dir()
+        self.default_config.Global.log_level = self.log_level
+
+    def log_default_config(self):
         self.log.debug('Default config loaded:')
         self.log.debug(repr(self.default_config))
 
+    def set_default_config_log_level(self):
+        try:
+            self.log_level = self.default_config.Global.log_level
+        except AttributeError:
+            # Fallback to the default_log_level class attribute
+            pass
+
     def create_command_line_config(self):
         """Create and return a command line config loader."""
-        return IPythonArgParseConfigLoader(description=self.name)
+        return BaseAppArgParseConfigLoader(
+            description=self.description, 
+            version=release.version
+        )
 
     def pre_load_command_line_config(self):
         """Do actions just before loading the command line config."""
         pass
 
     def load_command_line_config(self):
-        """Load the command line config.
-
-        This method also sets ``self.debug``.
-        """
-
+        """Load the command line config."""
         loader = self.create_command_line_config()
         self.command_line_config = loader.load_config()
         self.extra_args = loader.get_extra_args()
 
+    def set_command_line_config_log_level(self):
         try:
             self.log_level = self.command_line_config.Global.log_level
         except AttributeError:
-            pass # Use existing value which is set in Application.init_logger.
+            pass
+
+    def post_load_command_line_config(self):
+        """Do actions just after loading the command line config."""
+        pass
+
+    def log_command_line_config(self):
         self.log.debug("Command line config loaded:")
         self.log.debug(repr(self.command_line_config))
 
-    def post_load_command_line_config(self):
-        """Do actions just after loading the command line config."""
-        pass
-
-    def find_ipythondir(self):
+    def find_ipython_dir(self):
         """Set the IPython directory.
 
-        This sets ``self.ipythondir``, but the actual value that is passed
+        This sets ``self.ipython_dir``, but the actual value that is passed
         to the application is kept in either ``self.default_config`` or
-        ``self.command_line_config``.  This also added ``self.ipythondir`` to
+        ``self.command_line_config``.  This also adds ``self.ipython_dir`` to
         ``sys.path`` so config files there can be references by other config
         files.
         """
 
         try:
-            self.ipythondir = self.command_line_config.Global.ipythondir
+            self.ipython_dir = self.command_line_config.Global.ipython_dir
         except AttributeError:
-            self.ipythondir = self.default_config.Global.ipythondir
-        sys.path.append(os.path.abspath(self.ipythondir))
-        if not os.path.isdir(self.ipythondir):
-            os.makedirs(self.ipythondir, mode = 0777)
-        self.log.debug("IPYTHONDIR set to: %s" % self.ipythondir)
+            self.ipython_dir = self.default_config.Global.ipython_dir
+        sys.path.append(os.path.abspath(self.ipython_dir))
+        if not os.path.isdir(self.ipython_dir):
+            os.makedirs(self.ipython_dir, mode=0777)
+        self.log.debug("IPYTHON_DIR set to: %s" % self.ipython_dir)
+
+    def find_resources(self):
+        """Find other resources that need to be in place.
+
+        Things like cluster directories need to be in place to find the
+        config file.  These happen right after the IPython directory has
+        been set.
+        """
+        pass
 
     def find_config_file_name(self):
         """Find the config file name for this application.
 
+        This must set ``self.config_file_name`` to the filename of the
+        config file to use (just the filename). The search paths for the
+        config file are set in :meth:`find_config_file_paths` and then passed
+        to the config file loader where they are resolved to an absolute path.
+
         If a profile has been set at the command line, this will resolve
-        it.  The search paths for the config file are set in
-        :meth:`find_config_file_paths` and then passed to the config file
-        loader where they are resolved to an absolute path.
+        it.
         """
 
         try:
@@ -196,14 +242,18 @@
         try:
             self.profile_name = self.command_line_config.Global.profile
             name_parts = self.config_file_name.split('.')
-            name_parts.insert(1, '_' + self.profile_name + '.')
+            name_parts.insert(1, u'_' + self.profile_name + u'.')
             self.config_file_name = ''.join(name_parts)
         except AttributeError:
             pass
 
     def find_config_file_paths(self):
-        """Set the search paths for resolving the config file."""
-        self.config_file_paths = (os.getcwd(), self.ipythondir)
+        """Set the search paths for resolving the config file.
+
+        This must set ``self.config_file_paths`` to a sequence of search
+        paths to pass to the config file loader.
+        """
+        self.config_file_paths = (os.getcwd(), self.ipython_dir)
 
     def pre_load_file_config(self):
         """Do actions before the config file is loaded."""
@@ -216,7 +266,7 @@
         ``CONFIG_FILE`` config variable is set to the resolved config file
         location.  If not successful, an empty config is used.
         """
-        self.log.debug("Attempting to load config file: <%s>" % self.config_file_name)
+        self.log.debug("Attempting to load config file: %s" % self.config_file_name)
         loader = PyFileConfigLoader(self.config_file_name,
                                     path=self.config_file_paths)
         try:
@@ -225,19 +275,18 @@
         except IOError:
             # Only warn if the default config file was NOT being used.
             if not self.config_file_name==self.default_config_file_name:
-                self.log.warn("Config file not found, skipping: <%s>" % \
+                self.log.warn("Config file not found, skipping: %s" % \
                                self.config_file_name, exc_info=True)
             self.file_config = Config()
         except:
-            self.log.warn("Error loading config file: <%s>" % \
-                           self.config_file_name, exc_info=True)
+            self.log.warn("Error loading config file: %s" % \
+                          self.config_file_name, exc_info=True)
             self.file_config = Config()
-        else:
-            self.log.debug("Config file loaded: <%s>" % loader.full_filename)
-            self.log.debug(repr(self.file_config))
+
+    def set_file_config_log_level(self):
         # We need to keeep self.log_level updated.  But we only use the value
         # of the file_config if a value was not specified at the command
-        # line.
+        # line, because the command line overrides everything.
         if not hasattr(self.command_line_config.Global, 'log_level'):
             try:
                 self.log_level = self.file_config.Global.log_level
@@ -248,6 +297,11 @@
         """Do actions after the config file is loaded."""
         pass
 
+    def log_file_config(self):
+        if hasattr(self.file_config.Global, 'config_file'):
+            self.log.debug("Config file loaded: %s" % self.file_config.Global.config_file)
+            self.log.debug(repr(self.file_config))
+
     def merge_configs(self):
         """Merge the default, command line and file config objects."""
         config = Config()
@@ -255,6 +309,8 @@
         config._merge(self.file_config)
         config._merge(self.command_line_config)
         self.master_config = config
+
+    def log_master_config(self):
         self.log.debug("Master config created:")
         self.log.debug(repr(self.master_config))
 
@@ -280,21 +336,29 @@
 
     def abort(self):
         """Abort the starting of the application."""
-        self.log.critical("Aborting application: %s" % self.name, exc_info=True)
-        sys.exit(1)
+        if self._exiting:
+            pass
+        else:
+            self.log.critical("Aborting application: %s" % self.name, exc_info=True)
+            self._exiting = True
+            sys.exit(1)
 
-    def exit(self):
-        self.log.critical("Aborting application: %s" % self.name)
-        sys.exit(1)
+    def exit(self, exit_status=0):
+        if self._exiting:
+            pass
+        else:
+            self.log.debug("Exiting application: %s" % self.name)
+            self._exiting = True
+            sys.exit(exit_status)
 
     def attempt(self, func, action='abort'):
         try:
             func()
         except SystemExit:
-            self.exit()
+            raise
         except:
             if action == 'abort':
                 self.abort()
             elif action == 'exit':
-                self.exit()
-      
\ No newline at end of file
+                self.exit(0)
+

=== modified file 'IPython/core/builtin_trap.py'
--- IPython/core/builtin_trap.py	2009-10-09 22:54:40 +0000
+++ IPython/core/builtin_trap.py	2009-11-21 00:11:10 +0000
@@ -86,6 +86,7 @@
         """Store ipython references in the __builtin__ namespace."""
         self.add_builtin('exit', Quitter(self.shell, 'exit'))
         self.add_builtin('quit', Quitter(self.shell, 'quit'))
+        self.add_builtin('get_ipython', self.shell.get_ipython)
 
         # Recursive reload function
         try:

=== modified file 'IPython/core/component.py'
--- IPython/core/component.py	2009-09-28 05:57:44 +0000
+++ IPython/core/component.py	2009-11-21 00:11:11 +0000
@@ -237,14 +237,20 @@
             self.config = config
             # We used to deepcopy, but for now we are trying to just save
             # by reference.  This *could* have side effects as all components
-            # will share config.
+            # will share config. In fact, I did find such a side effect in
+            # _config_changed below. If a config attribute value was a mutable type
+            # all instances of a component were getting the same copy, effectively
+            # making that a class attribute.
             # self.config = deepcopy(config)
         else:
             if self.parent is not None:
                 self.config = self.parent.config
                 # We used to deepcopy, but for now we are trying to just save
                 # by reference.  This *could* have side effects as all components
-                # will share config.
+                # will share config. In fact, I did find such a side effect in
+                # _config_changed below. If a config attribute value was a mutable type
+                # all instances of a component were getting the same copy, effectively
+                # making that a class attribute.
                 # self.config = deepcopy(self.parent.config)
 
         self.created = datetime.datetime.now()
@@ -296,14 +302,29 @@
             if new._has_section(sname):
                 my_config = new[sname]
                 for k, v in traitlets.items():
+                    # Don't allow traitlets with config=True to start with
+                    # uppercase.  Otherwise, they are confused with Config
+                    # subsections.  But, developers shouldn't have uppercase
+                    # attributes anyways! (PEP 6)
+                    if k[0].upper()==k[0] and not k.startswith('_'):
+                        raise ComponentError('Component traitlets with '
+                        'config=True must start with a lowercase so they are '
+                        'not confused with Config subsections: %s.%s' % \
+                        (self.__class__.__name__, k))
                     try:
+                        # Here we grab the value from the config
+                        # If k has the naming convention of a config
+                        # section, it will be auto created.
                         config_value = my_config[k]
                     except KeyError:
                         pass
                     else:
                         # print "Setting %s.%s from %s.%s=%r" % \
                         #     (self.__class__.__name__,k,sname,k,config_value)
-                        setattr(self, k, config_value)
+                        # We have to do a deepcopy here if we don't deepcopy the entire
+                        # config object. If we don't, a mutable config_value will be
+                        # shared by all instances, effectively making it a class attribute.
+                        setattr(self, k, deepcopy(config_value))
 
     @property
     def children(self):

=== modified file 'IPython/core/crashhandler.py'
--- IPython/core/crashhandler.py	2009-08-26 20:54:07 +0000
+++ IPython/core/crashhandler.py	2009-11-21 00:11:11 +0000
@@ -124,7 +124,7 @@
         #color_scheme = 'Linux'   # dbg
         
         try:
-            rptdir = self.IP.config.IPYTHONDIR
+            rptdir = self.IP.ipython_dir
         except:
             rptdir = os.getcwd()
         if not os.path.isdir(rptdir):

=== modified file 'IPython/core/debugger.py'
--- IPython/core/debugger.py	2009-08-19 21:56:41 +0000
+++ IPython/core/debugger.py	2009-11-21 00:11:11 +0000
@@ -70,6 +70,7 @@
 def BdbQuit_IPython_excepthook(self,et,ev,tb):
     print 'Exiting Debugger.'
 
+
 class Tracer(object):
     """Class for local debugging, similar to pdb.set_trace.
 
@@ -105,12 +106,10 @@
         from the Python standard library for usage details.
         """
 
-        global __IPYTHON__
         try:
-            __IPYTHON__
-        except NameError:
+            ip = ipapi.get()
+        except:
             # Outside of ipython, we set our own exception hook manually
-            __IPYTHON__ = ipapi.get()
             BdbQuit_excepthook.excepthook_ori = sys.excepthook
             sys.excepthook = BdbQuit_excepthook
             def_colors = 'NoColor'
@@ -122,9 +121,8 @@
                 pass
         else:
             # In ipython, we use its custom exception handler mechanism
-            ip = ipapi.get()
             def_colors = ip.colors
-            ip.set_custom_exc((bdb.BdbQuit,),BdbQuit_IPython_excepthook)
+            ip.set_custom_exc((bdb.BdbQuit,), BdbQuit_IPython_excepthook)
 
         if colors is None:
             colors = def_colors
@@ -138,6 +136,7 @@
         
         self.debugger.set_trace(sys._getframe().f_back)
 
+
 def decorate_fn_with_doc(new_fn, old_fn, additional_text=""):
     """Make new_fn have old_fn's doc string. This is particularly useful
     for the do_... commands that hook into the help system.
@@ -149,6 +148,7 @@
         wrapper.__doc__ = old_fn.__doc__ + additional_text
     return wrapper
 
+
 def _file_lines(fname):
     """Return the contents of a named file as a list of lines.
 
@@ -164,143 +164,98 @@
         outfile.close()
         return out
 
+
 class Pdb(OldPdb):
     """Modified Pdb class, does not load readline."""
 
-    if sys.version[:3] >= '2.5' or has_pydb:
-        def __init__(self,color_scheme='NoColor',completekey=None,
-                     stdin=None, stdout=None):
+    def __init__(self,color_scheme='NoColor',completekey=None,
+                 stdin=None, stdout=None):
 
-            # Parent constructor:
-            if has_pydb and completekey is None:
-                OldPdb.__init__(self,stdin=stdin,stdout=Term.cout)
-            else:
-                OldPdb.__init__(self,completekey,stdin,stdout)
-                
-            self.prompt = prompt # The default prompt is '(Pdb)'
+        # Parent constructor:
+        if has_pydb and completekey is None:
+            OldPdb.__init__(self,stdin=stdin,stdout=Term.cout)
+        else:
+            OldPdb.__init__(self,completekey,stdin,stdout)
             
-            # IPython changes...
-            self.is_pydb = has_pydb
-
-            if self.is_pydb:
-
-                # iplib.py's ipalias seems to want pdb's checkline
-                # which located in pydb.fn
-                import pydb.fns
-                self.checkline = lambda filename, lineno: \
-                                 pydb.fns.checkline(self, filename, lineno)
-
-                self.curframe = None
-                self.do_restart = self.new_do_restart
-
-                self.old_all_completions = __IPYTHON__.Completer.all_completions
-                __IPYTHON__.Completer.all_completions=self.all_completions
-
-                self.do_list = decorate_fn_with_doc(self.list_command_pydb,
-                                                    OldPdb.do_list)
-                self.do_l     = self.do_list
-                self.do_frame = decorate_fn_with_doc(self.new_do_frame,
-                                                     OldPdb.do_frame)
-
-            self.aliases = {}
-
-            # Create color table: we copy the default one from the traceback
-            # module and add a few attributes needed for debugging
-            self.color_scheme_table = exception_colors()
-
-            # shorthands 
-            C = coloransi.TermColors
-            cst = self.color_scheme_table
-
-            cst['NoColor'].colors.breakpoint_enabled = C.NoColor
-            cst['NoColor'].colors.breakpoint_disabled = C.NoColor
-
-            cst['Linux'].colors.breakpoint_enabled = C.LightRed
-            cst['Linux'].colors.breakpoint_disabled = C.Red
-
-            cst['LightBG'].colors.breakpoint_enabled = C.LightRed
-            cst['LightBG'].colors.breakpoint_disabled = C.Red
-
-            self.set_colors(color_scheme)
-
-            # Add a python parser so we can syntax highlight source while
-            # debugging.
-            self.parser = PyColorize.Parser()
-
-
-    else:
-        # Ugly hack: for Python 2.3-2.4, we can't call the parent constructor,
-        # because it binds readline and breaks tab-completion.  This means we
-        # have to COPY the constructor here.
-        def __init__(self,color_scheme='NoColor'):
-            bdb.Bdb.__init__(self)
-            cmd.Cmd.__init__(self,completekey=None) # don't load readline
-            self.prompt = 'ipdb> ' # The default prompt is '(Pdb)'
-            self.aliases = {}
-
-            # These two lines are part of the py2.4 constructor, let's put them
-            # unconditionally here as they won't cause any problems in 2.3.
-            self.mainpyfile = ''
-            self._wait_for_mainpyfile = 0
-
-            # Read $HOME/.pdbrc and ./.pdbrc
-            try:
-                self.rcLines = _file_lines(os.path.join(os.environ['HOME'],
-                                                        ".pdbrc"))
-            except KeyError:
-                self.rcLines = []
-            self.rcLines.extend(_file_lines(".pdbrc"))
-
-            # Create color table: we copy the default one from the traceback
-            # module and add a few attributes needed for debugging
-            self.color_scheme_table = exception_colors()
-
-            # shorthands 
-            C = coloransi.TermColors
-            cst = self.color_scheme_table
-
-            cst['NoColor'].colors.breakpoint_enabled = C.NoColor
-            cst['NoColor'].colors.breakpoint_disabled = C.NoColor
-
-            cst['Linux'].colors.breakpoint_enabled = C.LightRed
-            cst['Linux'].colors.breakpoint_disabled = C.Red
-
-            cst['LightBG'].colors.breakpoint_enabled = C.LightRed
-            cst['LightBG'].colors.breakpoint_disabled = C.Red
-
-            self.set_colors(color_scheme)
-
-            # Add a python parser so we can syntax highlight source while
-            # debugging.
-            self.parser = PyColorize.Parser()
+        self.prompt = prompt # The default prompt is '(Pdb)'
         
+        # IPython changes...
+        self.is_pydb = has_pydb
+
+        self.shell = ipapi.get()
+
+        if self.is_pydb:
+
+            # iplib.py's ipalias seems to want pdb's checkline
+            # which located in pydb.fn
+            import pydb.fns
+            self.checkline = lambda filename, lineno: \
+                             pydb.fns.checkline(self, filename, lineno)
+
+            self.curframe = None
+            self.do_restart = self.new_do_restart
+
+            self.old_all_completions = self.shell.Completer.all_completions
+            self.shell.Completer.all_completions=self.all_completions
+
+            self.do_list = decorate_fn_with_doc(self.list_command_pydb,
+                                                OldPdb.do_list)
+            self.do_l     = self.do_list
+            self.do_frame = decorate_fn_with_doc(self.new_do_frame,
+                                                 OldPdb.do_frame)
+
+        self.aliases = {}
+
+        # Create color table: we copy the default one from the traceback
+        # module and add a few attributes needed for debugging
+        self.color_scheme_table = exception_colors()
+
+        # shorthands 
+        C = coloransi.TermColors
+        cst = self.color_scheme_table
+
+        cst['NoColor'].colors.breakpoint_enabled = C.NoColor
+        cst['NoColor'].colors.breakpoint_disabled = C.NoColor
+
+        cst['Linux'].colors.breakpoint_enabled = C.LightRed
+        cst['Linux'].colors.breakpoint_disabled = C.Red
+
+        cst['LightBG'].colors.breakpoint_enabled = C.LightRed
+        cst['LightBG'].colors.breakpoint_disabled = C.Red
+
+        self.set_colors(color_scheme)
+
+        # Add a python parser so we can syntax highlight source while
+        # debugging.
+        self.parser = PyColorize.Parser()
+
     def set_colors(self, scheme):
         """Shorthand access to the color table scheme selector method."""
         self.color_scheme_table.set_active_scheme(scheme)
 
     def interaction(self, frame, traceback):
-        __IPYTHON__.set_completer_frame(frame)
+        self.shell.set_completer_frame(frame)
         OldPdb.interaction(self, frame, traceback)
 
     def new_do_up(self, arg):
         OldPdb.do_up(self, arg)
-        __IPYTHON__.set_completer_frame(self.curframe)
+        self.shell.set_completer_frame(self.curframe)
     do_u = do_up = decorate_fn_with_doc(new_do_up, OldPdb.do_up)
 
     def new_do_down(self, arg):
         OldPdb.do_down(self, arg)
-        __IPYTHON__.set_completer_frame(self.curframe)
+        self.shell.set_completer_frame(self.curframe)
 
     do_d = do_down = decorate_fn_with_doc(new_do_down, OldPdb.do_down)
 
     def new_do_frame(self, arg):
         OldPdb.do_frame(self, arg)
-        __IPYTHON__.set_completer_frame(self.curframe)
+        self.shell.set_completer_frame(self.curframe)
 
     def new_do_quit(self, arg):
         
         if hasattr(self, 'old_all_completions'):
-            __IPYTHON__.Completer.all_completions=self.old_all_completions
+            self.shell.Completer.all_completions=self.old_all_completions
         
         
         return OldPdb.do_quit(self, arg)
@@ -314,7 +269,7 @@
         return self.do_quit(arg)
 
     def postloop(self):
-        __IPYTHON__.set_completer_frame(None)
+        self.shell.set_completer_frame(None)
 
     def print_stack_trace(self):
         try:
@@ -331,7 +286,7 @@
         # vds: >>
         frame, lineno = frame_lineno
         filename = frame.f_code.co_filename
-        __IPYTHON__.hooks.synchronize_with_editor(filename, lineno, 0)
+        self.shell.hooks.synchronize_with_editor(filename, lineno, 0)
         # vds: <<
 
     def format_stack_entry(self, frame_lineno, lprefix=': ', context = 3):
@@ -500,7 +455,7 @@
         # vds: >>
         lineno = first
         filename = self.curframe.f_code.co_filename
-        __IPYTHON__.hooks.synchronize_with_editor(filename, lineno, 0)
+        self.shell.hooks.synchronize_with_editor(filename, lineno, 0)
         # vds: <<
 
     do_l = do_list
@@ -509,16 +464,16 @@
         """The debugger interface to magic_pdef"""
         namespaces = [('Locals', self.curframe.f_locals),
                       ('Globals', self.curframe.f_globals)]
-        __IPYTHON__.magic_pdef(arg, namespaces=namespaces)
+        self.shell.magic_pdef(arg, namespaces=namespaces)
 
     def do_pdoc(self, arg):
         """The debugger interface to magic_pdoc"""
         namespaces = [('Locals', self.curframe.f_locals),
                       ('Globals', self.curframe.f_globals)]
-        __IPYTHON__.magic_pdoc(arg, namespaces=namespaces)
+        self.shell.magic_pdoc(arg, namespaces=namespaces)
 
     def do_pinfo(self, arg):
         """The debugger equivalant of ?obj"""
         namespaces = [('Locals', self.curframe.f_locals),
                       ('Globals', self.curframe.f_globals)]
-        __IPYTHON__.magic_pinfo("pinfo %s" % arg, namespaces=namespaces)
+        self.shell.magic_pinfo("pinfo %s" % arg, namespaces=namespaces)

=== modified file 'IPython/core/display_trap.py'
--- IPython/core/display_trap.py	2009-10-09 22:54:40 +0000
+++ IPython/core/display_trap.py	2009-11-21 00:11:11 +0000
@@ -46,11 +46,11 @@
         # Only turn off the trap when the outermost call to __exit__ is made.
         self._nested_level = 0
 
-    @auto_attr
-    def shell(self):
-        return Component.get_instances(
-            root=self.root,
-            klass='IPython.core.iplib.InteractiveShell')[0]
+    # @auto_attr
+    # def shell(self):
+    #     return Component.get_instances(
+    #         root=self.root,
+    #         klass='IPython.core.iplib.InteractiveShell')[0]
 
     def __enter__(self):
         if self._nested_level == 0:

=== modified file 'IPython/core/embed.py'
--- IPython/core/embed.py	2009-09-13 22:06:04 +0000
+++ IPython/core/embed.py	2009-11-21 00:11:11 +0000
@@ -68,7 +68,7 @@
     # is True by default.
     display_banner = CBool(True)
 
-    def __init__(self, parent=None, config=None, ipythondir=None, usage=None,
+    def __init__(self, parent=None, config=None, ipython_dir=None, usage=None,
                  user_ns=None, user_global_ns=None,
                  banner1=None, banner2=None, display_banner=None,
                  custom_exceptions=((),None), exit_msg=''):
@@ -76,7 +76,7 @@
         self.save_sys_ipcompleter()
 
         super(InteractiveShellEmbed,self).__init__(
-            parent=parent, config=config, ipythondir=ipythondir, usage=usage, 
+            parent=parent, config=config, ipython_dir=ipython_dir, usage=usage, 
             user_ns=user_ns, user_global_ns=user_global_ns,
             banner1=banner1, banner2=banner2, display_banner=display_banner,
             custom_exceptions=custom_exceptions)
@@ -233,14 +233,6 @@
             for var in local_varnames:
                 delvar(var,None)
 
-    def set_completer_frame(self, frame=None):
-        if frame:
-            self.Completer.namespace = frame.f_locals
-            self.Completer.global_namespace = frame.f_globals
-        else:
-            self.Completer.namespace = self.user_ns
-            self.Completer.global_namespace = self.user_global_ns
-
 
 _embedded_shell = None
 

=== modified file 'IPython/core/ipapi.py'
--- IPython/core/ipapi.py	2009-09-18 02:59:36 +0000
+++ IPython/core/ipapi.py	2009-11-21 00:11:11 +0000
@@ -18,16 +18,19 @@
 # Imports
 #-----------------------------------------------------------------------------
 
-from IPython.core.error import TryNext, UsageError
+from IPython.core.error import TryNext, UsageError, IPythonCoreError
 
 #-----------------------------------------------------------------------------
 # Classes and functions
 #-----------------------------------------------------------------------------
 
+
 def get():
     """Get the most recently created InteractiveShell instance."""
     from IPython.core.iplib import InteractiveShell
     insts = InteractiveShell.get_instances()
+    if len(insts)==0:
+        return None
     most_recent = insts[0]
     for inst in insts[1:]:
         if inst.created > most_recent.created:

=== modified file 'IPython/core/ipapp.py'
--- IPython/core/ipapp.py	2009-09-18 03:26:43 +0000
+++ IPython/core/ipapp.py	2009-11-21 00:11:11 +0000
@@ -1,7 +1,8 @@
 #!/usr/bin/env python
 # encoding: utf-8
 """
-The main IPython application object
+The :class:`~IPython.core.application.Application` object for the command
+line :command:`ipython` program.
 
 Authors:
 
@@ -28,19 +29,17 @@
 import sys
 import warnings
 
-from IPython.core.application import Application, IPythonArgParseConfigLoader
+from IPython.core.application import Application, BaseAppArgParseConfigLoader
 from IPython.core import release
 from IPython.core.iplib import InteractiveShell
 from IPython.config.loader import (
     NoConfigDefault, 
     Config,
-    ConfigError,
     PyFileConfigLoader
 )
 
 from IPython.lib import inputhook
 
-from IPython.utils.ipstruct import Struct
 from IPython.utils.genutils import filefind, get_ipython_dir
 
 #-----------------------------------------------------------------------------
@@ -80,181 +79,181 @@
 #-----------------------------------------------------------------------------
 
 cl_args = (
-    (('-autocall',), dict(
+    (('--autocall',), dict(
         type=int, dest='InteractiveShell.autocall', default=NoConfigDefault,
         help='Set the autocall value (0,1,2).',
         metavar='InteractiveShell.autocall')
     ),
-    (('-autoindent',), dict(
+    (('--autoindent',), dict(
         action='store_true', dest='InteractiveShell.autoindent', default=NoConfigDefault,
         help='Turn on autoindenting.')
     ),
-    (('-noautoindent',), dict(
+    (('--no-autoindent',), dict(
         action='store_false', dest='InteractiveShell.autoindent', default=NoConfigDefault,
         help='Turn off autoindenting.')
     ),
-    (('-automagic',), dict(
+    (('--automagic',), dict(
         action='store_true', dest='InteractiveShell.automagic', default=NoConfigDefault,
         help='Turn on the auto calling of magic commands.')
     ),
-    (('-noautomagic',), dict(
+    (('--no-automagic',), dict(
         action='store_false', dest='InteractiveShell.automagic', default=NoConfigDefault,
         help='Turn off the auto calling of magic commands.')
     ),
-    (('-autoedit_syntax',), dict(
+    (('--autoedit-syntax',), dict(
         action='store_true', dest='InteractiveShell.autoedit_syntax', default=NoConfigDefault,
         help='Turn on auto editing of files with syntax errors.')
     ),
-    (('-noautoedit_syntax',), dict(
+    (('--no-autoedit-syntax',), dict(
         action='store_false', dest='InteractiveShell.autoedit_syntax', default=NoConfigDefault,
         help='Turn off auto editing of files with syntax errors.')
     ),
-    (('-banner',), dict(
+    (('--banner',), dict(
         action='store_true', dest='Global.display_banner', default=NoConfigDefault,
         help='Display a banner upon starting IPython.')
     ),
-    (('-nobanner',), dict(
+    (('--no-banner',), dict(
         action='store_false', dest='Global.display_banner', default=NoConfigDefault,
         help="Don't display a banner upon starting IPython.")
     ),
-    (('-cache_size',), dict(
+    (('--cache-size',), dict(
         type=int, dest='InteractiveShell.cache_size', default=NoConfigDefault,
         help="Set the size of the output cache.",
         metavar='InteractiveShell.cache_size')
     ),
-    (('-classic',), dict(
+    (('--classic',), dict(
         action='store_true', dest='Global.classic', default=NoConfigDefault,
         help="Gives IPython a similar feel to the classic Python prompt.")
     ),
-    (('-colors',), dict(
+    (('--colors',), dict(
         type=str, dest='InteractiveShell.colors', default=NoConfigDefault,
         help="Set the color scheme (NoColor, Linux, and LightBG).",
         metavar='InteractiveShell.colors')
     ),
-    (('-color_info',), dict(
+    (('--color-info',), dict(
         action='store_true', dest='InteractiveShell.color_info', default=NoConfigDefault,
         help="Enable using colors for info related things.")
     ),
-    (('-nocolor_info',), dict(
+    (('--no-color-info',), dict(
         action='store_false', dest='InteractiveShell.color_info', default=NoConfigDefault,
         help="Disable using colors for info related things.")
     ),
-    (('-confirm_exit',), dict(
+    (('--confirm-exit',), dict(
         action='store_true', dest='InteractiveShell.confirm_exit', default=NoConfigDefault,
         help="Prompt the user when existing.")
     ),
-    (('-noconfirm_exit',), dict(
+    (('--no-confirm-exit',), dict(
         action='store_false', dest='InteractiveShell.confirm_exit', default=NoConfigDefault,
         help="Don't prompt the user when existing.")
     ),
-    (('-deep_reload',), dict(
+    (('--deep-reload',), dict(
         action='store_true', dest='InteractiveShell.deep_reload', default=NoConfigDefault,
         help="Enable deep (recursive) reloading by default.")
     ),
-    (('-nodeep_reload',), dict(
+    (('--no-deep-reload',), dict(
         action='store_false', dest='InteractiveShell.deep_reload', default=NoConfigDefault,
         help="Disable deep (recursive) reloading by default.")
     ),
-    (('-editor',), dict(
+    (('--editor',), dict(
         type=str, dest='InteractiveShell.editor', default=NoConfigDefault,
         help="Set the editor used by IPython (default to $EDITOR/vi/notepad).",
         metavar='InteractiveShell.editor')
     ),
-    (('-log','-l'), dict(
+    (('--log','-l'), dict(
         action='store_true', dest='InteractiveShell.logstart', default=NoConfigDefault,
         help="Start logging to the default file (./ipython_log.py).")
     ),
-    (('-logfile','-lf'), dict(
-        type=str, dest='InteractiveShell.logfile', default=NoConfigDefault,
+    (('--logfile','-lf'), dict(
+        type=unicode, dest='InteractiveShell.logfile', default=NoConfigDefault,
         help="Start logging to logfile.",
         metavar='InteractiveShell.logfile')
     ),
-    (('-logappend','-la'), dict(
-        type=str, dest='InteractiveShell.logappend', default=NoConfigDefault,
-        help="Start logging to logappend in append mode.",
+    (('--log-append','-la'), dict(
+        type=unicode, dest='InteractiveShell.logappend', default=NoConfigDefault,
+        help="Start logging to the give file in append mode.",
         metavar='InteractiveShell.logfile')
     ),
-    (('-pdb',), dict(
+    (('--pdb',), dict(
         action='store_true', dest='InteractiveShell.pdb', default=NoConfigDefault,
         help="Enable auto calling the pdb debugger after every exception.")
     ),
-    (('-nopdb',), dict(
+    (('--no-pdb',), dict(
         action='store_false', dest='InteractiveShell.pdb', default=NoConfigDefault,
         help="Disable auto calling the pdb debugger after every exception.")
     ),
-    (('-pprint',), dict(
+    (('--pprint',), dict(
         action='store_true', dest='InteractiveShell.pprint', default=NoConfigDefault,
         help="Enable auto pretty printing of results.")
     ),
-    (('-nopprint',), dict(
+    (('--no-pprint',), dict(
         action='store_false', dest='InteractiveShell.pprint', default=NoConfigDefault,
         help="Disable auto auto pretty printing of results.")
     ),
-    (('-prompt_in1','-pi1'), dict(
+    (('--prompt-in1','-pi1'), dict(
         type=str, dest='InteractiveShell.prompt_in1', default=NoConfigDefault,
         help="Set the main input prompt ('In [\#]: ')",
         metavar='InteractiveShell.prompt_in1')
     ),
-    (('-prompt_in2','-pi2'), dict(
+    (('--prompt-in2','-pi2'), dict(
         type=str, dest='InteractiveShell.prompt_in2', default=NoConfigDefault,
         help="Set the secondary input prompt (' .\D.: ')",
         metavar='InteractiveShell.prompt_in2')
     ),
-    (('-prompt_out','-po'), dict(
+    (('--prompt-out','-po'), dict(
         type=str, dest='InteractiveShell.prompt_out', default=NoConfigDefault,
         help="Set the output prompt ('Out[\#]:')",
         metavar='InteractiveShell.prompt_out')
     ),
-    (('-quick',), dict(
+    (('--quick',), dict(
         action='store_true', dest='Global.quick', default=NoConfigDefault,
         help="Enable quick startup with no config files.")
     ),
-    (('-readline',), dict(
+    (('--readline',), dict(
         action='store_true', dest='InteractiveShell.readline_use', default=NoConfigDefault,
         help="Enable readline for command line usage.")
     ),
-    (('-noreadline',), dict(
+    (('--no-readline',), dict(
         action='store_false', dest='InteractiveShell.readline_use', default=NoConfigDefault,
         help="Disable readline for command line usage.")
     ),
-    (('-screen_length','-sl'), dict(
+    (('--screen-length','-sl'), dict(
         type=int, dest='InteractiveShell.screen_length', default=NoConfigDefault,
         help='Number of lines on screen, used to control printing of long strings.',
         metavar='InteractiveShell.screen_length')
     ),
-    (('-separate_in','-si'), dict(
+    (('--separate-in','-si'), dict(
         type=str, dest='InteractiveShell.separate_in', default=NoConfigDefault,
         help="Separator before input prompts.  Default '\n'.",
         metavar='InteractiveShell.separate_in')
     ),
-    (('-separate_out','-so'), dict(
+    (('--separate-out','-so'), dict(
         type=str, dest='InteractiveShell.separate_out', default=NoConfigDefault,
         help="Separator before output prompts.  Default 0 (nothing).",
         metavar='InteractiveShell.separate_out')
     ),
-    (('-separate_out2','-so2'), dict(
+    (('--separate-out2','-so2'), dict(
         type=str, dest='InteractiveShell.separate_out2', default=NoConfigDefault,
         help="Separator after output prompts.  Default 0 (nonight).",
         metavar='InteractiveShell.separate_out2')
     ),
-    (('-nosep',), dict(
+    (('-no-sep',), dict(
         action='store_true', dest='Global.nosep', default=NoConfigDefault,
         help="Eliminate all spacing between prompts.")
     ),
-    (('-term_title',), dict(
+    (('--term-title',), dict(
         action='store_true', dest='InteractiveShell.term_title', default=NoConfigDefault,
         help="Enable auto setting the terminal title.")
     ),
-    (('-noterm_title',), dict(
+    (('--no-term-title',), dict(
         action='store_false', dest='InteractiveShell.term_title', default=NoConfigDefault,
         help="Disable auto setting the terminal title.")
     ),
-    (('-xmode',), dict(
+    (('--xmode',), dict(
         type=str, dest='InteractiveShell.xmode', default=NoConfigDefault,
         help="Exception mode ('Plain','Context','Verbose')",
         metavar='InteractiveShell.xmode')
     ),
-    (('-ext',), dict(
+    (('--ext',), dict(
         type=str, dest='Global.extra_extension', default=NoConfigDefault,
         help="The dotted module name of an IPython extension to load.",
         metavar='Global.extra_extension')
@@ -268,36 +267,39 @@
         action='store_true', dest='Global.force_interact', default=NoConfigDefault,
         help="If running code from the command line, become interactive afterwards.")
     ),
-    (('-wthread',), dict(
+    (('--wthread',), dict(
         action='store_true', dest='Global.wthread', default=NoConfigDefault,
         help="Enable wxPython event loop integration.")
     ),
-    (('-q4thread','-qthread'), dict(
+    (('--q4thread','--qthread'), dict(
         action='store_true', dest='Global.q4thread', default=NoConfigDefault,
         help="Enable Qt4 event loop integration. Qt3 is no longer supported.")
     ),
-    (('-gthread',), dict(
+    (('--gthread',), dict(
         action='store_true', dest='Global.gthread', default=NoConfigDefault,
         help="Enable GTK event loop integration.")
     ),
     # # These are only here to get the proper deprecation warnings
-    (('-pylab',), dict(
+    (('--pylab',), dict(
         action='store_true', dest='Global.pylab', default=NoConfigDefault,
-        help="Disabled.  Pylab has been disabled until matplotlib supports this version of IPython.")
+        help="Disabled.  Pylab has been disabled until matplotlib "
+        "supports this version of IPython.")
     )
 )
 
 
-class IPythonAppCLConfigLoader(IPythonArgParseConfigLoader):
+class IPythonAppCLConfigLoader(BaseAppArgParseConfigLoader):
 
     arguments = cl_args
 
 
-_default_config_file_name = 'ipython_config.py'
+default_config_file_name = u'ipython_config.py'
+
 
 class IPythonApp(Application):
-    name = 'ipython'
-    config_file_name = _default_config_file_name
+    name = u'ipython'
+    description = 'IPython: an enhanced interactive Python shell.'
+    config_file_name = default_config_file_name
 
     def create_default_config(self):
         super(IPythonApp, self).create_default_config()
@@ -313,11 +315,6 @@
         # By default always interact by starting the IPython mainloop.
         self.default_config.Global.interact = True
 
-        # Let the parent class set the default, but each time log_level
-        # changes from config, we need to update self.log_level as that is
-        # what updates the actual log level in self.log.
-        self.default_config.Global.log_level = self.log_level
-
         # No GUI integration by default
         self.default_config.Global.wthread = False
         self.default_config.Global.q4thread = False
@@ -326,8 +323,9 @@
     def create_command_line_config(self):
         """Create and return a command line config loader."""
         return IPythonAppCLConfigLoader(
-            description=ipython_desc,
-            version=release.version)
+            description=self.description, 
+            version=release.version
+        )
 
     def post_load_command_line_config(self):
         """Do actions after loading cl config."""
@@ -477,9 +475,9 @@
             self.shell.showtraceback()
 
     def _exec_file(self, fname):
-        full_filename = filefind(fname, ['.', self.ipythondir])
+        full_filename = filefind(fname, [u'.', self.ipython_dir])
         if os.path.isfile(full_filename):
-            if full_filename.endswith('.py'):
+            if full_filename.endswith(u'.py'):
                 self.log.info("Running file in user namespace: %s" % full_filename)
                 self.shell.safe_execfile(full_filename, self.shell.user_ns)
             elif full_filename.endswith('.ipy'):
@@ -527,20 +525,20 @@
             self.shell.mainloop()
 
 
-def load_default_config(ipythondir=None):
-    """Load the default config file from the default ipythondir.
+def load_default_config(ipython_dir=None):
+    """Load the default config file from the default ipython_dir.
 
     This is useful for embedded shells.
     """
-    if ipythondir is None:
-        ipythondir = get_ipython_dir()
-    cl = PyFileConfigLoader(_default_config_file_name, ipythondir)
+    if ipython_dir is None:
+        ipython_dir = get_ipython_dir()
+    cl = PyFileConfigLoader(default_config_file_name, ipython_dir)
     config = cl.load_config()
     return config
 
 
 def launch_new_instance():
-    """Create a run a full blown IPython instance"""
+    """Create and run a full blown IPython instance"""
     app = IPythonApp()
     app.start()
 

=== modified file 'IPython/core/iplib.py'
--- IPython/core/iplib.py	2009-10-09 22:54:40 +0000
+++ IPython/core/iplib.py	2009-11-21 00:11:11 +0000
@@ -162,6 +162,15 @@
     return ed
 
 
+def get_default_colors():
+    if sys.platform=='darwin':
+        return "LightBG"
+    elif os.name=='nt':
+        return 'Linux'
+    else:
+        return 'Linux'
+
+
 class SeparateStr(Str):
     """A Str subclass to validate separate_in, separate_out, etc.
 
@@ -182,7 +191,7 @@
 class InteractiveShell(Component, Magic):
     """An enhanced, interactive shell for Python."""
 
-    autocall = Enum((0,1,2), config=True)
+    autocall = Enum((0,1,2), default_value=1, config=True)
     autoedit_syntax = CBool(False, config=True)
     autoindent = CBool(True, config=True)
     automagic = CBool(True, config=True)
@@ -192,7 +201,7 @@
     cache_size = Int(1000, config=True)
     color_info = CBool(True, config=True)
     colors = CaselessStrEnum(('NoColor','LightBG','Linux'), 
-                             default_value='LightBG', config=True)
+                             default_value=get_default_colors(), config=True)
     confirm_exit = CBool(True, config=True)
     debug = CBool(False, config=True)
     deep_reload = CBool(False, config=True)
@@ -206,7 +215,7 @@
     embedded_active = CBool(False)
     editor = Str(get_default_editor(), config=True)
     filename = Str("<ipython console>")
-    ipythondir= Unicode('', config=True) # Set to get_ipython_dir() in __init__
+    ipython_dir= Unicode('', config=True) # Set to get_ipython_dir() in __init__
     logstart = CBool(False, config=True)
     logfile = Str('', config=True)
     logappend = Str('', config=True)
@@ -264,7 +273,7 @@
     # Subclasses with thread support should override this as needed.
     isthreaded = False
 
-    def __init__(self, parent=None, config=None, ipythondir=None, usage=None,
+    def __init__(self, parent=None, config=None, ipython_dir=None, usage=None,
                  user_ns=None, user_global_ns=None,
                  banner1=None, banner2=None, display_banner=None,
                  custom_exceptions=((),None)):
@@ -274,7 +283,7 @@
         super(InteractiveShell, self).__init__(parent, config=config)
 
         # These are relatively independent and stateless
-        self.init_ipythondir(ipythondir)
+        self.init_ipython_dir(ipython_dir)
         self.init_instance_attrs()
         self.init_term_title()
         self.init_usage(usage)
@@ -332,7 +341,7 @@
     def _banner2_changed(self):
         self.compute_banner()
 
-    def _ipythondir_changed(self, name, new):
+    def _ipython_dir_changed(self, name, new):
         if not os.path.isdir(new):
             os.makedirs(new, mode = 0777)
         if not os.path.isdir(self.ipython_extension_dir):
@@ -340,7 +349,7 @@
 
     @property
     def ipython_extension_dir(self):
-        return os.path.join(self.ipythondir, 'extensions')
+        return os.path.join(self.ipython_dir, 'extensions')
 
     @property
     def usable_screen_length(self):
@@ -372,19 +381,19 @@
     # init_* methods called by __init__
     #-------------------------------------------------------------------------
 
-    def init_ipythondir(self, ipythondir):
-        if ipythondir is not None:
-            self.ipythondir = ipythondir
-            self.config.Global.ipythondir = self.ipythondir
+    def init_ipython_dir(self, ipython_dir):
+        if ipython_dir is not None:
+            self.ipython_dir = ipython_dir
+            self.config.Global.ipython_dir = self.ipython_dir
             return
 
-        if hasattr(self.config.Global, 'ipythondir'):
-            self.ipythondir = self.config.Global.ipythondir
+        if hasattr(self.config.Global, 'ipython_dir'):
+            self.ipython_dir = self.config.Global.ipython_dir
         else:
-            self.ipythondir = get_ipython_dir()
+            self.ipython_dir = get_ipython_dir()
 
         # All children can just read this
-        self.config.Global.ipythondir = self.ipythondir
+        self.config.Global.ipython_dir = self.ipython_dir
 
     def init_instance_attrs(self):
         self.jobs = BackgroundJobManager()
@@ -1070,7 +1079,7 @@
             histfname = 'history-%s' % self.profile
         else:
             histfname = 'history'
-        self.histfile = os.path.join(self.ipythondir, histfname)
+        self.histfile = os.path.join(self.ipython_dir, histfname)
 
         # Fill the history zero entry, user counter starts at 1
         self.input_hist.append('\n')
@@ -1078,12 +1087,12 @@
 
     def init_shadow_hist(self):
         try:
-            self.db = pickleshare.PickleShareDB(self.ipythondir + "/db")
+            self.db = pickleshare.PickleShareDB(self.ipython_dir + "/db")
         except exceptions.UnicodeDecodeError:
-            print "Your ipythondir can't be decoded to unicode!"
+            print "Your ipython_dir can't be decoded to unicode!"
             print "Please set HOME environment variable to something that"
             print r"only has ASCII characters, e.g. c:\home"
-            print "Now it is", self.ipythondir
+            print "Now it is", self.ipython_dir
             sys.exit()
         self.shadowhist = ipcorehist.ShadowHist(self.db)
 
@@ -1426,9 +1435,7 @@
             return outcomps
 
     def set_custom_completer(self,completer,pos=0):
-        """set_custom_completer(completer,pos=0)
-
-        Adds a new custom completer function.
+        """Adds a new custom completer function.
 
         The position argument (defaults to 0) is the index in the completers
         list where you want the completer to be inserted."""
@@ -1438,9 +1445,18 @@
         self.Completer.matchers.insert(pos,newcomp)
 
     def set_completer(self):
-        """reset readline's completer to be our own."""
+        """Reset readline's completer to be our own."""
         self.readline.set_completer(self.Completer.complete)
 
+    def set_completer_frame(self, frame=None):
+        """Set the frame of the completer."""
+        if frame:
+            self.Completer.namespace = frame.f_locals
+            self.Completer.global_namespace = frame.f_globals
+        else:
+            self.Completer.namespace = self.user_ns
+            self.Completer.global_namespace = self.user_global_ns
+
     #-------------------------------------------------------------------------
     # Things related to readline
     #-------------------------------------------------------------------------
@@ -1913,7 +1929,7 @@
                 # SystemExit exception changed between Python 2.4 and 2.5, so
                 # the checks must be done in a version-dependent way.
                 show = False
-                if status.message!=0 and not kw['exit_ignore']:
+                if status.args[0]==0 and not kw['exit_ignore']:
                     show = True
                 if show:
                     self.showtraceback()
@@ -2278,6 +2294,8 @@
     def get_component(self, name=None, klass=None):
         """Fetch a component by name and klass in my tree."""
         c = Component.get_instances(root=self, name=name, klass=klass)
+        if len(c) == 0:
+            return None
         if len(c) == 1:
             return c[0]
         else:
@@ -2309,7 +2327,7 @@
         You can put your extension modules anywhere you want, as long as
         they can be imported by Python's standard import mechanism.  However,
         to make it easy to write extensions, you can also put your extensions
-        in ``os.path.join(self.ipythondir, 'extensions')``.  This directory
+        in ``os.path.join(self.ipython_dir, 'extensions')``.  This directory
         is added to ``sys.path`` automatically.
         """
         from IPython.utils.syspathcontext import prepended_to_syspath

=== modified file 'IPython/core/magic.py'
--- IPython/core/magic.py	2009-10-09 22:54:40 +0000
+++ IPython/core/magic.py	2009-11-21 00:11:11 +0000
@@ -21,6 +21,7 @@
 import pdb
 import pydoc
 import sys
+import shutil
 import re
 import tempfile
 import time
@@ -1268,7 +1269,6 @@
         If you want IPython to automatically do this on every exception, see
         the %pdb magic for more details.
         """
-        
         self.shell.debugger(force=True)
 
     @testdec.skip_doctest
@@ -3378,34 +3378,6 @@
         qr = IPython.core.usage.quick_reference + self.magic_magic('-brief')
         
         page(qr)
-        
-    def magic_upgrade(self,arg):
-        """ Upgrade your IPython installation
-        
-        This will copy the config files that don't yet exist in your 
-        ipython dir from the system config dir. Use this after upgrading 
-        IPython if you don't wish to delete your .ipython dir.
-
-        Call with -nolegacy to get rid of ipythonrc* files (recommended for
-        new users)
-
-        """
-        ip = self.getapi()
-        ipinstallation = path(IPython.__file__).dirname()
-        upgrade_script = '%s "%s"' % (sys.executable,ipinstallation / 'utils' / 'upgradedir.py')
-        src_config = ipinstallation / 'config' / 'userconfig'
-        userdir = path(ip.config.IPYTHONDIR)
-        cmd = '%s "%s" "%s"' % (upgrade_script, src_config, userdir)
-        print ">",cmd
-        shell(cmd)
-        if arg == '-nolegacy':
-            legacy = userdir.files('ipythonrc*')
-            print "Nuking legacy files:",legacy
-            
-            [p.remove() for p in legacy]
-            suffix = (sys.platform == 'win32' and '.ini' or '')
-            (userdir / ('ipythonrc' + suffix)).write_text('# Empty, see ipy_user_conf.py\n')
-
 
     def magic_doctest_mode(self,parameter_s=''):
         """Toggle doctest mode on and off.
@@ -3550,4 +3522,59 @@
         """Reload an IPython extension by its module name."""
         self.reload_extension(module_str)
 
+    def magic_install_profiles(self, s):
+        """Install the default IPython profiles into the .ipython dir.
+
+        If the default profiles have already been installed, they will not
+        be overwritten. You can force overwriting them by using the ``-o``
+        option::
+
+            In [1]: %install_profiles -o
+        """
+        if '-o' in s:
+            overwrite = True
+        else:
+            overwrite = False
+        from IPython.config import profile
+        profile_dir = os.path.split(profile.__file__)[0]
+        ipython_dir = self.ipython_dir
+        files = os.listdir(profile_dir)
+
+        to_install = []
+        for f in files:
+            if f.startswith('ipython_config'):
+                src = os.path.join(profile_dir, f)
+                dst = os.path.join(ipython_dir, f)
+                if (not os.path.isfile(dst)) or overwrite:
+                    to_install.append((f, src, dst))
+        if len(to_install)>0:
+            print "Installing profiles to: ", ipython_dir
+            for (f, src, dst) in to_install:
+                shutil.copy(src, dst)
+                print "    %s" % f
+
+    def magic_install_default_config(self, s):
+        """Install IPython's default config file into the .ipython dir.
+
+        If the default config file (:file:`ipython_config.py`) is already
+        installed, it will not be overwritten. You can force overwriting
+        by using the ``-o`` option::
+
+            In [1]: %install_default_config
+        """
+        if '-o' in s:
+            overwrite = True
+        else:
+            overwrite = False
+        from IPython.config import default
+        config_dir = os.path.split(default.__file__)[0]
+        ipython_dir = self.ipython_dir
+        default_config_file_name = 'ipython_config.py'
+        src = os.path.join(config_dir, default_config_file_name)
+        dst = os.path.join(ipython_dir, default_config_file_name)
+        if (not os.path.isfile(dst)) or overwrite:
+            shutil.copy(src, dst)
+            print "Installing default config file: %s" % dst
+
+
 # end Magic

=== removed file 'IPython/core/oldusersetup.py'
--- IPython/core/oldusersetup.py	2009-08-18 06:55:22 +0000
+++ IPython/core/oldusersetup.py	1970-01-01 00:00:00 +0000
@@ -1,219 +0,0 @@
-# -*- coding: utf-8 -*-
-"""
-Main IPython Component
-"""
-
-#-----------------------------------------------------------------------------
-#  Copyright (C) 2001 Janko Hauser <[e-mail]>
-#  Copyright (C) 2001-2007 Fernando Perez. <[e-mail]>
-#  Copyright (C) 2008-2009  The IPython Development Team
-#
-#  Distributed under the terms of the BSD License.  The full license is in
-#  the file COPYING, distributed as part of this software.
-#-----------------------------------------------------------------------------
-
-#-----------------------------------------------------------------------------
-# Imports
-#-----------------------------------------------------------------------------
-
-import glob
-import os
-import shutil
-import sys
-
-from IPython.utils.genutils import *
-
-def user_setup(ipythondir,rc_suffix,mode='install',interactive=True):
-    """Install or upgrade the user configuration directory.
-
-    Can be called when running for the first time or to upgrade the user's
-    .ipython/ directory.
-
-    Parameters
-    ----------
-      ipythondir : path
-        The directory to be used for installation/upgrade.  In 'install' mode,
-        if this path already exists, the function exits immediately.
-
-      rc_suffix : str
-        Extension for the config files.  On *nix platforms it is typically the
-        empty string, while Windows normally uses '.ini'.
-
-      mode : str, optional
-        Valid modes are 'install' and 'upgrade'.
-
-      interactive : bool, optional
-        If False, do not wait for user input on any errors.  Normally after
-        printing its status information, this function waits for the user to
-        hit Return before proceeding.  This is because the default use case is
-        when first installing the IPython configuration, so we want the user to
-        acknowledge the initial message, which contains some useful
-        information.
-        """
-
-    # For automatic use, deactivate all i/o
-    if interactive:
-        def wait():
-            try:
-                raw_input("Please press <RETURN> to start IPython.")
-            except EOFError:
-                print >> Term.cout
-            print '*'*70
-
-        def printf(s):
-            print s
-    else:
-        wait = lambda : None
-        printf = lambda s : None
-
-    # Install mode should be re-entrant: if the install dir already exists,
-    # bail out cleanly.
-    # XXX.  This is too hasty to return.  We need to check to make sure that
-    # all the expected config files and directories are actually there. We
-    # currently have a failure mode if someone deletes a needed config file
-    # but still has the ipythondir.
-    if mode == 'install' and os.path.isdir(ipythondir):
-        return
-
-    cwd = os.getcwd()  # remember where we started
-    glb = glob.glob
-
-    printf('*'*70)
-    if mode == 'install':
-        printf(
-"""Welcome to IPython. I will try to create a personal configuration directory
-where you can customize many aspects of IPython's functionality in:\n""")
-    else:
-        printf('I am going to upgrade your configuration in:')
-
-    printf(ipythondir)
-
-    rcdirend = os.path.join('IPython','config','userconfig')
-    cfg = lambda d: os.path.join(d,rcdirend)
-    try:
-        rcdir = filter(os.path.isdir,map(cfg,sys.path))[0]
-        printf("Initializing from configuration: %s" % rcdir)
-    except IndexError:
-        warning = """
-Installation error. IPython's directory was not found.
-
-Check the following:
-
-The ipython/IPython directory should be in a directory belonging to your
-PYTHONPATH environment variable (that is, it should be in a directory
-belonging to sys.path). You can copy it explicitly there or just link to it.
-
-IPython will create a minimal default configuration for you.
-
-"""
-        warn(warning)
-        wait()
-
-        if sys.platform =='win32':
-            inif = 'ipythonrc.ini'
-        else:
-            inif = 'ipythonrc'
-        minimal_setup = {'ipy_user_conf.py' : 'import ipy_defaults',
-                         inif : '# intentionally left blank' }
-        os.makedirs(ipythondir, mode = 0777)
-        for f, cont in minimal_setup.items():
-            # In 2.5, this can be more cleanly done using 'with'
-            fobj = file(ipythondir + '/' + f,'w')
-            fobj.write(cont)
-            fobj.close()
-
-        return
-
-    if mode == 'install':
-        try:
-            shutil.copytree(rcdir,ipythondir)
-            os.chdir(ipythondir)
-            rc_files = glb("ipythonrc*")
-            for rc_file in rc_files:
-                os.rename(rc_file,rc_file+rc_suffix)
-        except:
-            warning = """
-
-There was a problem with the installation:
-%s
-Try to correct it or contact the developers if you think it's a bug.
-IPython will proceed with builtin defaults.""" % sys.exc_info()[1]
-            warn(warning)
-            wait()
-            return
-
-    elif mode == 'upgrade':
-        try:
-            os.chdir(ipythondir)
-        except:
-            printf("""
-Can not upgrade: changing to directory %s failed. Details:
-%s
-""" % (ipythondir,sys.exc_info()[1]) )
-            wait()
-            return
-        else:
-            sources = glb(os.path.join(rcdir,'[A-Za-z]*'))
-            for new_full_path in sources:
-                new_filename = os.path.basename(new_full_path)
-                if new_filename.startswith('ipythonrc'):
-                    new_filename = new_filename + rc_suffix
-                # The config directory should only contain files, skip any
-                # directories which may be there (like CVS)
-                if os.path.isdir(new_full_path):
-                    continue
-                if os.path.exists(new_filename):
-                    old_file = new_filename+'.old'
-                    if os.path.exists(old_file):
-                        os.remove(old_file)
-                    os.rename(new_filename,old_file)
-                shutil.copy(new_full_path,new_filename)
-    else:
-        raise ValueError('unrecognized mode for install: %r' % mode)
-
-    # Fix line-endings to those native to each platform in the config
-    # directory.
-    try:
-        os.chdir(ipythondir)
-    except:
-        printf("""
-Problem: changing to directory %s failed.
-Details:
-%s
-
-Some configuration files may have incorrect line endings.  This should not
-cause any problems during execution.  """ % (ipythondir,sys.exc_info()[1]) )
-        wait()
-    else:
-        for fname in glb('ipythonrc*'):
-            try:
-                native_line_ends(fname,backup=0)
-            except IOError:
-                pass
-
-    if mode == 'install':
-        printf("""
-Successful installation!
-
-Please read the sections 'Initial Configuration' and 'Quick Tips' in the
-IPython manual (there are both HTML and PDF versions supplied with the
-distribution) to make sure that your system environment is properly configured
-to take advantage of IPython's features.
-
-Important note: the configuration system has changed! The old system is
-still in place, but its setting may be partly overridden by the settings in 
-"~/.ipython/ipy_user_conf.py" config file. Please take a look at the file 
-if some of the new settings bother you. 
-
-""")
-    else:
-        printf("""
-Successful upgrade!
-
-All files in your directory:
-%(ipythondir)s
-which would have been overwritten by the upgrade were backed up with a .old
-extension.  If you had made particular customizations in those files you may
-want to merge them back into the new files.""" % locals() )
-    wait()
-    os.chdir(cwd)
\ No newline at end of file

=== modified file 'IPython/core/prefilter.py' (properties changed: -x to +x)
--- IPython/core/prefilter.py	2009-09-28 05:57:44 +0000
+++ IPython/core/prefilter.py	2009-11-21 00:11:11 +0000
@@ -39,7 +39,7 @@
 from IPython.core.page import page
 
 from IPython.utils.traitlets import List, Int, Any, Str, CBool, Bool
-from IPython.utils.genutils import make_quoted_expr
+from IPython.utils.genutils import make_quoted_expr, Term
 from IPython.utils.autoattr import auto_attr
 
 #-----------------------------------------------------------------------------

=== modified file 'IPython/core/tests/test_iplib.py'
--- IPython/core/tests/test_iplib.py	2009-08-19 21:56:41 +0000
+++ IPython/core/tests/test_iplib.py	2009-11-21 00:11:11 +0000
@@ -15,7 +15,7 @@
 # our own packages
 from IPython.core import iplib
 from IPython.core import ipapi
-from IPython.core.oldusersetup import user_setup
+
 
 #-----------------------------------------------------------------------------
 # Globals
@@ -54,27 +54,4 @@
             continue
         nt.assert_equals(len(ns),0)
 
-
-# make sure that user_setup can be run re-entrantly in 'install' mode.
-def test_user_setup():
-    # use a lambda to pass kwargs to the generator
-    user_setup = lambda a,k: user_setup(*a,**k)
-    kw = dict(mode='install', interactive=False)
-
-    # Call the user setup and verify that the directory exists
-    yield user_setup, (ip.config.IPYTHONDIR,''), kw
-    yield os.path.isdir, ip.config.IPYTHONDIR
-
-    # Now repeat the operation with a non-existent directory. Check both that
-    # the call succeeds and that the directory is created.
-    tmpdir = tempfile.mktemp(prefix='ipython-test-')
-    # Use a try with an empty except because try/finally doesn't work with a 
-    # yield in Python 2.4.
-    try:
-        yield user_setup, (tmpdir,''), kw
-        yield os.path.isdir, tmpdir
-    except:
-        pass
-    # Clean up the temp dir once done
-    shutil.rmtree(tmpdir)
     
\ No newline at end of file

=== modified file 'IPython/core/usage.py'
--- IPython/core/usage.py	2009-08-18 23:09:56 +0000
+++ IPython/core/usage.py	2009-11-21 00:11:11 +0000
@@ -40,7 +40,7 @@
        in directories.
 
        In the rest of this text, we will refer to this directory as
-       IPYTHONDIR.
+       IPYTHON_DIR.
 
 REGULAR OPTIONS
        After the above threading options have been given, regular options  can
@@ -150,9 +150,9 @@
               here (in case your default EDITOR is something like Emacs).
 
        -ipythondir <name>
-              The  name  of  your  IPython configuration directory IPYTHONDIR.
+              The  name  of  your  IPython configuration directory IPYTHON_DIR.
               This can also be  specified  through  the  environment  variable
-              IPYTHONDIR.
+              IPYTHON_DIR.
 
        -log|l Generate  a log file of all input. The file is named
               ipython_log.py in your current directory (which prevents logs
@@ -201,10 +201,10 @@
 
        -profile|p <name>
               Assume that your config file is ipythonrc-<name> (looks in  cur-
-              rent dir first, then in IPYTHONDIR). This is a quick way to keep
+              rent dir first, then in IPYTHON_DIR). This is a quick way to keep
               and load multiple config files for different  tasks,  especially
               if  you  use  the include option of config files. You can keep a
-              basic IPYTHONDIR/ipythonrc file and then have  other  'profiles'
+              basic IPYTHON_DIR/ipythonrc file and then have  other  'profiles'
               which  include  this  one  and  load extra things for particular
               tasks. For example:
 
@@ -245,7 +245,7 @@
        -rcfile <name>
               Name of your  IPython  resource  configuration  file.   normally
               IPython    loads   ipythonrc   (from   current   directory)   or
-              IPYTHONDIR/ipythonrc.  If the loading of your config file fails,
+              IPYTHON_DIR/ipythonrc.  If the loading of your config file fails,
               IPython  starts  with  a  bare  bones  configuration (no modules
               loaded at all).
 
@@ -284,7 +284,7 @@
               Simply removes all input/output separators.
 
        -upgrade
-              Allows you to upgrade your  IPYTHONDIR  configuration  when  you
+              Allows you to upgrade your  IPYTHON_DIR  configuration  when  you
               install  a  new  version  of  IPython.   Since  new versions may
               include new command lines options or example files, this  copies
               updated ipythonrc-type files.  However, it backs up (with a .old

=== renamed file 'IPython/kernel/magic.py' => 'IPython/extensions/parallelmagic.py'
--- IPython/kernel/magic.py	2009-09-12 15:58:55 +0000
+++ IPython/extensions/parallelmagic.py	2009-11-21 00:11:11 +0000
@@ -1,171 +1,205 @@
+#!/usr/bin/env python
 # encoding: utf-8
 
 """Magic command interface for interactive parallel work."""
 
-__docformat__ = "restructuredtext en"
-
-#-------------------------------------------------------------------------------
-#  Copyright (C) 2008  The IPython Development Team
+#-----------------------------------------------------------------------------
+#  Copyright (C) 2008-2009  The IPython Development Team
 #
 #  Distributed under the terms of the BSD License.  The full license is in
 #  the file COPYING, distributed as part of this software.
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 # Imports
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 
 import new
 
-from IPython.core.iplib import InteractiveShell
-from IPython.core.shell import MTInteractiveShell
-
-from twisted.internet.defer import Deferred
-
-
-#-------------------------------------------------------------------------------
+from IPython.core.component import Component
+from IPython.utils.traitlets import Bool, Any
+from IPython.utils.autoattr import auto_attr
+
+#-----------------------------------------------------------------------------
 # Definitions of magic functions for use with IPython
-#-------------------------------------------------------------------------------
-
-NO_ACTIVE_CONTROLLER = """
-Error:  No Controller is activated
-Use activate() on a RemoteController object to activate it for magics.
+#-----------------------------------------------------------------------------
+
+
+NO_ACTIVE_MULTIENGINE_CLIENT = """
+Use activate() on a MultiEngineClient object to activate it for magics.
 """
 
-def magic_result(self,parameter_s=''):
-    """Print the result of command i on all engines of the active controller.
-    
-    To activate a controller in IPython, first create it and then call
-    the activate() method.
-    
-    Then you can do the following:
-    
-    >>> result                                # Print the latest result
-    Printing result... 
-    [127.0.0.1:0] In [1]: b = 10
-    [127.0.0.1:1] In [1]: b = 10
-    
-    >>> result 0                              # Print result 0
-    In [14]: result 0
-    Printing result... 
-    [127.0.0.1:0] In [0]: a = 5
-    [127.0.0.1:1] In [0]: a = 5
-    """
-    try:
-        activeController = __IPYTHON__.activeController
-    except AttributeError:
-        print NO_ACTIVE_CONTROLLER
-    else:
+
+class ParalleMagicComponent(Component):
+    """A component to manage the %result, %px and %autopx magics."""
+
+    active_multiengine_client = Any()
+    verbose = Bool(False, config=True)
+
+    def __init__(self, parent, name=None, config=None):
+        super(ParalleMagicComponent, self).__init__(parent, name=name, config=config)
+        self._define_magics()
+        # A flag showing if autopx is activated or not
+        self.autopx = False
+
+    # Access other components like this rather than by a regular attribute.
+    # This won't lookup the InteractiveShell object until it is used and
+    # then it is cached.  This is both efficient and couples this class 
+    # more loosely to InteractiveShell.
+    @auto_attr
+    def shell(self):
+        return Component.get_instances(
+            root=self.root,
+            klass='IPython.core.iplib.InteractiveShell')[0]
+
+    def _define_magics(self):
+        """Define the magic functions."""
+        self.shell.define_magic('result', self.magic_result)
+        self.shell.define_magic('px', self.magic_px)
+        self.shell.define_magic('autopx', self.magic_autopx)
+
+    def magic_result(self, ipself, parameter_s=''):
+        """Print the result of command i on all engines..
+
+        To use this a :class:`MultiEngineClient` instance must be created 
+        and then activated by calling its :meth:`activate` method.
+
+        Then you can do the following::
+
+            In [23]: %result
+            Out[23]: 
+            <Results List>
+            [0] In [6]: a = 10
+            [1] In [6]: a = 10
+    
+            In [22]: %result 6
+            Out[22]: 
+            <Results List>
+            [0] In [6]: a = 10
+            [1] In [6]: a = 10
+        """
+        if self.active_multiengine_client is None:
+            print NO_ACTIVE_MULTIENGINE_CLIENT
+            return
+
         try:
             index = int(parameter_s)
         except:
             index = None
-        result = activeController.get_result(index)
-        return result
-
-def magic_px(self,parameter_s=''):
-    """Executes the given python command on the active IPython Controller.
-    
-    To activate a Controller in IPython, first create it and then call
-    the activate() method.
-    
-    Then you can do the following:
-     
-    >>> %px a = 5       # Runs a = 5 on all nodes
-    """
-
-    try:
-        activeController = __IPYTHON__.activeController
-    except AttributeError:
-        print NO_ACTIVE_CONTROLLER
-    else:
-        print "Parallel execution on engines: %s" % activeController.targets
-        result = activeController.execute(parameter_s)
-        return result
-
-def pxrunsource(self, source, filename="<input>", symbol="single"):
-
-    try:
-        code = self.compile(source, filename, symbol)
-    except (OverflowError, SyntaxError, ValueError):
-        # Case 1
-        self.showsyntaxerror(filename)
-        return None
-
-    if code is None:
-        # Case 2
-        return True
-
-    # Case 3
-    # Because autopx is enabled, we now call executeAll or disable autopx if
-    # %autopx or autopx has been called
-    if 'get_ipython().magic("%autopx' in source or 'get_ipython().magic("autopx' in source:
-        _disable_autopx(self)
-        return False
-    else:
-        try:
-            result = self.activeController.execute(source)
-        except:
-            self.showtraceback()
-        else:
-            print result.__repr__()
-        return False
+        result = self.active_multiengine_client.get_result(index)
+        return result
+
+    def magic_px(self, ipself, parameter_s=''):
+        """Executes the given python command in parallel.
+
+        To use this a :class:`MultiEngineClient` instance must be created 
+        and then activated by calling its :meth:`activate` method.
         
-def magic_autopx(self, parameter_s=''):
-    """Toggles auto parallel mode for the active IPython Controller.
-    
-    To activate a Controller in IPython, first create it and then call
-    the activate() method.
-    
-    Then you can do the following:
-     
-    >>> %autopx                    # Now all commands are executed in parallel
-    Auto Parallel Enabled
-    Type %autopx to disable
-    ...
-    >>> %autopx                    # Now all commands are locally executed
-    Auto Parallel Disabled
-    """
-  
-    if hasattr(self, 'autopx'):
-        if self.autopx == True:
-            _disable_autopx(self)
+        Then you can do the following::
+
+            In [24]: %px a = 5
+            Parallel execution on engines: all
+            Out[24]: 
+            <Results List>
+            [0] In [7]: a = 5
+            [1] In [7]: a = 5
+        """
+
+        if self.active_multiengine_client is None:
+            print NO_ACTIVE_MULTIENGINE_CLIENT
+            return
+        print "Parallel execution on engines: %s" % self.active_multiengine_client.targets
+        result = self.active_multiengine_client.execute(parameter_s)
+        return result
+
+    def magic_autopx(self, ipself, parameter_s=''):
+        """Toggles auto parallel mode.
+
+        To use this a :class:`MultiEngineClient` instance must be created 
+        and then activated by calling its :meth:`activate` method. Once this
+        is called, all commands typed at the command line are send to
+        the engines to be executed in parallel. To control which engine
+        are used, set the ``targets`` attributed of the multiengine client
+        before entering ``%autopx`` mode.
+
+        Then you can do the following::
+
+            In [25]: %autopx
+            %autopx to enabled
+
+            In [26]: a = 10
+            <Results List>
+            [0] In [8]: a = 10
+            [1] In [8]: a = 10
+
+
+            In [27]: %autopx
+            %autopx disabled
+        """
+        if self.autopx:
+            self._disable_autopx()
         else:
-            _enable_autopx(self)
-    else:
-        _enable_autopx(self)
-
-def _enable_autopx(self):
-    """Enable %autopx mode by saving the original runsource and installing 
-    pxrunsource.
-    """
-    try:
-        activeController = __IPYTHON__.activeController
-    except AttributeError:
-        print "No active RemoteController found, use RemoteController.activate()."
-    else:
-        self._original_runsource = self.runsource
-        self.runsource = new.instancemethod(pxrunsource, self, self.__class__)
+            self._enable_autopx()
+
+    def _enable_autopx(self):
+        """Enable %autopx mode by saving the original runsource and installing 
+        pxrunsource.
+        """
+        if self.active_multiengine_client is None:
+            print NO_ACTIVE_MULTIENGINE_CLIENT
+            return
+
+        self._original_runsource = self.shell.runsource
+        self.shell.runsource = new.instancemethod(
+            self.pxrunsource, self.shell, self.shell.__class__
+        )
         self.autopx = True
-        print "Auto Parallel Enabled\nType %autopx to disable"
-
-def _disable_autopx(self):
-    """Disable %autopx by restoring the original runsource."""
-    if hasattr(self, 'autopx'):
-        if self.autopx == True:
-            self.runsource = self._original_runsource
+        print "%autopx enabled"
+    
+    def _disable_autopx(self):
+        """Disable %autopx by restoring the original InteractiveShell.runsource."""
+        if self.autopx:
+            self.shell.runsource = self._original_runsource
             self.autopx = False
-            print "Auto Parallel Disabled"
-            
-# Add the new magic function to the class dict:
-
-InteractiveShell.magic_result = magic_result
-InteractiveShell.magic_px = magic_px
-InteractiveShell.magic_autopx = magic_autopx
-
-# And remove the global name to keep global namespace clean.  Don't worry, the
-# copy bound to IPython stays, we're just removing the global name.
-del magic_result
-del magic_px
-del magic_autopx
+            print "%autopx disabled"
+
+    def pxrunsource(self, ipself, source, filename="<input>", symbol="single"):
+        """A parallel replacement for InteractiveShell.runsource."""
+
+        try:
+            code = ipself.compile(source, filename, symbol)
+        except (OverflowError, SyntaxError, ValueError):
+            # Case 1
+            ipself.showsyntaxerror(filename)
+            return None
+
+        if code is None:
+            # Case 2
+            return True
+
+        # Case 3
+        # Because autopx is enabled, we now call executeAll or disable autopx if
+        # %autopx or autopx has been called
+        if 'get_ipython().magic("%autopx' in source or 'get_ipython().magic("autopx' in source:
+            self._disable_autopx()
+            return False
+        else:
+            try:
+                result = self.active_multiengine_client.execute(source)
+            except:
+                ipself.showtraceback()
+            else:
+                print result.__repr__()
+            return False
+
+
+_loaded = False
+
+
+def load_ipython_extension(ip):
+    """Load the extension in IPython."""
+    global _loaded
+    if not _loaded:
+        prd = ParalleMagicComponent(ip, name='parallel_magic')
+        _loaded = True
 

=== modified file 'IPython/frontend/wx/ipythonx.py'
--- IPython/frontend/wx/ipythonx.py	2009-03-29 01:05:30 +0000
+++ IPython/frontend/wx/ipythonx.py	2009-11-21 00:11:11 +0000
@@ -6,11 +6,10 @@
 try:
     import wx
 except ImportError, e:
-    e.message = """%s
+    e.args[0] = """%s
 ________________________________________________________________________________
 You need wxPython to run this application.
-""" % e.message
-    e.args = (e.message, ) + e.args[1:]
+""" % e.args[0]
     raise e
 
 from wx_frontend import WxController

=== modified file 'IPython/gui/wx/wxIPython.py'
--- IPython/gui/wx/wxIPython.py	2009-08-19 21:56:41 +0000
+++ IPython/gui/wx/wxIPython.py	2009-11-21 00:11:11 +0000
@@ -109,7 +109,7 @@
 
     def optionSave(self, name, value):
         ip = get()
-        path = ip.config.IPYTHONDIR
+        path = ip.ipython_dir
         opt = open(path + '/options.conf','w')
 
         try:
@@ -126,7 +126,7 @@
     def optionLoad(self):
         try:
             ip = get()
-            path = ip.config.IPYTHONDIR
+            path = ip.ipython_dir
             opt = open(path + '/options.conf','r')
             lines = opt.readlines()
             opt.close()

=== modified file 'IPython/kernel/asyncclient.py'
--- IPython/kernel/asyncclient.py	2008-07-22 02:39:52 +0000
+++ IPython/kernel/asyncclient.py	2009-11-21 00:11:11 +0000
@@ -1,3 +1,4 @@
+#!/usr/bin/env python
 # encoding: utf-8
 
 """Asynchronous clients for the IPython controller.
@@ -9,32 +10,32 @@
 
 The main methods are are `get_*_client` and `get_client`.
 """
-
-__docformat__ = "restructuredtext en"
-
-#-------------------------------------------------------------------------------
-#  Copyright (C) 2008  The IPython Development Team
+#-----------------------------------------------------------------------------
+#  Copyright (C) 2008-2009  The IPython Development Team
 #
 #  Distributed under the terms of the BSD License.  The full license is in
 #  the file COPYING, distributed as part of this software.
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 # Imports
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 
 from IPython.kernel import codeutil
-from IPython.kernel.clientconnector import ClientConnector
+from IPython.kernel.clientconnector import (
+    AsyncClientConnector,
+    AsyncCluster
+)
 
 # Other things that the user will need
 from IPython.kernel.task import MapTask, StringTask
 from IPython.kernel.error import CompositeError
 
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 # Code
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 
-_client_tub = ClientConnector()
+_client_tub = AsyncClientConnector()
 get_multiengine_client = _client_tub.get_multiengine_client
 get_task_client = _client_tub.get_task_client
 get_client = _client_tub.get_client

=== modified file 'IPython/kernel/client.py'
--- IPython/kernel/client.py	2009-07-02 21:35:36 +0000
+++ IPython/kernel/client.py	2009-11-21 00:11:11 +0000
@@ -1,3 +1,4 @@
+#!/usr/bin/env python
 # encoding: utf-8
 
 """This module contains blocking clients for the controller interfaces.
@@ -15,33 +16,36 @@
     * CompositeError
 """
 
-__docformat__ = "restructuredtext en"
-
-#-------------------------------------------------------------------------------
-#  Copyright (C) 2008  The IPython Development Team
+#-----------------------------------------------------------------------------
+#  Copyright (C) 2008-2009  The IPython Development Team
 #
 #  Distributed under the terms of the BSD License.  The full license is in
 #  the file COPYING, distributed as part of this software.
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 # Imports
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 
+from cStringIO import StringIO
 import sys
+import warnings
 
 # from IPython.utils import growl
 # growl.start("IPython1 Client")
 
 
 from twisted.internet import reactor
-from IPython.kernel.clientconnector import ClientConnector
+from twisted.internet.error import PotentialZombieWarning
+from twisted.python import log
+
+from IPython.kernel.clientconnector import ClientConnector, Cluster
 from IPython.kernel.twistedutil import ReactorInThread
 from IPython.kernel.twistedutil import blockingCallFromThread
 
 # These enable various things 
 from IPython.kernel import codeutil
-import IPython.kernel.magic
+# import IPython.kernel.magic
 
 # Other things that the user will need
 from IPython.kernel.task import MapTask, StringTask
@@ -51,46 +55,34 @@
 # Code
 #-------------------------------------------------------------------------------
 
+warnings.simplefilter('ignore', PotentialZombieWarning)
+
 _client_tub = ClientConnector()
 
-
-def get_multiengine_client(furl_or_file=''):
-    """Get the blocking MultiEngine client.
-    
-    :Parameters:
-        furl_or_file : str
-            A furl or a filename containing a furl.  If empty, the
-            default furl_file will be used
-            
-    :Returns:
-        The connected MultiEngineClient instance
-    """
-    client = blockingCallFromThread(_client_tub.get_multiengine_client, 
-        furl_or_file)
-    return client.adapt_to_blocking_client()
-
-def get_task_client(furl_or_file=''):
-    """Get the blocking Task client.
-    
-    :Parameters:
-        furl_or_file : str
-            A furl or a filename containing a furl.  If empty, the
-            default furl_file will be used
-            
-    :Returns:
-        The connected TaskClient instance
-    """
-    client = blockingCallFromThread(_client_tub.get_task_client, 
-        furl_or_file)
-    return client.adapt_to_blocking_client()
-
-
+get_multiengine_client = _client_tub.get_multiengine_client
+get_task_client = _client_tub.get_task_client
 MultiEngineClient = get_multiengine_client
 TaskClient = get_task_client
 
-
+# This isn't great.  I should probably set this up in the ReactorInThread
+# class below.  But, it does work for now.
+log.startLogging(sys.stdout, setStdout=0)
 
 # Now we start the reactor in a thread
 rit = ReactorInThread()
 rit.setDaemon(True)
-rit.start()
\ No newline at end of file
+rit.start()
+
+
+
+
+__all__ = [
+    'MapTask',
+    'StringTask',
+    'MultiEngineClient',
+    'TaskClient',
+    'CompositeError',
+    'get_task_client',
+    'get_multiengine_client',
+    'Cluster'
+]

=== modified file 'IPython/kernel/clientconnector.py'
--- IPython/kernel/clientconnector.py	2009-08-27 21:34:41 +0000
+++ IPython/kernel/clientconnector.py	2009-11-21 00:11:11 +0000
@@ -1,142 +1,268 @@
+#!/usr/bin/env python
 # encoding: utf-8
 
-"""A class for handling client connections to the controller."""
-
-__docformat__ = "restructuredtext en"
-
-#-------------------------------------------------------------------------------
-#  Copyright (C) 2008  The IPython Development Team
+"""Facilities for handling client connections to the controller."""
+
+#-----------------------------------------------------------------------------
+#  Copyright (C) 2008-2009  The IPython Development Team
 #
 #  Distributed under the terms of the BSD License.  The full license is in
 #  the file COPYING, distributed as part of this software.
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 # Imports
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
+
+from __future__ import with_statement
+import os
+
+from IPython.kernel.fcutil import (
+    Tub,
+    find_furl,
+    is_valid_furl_or_file,
+    validate_furl_or_file,
+    FURLError
+)
+from IPython.kernel.clusterdir import ClusterDir, ClusterDirError
+from IPython.kernel.launcher import IPClusterLauncher
+from IPython.kernel.twistedutil import (
+    gatherBoth, 
+    make_deferred,
+    blockingCallFromThread, 
+    sleep_deferred
+)
+from IPython.utils.importstring import import_item
+from IPython.utils.genutils import get_ipython_dir
 
 from twisted.internet import defer
-
-from IPython.kernel.fcutil import Tub, UnauthenticatedTub
-
-from IPython.kernel.config import config_manager as kernel_config_manager
-from IPython.utils.importstring import import_item
-from IPython.kernel.fcutil import find_furl
-
-co = kernel_config_manager.get_config_obj()
-client_co = co['client']
-
-#-------------------------------------------------------------------------------
+from twisted.internet.defer import inlineCallbacks, returnValue
+from twisted.python import failure, log
+
+#-----------------------------------------------------------------------------
 # The ClientConnector class
-#-------------------------------------------------------------------------------
-
-class ClientConnector(object):
-    """
-    This class gets remote references from furls and returns the wrapped clients.
-    
-    This class is also used in `client.py` and `asyncclient.py` to create 
-    a single per client-process Tub.
-    """
-    
+#-----------------------------------------------------------------------------
+
+DELAY = 0.2
+MAX_TRIES = 9
+
+
+class ClientConnectorError(Exception):
+    pass
+
+
+class AsyncClientConnector(object):
+    """A class for getting remote references and clients from furls.
+
+    This start a single :class:`Tub` for all remote reference and caches
+    references.
+    """
+
     def __init__(self):
         self._remote_refs = {}
         self.tub = Tub()
         self.tub.startService()
-    
+
+    def _find_furl(self, profile='default', cluster_dir=None, 
+                   furl_or_file=None, furl_file_name=None,
+                   ipython_dir=None):
+        """Find a FURL file by profile+ipython_dir or cluster dir.
+
+        This raises an :exc:`~IPython.kernel.fcutil.FURLError` exception 
+        if a FURL file can't be found.
+        """
+        # Try by furl_or_file
+        if furl_or_file is not None:
+            validate_furl_or_file(furl_or_file)
+            return furl_or_file
+
+        if furl_file_name is None:
+            raise FURLError('A furl_file_name must be provided')
+
+        # Try by cluster_dir
+        if cluster_dir is not None:
+            cluster_dir_obj = ClusterDir.find_cluster_dir(cluster_dir)
+            sdir = cluster_dir_obj.security_dir
+            furl_file = os.path.join(sdir, furl_file_name)
+            validate_furl_or_file(furl_file)
+            return furl_file
+
+        # Try by profile
+        if ipython_dir is None:
+            ipython_dir = get_ipython_dir()
+        if profile is not None:
+            cluster_dir_obj = ClusterDir.find_cluster_dir_by_profile(
+                ipython_dir, profile)
+            sdir = cluster_dir_obj.security_dir
+            furl_file = os.path.join(sdir, furl_file_name)
+            validate_furl_or_file(furl_file)
+            return furl_file
+
+        raise FURLError('Could not find a valid FURL file.')
+
     def get_reference(self, furl_or_file):
-        """
-        Get a remote reference using a furl or a file containing a furl.
-        
+        """Get a remote reference using a furl or a file containing a furl.
+
         Remote references are cached locally so once a remote reference
         has been retrieved for a given furl, the cached version is 
         returned.
-        
-        :Parameters:
-            furl_or_file : str
-                A furl or a filename containing a furl
-        
-        :Returns:
-            A deferred to a remote reference
+
+        Parameters
+        ----------
+        furl_or_file : str
+            A furl or a filename containing a furl. This should already be
+            validated, but might not yet exist.
+
+        Returns
+        -------
+        A deferred to a remote reference
         """
-        furl = find_furl(furl_or_file)
+        furl = furl_or_file
         if furl in self._remote_refs:
             d = defer.succeed(self._remote_refs[furl])
         else:
             d = self.tub.getReference(furl)
-            d.addCallback(self.save_ref, furl)
+            d.addCallback(self._save_ref, furl)
         return d
         
-    def save_ref(self, ref, furl):
-        """
-        Cache a remote reference by its furl.
-        """
+    def _save_ref(self, ref, furl):
+        """Cache a remote reference by its furl."""
         self._remote_refs[furl] = ref
         return ref
         
-    def get_task_client(self, furl_or_file=''):
-        """
-        Get the task controller client.
-        
-        This method is a simple wrapper around `get_client` that allow
-        `furl_or_file` to be empty, in which case, the furls is taken
-        from the default furl file given in the configuration.
-        
-        :Parameters:
-            furl_or_file : str
-                A furl or a filename containing a furl.  If empty, the
-                default furl_file will be used
-                
-        :Returns:
-            A deferred to the actual client class
-        """
-        task_co = client_co['client_interfaces']['task']
-        if furl_or_file:
-            ff = furl_or_file
-        else:
-            ff = task_co['furl_file']
-        return self.get_client(ff)
-
-    def get_multiengine_client(self, furl_or_file=''):
-        """
-        Get the multiengine controller client.
-        
-        This method is a simple wrapper around `get_client` that allow
-        `furl_or_file` to be empty, in which case, the furls is taken
-        from the default furl file given in the configuration.
-        
-        :Parameters:
-            furl_or_file : str
-                A furl or a filename containing a furl.  If empty, the
-                default furl_file will be used
-                
-        :Returns:
-            A deferred to the actual client class
-        """
-        task_co = client_co['client_interfaces']['multiengine']
-        if furl_or_file:
-            ff = furl_or_file
-        else:
-            ff = task_co['furl_file']
-        return self.get_client(ff)
+    def get_task_client(self, profile='default', cluster_dir=None,
+                        furl_or_file=None, ipython_dir=None,
+                        delay=DELAY, max_tries=MAX_TRIES):
+        """Get the task controller client.
+        
+        This method is a simple wrapper around `get_client` that passes in
+        the default name of the task client FURL file.  Usually only
+        the ``profile`` option will be needed.  If a FURL file can't be
+        found by its profile, use ``cluster_dir`` or ``furl_or_file``.
+        
+        Parameters
+        ----------
+        profile : str
+            The name of a cluster directory profile (default="default"). The
+            cluster directory "cluster_<profile>" will be searched for
+            in ``os.getcwd()``, the ipython_dir and then in the directories
+            listed in the :env:`IPCLUSTER_DIR_PATH` environment variable.
+        cluster_dir : str
+            The full path to a cluster directory.  This is useful if profiles
+            are not being used.
+        furl_or_file : str
+            A furl or a filename containing a FURLK. This is useful if you 
+            simply know the location of the FURL file.
+        ipython_dir : str
+            The location of the ipython_dir if different from the default.
+            This is used if the cluster directory is being found by profile.
+        delay : float
+            The initial delay between re-connection attempts. Susequent delays
+            get longer according to ``delay[i] = 1.5*delay[i-1]``.
+        max_tries : int
+            The max number of re-connection attempts.
+
+        Returns
+        -------
+        A deferred to the actual client class.
+        """
+        return self.get_client(
+            profile, cluster_dir, furl_or_file, 
+            'ipcontroller-tc.furl', ipython_dir,
+            delay, max_tries
+        )
+
+    def get_multiengine_client(self, profile='default', cluster_dir=None,
+                               furl_or_file=None, ipython_dir=None,
+                               delay=DELAY, max_tries=MAX_TRIES):
+        """Get the multiengine controller client.
+        
+        This method is a simple wrapper around `get_client` that passes in
+        the default name of the task client FURL file.  Usually only
+        the ``profile`` option will be needed.  If a FURL file can't be
+        found by its profile, use ``cluster_dir`` or ``furl_or_file``.
+        
+        Parameters
+        ----------
+        profile : str
+            The name of a cluster directory profile (default="default"). The
+            cluster directory "cluster_<profile>" will be searched for
+            in ``os.getcwd()``, the ipython_dir and then in the directories
+            listed in the :env:`IPCLUSTER_DIR_PATH` environment variable.
+        cluster_dir : str
+            The full path to a cluster directory.  This is useful if profiles
+            are not being used.
+        furl_or_file : str
+            A furl or a filename containing a FURLK. This is useful if you 
+            simply know the location of the FURL file.
+        ipython_dir : str
+            The location of the ipython_dir if different from the default.
+            This is used if the cluster directory is being found by profile.
+        delay : float
+            The initial delay between re-connection attempts. Susequent delays
+            get longer according to ``delay[i] = 1.5*delay[i-1]``.
+        max_tries : int
+            The max number of re-connection attempts.
+
+        Returns
+        -------
+        A deferred to the actual client class.
+        """
+        return self.get_client(
+            profile, cluster_dir, furl_or_file, 
+            'ipcontroller-mec.furl', ipython_dir,
+            delay, max_tries
+        )
     
-    def get_client(self, furl_or_file):
-        """
-        Get a remote reference and wrap it in a client by furl.
-        
-        This method first gets a remote reference and then calls its 
-        `get_client_name` method to find the apprpriate client class
-        that should be used to wrap the remote reference.
-        
-        :Parameters:
-            furl_or_file : str
-                A furl or a filename containing a furl
-        
-        :Returns:
-            A deferred to the actual client class
-        """
-        furl = find_furl(furl_or_file)
-        d = self.get_reference(furl)
-        def wrap_remote_reference(rr):
+    def get_client(self, profile='default', cluster_dir=None,
+                   furl_or_file=None, furl_file_name=None, ipython_dir=None,
+                   delay=DELAY, max_tries=MAX_TRIES):
+        """Get a remote reference and wrap it in a client by furl.
+
+        This method is a simple wrapper around `get_client` that passes in
+        the default name of the task client FURL file.  Usually only
+        the ``profile`` option will be needed.  If a FURL file can't be
+        found by its profile, use ``cluster_dir`` or ``furl_or_file``.
+        
+        Parameters
+        ----------
+        profile : str
+            The name of a cluster directory profile (default="default"). The
+            cluster directory "cluster_<profile>" will be searched for
+            in ``os.getcwd()``, the ipython_dir and then in the directories
+            listed in the :env:`IPCLUSTER_DIR_PATH` environment variable.
+        cluster_dir : str
+            The full path to a cluster directory.  This is useful if profiles
+            are not being used.
+        furl_or_file : str
+            A furl or a filename containing a FURL. This is useful if you 
+            simply know the location of the FURL file.
+        furl_file_name : str
+            The filename (not the full path) of the FURL. This must be
+            provided if ``furl_or_file`` is not.
+        ipython_dir : str
+            The location of the ipython_dir if different from the default.
+            This is used if the cluster directory is being found by profile.
+        delay : float
+            The initial delay between re-connection attempts. Susequent delays
+            get longer according to ``delay[i] = 1.5*delay[i-1]``.
+        max_tries : int
+            The max number of re-connection attempts.
+
+        Returns
+        -------
+        A deferred to the actual client class. Or a failure to a 
+        :exc:`FURLError`.
+        """
+        try:
+            furl_file = self._find_furl(
+                profile, cluster_dir, furl_or_file,
+                furl_file_name, ipython_dir
+            )
+        except FURLError:
+            return defer.fail(failure.Failure())
+
+        def _wrap_remote_reference(rr):
             d = rr.callRemote('get_client_name')
             d.addCallback(lambda name: import_item(name))
             def adapt(client_interface):
@@ -146,5 +272,502 @@
             d.addCallback(adapt)
 
             return d
-        d.addCallback(wrap_remote_reference)
-        return d
+
+        d = self._try_to_connect(furl_file, delay, max_tries, attempt=0)
+        d.addCallback(_wrap_remote_reference)
+        d.addErrback(self._handle_error, furl_file)
+        return d
+
+    def _handle_error(self, f, furl_file):
+        raise ClientConnectorError('Could not connect to the controller '
+            'using the FURL file. This usually means that i) the controller '
+            'was not started or ii) a firewall was blocking the client from '
+            'connecting to the controller: %s' % furl_file)
+
+    @inlineCallbacks
+    def _try_to_connect(self, furl_or_file, delay, max_tries, attempt):
+        """Try to connect to the controller with retry logic."""
+        if attempt < max_tries:
+            log.msg("Connecting [%r]" % attempt)
+            try:
+                self.furl = find_furl(furl_or_file)
+                # Uncomment this to see the FURL being tried.
+                # log.msg("FURL: %s" % self.furl)
+                rr = yield self.get_reference(self.furl)
+                log.msg("Connected: %s" % furl_or_file)
+            except:
+                if attempt==max_tries-1:
+                    # This will propagate the exception all the way to the top
+                    # where it can be handled.
+                    raise
+                else:
+                    yield sleep_deferred(delay)
+                    rr = yield self._try_to_connect(
+                        furl_or_file, 1.5*delay, max_tries, attempt+1
+                    )
+                    returnValue(rr)
+            else:
+                returnValue(rr)
+        else:
+            raise ClientConnectorError(
+                'Could not connect to controller, max_tries (%r) exceeded. '
+                'This usually means that i) the controller was not started, '
+                'or ii) a firewall was blocking the client from connecting '
+                'to the controller.' % max_tries
+            )
+
+
+class ClientConnector(object):
+    """A blocking version of a client connector.
+
+    This class creates a single :class:`Tub` instance and allows remote
+    references and client to be retrieved by their FURLs.  Remote references
+    are cached locally and FURL files can be found using profiles and cluster
+    directories.
+    """
+
+    def __init__(self):
+        self.async_cc = AsyncClientConnector()
+
+    def get_task_client(self, profile='default', cluster_dir=None,
+                        furl_or_file=None, ipython_dir=None,
+                        delay=DELAY, max_tries=MAX_TRIES):
+        """Get the task client.
+        
+        Usually only the ``profile`` option will be needed. If a FURL file
+        can't be found by its profile, use ``cluster_dir`` or
+        ``furl_or_file``.
+        
+        Parameters
+        ----------
+        profile : str
+            The name of a cluster directory profile (default="default"). The
+            cluster directory "cluster_<profile>" will be searched for
+            in ``os.getcwd()``, the ipython_dir and then in the directories
+            listed in the :env:`IPCLUSTER_DIR_PATH` environment variable.
+        cluster_dir : str
+            The full path to a cluster directory.  This is useful if profiles
+            are not being used.
+        furl_or_file : str
+            A furl or a filename containing a FURLK. This is useful if you 
+            simply know the location of the FURL file.
+        ipython_dir : str
+            The location of the ipython_dir if different from the default.
+            This is used if the cluster directory is being found by profile.
+        delay : float
+            The initial delay between re-connection attempts. Susequent delays
+            get longer according to ``delay[i] = 1.5*delay[i-1]``.
+        max_tries : int
+            The max number of re-connection attempts.
+
+        Returns
+        -------
+        The task client instance.
+        """
+        client = blockingCallFromThread(
+            self.async_cc.get_task_client, profile, cluster_dir,
+            furl_or_file, ipython_dir, delay, max_tries
+        )
+        return client.adapt_to_blocking_client()
+
+    def get_multiengine_client(self, profile='default', cluster_dir=None,
+                               furl_or_file=None, ipython_dir=None,
+                               delay=DELAY, max_tries=MAX_TRIES):
+        """Get the multiengine client.
+        
+        Usually only the ``profile`` option will be needed. If a FURL file
+        can't be found by its profile, use ``cluster_dir`` or
+        ``furl_or_file``.
+        
+        Parameters
+        ----------
+        profile : str
+            The name of a cluster directory profile (default="default"). The
+            cluster directory "cluster_<profile>" will be searched for
+            in ``os.getcwd()``, the ipython_dir and then in the directories
+            listed in the :env:`IPCLUSTER_DIR_PATH` environment variable.
+        cluster_dir : str
+            The full path to a cluster directory.  This is useful if profiles
+            are not being used.
+        furl_or_file : str
+            A furl or a filename containing a FURLK. This is useful if you 
+            simply know the location of the FURL file.
+        ipython_dir : str
+            The location of the ipython_dir if different from the default.
+            This is used if the cluster directory is being found by profile.
+        delay : float
+            The initial delay between re-connection attempts. Susequent delays
+            get longer according to ``delay[i] = 1.5*delay[i-1]``.
+        max_tries : int
+            The max number of re-connection attempts.
+
+        Returns
+        -------
+        The multiengine client instance.
+        """
+        client = blockingCallFromThread(
+            self.async_cc.get_multiengine_client, profile, cluster_dir,
+            furl_or_file, ipython_dir, delay, max_tries
+        )
+        return client.adapt_to_blocking_client()
+
+    def get_client(self, profile='default', cluster_dir=None,
+                   furl_or_file=None, ipython_dir=None,
+                   delay=DELAY, max_tries=MAX_TRIES):
+        client = blockingCallFromThread(
+            self.async_cc.get_client, profile, cluster_dir,
+            furl_or_file, ipython_dir,
+            delay, max_tries
+        )
+        return client.adapt_to_blocking_client()
+
+
+class ClusterStateError(Exception):
+    pass
+
+
+class AsyncCluster(object):
+    """An class that wraps the :command:`ipcluster` script."""
+
+    def __init__(self, profile='default', cluster_dir=None, ipython_dir=None,
+                 auto_create=False, auto_stop=True):
+        """Create a class to manage an IPython cluster.
+
+        This class calls the :command:`ipcluster` command with the right
+        options to start an IPython cluster.  Typically a cluster directory
+        must be created (:command:`ipcluster create`) and configured before 
+        using this class. Configuration is done by editing the 
+        configuration files in the top level of the cluster directory.
+
+        Parameters
+        ----------
+        profile : str
+            The name of a cluster directory profile (default="default"). The
+            cluster directory "cluster_<profile>" will be searched for
+            in ``os.getcwd()``, the ipython_dir and then in the directories
+            listed in the :env:`IPCLUSTER_DIR_PATH` environment variable.
+        cluster_dir : str
+            The full path to a cluster directory.  This is useful if profiles
+            are not being used.
+        ipython_dir : str
+            The location of the ipython_dir if different from the default.
+            This is used if the cluster directory is being found by profile.
+        auto_create : bool
+            Automatically create the cluster directory it is dones't exist.
+            This will usually only make sense if using a local cluster 
+            (default=False).
+        auto_stop : bool
+            Automatically stop the cluster when this instance is garbage 
+            collected (default=True).  This is useful if you want the cluster
+            to live beyond your current process. There is also an instance
+            attribute ``auto_stop`` to change this behavior.
+        """
+        self._setup_cluster_dir(profile, cluster_dir, ipython_dir, auto_create)
+        self.state = 'before'
+        self.launcher = None
+        self.client_connector = None
+        self.auto_stop = auto_stop
+
+    def __del__(self):
+        if self.auto_stop and self.state=='running':
+            print "Auto stopping the cluster..."
+            self.stop()
+
+    @property
+    def location(self):
+        if hasattr(self, 'cluster_dir_obj'):
+            return self.cluster_dir_obj.location
+        else:
+            return ''
+
+    @property
+    def running(self):
+        if self.state=='running':
+            return True
+        else:
+            return False
+
+    def _setup_cluster_dir(self, profile, cluster_dir, ipython_dir, auto_create):
+        if ipython_dir is None:
+            ipython_dir = get_ipython_dir()
+        if cluster_dir is not None:
+            try:
+                self.cluster_dir_obj = ClusterDir.find_cluster_dir(cluster_dir)
+            except ClusterDirError:
+                pass
+        if profile is not None:
+            try:
+                self.cluster_dir_obj = ClusterDir.find_cluster_dir_by_profile(
+                    ipython_dir, profile)
+            except ClusterDirError:
+                pass
+        if auto_create or profile=='default':
+            # This should call 'ipcluster create --profile default
+            self.cluster_dir_obj = ClusterDir.create_cluster_dir_by_profile(
+                ipython_dir, profile)
+        else:
+            raise ClusterDirError('Cluster dir not found.')
+
+    @make_deferred
+    def start(self, n=2):
+        """Start the IPython cluster with n engines.
+
+        Parameters
+        ----------
+        n : int
+            The number of engine to start.
+        """
+        # We might want to add logic to test if the cluster has started
+        # by another process....
+        if not self.state=='running':
+            self.launcher = IPClusterLauncher(os.getcwd())
+            self.launcher.ipcluster_n = n
+            self.launcher.ipcluster_subcommand = 'start'
+            d = self.launcher.start()
+            d.addCallback(self._handle_start)
+            return d
+        else:
+            raise ClusterStateError('Cluster is already running')
+
+    @make_deferred
+    def stop(self):
+        """Stop the IPython cluster if it is running."""
+        if self.state=='running':
+            d1 = self.launcher.observe_stop()
+            d1.addCallback(self._handle_stop)
+            d2 = self.launcher.stop()
+            return gatherBoth([d1, d2], consumeErrors=True)
+        else:
+            raise ClusterStateError("Cluster not running")
+
+    def get_multiengine_client(self, delay=DELAY, max_tries=MAX_TRIES):
+        """Get the multiengine client for the running cluster.
+
+        If this fails, it means that the cluster has not finished starting.
+        Usually waiting a few seconds are re-trying will solve this.    
+        """
+        if self.client_connector is None:
+            self.client_connector = AsyncClientConnector()
+        return self.client_connector.get_multiengine_client(
+            cluster_dir=self.cluster_dir_obj.location,
+            delay=delay, max_tries=max_tries
+        )
+
+    def get_task_client(self, delay=DELAY, max_tries=MAX_TRIES):
+        """Get the task client for the running cluster.
+
+        If this fails, it means that the cluster has not finished starting.
+        Usually waiting a few seconds are re-trying will solve this.    
+        """
+        if self.client_connector is None:
+            self.client_connector = AsyncClientConnector()
+        return self.client_connector.get_task_client(
+            cluster_dir=self.cluster_dir_obj.location,
+            delay=delay, max_tries=max_tries
+        )
+
+    def get_ipengine_logs(self):
+        return self.get_logs_by_name('ipengine')
+
+    def get_ipcontroller_logs(self):
+        return self.get_logs_by_name('ipcontroller')
+
+    def get_ipcluster_logs(self):
+        return self.get_logs_by_name('ipcluster')
+
+    def get_logs_by_name(self, name='ipcluster'):
+        log_dir = self.cluster_dir_obj.log_dir
+        logs = {}
+        for log in os.listdir(log_dir):
+            if log.startswith(name + '-') and log.endswith('.log'):
+                with open(os.path.join(log_dir, log), 'r') as f:
+                    logs[log] = f.read()
+        return logs
+
+    def get_logs(self):
+        d = self.get_ipcluster_logs()
+        d.update(self.get_ipengine_logs())
+        d.update(self.get_ipcontroller_logs())
+        return d
+
+    def _handle_start(self, r):
+        self.state = 'running'
+
+    def _handle_stop(self, r):
+        self.state = 'after'
+
+
+class Cluster(object):
+
+
+    def __init__(self, profile='default', cluster_dir=None, ipython_dir=None,
+                 auto_create=False, auto_stop=True):
+        """Create a class to manage an IPython cluster.
+
+        This class calls the :command:`ipcluster` command with the right
+        options to start an IPython cluster.  Typically a cluster directory
+        must be created (:command:`ipcluster create`) and configured before 
+        using this class. Configuration is done by editing the 
+        configuration files in the top level of the cluster directory.
+
+        Parameters
+        ----------
+        profile : str
+            The name of a cluster directory profile (default="default"). The
+            cluster directory "cluster_<profile>" will be searched for
+            in ``os.getcwd()``, the ipython_dir and then in the directories
+            listed in the :env:`IPCLUSTER_DIR_PATH` environment variable.
+        cluster_dir : str
+            The full path to a cluster directory.  This is useful if profiles
+            are not being used.
+        ipython_dir : str
+            The location of the ipython_dir if different from the default.
+            This is used if the cluster directory is being found by profile.
+        auto_create : bool
+            Automatically create the cluster directory it is dones't exist.
+            This will usually only make sense if using a local cluster 
+            (default=False).
+        auto_stop : bool
+            Automatically stop the cluster when this instance is garbage 
+            collected (default=True).  This is useful if you want the cluster
+            to live beyond your current process. There is also an instance
+            attribute ``auto_stop`` to change this behavior.
+        """
+        self.async_cluster = AsyncCluster(
+            profile, cluster_dir, ipython_dir, auto_create, auto_stop
+        )
+        self.cluster_dir_obj = self.async_cluster.cluster_dir_obj
+        self.client_connector = None
+
+    def _set_auto_stop(self, value):
+        self.async_cluster.auto_stop = value
+
+    def _get_auto_stop(self):
+        return self.async_cluster.auto_stop
+
+    auto_stop = property(_get_auto_stop, _set_auto_stop)
+
+    @property
+    def location(self):
+        return self.async_cluster.location
+
+    @property
+    def running(self):
+        return self.async_cluster.running
+
+    def start(self, n=2):
+        """Start the IPython cluster with n engines.
+
+        Parameters
+        ----------
+        n : int
+            The number of engine to start.
+        """
+        return blockingCallFromThread(self.async_cluster.start, n)
+
+    def stop(self):
+        """Stop the IPython cluster if it is running."""
+        return blockingCallFromThread(self.async_cluster.stop)
+
+    def get_multiengine_client(self, delay=DELAY, max_tries=MAX_TRIES):
+        """Get the multiengine client for the running cluster.
+
+        This will try to attempt to the controller multiple times. If this 
+        fails altogether, try looking at the following:
+        * Make sure the controller is starting properly by looking at its
+          log files.
+        * Make sure the controller is writing its FURL file in the location
+          expected by the client.
+        * Make sure a firewall on the controller's host is not blocking the
+          client from connecting.
+
+        Parameters
+        ----------
+        delay : float
+            The initial delay between re-connection attempts. Susequent delays
+            get longer according to ``delay[i] = 1.5*delay[i-1]``.
+        max_tries : int
+            The max number of re-connection attempts.
+        """
+        if self.client_connector is None:
+            self.client_connector = ClientConnector()
+        return self.client_connector.get_multiengine_client(
+            cluster_dir=self.cluster_dir_obj.location,
+            delay=delay, max_tries=max_tries
+        )
+
+    def get_task_client(self, delay=DELAY, max_tries=MAX_TRIES):
+        """Get the task client for the running cluster.
+
+        This will try to attempt to the controller multiple times. If this 
+        fails altogether, try looking at the following:
+        * Make sure the controller is starting properly by looking at its
+          log files.
+        * Make sure the controller is writing its FURL file in the location
+          expected by the client.
+        * Make sure a firewall on the controller's host is not blocking the
+          client from connecting.
+
+        Parameters
+        ----------
+        delay : float
+            The initial delay between re-connection attempts. Susequent delays
+            get longer according to ``delay[i] = 1.5*delay[i-1]``.
+        max_tries : int
+            The max number of re-connection attempts.
+        """
+        if self.client_connector is None:
+            self.client_connector = ClientConnector()
+        return self.client_connector.get_task_client(
+            cluster_dir=self.cluster_dir_obj.location,
+            delay=delay, max_tries=max_tries
+        )
+
+    def __repr__(self):
+        s = "<Cluster(running=%r, location=%s)" % (self.running, self.location)
+        return s
+
+    def get_logs_by_name(self, name='ipcluter'):
+        """Get a dict of logs by process name (ipcluster, ipengine, etc.)"""
+        return self.async_cluster.get_logs_by_name(name)
+
+    def get_ipengine_logs(self):
+        """Get a dict of logs for all engines in this cluster."""
+        return self.async_cluster.get_ipengine_logs()
+
+    def get_ipcontroller_logs(self):
+        """Get a dict of logs for the controller in this cluster."""
+        return self.async_cluster.get_ipcontroller_logs()
+
+    def get_ipcluster_logs(self):
+        """Get a dict of the ipcluster logs for this cluster."""
+        return self.async_cluster.get_ipcluster_logs()
+
+    def get_logs(self):
+        """Get a dict of all logs for this cluster."""
+        return self.async_cluster.get_logs()
+
+    def _print_logs(self, logs):
+        for k, v in logs.iteritems():
+            print "==================================="
+            print "Logfile: %s" % k
+            print "==================================="
+            print v
+            print
+
+    def print_ipengine_logs(self):
+        """Print the ipengine logs for this cluster to stdout."""
+        self._print_logs(self.get_ipengine_logs())
+
+    def print_ipcontroller_logs(self):
+        """Print the ipcontroller logs for this cluster to stdout."""
+        self._print_logs(self.get_ipcontroller_logs())
+
+    def print_ipcluster_logs(self):
+        """Print the ipcluster logs for this cluster to stdout."""
+        self._print_logs(self.get_ipcluster_logs())
+
+    def print_logs(self):
+        """Print all the logs for this cluster to stdout."""
+        self._print_logs(self.get_logs())
+

=== added file 'IPython/kernel/clusterdir.py'
--- IPython/kernel/clusterdir.py	1970-01-01 00:00:00 +0000
+++ IPython/kernel/clusterdir.py	2009-11-21 00:11:11 +0000
@@ -0,0 +1,475 @@
+#!/usr/bin/env python
+# encoding: utf-8
+"""
+The IPython cluster directory
+"""
+
+#-----------------------------------------------------------------------------
+#  Copyright (C) 2008-2009  The IPython Development Team
+#
+#  Distributed under the terms of the BSD License.  The full license is in
+#  the file COPYING, distributed as part of this software.
+#-----------------------------------------------------------------------------
+
+#-----------------------------------------------------------------------------
+# Imports
+#-----------------------------------------------------------------------------
+
+from __future__ import with_statement
+
+import os
+import shutil
+import sys
+
+from twisted.python import log
+
+from IPython.core import release
+from IPython.config.loader import PyFileConfigLoader
+from IPython.core.application import Application
+from IPython.core.component import Component
+from IPython.config.loader import ArgParseConfigLoader, NoConfigDefault
+from IPython.utils.traitlets import Unicode, Bool
+from IPython.utils import genutils
+
+#-----------------------------------------------------------------------------
+# Imports
+#-----------------------------------------------------------------------------
+
+
+class ClusterDirError(Exception):
+    pass
+
+
+class PIDFileError(Exception):
+    pass
+
+
+class ClusterDir(Component):
+    """An object to manage the cluster directory and its resources.
+
+    The cluster directory is used by :command:`ipcontroller`, 
+    :command:`ipcontroller` and :command:`ipcontroller` to manage the
+    configuration, logging and security of these applications.
+
+    This object knows how to find, create and manage these directories. This
+    should be used by any code that want's to handle cluster directories.
+    """
+
+    security_dir_name = Unicode('security')
+    log_dir_name = Unicode('log')
+    pid_dir_name = Unicode('pid')
+    security_dir = Unicode(u'')
+    log_dir = Unicode(u'')
+    pid_dir = Unicode(u'')
+    location = Unicode(u'')
+
+    def __init__(self, location):
+        super(ClusterDir, self).__init__(None)
+        self.location = location
+
+    def _location_changed(self, name, old, new):
+        if not os.path.isdir(new):
+            os.makedirs(new)
+        self.security_dir = os.path.join(new, self.security_dir_name)
+        self.log_dir = os.path.join(new, self.log_dir_name)
+        self.pid_dir = os.path.join(new, self.pid_dir_name)
+        self.check_dirs()
+
+    def _log_dir_changed(self, name, old, new):
+        self.check_log_dir()
+
+    def check_log_dir(self):
+        if not os.path.isdir(self.log_dir):
+            os.mkdir(self.log_dir)
+
+    def _security_dir_changed(self, name, old, new):
+        self.check_security_dir()
+
+    def check_security_dir(self):
+        if not os.path.isdir(self.security_dir):
+            os.mkdir(self.security_dir, 0700)
+        os.chmod(self.security_dir, 0700)
+
+    def _pid_dir_changed(self, name, old, new):
+        self.check_pid_dir()
+
+    def check_pid_dir(self):
+        if not os.path.isdir(self.pid_dir):
+            os.mkdir(self.pid_dir, 0700)
+        os.chmod(self.pid_dir, 0700)
+
+    def check_dirs(self):
+        self.check_security_dir()
+        self.check_log_dir()
+        self.check_pid_dir()
+
+    def load_config_file(self, filename):
+        """Load a config file from the top level of the cluster dir.
+
+        Parameters
+        ----------
+        filename : unicode or str
+            The filename only of the config file that must be located in
+            the top-level of the cluster directory.
+        """
+        loader = PyFileConfigLoader(filename, self.location)
+        return loader.load_config()
+
+    def copy_config_file(self, config_file, path=None, overwrite=False):
+        """Copy a default config file into the active cluster directory.
+
+        Default configuration files are kept in :mod:`IPython.config.default`.
+        This function moves these from that location to the working cluster
+        directory.
+        """
+        if path is None:
+            import IPython.config.default
+            path = IPython.config.default.__file__.split(os.path.sep)[:-1]
+            path = os.path.sep.join(path)
+        src = os.path.join(path, config_file)
+        dst = os.path.join(self.location, config_file)
+        if not os.path.isfile(dst) or overwrite:
+            shutil.copy(src, dst)
+
+    def copy_all_config_files(self, path=None, overwrite=False):
+        """Copy all config files into the active cluster directory."""
+        for f in [u'ipcontroller_config.py', u'ipengine_config.py',
+                  u'ipcluster_config.py']:
+            self.copy_config_file(f, path=path, overwrite=overwrite)
+
+    @classmethod
+    def create_cluster_dir(csl, cluster_dir):
+        """Create a new cluster directory given a full path.
+
+        Parameters
+        ----------
+        cluster_dir : str
+            The full path to the cluster directory.  If it does exist, it will
+            be used.  If not, it will be created.
+        """
+        return ClusterDir(cluster_dir)
+
+    @classmethod
+    def create_cluster_dir_by_profile(cls, path, profile=u'default'):
+        """Create a cluster dir by profile name and path.
+
+        Parameters
+        ----------
+        path : str
+            The path (directory) to put the cluster directory in.
+        profile : str
+            The name of the profile.  The name of the cluster directory will
+            be "cluster_<profile>".
+        """
+        if not os.path.isdir(path):
+            raise ClusterDirError('Directory not found: %s' % path)
+        cluster_dir = os.path.join(path, u'cluster_' + profile)
+        return ClusterDir(cluster_dir)
+
+    @classmethod
+    def find_cluster_dir_by_profile(cls, ipython_dir, profile=u'default'):
+        """Find an existing cluster dir by profile name, return its ClusterDir.
+
+        This searches through a sequence of paths for a cluster dir.  If it
+        is not found, a :class:`ClusterDirError` exception will be raised.
+
+        The search path algorithm is:
+        1. ``os.getcwd()``
+        2. ``ipython_dir``
+        3. The directories found in the ":" separated 
+           :env:`IPCLUSTER_DIR_PATH` environment variable.
+
+        Parameters
+        ----------
+        ipython_dir : unicode or str
+            The IPython directory to use.
+        profile : unicode or str
+            The name of the profile.  The name of the cluster directory
+            will be "cluster_<profile>".
+        """
+        dirname = u'cluster_' + profile
+        cluster_dir_paths = os.environ.get('IPCLUSTER_DIR_PATH','')
+        if cluster_dir_paths:
+            cluster_dir_paths = cluster_dir_paths.split(':')
+        else:
+            cluster_dir_paths = []
+        paths = [os.getcwd(), ipython_dir] + cluster_dir_paths
+        for p in paths:
+            cluster_dir = os.path.join(p, dirname)
+            if os.path.isdir(cluster_dir):
+                return ClusterDir(cluster_dir)
+        else:
+            raise ClusterDirError('Cluster directory not found in paths: %s' % dirname)
+
+    @classmethod
+    def find_cluster_dir(cls, cluster_dir):
+        """Find/create a cluster dir and return its ClusterDir.
+
+        This will create the cluster directory if it doesn't exist.
+
+        Parameters
+        ----------
+        cluster_dir : unicode or str
+            The path of the cluster directory.  This is expanded using
+            :func:`IPython.utils.genutils.expand_path`.
+        """
+        cluster_dir = genutils.expand_path(cluster_dir)
+        if not os.path.isdir(cluster_dir):
+            raise ClusterDirError('Cluster directory not found: %s' % cluster_dir)
+        return ClusterDir(cluster_dir)
+
+
+class AppWithClusterDirArgParseConfigLoader(ArgParseConfigLoader):
+    """Default command line options for IPython cluster applications."""
+
+    def _add_other_arguments(self):
+        self.parser.add_argument('--ipython-dir', 
+            dest='Global.ipython_dir',type=unicode,
+            help='Set to override default location of Global.ipython_dir.',
+            default=NoConfigDefault,
+            metavar='Global.ipython_dir'
+        )
+        self.parser.add_argument('-p', '--profile',
+            dest='Global.profile',type=unicode,
+            help='The string name of the profile to be used. This determines '
+            'the name of the cluster dir as: cluster_<profile>. The default profile '
+            'is named "default".  The cluster directory is resolve this way '
+            'if the --cluster-dir option is not used.',
+            default=NoConfigDefault,
+            metavar='Global.profile'
+        )
+        self.parser.add_argument('--log-level',
+            dest="Global.log_level",type=int,
+            help='Set the log level (0,10,20,30,40,50).  Default is 30.',
+            default=NoConfigDefault,
+            metavar="Global.log_level"
+        )
+        self.parser.add_argument('--cluster-dir',
+            dest='Global.cluster_dir',type=unicode,
+            help='Set the cluster dir. This overrides the logic used by the '
+            '--profile option.',
+            default=NoConfigDefault,
+            metavar='Global.cluster_dir'
+        ),
+        self.parser.add_argument('--work-dir',
+            dest='Global.work_dir',type=unicode,
+            help='Set the working dir for the process.',
+            default=NoConfigDefault,
+            metavar='Global.work_dir'
+        )
+        self.parser.add_argument('--clean-logs',
+            dest='Global.clean_logs', action='store_true',
+            help='Delete old log flies before starting.',
+            default=NoConfigDefault
+        )
+        self.parser.add_argument('--no-clean-logs',
+            dest='Global.clean_logs', action='store_false',
+            help="Don't Delete old log flies before starting.",
+            default=NoConfigDefault
+        )
+
+class ApplicationWithClusterDir(Application):
+    """An application that puts everything into a cluster directory.
+
+    Instead of looking for things in the ipython_dir, this type of application
+    will use its own private directory called the "cluster directory"
+    for things like config files, log files, etc.
+
+    The cluster directory is resolved as follows:
+
+    * If the ``--cluster-dir`` option is given, it is used.
+    * If ``--cluster-dir`` is not given, the application directory is 
+      resolve using the profile name as ``cluster_<profile>``. The search 
+      path for this directory is then i) cwd if it is found there
+      and ii) in ipython_dir otherwise.
+
+    The config file for the application is to be put in the cluster
+    dir and named the value of the ``config_file_name`` class attribute.
+    """
+
+    auto_create_cluster_dir = True
+
+    def create_default_config(self):
+        super(ApplicationWithClusterDir, self).create_default_config()
+        self.default_config.Global.profile = u'default'
+        self.default_config.Global.cluster_dir = u''
+        self.default_config.Global.work_dir = os.getcwd()
+        self.default_config.Global.log_to_file = False
+        self.default_config.Global.clean_logs = False
+
+    def create_command_line_config(self):
+        """Create and return a command line config loader."""
+        return AppWithClusterDirArgParseConfigLoader(
+            description=self.description, 
+            version=release.version
+        )
+
+    def find_resources(self):
+        """This resolves the cluster directory.
+
+        This tries to find the cluster directory and if successful, it will
+        have done:
+        * Sets ``self.cluster_dir_obj`` to the :class:`ClusterDir` object for 
+          the application.
+        * Sets ``self.cluster_dir`` attribute of the application and config
+          objects.
+
+        The algorithm used for this is as follows:
+        1. Try ``Global.cluster_dir``.
+        2. Try using ``Global.profile``.
+        3. If both of these fail and ``self.auto_create_cluster_dir`` is
+           ``True``, then create the new cluster dir in the IPython directory.
+        4. If all fails, then raise :class:`ClusterDirError`.
+        """
+
+        try:
+            cluster_dir = self.command_line_config.Global.cluster_dir
+        except AttributeError:
+            cluster_dir = self.default_config.Global.cluster_dir
+        cluster_dir = genutils.expand_path(cluster_dir)
+        try:
+            self.cluster_dir_obj = ClusterDir.find_cluster_dir(cluster_dir)
+        except ClusterDirError:
+            pass
+        else:
+            self.log.info('Using existing cluster dir: %s' % \
+                self.cluster_dir_obj.location
+            )
+            self.finish_cluster_dir()
+            return
+
+        try:
+            self.profile = self.command_line_config.Global.profile
+        except AttributeError:
+            self.profile = self.default_config.Global.profile
+        try:
+            self.cluster_dir_obj = ClusterDir.find_cluster_dir_by_profile(
+                self.ipython_dir, self.profile)
+        except ClusterDirError:
+            pass
+        else:
+            self.log.info('Using existing cluster dir: %s' % \
+                self.cluster_dir_obj.location
+            )
+            self.finish_cluster_dir()
+            return
+
+        if self.auto_create_cluster_dir:
+            self.cluster_dir_obj = ClusterDir.create_cluster_dir_by_profile(
+                self.ipython_dir, self.profile
+            )
+            self.log.info('Creating new cluster dir: %s' % \
+                self.cluster_dir_obj.location
+            )
+            self.finish_cluster_dir()
+        else:
+            raise ClusterDirError('Could not find a valid cluster directory.')
+
+    def finish_cluster_dir(self):
+        # Set the cluster directory
+        self.cluster_dir = self.cluster_dir_obj.location
+        
+        # These have to be set because they could be different from the one
+        # that we just computed.  Because command line has the highest
+        # priority, this will always end up in the master_config.
+        self.default_config.Global.cluster_dir = self.cluster_dir
+        self.command_line_config.Global.cluster_dir = self.cluster_dir
+
+        # Set the search path to the cluster directory
+        self.config_file_paths = (self.cluster_dir,)
+
+    def find_config_file_name(self):
+        """Find the config file name for this application."""
+        # For this type of Application it should be set as a class attribute.
+        if not hasattr(self, 'config_file_name'):
+            self.log.critical("No config filename found")
+
+    def find_config_file_paths(self):
+        # Set the search path to the cluster directory
+        self.config_file_paths = (self.cluster_dir,)
+
+    def pre_construct(self):
+        # The log and security dirs were set earlier, but here we put them
+        # into the config and log them.
+        config = self.master_config
+        sdir = self.cluster_dir_obj.security_dir
+        self.security_dir = config.Global.security_dir = sdir
+        ldir = self.cluster_dir_obj.log_dir
+        self.log_dir = config.Global.log_dir = ldir
+        pdir = self.cluster_dir_obj.pid_dir
+        self.pid_dir = config.Global.pid_dir = pdir
+        self.log.info("Cluster directory set to: %s" % self.cluster_dir)
+        config.Global.work_dir = unicode(genutils.expand_path(config.Global.work_dir))
+        # Change to the working directory. We do this just before construct
+        # is called so all the components there have the right working dir.
+        self.to_work_dir()
+
+    def to_work_dir(self):
+        wd = self.master_config.Global.work_dir
+        if unicode(wd) != unicode(os.getcwd()):
+            os.chdir(wd)
+            self.log.info("Changing to working dir: %s" % wd)
+
+    def start_logging(self):
+        # Remove old log files
+        if self.master_config.Global.clean_logs:
+            log_dir = self.master_config.Global.log_dir
+            for f in os.listdir(log_dir):
+                if f.startswith(self.name + u'-') and f.endswith('.log'):
+                    os.remove(os.path.join(log_dir, f))
+        # Start logging to the new log file
+        if self.master_config.Global.log_to_file:
+            log_filename = self.name + u'-' + str(os.getpid()) + u'.log'
+            logfile = os.path.join(self.log_dir, log_filename)
+            open_log_file = open(logfile, 'w')
+        else:
+            open_log_file = sys.stdout
+        log.startLogging(open_log_file)
+
+    def write_pid_file(self, overwrite=False):
+        """Create a .pid file in the pid_dir with my pid.
+
+        This must be called after pre_construct, which sets `self.pid_dir`.
+        This raises :exc:`PIDFileError` if the pid file exists already.
+        """
+        pid_file = os.path.join(self.pid_dir, self.name + u'.pid')
+        if os.path.isfile(pid_file):
+            pid = self.get_pid_from_file()
+            if not overwrite:
+                raise PIDFileError(
+                    'The pid file [%s] already exists. \nThis could mean that this '
+                    'server is already running with [pid=%s].' % (pid_file, pid)
+                )
+        with open(pid_file, 'w') as f:
+            self.log.info("Creating pid file: %s" % pid_file)
+            f.write(repr(os.getpid())+'\n')
+
+    def remove_pid_file(self):
+        """Remove the pid file.
+
+        This should be called at shutdown by registering a callback with
+        :func:`reactor.addSystemEventTrigger`. This needs to return
+        ``None``.
+        """
+        pid_file = os.path.join(self.pid_dir, self.name + u'.pid')
+        if os.path.isfile(pid_file):
+            try:
+                self.log.info("Removing pid file: %s" % pid_file)
+                os.remove(pid_file)
+            except:
+                self.log.warn("Error removing the pid file: %s" % pid_file)
+
+    def get_pid_from_file(self):
+        """Get the pid from the pid file.
+
+        If the  pid file doesn't exist a :exc:`PIDFileError` is raised.
+        """
+        pid_file = os.path.join(self.pid_dir, self.name + u'.pid')
+        if os.path.isfile(pid_file):
+            with open(pid_file, 'r') as f:
+                pid = int(f.read().strip())
+                return pid
+        else:
+            raise PIDFileError('pid file not found: %s' % pid_file)
+
+

=== removed directory 'IPython/kernel/config'
=== removed file 'IPython/kernel/config/__init__.py'
--- IPython/kernel/config/__init__.py	2009-07-02 02:49:32 +0000
+++ IPython/kernel/config/__init__.py	1970-01-01 00:00:00 +0000
@@ -1,126 +0,0 @@
-# encoding: utf-8
-
-"""Default kernel configuration."""
-
-__docformat__ = "restructuredtext en"
-
-#-------------------------------------------------------------------------------
-#  Copyright (C) 2008  The IPython Development Team
-#
-#  Distributed under the terms of the BSD License.  The full license is in
-#  the file COPYING, distributed as part of this software.
-#-------------------------------------------------------------------------------
-
-#-------------------------------------------------------------------------------
-# Imports
-#-------------------------------------------------------------------------------
-
-import os, sys
-from os.path import join as pjoin
-
-from IPython.external.configobj import ConfigObj
-from IPython.config.api import ConfigObjManager
-from IPython.utils.genutils import get_ipython_dir, get_security_dir
-
-default_kernel_config = ConfigObj()
-
-# This will raise OSError if ipythondir doesn't exist.
-security_dir = get_security_dir()
-
-#-------------------------------------------------------------------------------
-# Engine Configuration
-#-------------------------------------------------------------------------------
-
-engine_config = dict(
-    logfile = '',    # Empty means log to stdout
-    furl_file = pjoin(security_dir, 'ipcontroller-engine.furl')
-)
-
-#-------------------------------------------------------------------------------
-# MPI Configuration
-#-------------------------------------------------------------------------------
-
-mpi_config = dict(
-    mpi4py = """from mpi4py import MPI as mpi
-mpi.size = mpi.COMM_WORLD.Get_size()
-mpi.rank = mpi.COMM_WORLD.Get_rank()
-""",
-    pytrilinos = """from PyTrilinos import Epetra
-class SimpleStruct:
-    pass
-mpi = SimpleStruct()
-mpi.rank = 0
-mpi.size = 0
-""",
-    default = ''
-)
-
-#-------------------------------------------------------------------------------
-# Controller Configuration
-#-------------------------------------------------------------------------------
-
-controller_config = dict(
-
-    logfile = '',    # Empty means log to stdout
-    import_statement = '',
-    reuse_furls = False, # If False, old furl files are deleted
-
-    engine_tub = dict(
-        ip = '',         # Empty string means all interfaces
-        port = 0,        # 0 means pick a port for me
-        location = '',    # Empty string means try to set automatically
-        secure = True,
-        cert_file = pjoin(security_dir, 'ipcontroller-engine.pem'),
-    ),
-    engine_fc_interface = 'IPython.kernel.enginefc.IFCControllerBase',
-    engine_furl_file = pjoin(security_dir, 'ipcontroller-engine.furl'),
-    
-    controller_interfaces = dict(
-        # multiengine = dict(
-        #     controller_interface = 'IPython.kernel.multiengine.IMultiEngine',
-        #     fc_interface = 'IPython.kernel.multienginefc.IFCMultiEngine',
-        #     furl_file = 'ipcontroller-mec.furl'
-        # ),
-        task = dict(
-            controller_interface = 'IPython.kernel.task.ITaskController',
-            fc_interface = 'IPython.kernel.taskfc.IFCTaskController',
-            furl_file = pjoin(security_dir, 'ipcontroller-tc.furl')
-        ),
-        multiengine = dict(
-            controller_interface = 'IPython.kernel.multiengine.IMultiEngine',
-            fc_interface = 'IPython.kernel.multienginefc.IFCSynchronousMultiEngine',
-            furl_file = pjoin(security_dir, 'ipcontroller-mec.furl')
-        )
-    ),
-
-    client_tub = dict(
-        ip = '',         # Empty string means all interfaces
-        port = 0,        # 0 means pick a port for me
-        location = '',    # Empty string means try to set automatically
-        secure = True,
-        cert_file = pjoin(security_dir, 'ipcontroller-client.pem')
-    )
-)
-
-#-------------------------------------------------------------------------------
-# Client Configuration
-#-------------------------------------------------------------------------------
-
-client_config = dict(
-    client_interfaces = dict(
-        task = dict(
-            furl_file = pjoin(security_dir, 'ipcontroller-tc.furl')
-        ),
-        multiengine = dict(
-            furl_file = pjoin(security_dir, 'ipcontroller-mec.furl')
-        )
-    )
-)
-
-default_kernel_config['engine'] = engine_config
-default_kernel_config['mpi'] = mpi_config
-default_kernel_config['controller'] = controller_config
-default_kernel_config['client'] = client_config
-
-
-config_manager = ConfigObjManager(default_kernel_config, 'IPython.kernel.ini')
\ No newline at end of file

=== added file 'IPython/kernel/configobjfactory.py'
--- IPython/kernel/configobjfactory.py	1970-01-01 00:00:00 +0000
+++ IPython/kernel/configobjfactory.py	2009-11-21 00:11:11 +0000
@@ -0,0 +1,79 @@
+#!/usr/bin/env python
+# encoding: utf-8
+"""
+A class for creating a Twisted service that is configured using IPython's
+configuration system.
+"""
+
+#-----------------------------------------------------------------------------
+#  Copyright (C) 2008-2009  The IPython Development Team
+#
+#  Distributed under the terms of the BSD License.  The full license is in
+#  the file COPYING, distributed as part of this software.
+#-----------------------------------------------------------------------------
+
+#-----------------------------------------------------------------------------
+# Imports
+#-----------------------------------------------------------------------------
+
+import zope.interface as zi
+
+from IPython.core.component import Component
+
+#-----------------------------------------------------------------------------
+# Code
+#-----------------------------------------------------------------------------
+
+
+class IConfiguredObjectFactory(zi.Interface):
+    """I am a component that creates a configured object.
+
+    This class is useful if you want to configure a class that is not a 
+    subclass of :class:`IPython.core.component.Component`.
+    """
+
+    def __init__(config):
+        """Get ready to configure the object using config."""
+
+    def create():
+        """Return an instance of the configured object."""
+
+
+class ConfiguredObjectFactory(Component):
+
+    zi.implements(IConfiguredObjectFactory)
+
+    def __init__(self, config):
+        super(ConfiguredObjectFactory, self).__init__(None, config=config)
+
+    def create(self):
+        raise NotImplementedError('create must be implemented in a subclass')
+
+
+class IAdaptedConfiguredObjectFactory(zi.Interface):
+    """I am a component that adapts and configures an object.
+
+    This class is useful if you have the adapt an instance and configure it.
+    """
+
+    def __init__(config, adaptee=None):
+        """Get ready to adapt adaptee and then configure it using config."""
+
+    def create():
+        """Return an instance of the adapted and configured object."""
+
+
+class AdaptedConfiguredObjectFactory(Component):
+
+    # zi.implements(IAdaptedConfiguredObjectFactory)
+
+    def __init__(self, config, adaptee):
+        # print
+        # print "config pre:", config
+        super(AdaptedConfiguredObjectFactory, self).__init__(None, config=config)
+        # print
+        # print "config post:", config
+        self.adaptee = adaptee
+
+    def create(self):
+        raise NotImplementedError('create must be implemented in a subclass')
\ No newline at end of file

=== removed directory 'IPython/kernel/core/config'
=== removed file 'IPython/kernel/core/config/__init__.py'
--- IPython/kernel/core/config/__init__.py	2008-06-06 19:41:55 +0000
+++ IPython/kernel/core/config/__init__.py	1970-01-01 00:00:00 +0000
@@ -1,25 +0,0 @@
-# encoding: utf-8
-
-__docformat__ = "restructuredtext en"
-
-#-------------------------------------------------------------------------------
-#  Copyright (C) 2008  The IPython Development Team
-#
-#  Distributed under the terms of the BSD License.  The full license is in
-#  the file COPYING, distributed as part of this software.
-#-------------------------------------------------------------------------------
-
-#-------------------------------------------------------------------------------
-# Imports
-#-------------------------------------------------------------------------------
-
-from IPython.external.configobj import ConfigObj
-from IPython.config.api import ConfigObjManager
-
-default_core_config = ConfigObj()
-default_core_config['shell'] = dict(
-    shell_class = 'IPython.kernel.core.interpreter.Interpreter',
-    import_statement = ''
-)
-
-config_manager = ConfigObjManager(default_core_config, 'IPython.kernel.core.ini')
\ No newline at end of file

=== modified file 'IPython/kernel/engineconnector.py'
--- IPython/kernel/engineconnector.py	2009-04-18 21:53:37 +0000
+++ IPython/kernel/engineconnector.py	2009-11-21 00:11:11 +0000
@@ -1,32 +1,38 @@
+#!/usr/bin/env python
 # encoding: utf-8
 
 """A class that manages the engines connection to the controller."""
 
-__docformat__ = "restructuredtext en"
-
-#-------------------------------------------------------------------------------
-#  Copyright (C) 2008  The IPython Development Team
+#-----------------------------------------------------------------------------
+#  Copyright (C) 2008-2009  The IPython Development Team
 #
 #  Distributed under the terms of the BSD License.  The full license is in
 #  the file COPYING, distributed as part of this software.
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 # Imports
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 
 import os
 import cPickle as pickle
 
 from twisted.python import log, failure
 from twisted.internet import defer
+from twisted.internet.defer import inlineCallbacks, returnValue
 
-from IPython.kernel.fcutil import find_furl
+from IPython.kernel.fcutil import find_furl, validate_furl_or_file
 from IPython.kernel.enginefc import IFCEngine
+from IPython.kernel.twistedutil import sleep_deferred, make_deferred
 
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 # The ClientConnector class
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
+
+
+class EngineConnectorError(Exception):
+    pass
+
 
 class EngineConnector(object):
     """Manage an engines connection to a controller.
@@ -38,8 +44,10 @@
     
     def __init__(self, tub):
         self.tub = tub
-        
-    def connect_to_controller(self, engine_service, furl_or_file):
+
+    @make_deferred
+    def connect_to_controller(self, engine_service, furl_or_file,
+                              delay=0.1, max_tries=10):
         """
         Make a connection to a controller specified by a furl.
         
@@ -48,34 +56,73 @@
         foolscap URL contains all the information needed to connect to the 
         controller, including the ip and port as well as any encryption and 
         authentication information needed for the connection.
-        
+
         After getting a reference to the controller, this method calls the 
         `register_engine` method of the controller to actually register the 
         engine.
-        
-        :Parameters:
-            engine_service : IEngineBase
-                An instance of an `IEngineBase` implementer
-            furl_or_file : str
-                A furl or a filename containing a furl
+
+        This method will try to connect to the controller multiple times with
+        a delay in between.  Each time the FURL file is read anew.
+
+        Parameters
+        __________
+        engine_service : IEngineBase
+            An instance of an `IEngineBase` implementer
+        furl_or_file : str
+            A furl or a filename containing a furl
+        delay : float
+            The intial time to wait between connection attempts.  Subsequent
+            attempts have increasing delays.
+        max_tries : int
+            The maximum number of connection attempts.
+
+        Returns
+        -------
+        A deferred to the registered client or a failure to an error
+        like :exc:`FURLError`.
         """
         if not self.tub.running:
             self.tub.startService()
         self.engine_service = engine_service
         self.engine_reference = IFCEngine(self.engine_service)
-        try:
-            self.furl = find_furl(furl_or_file)
-        except ValueError:
-            return defer.fail(failure.Failure())
+
+        validate_furl_or_file(furl_or_file)
+        d = self._try_to_connect(furl_or_file, delay, max_tries, attempt=0)
+        d.addCallback(self._register)
+        return d
+
+    @inlineCallbacks
+    def _try_to_connect(self, furl_or_file, delay, max_tries, attempt):
+        """Try to connect to the controller with retry logic."""
+        if attempt < max_tries:
+            log.msg("Attempting to connect to controller [%r]: %s" % \
+                (attempt, furl_or_file))
+            try:
+                self.furl = find_furl(furl_or_file)
+                # Uncomment this to see the FURL being tried.
+                # log.msg("FURL: %s" % self.furl)
+                rr = yield self.tub.getReference(self.furl)
+            except:
+                if attempt==max_tries-1:
+                    # This will propagate the exception all the way to the top
+                    # where it can be handled.
+                    raise
+                else:
+                    yield sleep_deferred(delay)
+                    rr = yield self._try_to_connect(
+                        furl_or_file, 1.5*delay, max_tries, attempt+1
+                    )
+                    # rr becomes an int when there is a connection!!!
+                    returnValue(rr)
+            else:
+                returnValue(rr)
         else:
-            d = self.tub.getReference(self.furl)
-            d.addCallbacks(self._register, self._log_failure)
-            return d
-    
-    def _log_failure(self, reason):
-        log.err('EngineConnector: engine registration failed:')
-        log.err(reason)
-        return reason
+            raise EngineConnectorError(
+                'Could not connect to controller, max_tries (%r) exceeded. '
+                'This usually means that i) the controller was not started, '
+                'or ii) a firewall was blocking the engine from connecting '
+                'to the controller.' % max_tries
+            )
 
     def _register(self, rr):
         self.remote_ref = rr
@@ -83,7 +130,7 @@
         desired_id = self.engine_service.id
         d = self.remote_ref.callRemote('register_engine', self.engine_reference, 
             desired_id, os.getpid(), pickle.dumps(self.engine_service.properties,2))
-        return d.addCallbacks(self._reference_sent, self._log_failure)
+        return d.addCallback(self._reference_sent)
 
     def _reference_sent(self, registration_dict):
         self.engine_service.id = registration_dict['id']

=== modified file 'IPython/kernel/error.py'
--- IPython/kernel/error.py	2009-08-04 01:18:28 +0000
+++ IPython/kernel/error.py	2009-11-21 00:11:11 +0000
@@ -127,9 +127,11 @@
 class CompositeError(KernelError):
     def __init__(self, message, elist):
         Exception.__init__(self, *(message, elist))
-        self.message = message
+        # Don't use pack_exception because it will conflict with the .message
+        # attribute that is being deprecated in 2.6 and beyond.
+        self.msg = message
         self.elist = elist
-  
+
     def _get_engine_str(self, ev):
         try:
             ei = ev._ipython_engine_info
@@ -137,7 +139,7 @@
             return '[Engine Exception]'
         else:
             return '[%i:%s]: ' % (ei['engineid'], ei['method'])
-    
+
     def _get_traceback(self, ev):
         try:
             tb = ev._ipython_traceback_text
@@ -145,14 +147,14 @@
             return 'No traceback available'
         else:
             return tb
-  
+
     def __str__(self):
-        s = str(self.message)
+        s = str(self.msg)
         for et, ev, etb in self.elist:
             engine_str = self._get_engine_str(ev)
             s = s + '\n' + engine_str + str(et.__name__) + ': ' + str(ev)
         return s
-    
+
     def print_tracebacks(self, excid=None):
         if excid is None:
             for (et,ev,etb) in self.elist:

=== modified file 'IPython/kernel/fcutil.py'
--- IPython/kernel/fcutil.py	2008-06-06 19:41:55 +0000
+++ IPython/kernel/fcutil.py	2009-11-21 00:11:11 +0000
@@ -1,27 +1,62 @@
+#!/usr/bin/env python
 # encoding: utf-8
-
-"""Foolscap related utilities."""
-
-__docformat__ = "restructuredtext en"
-
-#-------------------------------------------------------------------------------
-#  Copyright (C) 2008  The IPython Development Team
+"""
+Foolscap related utilities.
+"""
+
+#-----------------------------------------------------------------------------
+#  Copyright (C) 2008-2009  The IPython Development Team
 #
 #  Distributed under the terms of the BSD License.  The full license is in
 #  the file COPYING, distributed as part of this software.
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
 # Imports
-#-------------------------------------------------------------------------------
+#-----------------------------------------------------------------------------
+
+from __future__ import with_statement
 
 import os
+import tempfile
+
+from twisted.internet import reactor, defer
+from twisted.python import log
 
 from foolscap import Tub, UnauthenticatedTub
 
+from IPython.config.loader import Config
+
+from IPython.kernel.configobjfactory import AdaptedConfiguredObjectFactory
+
+from IPython.kernel.error import SecurityError
+
+from IPython.utils.traitlets import Int, Str, Bool, Instance
+from IPython.utils.importstring import import_item
+
+#-----------------------------------------------------------------------------
+# Code
+#-----------------------------------------------------------------------------
+
+
+# We do this so if a user doesn't have OpenSSL installed, it will try to use
+# an UnauthenticatedTub.  But, they will still run into problems if they
+# try to use encrypted furls.
+try:
+    import OpenSSL
+except:
+    Tub = UnauthenticatedTub
+    have_crypto = False
+else:
+    have_crypto = True
+
+
+class FURLError(Exception):
+    pass
+
+
 def check_furl_file_security(furl_file, secure):
     """Remove the old furl_file if changing security modes."""
-    
     if os.path.isfile(furl_file):
         f = open(furl_file, 'r')
         oldfurl = f.read().strip()
@@ -29,41 +64,210 @@
         if (oldfurl.startswith('pb://') and not secure) or (oldfurl.startswith('pbu://') and secure):
             os.remove(furl_file)
 
+
 def is_secure(furl):
+    """Is the given FURL secure or not."""
     if is_valid(furl):
         if furl.startswith("pb://"):
             return True
         elif furl.startswith("pbu://"):
             return False
     else:
-        raise ValueError("invalid furl: %s" % furl)
+        raise FURLError("invalid FURL: %s" % furl)
+
 
 def is_valid(furl):
+    """Is the str a valid FURL or not."""
     if isinstance(furl, str):
         if furl.startswith("pb://") or furl.startswith("pbu://"):
             return True
     else:
         return False
 
+
 def find_furl(furl_or_file):
+    """Find, validate and return a FURL in a string or file."""
     if isinstance(furl_or_file, str):
         if is_valid(furl_or_file):
             return furl_or_file
     if os.path.isfile(furl_or_file):
-        furl = open(furl_or_file, 'r').read().strip()
+        with open(furl_or_file, 'r') as f:
+            furl = f.read().strip()
         if is_valid(furl):
             return furl
-    raise ValueError("not a furl or a file containing a furl: %s" % furl_or_file)
-
-# We do this so if a user doesn't have OpenSSL installed, it will try to use
-# an UnauthenticatedTub.  But, they will still run into problems if they
-# try to use encrypted furls.
-try: