Merge lp:~ellisonbg/ipython/bugfixes0411409 into lp:ipython/0.11
- bugfixes0411409
- Merge into trunk
| Reviewer | Review Type | Date Requested | Status |
|---|---|---|---|
| Fernando Perez | Needs Fixing | ||
|
Review via email:
|
|||
Commit message
Description of the change
| Brian Granger (ellisonbg) wrote : | # |
- 1176. By Brian Granger
-
Adding test_twisteduti
l.py. - 1177. By Brian Granger
-
Merged Gael's ipython-
sync-frontend branch with important mods. This branch showed conflicts with recent changes I made to:
IPython.
kernel. core.interprete r
IPython.kernel. core.tests. test_interprete r I resolved these conflicts and closed a ticket that Gael fixed in his
branch. BUT, there is a related ticket that I opened. Furthermore,
Gael was using nose in test_interpreter, but we need to keep things in
IPython.kernel free of nose! - 1178. By Brian Granger
-
Merging in Jorgen's changes to iplib.py for ticket 315392.
- 1179. By Brian Granger
-
Closing invalid ticket 364347.
This ticket is invalid (thanks to R. Kern) and I have changed the test
to test the correct thing. - 1180. By Brian Granger
-
Initial refactor of task dependency system.
We are thinking about refactoring the task dependency system. Currently
is is based on engine properties and a function sent with each task.In this commit, I have added a TaskRejectError that tasks can raise to
indicate that the engine doesn't have the required dependencies. For
now I have not removed any of the old stuff, but I have added warnings
that we might remove the older *_prop* methods in IMultiEngine.See this ticket for more info: https:/
/bugs.launchpad .net/bugs/ 361419
| Fernando Perez (fdo.perez) wrote : | # |
Not done with the review yet, but in the meantime, please merge the fixes I put in:
https:/
This fixes one of the broken tests. There's another one that I don't know how to fix, so I'll mention that in the actual review.
| Fernando Perez (fdo.perez) wrote : | # |
- The cocoa tests currently fail if I try to run the suite with Trial, because they use the nose skiptest form. I'm not sure if trial has a way to spell that out. After merging my fix (see previous comment) I get with
trial IPython
...
=======
[ERROR]: IPython.
Traceback (most recent call last):
File "/usr/lib/
module = modinfo.load()
File "/usr/lib/
return self.pathEntry.
File "/usr/lib/
return self._moduleLoa
File "/usr/lib/
topLevelPackage = _importAndCheck
File "/usr/lib/
return __import_
File "/home/
raise nose.SkipTest("This test requires zope.interface, Twisted, Foolscap and PyObjC")
nose.plugins.
-------
Ran 387 tests in 40.094s
FAILED (errors=1, successes=387)
Since it seems we'll be moving towards using trial for the twisted-using parts, I guess we'll need to mark these skips in a trial-compliant way, but I don't know if twisted has support for that out of the box. You may know...
More to come...
| Brian Granger (ellisonbg) wrote : | # |
I don't see any commits to your branch, should there be any?
On Tue, Apr 21, 2009 at 11:07 PM, Fernando Perez <email address hidden> wrote:
> Not done with the review yet, but in the meantime, please merge the fixes I put in:
>
> https:/
>
> This fixes one of the broken tests. There's another one that I don't know how to fix, so I'll mention that in the actual review.
> --
> https:/
> You are subscribed to branch lp:ipython.
>
--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>
| Brian Granger (ellisonbg) wrote : | # |
Yes, I think there is a way of getting twisted to skip tests, I will
look into this.
Brian
On Tue, Apr 21, 2009 at 11:11 PM, Fernando Perez <email address hidden> wrote:
> Review: Needs Fixing
> - The cocoa tests currently fail if I try to run the suite with Trial, because they use the nose skiptest form. I'm not sure if trial has a way to spell that out. After merging my fix (see previous comment) I get with
>
> trial IPython
>
> ...
>
> =======
> [ERROR]: IPython.
>
> Traceback (most recent call last):
> File "/usr/lib/
> module = modinfo.load()
> File "/usr/lib/
> return self.pathEntry.
> File "/usr/lib/
> return self._moduleLoa
> File "/usr/lib/
> topLevelPackage = _importAndCheck
> File "/usr/lib/
> return __import_
> File "/home/
> raise nose.SkipTest("This test requires zope.interface, Twisted, Foolscap and PyObjC")
> nose.plugins.
> -------
> Ran 387 tests in 40.094s
>
> FAILED (errors=1, successes=387)
>
>
> Since it seems we'll be moving towards using trial for the twisted-using parts, I guess we'll need to mark these skips in a trial-compliant way, but I don't know if twisted has support for that out of the box. You may know...
>
> More to come...
> --
> https:/
> You are subscribed to branch lp:ipython.
>
--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>
| Fernando Perez (fdo.perez) wrote : | # |
- ultraTB problems: I'm OK with the fixes in place now, I made a ticket:
https:/
so we fix this the right way once we're done with 0.10.
Please just make 'XXX' marks in those __IPYTHON__ sections so they stand out when I search for problems in the merge.
- in ipcluster.py, rather than
428 +from IPython.genutils import get_ipython_dir, get_log_dir, get_security_dir
429 +from IPython.genutils import num_cpus
I'd just use
+from IPython.genutils import ( get_ipython_dir, get_log_dir, get_security_dir,
+ num_cpus )
since now python allows that syntax (as of 2.4).
- We probably should refactor the entire ipython directory tree creation into a single get_ipython_dirs() function that makes them all and returns a dict. But that can be done later, just an idea for now...
- No docstrings in _err_and_stop and _delay_start. Even if they are private, they should have a docstring so others (like me :) actually know what they are meant to do. For private functions like this a more terse docstring is OK, but at least it should be enough so anyone reading the code has some idea of what's going on.
- In ipcontroller:
589 import sys, time, os
590 +import tempfile
I think we're now using the style of one import per line, sorted alphabetically. This makes it also easier to spot unneded imports via pyflakes and remove them by just nuking the line.
- In the docs:
930 +This is possible. but before you do this, you **must** remove any old FURL
should read:
930 +This is possible, but before you do this, you **must** remove any old FURL
- Overall, the changes look great. Once these minor things are fixed, and the Cocoa/trial skip problem referred in my previous message is addressed, as well as merging that fix I made for test_iplib, this has my approval.
Thanks a lot for all this work, it's excellent and gets us much closer to 0.10!
| Fernando Perez (fdo.perez) wrote : | # |
> I don't see any commits to your branch, should there be any?
Oops, forgot to push. There now:
http://
Sorry 'bout that :)
f
| Vishal Vatsa (vvatsa) wrote : | # |
I downloaded and ran trial also, I have one other test failure:
[ERROR]: IPython.
Traceback (most recent call last):
File "/usr/lib/
module = modinfo.load()
File "/usr/lib/
return self.pathEntry.
File "/usr/lib/
return self._moduleLoa
File "/usr/lib/
topLevelPackage = _importAndCheck
File "/usr/lib/
return __import_
File "/home/
ip = _ip # This is the ipapi instance
exceptions.
=======
Did you guys also get this?
| Vishal Vatsa (vvatsa) wrote : | # |
Hey,
Another thing, I see that you are removing the --ipythondir option,
since it did not work any way thats fine.
But could we make the IPYTHONDIR environmental var. more obvious in the docs.
-v
| Brian Granger (ellisonbg) wrote : | # |
Thanks for the review. I will fix all these things and merge in the
next few hours.
On Tue, Apr 21, 2009 at 11:35 PM, Fernando Perez <email address hidden> wrote:
> Review: Needs Fixing
> - ultraTB problems: I'm OK with the fixes in place now, I made a ticket:
>
> https:/
>
> so we fix this the right way once we're done with 0.10.
>
> Please just make 'XXX' marks in those __IPYTHON__ sections so they stand out when I search for problems in the merge.
>
> - in ipcluster.py, rather than
> 428 +from IPython.genutils import get_ipython_dir, get_log_dir, get_security_dir
> 429 +from IPython.genutils import num_cpus
>
> I'd just use
>
> +from IPython.genutils import ( get_ipython_dir, get_log_dir, get_security_dir,
> + num_cpus )
>
> since now python allows that syntax (as of 2.4).
>
>
> - We probably should refactor the entire ipython directory tree creation into a single get_ipython_dirs() function that makes them all and returns a dict. But that can be done later, just an idea for now...
>
>
> - No docstrings in _err_and_stop and _delay_start. Even if they are private, they should have a docstring so others (like me :) actually know what they are meant to do. For private functions like this a more terse docstring is OK, but at least it should be enough so anyone reading the code has some idea of what's going on.
>
>
> - In ipcontroller:
>
> 589 import sys, time, os
> 590 +import tempfile
>
> I think we're now using the style of one import per line, sorted alphabetically. This makes it also easier to spot unneded imports via pyflakes and remove them by just nuking the line.
>
>
> - In the docs:
> 930 +This is possible. but before you do this, you **must** remove any old FURL
>
> should read:
> 930 +This is possible, but before you do this, you **must** remove any old FURL
>
>
> - Overall, the changes look great. Once these minor things are fixed, and the Cocoa/trial skip problem referred in my previous message is addressed, as well as merging that fix I made for test_iplib, this has my approval.
>
>
> Thanks a lot for all this work, it's excellent and gets us much closer to 0.10!
> --
> https:/
> You are subscribed to branch lp:ipython.
>
--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>
| Brian Granger (ellisonbg) wrote : | # |
This fails because this test needs to be run using nosetests rather
than trial. We are still trying to figure out the best way of running
the test suite, but for now, just do:
trial IPython.kernel
Fernando and I are putting some flags in place that will make all of
this work better though. This should be in place by the end of the
day today.
Cheers,
Brian
On Wed, Apr 22, 2009 at 2:08 AM, Vishal Vatsa <email address hidden> wrote:
> I downloaded and ran trial also, I have one other test failure:
>
> [ERROR]: IPython.
>
> Traceback (most recent call last):
> File "/usr/lib/
> module = modinfo.load()
> File "/usr/lib/
> return self.pathEntry.
> File "/usr/lib/
> return self._moduleLoa
> File "/usr/lib/
> topLevelPackage = _importAndCheck
> File "/usr/lib/
> return __import_
> File "/home/
> ip = _ip # This is the ipapi instance
> exceptions.
>
> =======
> Did you guys also get this?
> --
> https:/
> You are subscribed to branch lp:ipython.
>
--
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
<email address hidden>
<email address hidden>
| Fernando Perez (fdo.perez) wrote : | # |
On Wed, Apr 22, 2009 at 2:08 AM, Vishal Vatsa <email address hidden> wrote:
> I downloaded and ran trial also, I have one other test failure:
>
> [ERROR]: IPython.
> Did you guys also get this?
Yes, but there's a fix for it in my copy of Brian's branch that he can
pull in before merging, so this one will be fine. The one about nose
vs trial will take a little more work, but we should have it sorted
out later.
Thanks!
f
- 1181. By Brian Granger
-
Merging Fernando's fixes to test_iplib.py.
| Brian Granger (ellisonbg) wrote : | # |
In trial, any module, TestCase or TestCase method that has a skip
attribute=True will be skipped. However, we need to think more
carefully about exactly what logic needs to be in place in these
situations that takes into account all of the following:
* Is nose installed
* Is Twisted installed
* Is nose or trial running the test suite
* Are the other deps installed (PyObjC, Wx, etc.)
The current logic in our test files looks like this:
try:
from IPython.
import IPython.
from IPython.
from twisted.
from IPython.
IPythonCocoaCon
from Foundation import NSMakeRect
from AppKit import NSTextView, NSScrollView
except ImportError:
import nose
raise nose.SkipTest("This test requires zope.interface, Twisted,
Foolscap and PyObjC")
Is simply not sufficient to properly handle all the various
possibilities. We need to think more carefully about the logic that
is needed. Honestly, I have a feeling that we are simply going to get
rid off all this logic and put it into our test runner.
On Tue, Apr 21, 2009 at 11:11 PM, Fernando Perez <email address hidden> wrote:
> Review: Needs Fixing
> - The cocoa tests currently fail if I try to run the suite with Trial, because they use the nose skiptest form. I'm not sure if trial has a way to spell that out. After merging my fix (see previous comment) I get with
>
> trial IPython
>
> ...
>
> =======
> [ERROR]: IPython.
>
> Traceback (most recent call last):
> File "/usr/lib/
> module = modinfo.load()
> File "/usr/lib/
> return self.pathEntry.
> File "/usr/lib/
> return self._moduleLoa
> File "/usr/lib/
> topLevelPackage = _importAndCheck
> File "/usr/lib/
> return __import_
> File "/home/
> raise nose.SkipTest("This test requires zope.interface, Twisted, Foolscap and PyObjC")
> nose.plugins.
> -------
> Ran 387 tests in 40.094s
>
> FAILED (errors=1, successes=387)
>
>
> Since it seems we'll be moving towards using trial for the twisted-using parts, I guess we'll need to mark these skips in a trial-compliant way, but I don't know if twisted has support for that out of the box. You may know...
>
> More to come...
> --
> https:/
- 1182. By Brian Granger
-
Addressing various review comments.
- 1183. By Brian Granger
-
Adding information about IPYTHONDIR to usage of ipcluster and friends.
- 1184. By Brian Granger
-
Fixing misc testing related things.
- 1185. By Brian Granger
-
Commiting fixes for running our test suite using trial and nose.
We are moving towards a model that allows trial and nose to each run
different parts of our test suite. I have added __test__ = {} to
modules that nose should skip. I have added new decorators that trial
can use to skip tests. See this ticket for more information: - 1186. By Brian Granger
-
Fix for ticket https:/
/bugs.launchpad .net/bugs/ 362458 Python 2.4 does not support a try/finally with a yield in the try. I
have changed this to a try with an empty except, followed by the
statement that was in the finally. - 1187. By Brian Granger
-
Merging fixes from Fernando's branch.
- 1188. By Brian Granger
-
Refactored iptest to include the iptestall capabilities.
We not have a test suite that can be run with `iptest all`. Give it a
shot! - 1189. By Brian Granger
-
Merging Fernando's fixes from his trunk-dev and fixing testing things.
I have merged his changes with a few modifications. I have also
removed IPython.kernel from the list of packages that nose will test. - 1190. By Brian Granger
-
Run the top level module tests in a single process.
- 1191. By Brian Granger
-
Moving find_exe -> platutils.find_cmd and making is cross platform.
The find_cmd function used to be in ipcluster.py to find executables
on win32. We need this on all platforms and now we have it! - 1192. By Brian Granger
-
Adding test_platutils.py.
- 1193. By Administrator <Administrator@brian-cda46cdbf>
-
Adding new tests for test_magic for win32.
- 1194. By Brian Granger
-
Fixing ticket https:/
/bugs.launchpad .net/bugs/ 366334 Now we find ipython using platutils.find_cmd
- 1195. By Brian Granger
-
Making the doctest exclude paths os independent.
- 1196. By Administrator <Administrator@brian-cda46cdbf>
-
Fixing doctest EXCLUDES in iptest on win32.
The regular expressions we were using on win32 were not matching. The
trick is to construct the regular expression with double \\ like this::re.
compile( 'foo\\\ \bar') Arggg!!
- 1197. By Administrator <Administrator@brian-cda46cdbf>
-
Fixing bugs with the testing system.
- 1198. By Brian Granger
-
Fixing tests in IPython.testing.
- 1199. By Brian Granger
-
Adding test_tools.py
- 1200. By Administrator <Administrator@brian-cda46cdbf>
-
More fixes for testing on win32.
- 1201. By Administrator <Administrator@brian-cda46cdbf>
-
Added platutils.
get_long_ path_name to expand paths with "~" on win32. This might be needed to fix ticket https:/
/bugs.launchpad .net/bugs/ 366353
But, there are still problems with Magic.parse_options - 1202. By Administrator <Administrator@brian-cda46cdbf>
-
Merging changes from above.
- 1203. By Administrator <Administrator@brian-cda46cdbf>
-
Changing comment in test_magic to reflect bug status.
Preview Diff
| 1 | === modified file 'IPython/frontend/tests/test_asyncfrontendbase.py' |
| 2 | --- IPython/frontend/tests/test_asyncfrontendbase.py 2008-09-14 22:13:51 +0000 |
| 3 | +++ IPython/frontend/tests/test_asyncfrontendbase.py 2009-04-16 05:08:03 +0000 |
| 4 | @@ -1,6 +1,6 @@ |
| 5 | # encoding: utf-8 |
| 6 | |
| 7 | -"""This file contains unittests for the frontendbase module.""" |
| 8 | +"""This file contains unittests for the asyncfrontendbase module.""" |
| 9 | |
| 10 | __docformat__ = "restructuredtext en" |
| 11 | |
| 12 | @@ -15,17 +15,17 @@ |
| 13 | # Imports |
| 14 | #--------------------------------------------------------------------------- |
| 15 | |
| 16 | -import unittest |
| 17 | |
| 18 | try: |
| 19 | + from twisted.trial import unittest |
| 20 | from IPython.frontend.asyncfrontendbase import AsyncFrontEndBase |
| 21 | from IPython.frontend import frontendbase |
| 22 | from IPython.kernel.engineservice import EngineService |
| 23 | + from IPython.testing.parametric import Parametric, parametric |
| 24 | except ImportError: |
| 25 | import nose |
| 26 | raise nose.SkipTest("This test requires zope.interface, Twisted and Foolscap") |
| 27 | |
| 28 | -from IPython.testing.decorators import skip |
| 29 | |
| 30 | class FrontEndCallbackChecker(AsyncFrontEndBase): |
| 31 | """FrontEndBase subclass for checking callbacks""" |
| 32 | @@ -44,14 +44,11 @@ |
| 33 | self.renderResultCalled = True |
| 34 | return result |
| 35 | |
| 36 | - |
| 37 | def render_error(self, failure): |
| 38 | self.renderErrorCalled = True |
| 39 | return failure |
| 40 | - |
| 41 | - |
| 42 | - |
| 43 | - |
| 44 | + |
| 45 | + |
| 46 | class TestAsyncFrontendBase(unittest.TestCase): |
| 47 | def setUp(self): |
| 48 | """Setup the EngineService and FrontEndBase""" |
| 49 | @@ -59,97 +56,56 @@ |
| 50 | self.fb = FrontEndCallbackChecker(engine=EngineService()) |
| 51 | |
| 52 | def test_implements_IFrontEnd(self): |
| 53 | - assert(frontendbase.IFrontEnd.implementedBy( |
| 54 | + self.assert_(frontendbase.IFrontEnd.implementedBy( |
| 55 | AsyncFrontEndBase)) |
| 56 | |
| 57 | def test_is_complete_returns_False_for_incomplete_block(self): |
| 58 | - """""" |
| 59 | - |
| 60 | block = """def test(a):""" |
| 61 | - |
| 62 | - assert(self.fb.is_complete(block) == False) |
| 63 | + self.assert_(self.fb.is_complete(block) == False) |
| 64 | |
| 65 | def test_is_complete_returns_True_for_complete_block(self): |
| 66 | - """""" |
| 67 | - |
| 68 | block = """def test(a): pass""" |
| 69 | - |
| 70 | - assert(self.fb.is_complete(block)) |
| 71 | - |
| 72 | + self.assert_(self.fb.is_complete(block)) |
| 73 | block = """a=3""" |
| 74 | - |
| 75 | - assert(self.fb.is_complete(block)) |
| 76 | + self.assert_(self.fb.is_complete(block)) |
| 77 | |
| 78 | def test_blockID_added_to_result(self): |
| 79 | block = """3+3""" |
| 80 | - |
| 81 | d = self.fb.execute(block, blockID='TEST_ID') |
| 82 | - |
| 83 | - d.addCallback(self.checkBlockID, expected='TEST_ID') |
| 84 | + d.addCallback(lambda r: self.assert_(r['blockID']=='TEST_ID')) |
| 85 | + return d |
| 86 | |
| 87 | def test_blockID_added_to_failure(self): |
| 88 | block = "raise Exception()" |
| 89 | - |
| 90 | d = self.fb.execute(block,blockID='TEST_ID') |
| 91 | - d.addErrback(self.checkFailureID, expected='TEST_ID') |
| 92 | - |
| 93 | - def checkBlockID(self, result, expected=""): |
| 94 | - assert(result['blockID'] == expected) |
| 95 | - |
| 96 | - |
| 97 | - def checkFailureID(self, failure, expected=""): |
| 98 | - assert(failure.blockID == expected) |
| 99 | - |
| 100 | + d.addErrback(lambda f: self.assert_(f.blockID=='TEST_ID')) |
| 101 | + return d |
| 102 | |
| 103 | def test_callbacks_added_to_execute(self): |
| 104 | - """test that |
| 105 | - update_cell_prompt |
| 106 | - render_result |
| 107 | - |
| 108 | - are added to execute request |
| 109 | - """ |
| 110 | - |
| 111 | d = self.fb.execute("10+10") |
| 112 | - d.addCallback(self.checkCallbacks) |
| 113 | - |
| 114 | - def checkCallbacks(self, result): |
| 115 | - assert(self.fb.updateCalled) |
| 116 | - assert(self.fb.renderResultCalled) |
| 117 | - |
| 118 | - @skip("This test fails and lead to an unhandled error in a Deferred.") |
| 119 | + d.addCallback(lambda r: self.assert_(self.fb.updateCalled and self.fb.renderResultCalled)) |
| 120 | + return d |
| 121 | + |
| 122 | def test_error_callback_added_to_execute(self): |
| 123 | - """test that render_error called on execution error""" |
| 124 | + """Test that render_error called on execution error.""" |
| 125 | |
| 126 | d = self.fb.execute("raise Exception()") |
| 127 | - d.addCallback(self.checkRenderError) |
| 128 | - |
| 129 | - def checkRenderError(self, result): |
| 130 | - assert(self.fb.renderErrorCalled) |
| 131 | + d.addErrback(lambda f: self.assert_(self.fb.renderErrorCalled)) |
| 132 | + return d |
| 133 | |
| 134 | def test_history_returns_expected_block(self): |
| 135 | - """Make sure history browsing doesn't fail""" |
| 136 | + """Make sure history browsing doesn't fail.""" |
| 137 | |
| 138 | blocks = ["a=1","a=2","a=3"] |
| 139 | - for b in blocks: |
| 140 | - d = self.fb.execute(b) |
| 141 | - |
| 142 | - # d is now the deferred for the last executed block |
| 143 | - d.addCallback(self.historyTests, blocks) |
| 144 | - |
| 145 | - |
| 146 | - def historyTests(self, result, blocks): |
| 147 | - """historyTests""" |
| 148 | - |
| 149 | - assert(len(blocks) >= 3) |
| 150 | - assert(self.fb.get_history_previous("") == blocks[-2]) |
| 151 | - assert(self.fb.get_history_previous("") == blocks[-3]) |
| 152 | - assert(self.fb.get_history_next() == blocks[-2]) |
| 153 | - |
| 154 | - |
| 155 | - def test_history_returns_none_at_startup(self): |
| 156 | - """test_history_returns_none_at_startup""" |
| 157 | - |
| 158 | - assert(self.fb.get_history_previous("")==None) |
| 159 | - assert(self.fb.get_history_next()==None) |
| 160 | - |
| 161 | - |
| 162 | + d = self.fb.execute(blocks[0]) |
| 163 | + d.addCallback(lambda _: self.fb.execute(blocks[1])) |
| 164 | + d.addCallback(lambda _: self.fb.execute(blocks[2])) |
| 165 | + d.addCallback(lambda _: self.assert_(self.fb.get_history_previous("")==blocks[-2])) |
| 166 | + d.addCallback(lambda _: self.assert_(self.fb.get_history_previous("")==blocks[-3])) |
| 167 | + d.addCallback(lambda _: self.assert_(self.fb.get_history_next()==blocks[-2])) |
| 168 | + return d |
| 169 | + |
| 170 | + def test_history_returns_none_at_startup(self): |
| 171 | + self.assert_(self.fb.get_history_previous("")==None) |
| 172 | + self.assert_(self.fb.get_history_next()==None) |
| 173 | + |
| 174 | |
| 175 | === modified file 'IPython/genutils.py' |
| 176 | --- IPython/genutils.py 2009-03-16 23:58:42 +0000 |
| 177 | +++ IPython/genutils.py 2009-04-18 23:24:22 +0000 |
| 178 | @@ -1007,7 +1007,17 @@ |
| 179 | else: |
| 180 | os.chmod(security_dir, 0700) |
| 181 | return security_dir |
| 182 | - |
| 183 | + |
| 184 | +def get_log_dir(): |
| 185 | + """Get the IPython log directory. |
| 186 | + |
| 187 | + If the log directory does not exist, it is created. |
| 188 | + """ |
| 189 | + log_dir = os.path.join(get_ipython_dir(), 'log') |
| 190 | + if not os.path.isdir(log_dir): |
| 191 | + os.mkdir(log_dir, 0777) |
| 192 | + return log_dir |
| 193 | + |
| 194 | #**************************************************************************** |
| 195 | # strings and text |
| 196 | |
| 197 | |
| 198 | === modified file 'IPython/kernel/__init__.py' |
| 199 | --- IPython/kernel/__init__.py 2008-06-08 22:25:02 +0000 |
| 200 | +++ IPython/kernel/__init__.py 2009-04-18 23:24:22 +0000 |
| 201 | @@ -15,10 +15,9 @@ |
| 202 | deferred/Twisted using clients. |
| 203 | """ |
| 204 | __docformat__ = "restructuredtext en" |
| 205 | -#------------------------------------------------------------------------------- |
| 206 | +#----------------------------------------------------------------------------- |
| 207 | # Copyright (C) 2008 The IPython Development Team |
| 208 | # |
| 209 | # Distributed under the terms of the BSD License. The full license is in |
| 210 | # the file COPYING, distributed as part of this software. |
| 211 | -#------------------------------------------------------------------------------- |
| 212 | - |
| 213 | \ No newline at end of file |
| 214 | +#----------------------------------------------------------------------------- |
| 215 | \ No newline at end of file |
| 216 | |
| 217 | === modified file 'IPython/kernel/config/__init__.py' |
| 218 | --- IPython/kernel/config/__init__.py 2008-11-14 23:27:13 +0000 |
| 219 | +++ IPython/kernel/config/__init__.py 2009-04-18 23:24:22 +0000 |
| 220 | @@ -15,6 +15,7 @@ |
| 221 | # Imports |
| 222 | #------------------------------------------------------------------------------- |
| 223 | |
| 224 | +import os, sys |
| 225 | from os.path import join as pjoin |
| 226 | |
| 227 | from IPython.external.configobj import ConfigObj |
| 228 | @@ -23,6 +24,7 @@ |
| 229 | |
| 230 | default_kernel_config = ConfigObj() |
| 231 | |
| 232 | +# This will raise OSError if ipythondir doesn't exist. |
| 233 | security_dir = get_security_dir() |
| 234 | |
| 235 | #------------------------------------------------------------------------------- |
| 236 | |
| 237 | === modified file 'IPython/kernel/core/interpreter.py' |
| 238 | --- IPython/kernel/core/interpreter.py 2008-09-06 02:26:28 +0000 |
| 239 | +++ IPython/kernel/core/interpreter.py 2009-04-19 00:13:57 +0000 |
| 240 | @@ -679,21 +679,22 @@ |
| 241 | # to exec will fail however. There seems to be some inconsistency in |
| 242 | # how trailing whitespace is handled, but this seems to work. |
| 243 | python = python.strip() |
| 244 | - |
| 245 | + |
| 246 | # The compiler module does not like unicode. We need to convert |
| 247 | # it encode it: |
| 248 | if isinstance(python, unicode): |
| 249 | # Use the utf-8-sig BOM so the compiler detects this a UTF-8 |
| 250 | # encode string. |
| 251 | python = '\xef\xbb\xbf' + python.encode('utf-8') |
| 252 | - |
| 253 | + |
| 254 | # The compiler module will parse the code into an abstract syntax tree. |
| 255 | + # This has a bug with str("a\nb"), but not str("""a\nb""")!!! |
| 256 | ast = compiler.parse(python) |
| 257 | - |
| 258 | + |
| 259 | # Uncomment to help debug the ast tree |
| 260 | # for n in ast.node: |
| 261 | # print n.lineno,'->',n |
| 262 | - |
| 263 | + |
| 264 | # Each separate command is available by iterating over ast.node. The |
| 265 | # lineno attribute is the line number (1-indexed) beginning the commands |
| 266 | # suite. |
| 267 | @@ -703,20 +704,23 @@ |
| 268 | # We might eventually discover other cases where lineno is None and have |
| 269 | # to put in a more sophisticated test. |
| 270 | linenos = [x.lineno-1 for x in ast.node if x.lineno is not None] |
| 271 | - |
| 272 | + |
| 273 | # When we finally get the slices, we will need to slice all the way to |
| 274 | # the end even though we don't have a line number for it. Fortunately, |
| 275 | # None does the job nicely. |
| 276 | linenos.append(None) |
| 277 | + |
| 278 | + # This is causing problems with commands that have a \n embedded in |
| 279 | + # a string, such as str("""a\nb""") |
| 280 | lines = python.splitlines() |
| 281 | - |
| 282 | + |
| 283 | # Create a list of atomic commands. |
| 284 | cmds = [] |
| 285 | for i, j in zip(linenos[:-1], linenos[1:]): |
| 286 | cmd = lines[i:j] |
| 287 | if cmd: |
| 288 | cmds.append('\n'.join(cmd)+'\n') |
| 289 | - |
| 290 | + |
| 291 | return cmds |
| 292 | |
| 293 | def error(self, text): |
| 294 | |
| 295 | === modified file 'IPython/kernel/core/tests/test_interpreter.py' |
| 296 | --- IPython/kernel/core/tests/test_interpreter.py 2008-09-06 03:26:50 +0000 |
| 297 | +++ IPython/kernel/core/tests/test_interpreter.py 2009-04-19 00:13:57 +0000 |
| 298 | @@ -15,12 +15,22 @@ |
| 299 | # Imports |
| 300 | #----------------------------------------------------------------------------- |
| 301 | |
| 302 | +import unittest |
| 303 | from IPython.kernel.core.interpreter import Interpreter |
| 304 | |
| 305 | -def test_unicode(): |
| 306 | - """ Test unicode handling with the interpreter. |
| 307 | - """ |
| 308 | - i = Interpreter() |
| 309 | - i.execute_python(u'print "ù"') |
| 310 | - i.execute_python('print "ù"') |
| 311 | +class TestInterpreter(unittest.TestCase): |
| 312 | + |
| 313 | + def test_unicode(self): |
| 314 | + """ Test unicode handling with the interpreter. |
| 315 | + """ |
| 316 | + i = Interpreter() |
| 317 | + i.execute_python(u'print "ù"') |
| 318 | + i.execute_python('print "ù"') |
| 319 | + |
| 320 | + def test_ticket266993_1(self): |
| 321 | + i = Interpreter() |
| 322 | + i.execute('str("""a\nb""")') |
| 323 | |
| 324 | + def test_ticket266993_2(self): |
| 325 | + i = Interpreter() |
| 326 | + i.execute('str("a\nb")') |
| 327 | \ No newline at end of file |
| 328 | |
| 329 | === modified file 'IPython/kernel/core/ultraTB.py' |
| 330 | --- IPython/kernel/core/ultraTB.py 2009-03-11 06:53:19 +0000 |
| 331 | +++ IPython/kernel/core/ultraTB.py 2009-04-16 05:22:09 +0000 |
| 332 | @@ -487,10 +487,14 @@ |
| 333 | else: |
| 334 | list.append('%s\n' % str(stype)) |
| 335 | |
| 336 | - # vds:>> |
| 337 | - if have_filedata: |
| 338 | - __IPYTHON__.hooks.synchronize_with_editor(filename, lineno, 0) |
| 339 | - # vds:<< |
| 340 | + # This is being commented out for now as the __IPYTHON__ variable |
| 341 | + # referenced here is not resolved and causes massive test failures |
| 342 | + # and errors. B. Granger, 04/2009. |
| 343 | + # See https://bugs.launchpad.net/bugs/362137 |
| 344 | + # # vds:>> |
| 345 | + # if have_filedata: |
| 346 | + # __IPYTHON__.hooks.synchronize_with_editor(filename, lineno, 0) |
| 347 | + # # vds:<< |
| 348 | |
| 349 | return list |
| 350 | |
| 351 | @@ -804,13 +808,17 @@ |
| 352 | value = text_repr(getattr(evalue, name)) |
| 353 | exception.append('\n%s%s = %s' % (indent, name, value)) |
| 354 | |
| 355 | - # vds: >> |
| 356 | - if records: |
| 357 | - filepath, lnum = records[-1][1:3] |
| 358 | - #print "file:", str(file), "linenb", str(lnum) # dbg |
| 359 | - filepath = os.path.abspath(filepath) |
| 360 | - __IPYTHON__.hooks.synchronize_with_editor(filepath, lnum, 0) |
| 361 | - # vds: << |
| 362 | + # This is being commented out for now as the __IPYTHON__ variable |
| 363 | + # referenced here is not resolved and causes massive test failures |
| 364 | + # and errors. B. Granger, 04/2009. |
| 365 | + # See https://bugs.launchpad.net/bugs/362137 |
| 366 | + # # vds: >> |
| 367 | + # if records: |
| 368 | + # filepath, lnum = records[-1][1:3] |
| 369 | + # #print "file:", str(file), "linenb", str(lnum) # dbg |
| 370 | + # filepath = os.path.abspath(filepath) |
| 371 | + # __IPYTHON__.hooks.synchronize_with_editor(filepath, lnum, 0) |
| 372 | + # # vds: << |
| 373 | |
| 374 | # return all our info assembled as a single string |
| 375 | return '%s\n\n%s\n%s' % (head,'\n'.join(frames),''.join(exception[0]) ) |
| 376 | |
| 377 | === modified file 'IPython/kernel/engineconnector.py' |
| 378 | --- IPython/kernel/engineconnector.py 2008-11-14 23:27:13 +0000 |
| 379 | +++ IPython/kernel/engineconnector.py 2009-04-18 21:53:37 +0000 |
| 380 | @@ -67,10 +67,10 @@ |
| 381 | self.furl = find_furl(furl_or_file) |
| 382 | except ValueError: |
| 383 | return defer.fail(failure.Failure()) |
| 384 | - # return defer.fail(failure.Failure(ValueError('not a valid furl or furl file: %r' % furl_or_file))) |
| 385 | - d = self.tub.getReference(self.furl) |
| 386 | - d.addCallbacks(self._register, self._log_failure) |
| 387 | - return d |
| 388 | + else: |
| 389 | + d = self.tub.getReference(self.furl) |
| 390 | + d.addCallbacks(self._register, self._log_failure) |
| 391 | + return d |
| 392 | |
| 393 | def _log_failure(self, reason): |
| 394 | log.err('EngineConnector: engine registration failed:') |
| 395 | |
| 396 | === modified file 'IPython/kernel/error.py' |
| 397 | --- IPython/kernel/error.py 2008-06-06 19:41:55 +0000 |
| 398 | +++ IPython/kernel/error.py 2009-04-18 21:53:37 +0000 |
| 399 | @@ -104,6 +104,9 @@ |
| 400 | class SecurityError(KernelError): |
| 401 | pass |
| 402 | |
| 403 | +class FileTimeoutError(KernelError): |
| 404 | + pass |
| 405 | + |
| 406 | class CompositeError(KernelError): |
| 407 | def __init__(self, message, elist): |
| 408 | Exception.__init__(self, *(message, elist)) |
| 409 | |
| 410 | === modified file 'IPython/kernel/scripts/ipcluster.py' |
| 411 | --- IPython/kernel/scripts/ipcluster.py 2009-03-23 15:54:00 +0000 |
| 412 | +++ IPython/kernel/scripts/ipcluster.py 2009-04-18 23:24:22 +0000 |
| 413 | @@ -1,4 +1,4 @@ |
| 414 | -#!/usr/bin/env python |
| 415 | + #!/usr/bin/env python |
| 416 | # encoding: utf-8 |
| 417 | |
| 418 | """Start an IPython cluster = (controller + engines).""" |
| 419 | @@ -29,11 +29,25 @@ |
| 420 | |
| 421 | from IPython.external import argparse |
| 422 | from IPython.external import Itpl |
| 423 | -from IPython.genutils import get_ipython_dir, num_cpus |
| 424 | -from IPython.kernel.fcutil import have_crypto |
| 425 | -from IPython.kernel.error import SecurityError |
| 426 | -from IPython.kernel.fcutil import have_crypto |
| 427 | -from IPython.kernel.twistedutil import gatherBoth |
| 428 | +from IPython.genutils import get_ipython_dir, get_log_dir, get_security_dir |
| 429 | +from IPython.genutils import num_cpus |
| 430 | +from IPython.kernel.fcutil import have_crypto |
| 431 | + |
| 432 | +# Create various ipython directories if they don't exist. |
| 433 | +# This must be done before IPython.kernel.config is imported. |
| 434 | +from IPython.iplib import user_setup |
| 435 | +if os.name == 'posix': |
| 436 | + rc_suffix = '' |
| 437 | +else: |
| 438 | + rc_suffix = '.ini' |
| 439 | +user_setup(get_ipython_dir(), rc_suffix, mode='install', interactive=False) |
| 440 | +get_log_dir() |
| 441 | +get_security_dir() |
| 442 | + |
| 443 | +from IPython.kernel.config import config_manager as kernel_config_manager |
| 444 | +from IPython.kernel.error import SecurityError, FileTimeoutError |
| 445 | +from IPython.kernel.fcutil import have_crypto |
| 446 | +from IPython.kernel.twistedutil import gatherBoth, wait_for_file |
| 447 | from IPython.kernel.util import printer |
| 448 | |
| 449 | |
| 450 | @@ -184,8 +198,10 @@ |
| 451 | from IPython.kernel.scripts import ipcontroller |
| 452 | script_location = ipcontroller.__file__.replace('.pyc', '.py') |
| 453 | # The -u option here turns on unbuffered output, which is required |
| 454 | - # on Win32 to prevent wierd conflict and problems with Twisted |
| 455 | - args = [find_exe('python'), '-u', script_location] |
| 456 | + # on Win32 to prevent wierd conflict and problems with Twisted. |
| 457 | + # Also, use sys.executable to make sure we are picking up the |
| 458 | + # right python exe. |
| 459 | + args = [sys.executable, '-u', script_location] |
| 460 | else: |
| 461 | args = ['ipcontroller'] |
| 462 | self.extra_args = extra_args |
| 463 | @@ -204,8 +220,10 @@ |
| 464 | from IPython.kernel.scripts import ipengine |
| 465 | script_location = ipengine.__file__.replace('.pyc', '.py') |
| 466 | # The -u option here turns on unbuffered output, which is required |
| 467 | - # on Win32 to prevent wierd conflict and problems with Twisted |
| 468 | - args = [find_exe('python'), '-u', script_location] |
| 469 | + # on Win32 to prevent wierd conflict and problems with Twisted. |
| 470 | + # Also, use sys.executable to make sure we are picking up the |
| 471 | + # right python exe. |
| 472 | + args = [sys.executable, '-u', script_location] |
| 473 | else: |
| 474 | args = ['ipengine'] |
| 475 | self.extra_args = extra_args |
| 476 | @@ -465,6 +483,7 @@ |
| 477 | # The main functions should then just parse the command line arguments, create |
| 478 | # the appropriate class and call a 'start' method. |
| 479 | |
| 480 | + |
| 481 | def check_security(args, cont_args): |
| 482 | if (not args.x or not args.y) and not have_crypto: |
| 483 | log.err(""" |
| 484 | @@ -478,6 +497,7 @@ |
| 485 | cont_args.append('-y') |
| 486 | return True |
| 487 | |
| 488 | + |
| 489 | def check_reuse(args, cont_args): |
| 490 | if args.r: |
| 491 | cont_args.append('-r') |
| 492 | @@ -491,6 +511,23 @@ |
| 493 | cont_args.append('--engine-port=%i' % args.engine_port) |
| 494 | return True |
| 495 | |
| 496 | + |
| 497 | +def _err_and_stop(f): |
| 498 | + log.err(f) |
| 499 | + reactor.stop() |
| 500 | + |
| 501 | + |
| 502 | +def _delay_start(cont_pid, start_engines, furl_file, reuse): |
| 503 | + if not reuse: |
| 504 | + if os.path.isfile(furl_file): |
| 505 | + os.unlink(furl_file) |
| 506 | + log.msg('Waiting for controller to finish starting...') |
| 507 | + d = wait_for_file(furl_file, delay=0.2, max_tries=50) |
| 508 | + d.addCallback(lambda _: log.msg('Controller started')) |
| 509 | + d.addCallback(lambda _: start_engines(cont_pid)) |
| 510 | + return d |
| 511 | + |
| 512 | + |
| 513 | def main_local(args): |
| 514 | cont_args = [] |
| 515 | cont_args.append('--logfile=%s' % pjoin(args.logdir,'ipcontroller')) |
| 516 | @@ -520,13 +557,10 @@ |
| 517 | signal.signal(signal.SIGINT,shutdown) |
| 518 | d = eset.start(args.n) |
| 519 | return d |
| 520 | - def delay_start(cont_pid): |
| 521 | - # This is needed because the controller doesn't start listening |
| 522 | - # right when it starts and the controller needs to write |
| 523 | - # furl files for the engine to pick up |
| 524 | - reactor.callLater(1.0, start_engines, cont_pid) |
| 525 | - dstart.addCallback(delay_start) |
| 526 | - dstart.addErrback(lambda f: f.raiseException()) |
| 527 | + config = kernel_config_manager.get_config_obj() |
| 528 | + furl_file = config['controller']['engine_furl_file'] |
| 529 | + dstart.addCallback(_delay_start, start_engines, furl_file, args.r) |
| 530 | + dstart.addErrback(_err_and_stop) |
| 531 | |
| 532 | |
| 533 | def main_mpi(args): |
| 534 | @@ -562,13 +596,10 @@ |
| 535 | signal.signal(signal.SIGINT,shutdown) |
| 536 | d = eset.start() |
| 537 | return d |
| 538 | - def delay_start(cont_pid): |
| 539 | - # This is needed because the controller doesn't start listening |
| 540 | - # right when it starts and the controller needs to write |
| 541 | - # furl files for the engine to pick up |
| 542 | - reactor.callLater(1.0, start_engines, cont_pid) |
| 543 | - dstart.addCallback(delay_start) |
| 544 | - dstart.addErrback(lambda f: f.raiseException()) |
| 545 | + config = kernel_config_manager.get_config_obj() |
| 546 | + furl_file = config['controller']['engine_furl_file'] |
| 547 | + dstart.addCallback(_delay_start, start_engines, furl_file, args.r) |
| 548 | + dstart.addErrback(_err_and_stop) |
| 549 | |
| 550 | |
| 551 | def main_pbs(args): |
| 552 | @@ -595,8 +626,10 @@ |
| 553 | signal.signal(signal.SIGINT,shutdown) |
| 554 | d = pbs_set.start(args.n) |
| 555 | return d |
| 556 | - dstart.addCallback(start_engines) |
| 557 | - dstart.addErrback(lambda f: f.raiseException()) |
| 558 | + config = kernel_config_manager.get_config_obj() |
| 559 | + furl_file = config['controller']['engine_furl_file'] |
| 560 | + dstart.addCallback(_delay_start, start_engines, furl_file, args.r) |
| 561 | + dstart.addErrback(_err_and_stop) |
| 562 | |
| 563 | |
| 564 | def main_ssh(args): |
| 565 | @@ -637,12 +670,10 @@ |
| 566 | signal.signal(signal.SIGINT,shutdown) |
| 567 | d = ssh_set.start(clusterfile['send_furl']) |
| 568 | return d |
| 569 | - |
| 570 | - def delay_start(cont_pid): |
| 571 | - reactor.callLater(1.0, start_engines, cont_pid) |
| 572 | - |
| 573 | - dstart.addCallback(delay_start) |
| 574 | - dstart.addErrback(lambda f: f.raiseException()) |
| 575 | + config = kernel_config_manager.get_config_obj() |
| 576 | + furl_file = config['controller']['engine_furl_file'] |
| 577 | + dstart.addCallback(_delay_start, start_engines, furl_file, args.r) |
| 578 | + dstart.addErrback(_err_and_stop) |
| 579 | |
| 580 | |
| 581 | def get_args(): |
| 582 | |
| 583 | === modified file 'IPython/kernel/scripts/ipcontroller.py' |
| 584 | --- IPython/kernel/scripts/ipcontroller.py 2009-03-15 01:38:00 +0000 |
| 585 | +++ IPython/kernel/scripts/ipcontroller.py 2009-04-18 23:24:22 +0000 |
| 586 | @@ -22,6 +22,7 @@ |
| 587 | sys.path.insert(0, '') |
| 588 | |
| 589 | import sys, time, os |
| 590 | +import tempfile |
| 591 | from optparse import OptionParser |
| 592 | |
| 593 | from twisted.application import internet, service |
| 594 | @@ -37,6 +38,18 @@ |
| 595 | from IPython.kernel import controllerservice |
| 596 | from IPython.kernel.fcutil import check_furl_file_security |
| 597 | |
| 598 | +# Create various ipython directories if they don't exist. |
| 599 | +# This must be done before IPython.kernel.config is imported. |
| 600 | +from IPython.iplib import user_setup |
| 601 | +from IPython.genutils import get_ipython_dir, get_log_dir, get_security_dir |
| 602 | +if os.name == 'posix': |
| 603 | + rc_suffix = '' |
| 604 | +else: |
| 605 | + rc_suffix = '.ini' |
| 606 | +user_setup(get_ipython_dir(), rc_suffix, mode='install', interactive=False) |
| 607 | +get_log_dir() |
| 608 | +get_security_dir() |
| 609 | + |
| 610 | from IPython.kernel.config import config_manager as kernel_config_manager |
| 611 | from IPython.config.cutils import import_item |
| 612 | |
| 613 | @@ -45,6 +58,10 @@ |
| 614 | # Code |
| 615 | #------------------------------------------------------------------------------- |
| 616 | |
| 617 | +def get_temp_furlfile(filename): |
| 618 | + return tempfile.mktemp(dir=os.path.dirname(filename), |
| 619 | + prefix=os.path.basename(filename)) |
| 620 | + |
| 621 | def make_tub(ip, port, secure, cert_file): |
| 622 | """ |
| 623 | Create a listening tub given an ip, port, and cert_file location. |
| 624 | @@ -107,13 +124,18 @@ |
| 625 | """Set the location for the tub and return a deferred.""" |
| 626 | |
| 627 | def register(empty, ref, furl_file): |
| 628 | - client_tub.registerReference(ref, furlFile=furl_file) |
| 629 | + # We create and then move to make sure that when the file |
| 630 | + # appears to other processes, the buffer has the flushed |
| 631 | + # and the file has been closed |
| 632 | + temp_furl_file = get_temp_furlfile(furl_file) |
| 633 | + client_tub.registerReference(ref, furlFile=temp_furl_file) |
| 634 | + os.rename(temp_furl_file, furl_file) |
| 635 | |
| 636 | if location == '': |
| 637 | d = client_tub.setLocationAutomatically() |
| 638 | else: |
| 639 | d = defer.maybeDeferred(client_tub.setLocation, "%s:%i" % (location, client_listener.getPortnum())) |
| 640 | - |
| 641 | + |
| 642 | for ciname, ci in config['controller']['controller_interfaces'].iteritems(): |
| 643 | log.msg("Adapting Controller to interface: %s" % ciname) |
| 644 | furl_file = ci['furl_file'] |
| 645 | @@ -154,7 +176,12 @@ |
| 646 | """Set the location for the tub and return a deferred.""" |
| 647 | |
| 648 | def register(empty, ref, furl_file): |
| 649 | - engine_tub.registerReference(ref, furlFile=furl_file) |
| 650 | + # We create and then move to make sure that when the file |
| 651 | + # appears to other processes, the buffer has the flushed |
| 652 | + # and the file has been closed |
| 653 | + temp_furl_file = get_temp_furlfile(furl_file) |
| 654 | + engine_tub.registerReference(ref, furlFile=temp_furl_file) |
| 655 | + os.rename(temp_furl_file, furl_file) |
| 656 | |
| 657 | if location == '': |
| 658 | d = engine_tub.setLocationAutomatically() |
| 659 | @@ -325,12 +352,6 @@ |
| 660 | help="log file name (default is stdout)" |
| 661 | ) |
| 662 | parser.add_option( |
| 663 | - "--ipythondir", |
| 664 | - type="string", |
| 665 | - dest="ipythondir", |
| 666 | - help="look for config files and profiles in this directory" |
| 667 | - ) |
| 668 | - parser.add_option( |
| 669 | "-r", |
| 670 | action="store_true", |
| 671 | dest="reuse_furls", |
| 672 | @@ -339,7 +360,6 @@ |
| 673 | |
| 674 | (options, args) = parser.parse_args() |
| 675 | |
| 676 | - kernel_config_manager.update_config_obj_from_default_file(options.ipythondir) |
| 677 | config = kernel_config_manager.get_config_obj() |
| 678 | |
| 679 | # Update with command line options |
| 680 | |
| 681 | === modified file 'IPython/kernel/scripts/ipengine.py' |
| 682 | --- IPython/kernel/scripts/ipengine.py 2009-03-15 01:38:00 +0000 |
| 683 | +++ IPython/kernel/scripts/ipengine.py 2009-04-18 23:24:22 +0000 |
| 684 | @@ -33,6 +33,19 @@ |
| 685 | from IPython.kernel.core.config import config_manager as core_config_manager |
| 686 | from IPython.config.cutils import import_item |
| 687 | from IPython.kernel.engineservice import EngineService |
| 688 | + |
| 689 | +# Create various ipython directories if they don't exist. |
| 690 | +# This must be done before IPython.kernel.config is imported. |
| 691 | +from IPython.iplib import user_setup |
| 692 | +from IPython.genutils import get_ipython_dir, get_log_dir, get_security_dir |
| 693 | +if os.name == 'posix': |
| 694 | + rc_suffix = '' |
| 695 | +else: |
| 696 | + rc_suffix = '.ini' |
| 697 | +user_setup(get_ipython_dir(), rc_suffix, mode='install', interactive=False) |
| 698 | +get_log_dir() |
| 699 | +get_security_dir() |
| 700 | + |
| 701 | from IPython.kernel.config import config_manager as kernel_config_manager |
| 702 | from IPython.kernel.engineconnector import EngineConnector |
| 703 | |
| 704 | @@ -106,13 +119,19 @@ |
| 705 | engine_connector = EngineConnector(tub_service) |
| 706 | furl_file = kernel_config['engine']['furl_file'] |
| 707 | log.msg("Using furl file: %s" % furl_file) |
| 708 | - d = engine_connector.connect_to_controller(engine_service, furl_file) |
| 709 | - def handle_error(f): |
| 710 | - log.err(f) |
| 711 | - if reactor.running: |
| 712 | - reactor.stop() |
| 713 | - d.addErrback(handle_error) |
| 714 | - |
| 715 | + |
| 716 | + def call_connect(engine_service, furl_file): |
| 717 | + d = engine_connector.connect_to_controller(engine_service, furl_file) |
| 718 | + def handle_error(f): |
| 719 | + # If this print statement is replaced by a log.err(f) I get |
| 720 | + # an unhandled error, which makes no sense. I shouldn't have |
| 721 | + # to use a print statement here. My only thought is that |
| 722 | + # at the beginning of the process the logging is still starting up |
| 723 | + print "error connecting to controller:", f.getErrorMessage() |
| 724 | + reactor.callLater(0.1, reactor.stop) |
| 725 | + d.addErrback(handle_error) |
| 726 | + |
| 727 | + reactor.callWhenRunning(call_connect, engine_service, furl_file) |
| 728 | reactor.run() |
| 729 | |
| 730 | |
| 731 | @@ -142,18 +161,9 @@ |
| 732 | dest="logfile", |
| 733 | help="log file name (default is stdout)" |
| 734 | ) |
| 735 | - parser.add_option( |
| 736 | - "--ipythondir", |
| 737 | - type="string", |
| 738 | - dest="ipythondir", |
| 739 | - help="look for config files and profiles in this directory" |
| 740 | - ) |
| 741 | |
| 742 | (options, args) = parser.parse_args() |
| 743 | |
| 744 | - kernel_config_manager.update_config_obj_from_default_file(options.ipythondir) |
| 745 | - core_config_manager.update_config_obj_from_default_file(options.ipythondir) |
| 746 | - |
| 747 | kernel_config = kernel_config_manager.get_config_obj() |
| 748 | # Now override with command line options |
| 749 | if options.furl_file is not None: |
| 750 | |
| 751 | === modified file 'IPython/kernel/tests/test_multiengine.py' |
| 752 | --- IPython/kernel/tests/test_multiengine.py 2008-08-23 23:06:36 +0000 |
| 753 | +++ IPython/kernel/tests/test_multiengine.py 2009-04-18 21:53:37 +0000 |
| 754 | @@ -4,16 +4,16 @@ |
| 755 | |
| 756 | __docformat__ = "restructuredtext en" |
| 757 | |
| 758 | -#------------------------------------------------------------------------------- |
| 759 | +#----------------------------------------------------------------------------- |
| 760 | # Copyright (C) 2008 The IPython Development Team |
| 761 | # |
| 762 | # Distributed under the terms of the BSD License. The full license is in |
| 763 | # the file COPYING, distributed as part of this software. |
| 764 | -#------------------------------------------------------------------------------- |
| 765 | +#----------------------------------------------------------------------------- |
| 766 | |
| 767 | -#------------------------------------------------------------------------------- |
| 768 | +#----------------------------------------------------------------------------- |
| 769 | # Imports |
| 770 | -#------------------------------------------------------------------------------- |
| 771 | +#----------------------------------------------------------------------------- |
| 772 | |
| 773 | try: |
| 774 | from twisted.internet import defer |
| 775 | |
| 776 | === modified file 'IPython/kernel/tests/test_newserialized.py' |
| 777 | --- IPython/kernel/tests/test_newserialized.py 2008-08-23 23:06:36 +0000 |
| 778 | +++ IPython/kernel/tests/test_newserialized.py 2009-04-18 21:53:37 +0000 |
| 779 | @@ -4,16 +4,16 @@ |
| 780 | |
| 781 | __docformat__ = "restructuredtext en" |
| 782 | |
| 783 | -#------------------------------------------------------------------------------- |
| 784 | +#----------------------------------------------------------------------------- |
| 785 | # Copyright (C) 2008 The IPython Development Team |
| 786 | # |
| 787 | # Distributed under the terms of the BSD License. The full license is in |
| 788 | # the file COPYING, distributed as part of this software. |
| 789 | -#------------------------------------------------------------------------------- |
| 790 | +#----------------------------------------------------------------------------- |
| 791 | |
| 792 | -#------------------------------------------------------------------------------- |
| 793 | +#----------------------------------------------------------------------------- |
| 794 | # Imports |
| 795 | -#------------------------------------------------------------------------------- |
| 796 | +#----------------------------------------------------------------------------- |
| 797 | |
| 798 | try: |
| 799 | import zope.interface as zi |
| 800 | @@ -31,9 +31,9 @@ |
| 801 | import nose |
| 802 | raise nose.SkipTest("This test requires zope.interface, Twisted and Foolscap") |
| 803 | |
| 804 | -#------------------------------------------------------------------------------- |
| 805 | +#----------------------------------------------------------------------------- |
| 806 | # Tests |
| 807 | -#------------------------------------------------------------------------------- |
| 808 | +#----------------------------------------------------------------------------- |
| 809 | |
| 810 | class SerializedTestCase(unittest.TestCase): |
| 811 | |
| 812 | |
| 813 | === modified file 'IPython/kernel/twistedutil.py' |
| 814 | --- IPython/kernel/twistedutil.py 2008-06-06 19:41:55 +0000 |
| 815 | +++ IPython/kernel/twistedutil.py 2009-04-18 21:53:37 +0000 |
| 816 | @@ -16,12 +16,15 @@ |
| 817 | # Imports |
| 818 | #------------------------------------------------------------------------------- |
| 819 | |
| 820 | +import os, sys |
| 821 | import threading, Queue, atexit |
| 822 | + |
| 823 | import twisted |
| 824 | - |
| 825 | from twisted.internet import defer, reactor |
| 826 | from twisted.python import log, failure |
| 827 | |
| 828 | +from IPython.kernel.error import FileTimeoutError |
| 829 | + |
| 830 | #------------------------------------------------------------------------------- |
| 831 | # Classes related to twisted and threads |
| 832 | #------------------------------------------------------------------------------- |
| 833 | @@ -204,3 +207,43 @@ |
| 834 | result = None |
| 835 | |
| 836 | return result |
| 837 | + |
| 838 | + |
| 839 | +def wait_for_file(filename, delay=0.1, max_tries=10): |
| 840 | + """Wait (poll) for a file to be created. |
| 841 | + |
| 842 | + This method returns a Deferred that will fire when a file exists. It |
| 843 | + works by polling os.path.isfile in time intervals specified by the |
| 844 | + delay argument. If `max_tries` is reached, it will errback with a |
| 845 | + `FileTimeoutError`. |
| 846 | + |
| 847 | + Parameters |
| 848 | + ---------- |
| 849 | + filename : str |
| 850 | + The name of the file to wait for. |
| 851 | + delay : float |
| 852 | + The time to wait between polls. |
| 853 | + max_tries : int |
| 854 | + The max number of attempts before raising `FileTimeoutError` |
| 855 | + |
| 856 | + Returns |
| 857 | + ------- |
| 858 | + d : Deferred |
| 859 | + A Deferred instance that will fire when the file exists. |
| 860 | + """ |
| 861 | + |
| 862 | + d = defer.Deferred() |
| 863 | + |
| 864 | + def _test_for_file(filename, attempt=0): |
| 865 | + if attempt >= max_tries: |
| 866 | + d.errback(FileTimeoutError( |
| 867 | + 'timeout waiting for file to be created: %s' % filename |
| 868 | + )) |
| 869 | + else: |
| 870 | + if os.path.isfile(filename): |
| 871 | + d.callback(True) |
| 872 | + else: |
| 873 | + reactor.callLater(delay, _test_for_file, filename, attempt+1) |
| 874 | + |
| 875 | + _test_for_file(filename) |
| 876 | + return d |
| 877 | |
| 878 | === modified file 'IPython/tests/test_genutils.py' |
| 879 | --- IPython/tests/test_genutils.py 2009-03-11 17:44:53 +0000 |
| 880 | +++ IPython/tests/test_genutils.py 2009-04-18 23:24:22 +0000 |
| 881 | @@ -210,7 +210,6 @@ |
| 882 | home_dir = genutils.get_home_dir() |
| 883 | nt.assert_equal(home_dir, abspath(HOME_TEST_DIR)) |
| 884 | |
| 885 | - |
| 886 | # |
| 887 | # Tests for get_ipython_dir |
| 888 | # |
| 889 | @@ -239,7 +238,6 @@ |
| 890 | ipdir = genutils.get_ipython_dir() |
| 891 | nt.assert_equal(ipdir, os.path.abspath(os.path.join("someplace", "_ipython"))) |
| 892 | |
| 893 | - |
| 894 | # |
| 895 | # Tests for get_security_dir |
| 896 | # |
| 897 | @@ -249,6 +247,14 @@ |
| 898 | """Testcase to see if we can call get_security_dir without Exceptions.""" |
| 899 | sdir = genutils.get_security_dir() |
| 900 | |
| 901 | +# |
| 902 | +# Tests for get_log_dir |
| 903 | +# |
| 904 | + |
| 905 | +@with_enivronment |
| 906 | +def test_get_log_dir(): |
| 907 | + """Testcase to see if we can call get_log_dir without Exceptions.""" |
| 908 | + sdir = genutils.get_log_dir() |
| 909 | |
| 910 | # |
| 911 | # Tests for popkey |
| 912 | |
| 913 | === modified file 'docs/source/parallel/parallel_process.txt' |
| 914 | --- docs/source/parallel/parallel_process.txt 2009-03-18 18:45:20 +0000 |
| 915 | +++ docs/source/parallel/parallel_process.txt 2009-04-18 21:53:37 +0000 |
| 916 | @@ -302,18 +302,33 @@ |
| 917 | Make FURL files persistent |
| 918 | --------------------------- |
| 919 | |
| 920 | -At fist glance it may seem that that managing the FURL files is a bit annoying. Going back to the house and key analogy, copying the FURL around each time you start the controller is like having to make a new key every time you want to unlock the door and enter your house. As with your house, you want to be able to create the key (or FURL file) once, and then simply use it at any point in the future. |
| 921 | - |
| 922 | -This is possible. The only thing you have to do is decide what ports the controller will listen on for the engines and clients. This is done as follows:: |
| 923 | +At fist glance it may seem that that managing the FURL files is a bit |
| 924 | +annoying. Going back to the house and key analogy, copying the FURL around |
| 925 | +each time you start the controller is like having to make a new key every time |
| 926 | +you want to unlock the door and enter your house. As with your house, you want |
| 927 | +to be able to create the key (or FURL file) once, and then simply use it at |
| 928 | +any point in the future. |
| 929 | + |
| 930 | +This is possible. but before you do this, you **must** remove any old FURL |
| 931 | +files in the :file:`~/.ipython/security` directory. |
| 932 | + |
| 933 | +.. warning:: |
| 934 | + |
| 935 | + You **must** remove old FURL files before using persistent FURL files. |
| 936 | + |
| 937 | +Then, The only thing you have to do is decide what ports the controller will |
| 938 | +listen on for the engines and clients. This is done as follows:: |
| 939 | |
| 940 | $ ipcontroller -r --client-port=10101 --engine-port=10102 |
| 941 | |
| 942 | -These options also work with all of the various modes of |
| 943 | +These options also work with all of the various modes of |
| 944 | :command:`ipcluster`:: |
| 945 | |
| 946 | $ ipcluster local -n 2 -r --client-port=10101 --engine-port=10102 |
| 947 | |
| 948 | -Then, just copy the furl files over the first time and you are set. You can start and stop the controller and engines any many times as you want in the future, just make sure to tell the controller to use the *same* ports. |
| 949 | +Then, just copy the furl files over the first time and you are set. You can |
| 950 | +start and stop the controller and engines any many times as you want in the |
| 951 | +future, just make sure to tell the controller to use the *same* ports. |
| 952 | |
| 953 | .. note:: |
| 954 |
This has a number of bug fixes that are targeted for the 0.10 release. Many of the bug fixes are trivial, but a few are quite subtle and need to be carefully reviewed.