Merge lp:~gz/pyjuju/openstack_provider into lp:pyjuju

[Martin Packman]

OpenStack Provider

Implementation of a provider using native OpenStack apis, and using swift for file storage.
There is still some important features to complete here, but the core is done and the
remaining gaps documented.

Also included is an openstack_s3 provider fudge that uses the Nova api, with the S3 api
for storage. This allows for running against deployments using nova-objectstore rather
than swift, which currently includes canonistack.

The basics are modelled on the EC2 provider implementation, with the addition of a client
module that handles the details of the OpenStack api in the manner of txaws. Along the way
I also refactored the security_groups module into helper class for port management.

Also tested against hpcloud. rackspace public cloud support for a future branch.

https://codereview.appspot.com/6312050/

[Kapil Thangavelu (hazmat)]

Hi martin, would you mind resubmitting this with the lbox tool that's in the juju ppa, its integrates with reitveld for inline code reviews. it should just be a matter of installing it and running lbox propose within the branch.

[Kapil Thangavelu (hazmat)]

Oh.. and btw this is awesome! :-)

[Martin Packman (gz)]

Please take a look.

[Martin Packman (gz)]

Reviewers: mp+110860_code.launchpad.net,

Message:
Please take a look.

Description:
Implementation of a provider using native OpenStack apis, and using
swift for file storage. There is still some important features to
complete here, but the core is done and the remaining gaps documented.

Also included is an openstack_s3 provider fudge that uses the Nova api,
with the S3 api for storage. This allows for running against deployments
using nova-objectstore rather than swift, which currently includes
canonistack.

The basics are modelled on the EC2 provider implementation, with the
addition of a client module that handles the details of the OpenStack
api in the manner of txaws. Along the way I also refactored the
security_groups module into helper class for port management.

I've made an effort to document all the tricky bits as clearly as
possible, to guard against important details getting lost in
translation. See those docstrings for a deeper description of the
implementation.

Apologies for the size of the delta, all feedback welcome.

https://code.launchpad.net/~gz/juju/openstack_provider/+merge/110860

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/6312050/

Affected files:
   A [revision details]
   M juju/environment/config.py
   M juju/errors.py
   M juju/providers/common/findzookeepers.py
   M juju/providers/ec2/tests/test_launch.py
   A juju/providers/openstack/__init__.py
   A juju/providers/openstack/client.py
   A juju/providers/openstack/files.py
   A juju/providers/openstack/launch.py
   A juju/providers/openstack/machine.py
   A juju/providers/openstack/ports.py
   A juju/providers/openstack/provider.py
   A juju/providers/openstack/tests/__init__.py
   A juju/providers/openstack/tests/test_bootstrap.py
   A juju/providers/openstack/tests/test_client.py
   A juju/providers/openstack/tests/test_files.py
   A juju/providers/openstack/tests/test_getmachines.py
   A juju/providers/openstack/tests/test_machine.py
   A juju/providers/openstack/tests/test_ports.py
   A juju/providers/openstack/tests/test_provider.py
   A juju/providers/openstack/tests/test_shutdown.py
   A juju/providers/openstack/tests/test_state.py
   A juju/providers/openstack_s3/__init__.py
   A juju/providers/openstack_s3/tests/__init__.py
   A juju/providers/openstack_s3/tests/test_provider.py
   M juju/unit/address.py
   M juju/unit/tests/test_address.py

[Kapil Thangavelu (hazmat)]

Hi Martin,

this is a big branch so i'm going to break up the review into multiple
passes. Overall it looks good, but i have a few questions about some of
the behavior ic wrt to waiting for post-boot network setup and md server
availability (bootstrap node instance id access) that i'd like to
discuss with you and document.

https://codereview.appspot.com/6312050/diff/1/juju/environment/config.py
File juju/environment/config.py (right):

https://codereview.appspot.com/6312050/diff/1/juju/environment/config.py#newcode57
juju/environment/config.py:57: "default-instance-type": String(),
given constraint support default-instance-type shouldn't be needed as a
config option.

https://codereview.appspot.com/6312050/diff/1/juju/errors.py
File juju/errors.py (right):

https://codereview.appspot.com/6312050/diff/1/juju/errors.py#newcode162
juju/errors.py:162: ", ".join(map(str, self.instance_ids)))
looks fine, but curious why needed, instance ids should always be
integers.

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/client.py
File juju/providers/openstack/client.py (right):

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/client.py#newcode158
juju/providers/openstack/client.py:158: response, body = yield deferred
style minors, deferred transient isn't needed.

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/client.py#newcode166
juju/providers/openstack/client.py:166: [self.token] =
response.headers.getRawHeaders("X-Auth-Token")
haven't seen this style before, preferable to just index 0 off the
result.

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/client.py#newcode295
juju/providers/openstack/client.py:295: flavor_id = yield
self._lookup_flavor(flavor_name)
please document the kw params supported (sec groups and ipv4).

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/client.py#newcode309
juju/providers/openstack/client.py:309: server["accessIPv4"] = ip
what's this for?

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/files.py
File juju/providers/openstack/files.py (right):

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/files.py#newcode78
juju/providers/openstack/files.py:78: if response.code != 201:
its likely the original error is more interesting at this point.

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/launch.py
File juju/providers/openstack/launch.py (right):

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/launch.py#newcode61
juju/providers/openstack/launch.py:61: if self._master:
is the md service not available?

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/launch.py#newcode89
juju/providers/openstack/launch.py:89: yield filestorage.put(id_name,
StringIO(server['id']))
have you run into this in practice?, it seems hard to achieve, but good
to note nonetheless.

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/launch.py#newcode99
juju/providers/openstack/launch.py:99: reactor.callLater(5,
deferred.callback, None)
ouch!.. you have to wait for the servers to boot to attach a public...

[Martin Packman (gz)]

Thanks for the comments! Have replied to some below, will act on the
others.

https://codereview.appspot.com/6312050/diff/1/juju/errors.py
File juju/errors.py (right):

https://codereview.appspot.com/6312050/diff/1/juju/errors.py#newcode162
juju/errors.py:162: ", ".join(map(str, self.instance_ids)))
Instance ids are normally UUIDs and stored as strings, but Nova
currently also allows the integer form of the ec2 id. This shouldn't
come up in production, but is exercised in tests as writing out UUIDs is
annoying.

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/client.py
File juju/providers/openstack/client.py (right):

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/client.py#newcode309
juju/providers/openstack/client.py:309: server["accessIPv4"] = ip
Serves no purpose now, was added when I was experimenting with ways to
solve the problem of juju needing a public ip.

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/files.py
File juju/providers/openstack/files.py (right):

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/files.py#newcode78
juju/providers/openstack/files.py:78: if response.code != 201:
Right, the swift backend should probably raise exceptions then this
block could reraise the original one.

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/launch.py
File juju/providers/openstack/launch.py (right):

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/launch.py#newcode61
juju/providers/openstack/launch.py:61: if self._master:
The metadata service only exposes the EC2 form of the server id, but the
rest of the code deals with the UUID form. Sticking the UUID in file
storage stops juju getting confused with two different ids for the same
server. Trick should no longer be required with the Folsom release, as
the metadata service will be adding some OpenStack values.

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/launch.py#newcode89
juju/providers/openstack/launch.py:89: yield filestorage.put(id_name,
StringIO(server['id']))
Not had any problems in testing, the conditions should heavily favour
the correct outcome. Sticking a small string in the object store should
beat the server boot process even giving it a head start.

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/launch.py#newcode99
juju/providers/openstack/launch.py:99: reactor.callLater(5,
deferred.callback, None)
Doesn't need to finish booting, but doesn't work till the network setup
has been done, which is not synchronous on creating the server.

This is one the reasons I'd really like the public ip assignment to move
to the port handling code, and juju to not expect a public address for
all units.

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/provider.py
File juju/providers/openstack/provider.py (right):

https://codereview.appspot.com/6312050/diff/1/juju/providers/openstack/provider.py#newcode198
juju/providers/openstack/provider.py:198: return defer.gatherResults(
I saw that helper but it wasn't obvious what it was adding. I think the
correct behaviour here requires usi...

[Clint Byrum (clint-fewbar)]

Just wanted to add some testing results..

1) openstack_s3 works perfectly w/ canonistack. Need to provide a map in documentation of Env Var -> environments.yaml values, as they are not clear.

2) I was able to coax the branch with some tweaks to work with hpcloud (which, I believe, is a very old OpenStack:

lp:~clint-fewbar/juju/openstack_provider_fixes

Note that there's a stupid amount of debug logging that should be removed.

Even w/ that branch, one still must manually delete everything on destroy-environment, and for some reason machine 0 doesn't get hostname details.

Anyway, nice work, I'm sure there is all kinds of improvement to it, but in general it works GREAT. I'd love to see this in juju AS-IS.

[Martin Packman (gz)]

> Just wanted to add some testing results..

Thanks for doing this Clint!

> 1) openstack_s3 works perfectly w/ canonistack. Need to provide a map in
> documentation of Env Var -> environments.yaml values, as they are not clear.

Yes. Is there a good place to document these? With some recent changes, most should be taken from the local environment rather than needing to be copied into the juju yaml file at least.

> 2) I was able to coax the branch with some tweaks to work with hpcloud (which,
> I believe, is a very old OpenStack:
>
> lp:~clint-fewbar/juju/openstack_provider_fixes

This use useful, these are areas I knew the code was weak but hadn't seen it breaking. A couple of queries to help me improve the coverage:

* What exactly went wrong when wrongly setting Accept as json all the time? Swift I expect?
* What is the configured OS_REGION vs the one returned from keystone in the serviceCatalog?

> Note that there's a stupid amount of debug logging that should be removed.

Looking at where you've added that logging is actually useful. Currently the client logs requests and responses, but only after the authentication step to avoid leaking passwords. More targetted debug logging there, given the number of times auth has needed tweaking, would be a good addition.

> Even w/ that branch, one still must manually delete everything on destroy-
> environment, and for some reason machine 0 doesn't get hostname details.

There are some failing tests for the shutdown code, hopefully those cover the issue here but I'll make a note to check when they're passing and if there's more to be done.

[Martin Packman (gz)]

Please take a look.

https://codereview.appspot.com/6312050/

[Kapil Thangavelu (hazmat)]

review round 2

https://codereview.appspot.com/6312050/diff/9001/juju/environment/config.py
File juju/environment/config.py (right):

https://codereview.appspot.com/6312050/diff/9001/juju/environment/config.py#newcode26
juju/environment/config.py:26: _OPENSTACK_AUTH_MODE = OneOf(
This could use a doc string, at least describing the correspondence to
ostack rels and what it entails wrt to the ostack setup, ie. is this a
user choice or a provider configuration matter. Eventually we'll start
dropping support for old ostack releases, and it will be good to
understand what bits correspond to what so we can keep this tidy.

EDIT: a ref to credentials.py for details would work

https://codereview.appspot.com/6312050/diff/9001/juju/providers/common/findzookeepers.py
File juju/providers/common/findzookeepers.py (right):

https://codereview.appspot.com/6312050/diff/9001/juju/providers/common/findzookeepers.py#newcode40
juju/providers/common/findzookeepers.py:40: % missing_instance_ids)
There's a broken test with this change not addressed by the branch. Its
not clear if this is adding value, rather than just leaking impl
details. i assume this is to help distinguish the int/str form of
openstack id provider.

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/client.py
File juju/providers/openstack/client.py (right):

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/client.py#newcode115
juju/providers/openstack/client.py:115: body = yield
reader.onConnectionLost
please don't name instance attributes the same as framework methods,
thats very confusing, it looks like your yielding on a method instead of
its invocation here.

its good to keep in mind and perhaps note here, if connectionLost method
callstack should return a deferred twisted will not wait on it before
proceeding and processing other things.

this method is also lacking any tests.

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/client.py#newcode131
juju/providers/openstack/client.py:131: def make_url(self, service,
parts):
minor. make_url should be prolly be private, since it requires
self.services which is not initialized in the constructor, so requires
internal knowledge to use properly in the correct initialization order.

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/credentials.py
File juju/providers/openstack/credentials.py (right):

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/credentials.py#newcode8
juju/providers/openstack/credentials.py:8: TODO: rename 'nova-uri' to
'auth-url'
re TODO b/c?

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/files.py
File juju/providers/openstack/files.py (right):

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/files.py#newcode41
juju/providers/openstack/files.py:41: # XXX: what exactly is the
expectation here?
already documented in the module doc string.

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/launch.py
File juju/providers/openstack/launch.py (right):

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/launch.py#...

[Martin Packman (gz)]

Thanks Kapil, is really useful input.

https://codereview.appspot.com/6312050/diff/9001/juju/environment/config.py
File juju/environment/config.py (right):

https://codereview.appspot.com/6312050/diff/9001/juju/environment/config.py#newcode26
juju/environment/config.py:26: _OPENSTACK_AUTH_MODE = OneOf(
On 2012/07/03 17:25:44, hazmat wrote:
> EDIT: a ref to credentials.py for details would work

Will add, most users should be able to ignore this setting and have the
right thing autodetected.

https://codereview.appspot.com/6312050/diff/9001/juju/providers/common/findzookeepers.py
File juju/providers/common/findzookeepers.py (right):

https://codereview.appspot.com/6312050/diff/9001/juju/providers/common/findzookeepers.py#newcode40
juju/providers/common/findzookeepers.py:40: % missing_instance_ids)
On 2012/07/03 17:25:44, hazmat wrote:
> There's a broken test with this change not addressed by the branch.
Its not
> clear if this is adding value, rather than just leaking impl details.
i assume
> this is to help distinguish the int/str form of openstack id provider.

Have changed this back and forth as I saw that test fail before then
forgot about it and needed to fix int ids. Will resolve in some sensible
manner.

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/client.py
File juju/providers/openstack/client.py (right):

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/client.py#newcode115
juju/providers/openstack/client.py:115: body = yield
reader.onConnectionLost
Right, still some rearrangements needed here for sanity.

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/client.py#newcode131
juju/providers/openstack/client.py:131: def make_url(self, service,
parts):
On 2012/07/03 17:25:44, hazmat wrote:
> minor. make_url should be prolly be private, since it requires
self.services
> which is not initialized in the constructor, so requires internal
knowledge to
> use properly in the correct initialization order.

Sounds sensible. _SwiftClient/public_object_url needs it but that's a
somewhat special case.

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/credentials.py
File juju/providers/openstack/credentials.py (right):

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/credentials.py#newcode8
juju/providers/openstack/credentials.py:8: TODO: rename 'nova-uri' to
'auth-url'
On 2012/07/03 17:25:44, hazmat wrote:
> re TODO b/c?

Because 'nova-uri' is wrong, in a modern openstack deployment it's a
keystone identity service url and used to get credentials for swift as
well. There aren't legacy compat issues in juju so we may as well use a
less misleading name.

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/files.py
File juju/providers/openstack/files.py (right):

https://codereview.appspot.com/6312050/diff/9001/juju/providers/openstack/files.py#newcode41
juju/providers/openstack/files.py:41: # XXX: what exactly is the
expectation here?
On 2012/07/03 17:25:44, hazmat wrote:
> already documented in the module doc string.

Right, will remove this.

https://codereview.appspot.com/6312050/diff/9001/juju/pr...

[Martin Packman (gz)]

Please take a look.

https://codereview.appspot.com/6312050/

[lp:~gz/pyjuju/openstack_provider]

537. By Martin Packman on 2012-07-11

Tweaks to test_launch

538. By Martin Packman on 2012-07-11

Rename _NovaClient method post_empty to clearer post_no_data

539. By Martin Packman on 2012-07-12

Make default-series required for openstack providers too

[Kapil Thangavelu (hazmat)]

i've put together lp:~hazmat/juju/openstack_provider which fixes a few issues i found in testing on hpcloud and canonistack. Floating ip usage is off by default, as public clouds provide them by default, and speeds things up a bit. Internal access/private net secgroup was missing. Also includes the hp tweak branch with compatibility/preference for essex style. Shutdown is more robust and simpler now with yanking the sec groups manipulation, trunk already supports group reuse, which obsoletes maintenance to a large extent (quotas not withstanding, but unused cleanup can be a jitsu thing). hpcloud's int instance ids are better supported (really only the bootstrap node was an issue). also includes rackspace auth support, i've left off on going much further with that, as its going to require a large refactoring of the client api interaction to be capabilities driven via service catalogs and extensions discovery.

anyways, with that branch and trunk merged and tests passing, i'm good with this. it works flawlessly with hpcloud and canonistack to a lesser extent due to some incredible floating ip behavior vagaries inspite of correct state. the rspace stuff can wait for the future against trunk.

thanks for persevering with this martin, it rocks!

review: Approve

[lp:~gz/pyjuju/openstack_provider]

540. By Martin Packman on 2012-07-18

Merge work in progress changes by hazmat

541. By Martin Packman on 2012-07-18

Fix tests after changes to shutdown and security groups

542. By Martin Packman on 2012-07-18

Teeny cleanups of merged changes

543. By Martin Packman on 2012-07-18

Revert change to test_findzookeepers now the old form is used again

=== modified file 'juju/agents/provision.py'
--- juju/agents/provision.py	2012-03-20 10:13:22 +0000
+++ juju/agents/provision.py	2012-07-18 19:47:20 +0000
@@ -195,6 +195,7 @@
         for instance_id in unused:
             log.info("Shutting down machine id:%s ...", instance_id)
             machine = provider_machines[instance_id]
+
             try:
                 yield self.provider.shutdown_machine(machine)
             except ProviderError:

=== modified file 'juju/control/status.py'
--- juju/control/status.py	2012-05-18 21:41:15 +0000
+++ juju/control/status.py	2012-07-18 19:47:20 +0000
@@ -543,7 +543,7 @@
                         m["agent-state"] = "not-started"
             except ProviderError:
                 # The provider doesn't have machine information
-                self.log.error(
+                self.log.exception(
                     "Machine provider information missing: machine %s" % (
                         machine_state.id))
 

=== modified file 'juju/environment/config.py'
--- juju/environment/config.py	2012-04-11 01:17:53 +0000
+++ juju/environment/config.py	2012-07-18 19:47:20 +0000
@@ -6,8 +6,8 @@
 from juju.environment.errors import EnvironmentsConfigError
 from juju.errors import FileAlreadyExists, FileNotFound
 from juju.lib.schema import (
-    Constant, Dict, KeyDict, OAuthString, OneOf, SchemaError, SelectDict,
-    String)
+    Constant, Dict, Int, KeyDict, OAuthString, OneOf, SchemaError, SelectDict,
+    String, Bool)
 
 DEFAULT_CONFIG_PATH = "~/.juju/environments.yaml"
 
@@ -23,6 +23,14 @@
 
 _EITHER_PLACEMENT = OneOf(Constant("unassigned"), Constant("local"))
 
+# See juju.providers.openstack.credentials for definition and more details
+_OPENSTACK_AUTH_MODE = OneOf(
+    Constant("userpass"),
+    Constant("keypair"),
+    Constant("legacy"),
+    Constant("rax"),
+    )
+
 SCHEMA = KeyDict({
     "default": String(),
     "environments": Dict(String(), SelectDict("type", {
@@ -49,6 +57,44 @@
             optional=[
                 "access-key", "secret-key", "region", "ec2-uri", "s3-uri",
                 "placement", "ssl-hostname-verification"]),
+        "openstack": KeyDict({
+            "control-bucket": String(),
+            "admin-secret": String(),
+            "access-key": String(),
+            "secret-key": String(),
+            "default-instance-type": String(),
+            "default-image-id": OneOf(String(), Int()),
+            "auth-url": String(),
+            "project-name": String(),
+            "use-floating-ip": Bool(),
+            "placement": _EITHER_PLACEMENT,
+            "auth-mode": _OPENSTACK_AUTH_MODE,
+            "region": String(),
+            "default-series": String(),
+            },
+            optional=[
+                "access-key", "secret-key", "auth-url", "project-name",
+                "placement", "auth-mode", "region", "use-floating-ip"]),
+        "openstack_s3": KeyDict({
+            "control-bucket": String(),
+            "admin-secret": String(),
+            "access-key": String(),
+            "secret-key": String(),
+            "default-instance-type": String(),
+            "default-image-id": OneOf(String(), Int()),
+            "auth-url": String(),
+            "placement": _EITHER_PLACEMENT,
+            "combined-key": String(),
+            "s3-uri": String(),
+            "use-floating-ip": Bool(),
+            "auth-mode": _OPENSTACK_AUTH_MODE,
+            "region": String(),
+            "default-series": String(),
+            },
+            optional=[
+                "access-key", "secret-key", "combined-key", "auth-url",
+                "s3-uri", "project-name", "placement", "auth-mode", "region",
+                "use-floating-ip"]),
         "orchestra": KeyDict({
             "orchestra-server": String(),
             "orchestra-user": String(),

=== modified file 'juju/errors.py'
--- juju/errors.py	2012-03-28 07:33:22 +0000
+++ juju/errors.py	2012-07-18 19:47:20 +0000
@@ -159,7 +159,7 @@
     def __str__(self):
         return "Cannot find machine%s: %s" % (
             "" if len(self.instance_ids) == 1 else "s",
-            ", ".join(self.instance_ids))
+            ", ".join(map(str, self.instance_ids)))
 
 
 class ProviderInteractionError(ProviderError):

=== modified file 'juju/providers/common/findzookeepers.py'
--- juju/providers/common/findzookeepers.py	2011-09-15 18:50:23 +0000
+++ juju/providers/common/findzookeepers.py	2012-07-18 19:47:20 +0000
@@ -36,5 +36,6 @@
 
     if machines:
         returnValue(machines)
+
     raise EnvironmentNotFound("machines are not running (%s)"
-                              % ", ".join(missing_instance_ids))
+                             % ", ".join(map(str, missing_instance_ids)))

=== modified file 'juju/providers/ec2/tests/test_launch.py'
--- juju/providers/ec2/tests/test_launch.py	2012-07-05 21:49:12 +0000
+++ juju/providers/ec2/tests/test_launch.py	2012-07-18 19:47:20 +0000
@@ -67,7 +67,7 @@
     def test_provider_launch(self):
         """
         The provider can be used to launch a machine with a minimal set of
-        required packages, repositories, and and security groups.
+        required packages, repositories, and security groups.
         """
         self.ec2.describe_security_groups()
         self.mocker.result(succeed([]))

=== added directory 'juju/providers/openstack'
=== added file 'juju/providers/openstack/__init__.py'
--- juju/providers/openstack/__init__.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/__init__.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,3 @@
+"""Support for using OpenStack as a cloud provider for juju"""
+
+from .provider import MachineProvider

=== added file 'juju/providers/openstack/client.py'
--- juju/providers/openstack/client.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/client.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,470 @@
+"""Client for talking to OpenStack APIs using twisted
+
+This is not a complete implemention of all interfaces, just what juju needs.
+
+There is a fair bit of code cleanup and feature implementation to do here
+still.
+
+* Must check https certificates, can use code in txaws to do this.
+* Must support user/password authentication with keystone as well as keypair.
+* Want a ProviderInteractionError subclass that can include the extra details
+  returned in json form when something goes wrong and is raised by clients.
+* Request flow and json handling in general needs polish.
+* Need to prevent concurrent authentication attempts.
+* Need to limit concurrent http api requests to 4 or something reasonable,
+  can use DeferredSemaphore for this.
+* Should really have authentication retry logic in case the token expires.
+* Would be nice to use Agent keep alive support that twisted 12.1.0 added.
+"""
+
+import base64
+import json
+import logging
+import operator
+import urllib
+
+import twisted
+from twisted.internet import (
+    defer,
+    interfaces,
+    protocol,
+    reactor,
+    )
+from twisted.web import (
+    client,
+    http_headers,
+    )
+from zope.interface import implements
+
+from juju import errors
+
+
+log = logging.getLogger("juju.openstack")
+
+
+# Need the right juju version number, not exposed within python package.
+_USER_AGENT = "juju/%s twisted/%s" % ("0.5", twisted.__version__)
+
+
+class BytestringProducer(object):
+    """Wrap basic bytestring as a needlessly fancy twisted producer."""
+
+    implements(interfaces.IProducer)
+
+    def __init__(self, bytestring):
+        self.content = bytestring
+        self.length = len(bytestring)
+
+    def pauseProducing(self):
+        """Nothing to do if production is paused"""
+
+    def startProducing(self, consumer):
+        """Write entire contents when production starts"""
+        consumer.write(self.content)
+        return defer.succeed(None)
+
+    def stopProducing(self):
+        """Nothing to do when production halts"""
+
+
+class ResponseReader(protocol.Protocol):
+    """Protocol object suitable for use with Response.deliverBody
+
+    The 'onConnectionLost' deferred will be called back once the connection
+    is shut down with all the bytes from the body collected at that point.
+    """
+
+    def __init__(self):
+        self.onConnectionLost = defer.Deferred()
+
+    def connectionMade(self):
+        self.data = []
+
+    def dataReceived(self, data):
+        self.data.append(data)
+
+    def connectionLost(self, reason):
+        """Called on connection shut down
+
+        Here 'reason' can be one of ResponseDone, PotentialDataLost, or
+        ResponseFailed, but currently there is no fancy handling of these.
+        """
+        self.onConnectionLost.callback("".join(self.data))
+
+
[e-mail]
+def request(method, url, extra_headers=(), body=None):
+    headers = http_headers.Headers({
+        # GZ 2012-07-03: Previously passed Accept: application/json header
+        #                here, but not always the right thing. Bad for swift?
+        "User-Agent": [_USER_AGENT],
+        })
+    for header, value in extra_headers:
+        headers.setRawHeaders(header, [value])
+    if body is not None:
+        if isinstance(body, dict):
+            content_type = "application/json"
+            body = json.dumps(body)
+        elif isinstance(body, str):
+            content_type = "application/octet-stream"
+        headers.setRawHeaders("Content-Type", [content_type])
+        body = BytestringProducer(body)
+    response = yield client.Agent(reactor).request(method, url, headers, body)
+    if response.length == 0:
+        defer.returnValue((response, ""))
+    reader = ResponseReader()
+    response.deliverBody(reader)
+    body = yield reader.onConnectionLost
+    defer.returnValue((response, body))
+
+
+class _OpenStackClient(object):
+
+    def __init__(self, credentials):
+        self.credentials = credentials
+        log.debug("openstack: using auth-mode %r with %s", credentials.mode,
+            credentials.url)
+        if credentials.mode == "keypair":
+            self.authenticate = self.authenticate_v2_keypair
+        elif credentials.mode == "legacy":
+            self.authenticate = self.authenticate_v1
+        elif credentials.mode == "rax":
+            self.authenticate = self.authenticate_rax_auth
+        else:
+            self.authenticate = self.authenticate_v2_userpass
+        self.token = None
+
+    def _make_url(self, service, parts):
+        """Form full url from path components to service endpoint url"""
+        # GZ 2012-07-03: Need to ensure either services is populated or catch
+        #                error here and propogate as one useful for users.
+        endpoint = self.services[service]
+        if not endpoint[-1] == "/":
+            endpoint += "/"
+        if isinstance(parts, str):
+            return endpoint + parts
+        quoted_parts = []
+        for part in parts:
+            if not isinstance(part, str):
+                part = urllib.quote(unicode(part).encode("utf-8"), "/~")
+            quoted_parts.append(part)
+        url = endpoint + "/".join(quoted_parts)
+        log.debug('access %s @ %s', service, url)
+        return url
+
+    @defer.inlineCallbacks
+    def authenticate_v1(self):
+        deferred = request(
+            "GET",
+            self.credentials.url,
+            extra_headers=[
+                ("X-Auth-User", self.credentials.username),
+                ("X-Auth-Key", self.credentials.access_key),
+                ],
+            )
+        response, body = yield deferred
+        if response.code != 204:
+            raise errors.ProviderInteractionError("Failed to authenticate")
+        # TODO: check response has right headers
+        [nova_url] = response.headers.getRawHeaders("X-Server-Management-Url")
+        self.services = {"compute": self.nova_url}
+        # No swift_url set as that is not supported
+        [self.token] = response.headers.getRawHeaders("X-Auth-Token")
+
+    def authenticate_v2_keypair(self):
+        deferred = request(
+            "POST",
+            self.credentials.url + "tokens",
+            body={"auth": {
+                  "apiAccessKeyCredentials": {
+                    "accessKey": self.credentials.access_key,
+                    "secretKey": self.credentials.secret_key,
+                  },
+                  "tenantName": self.credentials.project_name,
+                }}
+            )
+        return deferred.addCallback(self._handle_v2_auth)
+
+    def authenticate_v2_userpass(self):
+        deferred = request(
+            "POST",
+            self.credentials.url + "tokens",
+            body={"auth": {
+                  "passwordCredentials": {
+                    "username": self.credentials.username,
+                    "password": self.credentials.password,
+                  },
+                  "tenantName": self.credentials.project_name,
+                }}
+            )
+        return deferred.addCallback(self._handle_v2_auth)
+
+    def authenticate_rax_auth(self):
+        # openstack is not a product, but a kit for making snowflakes.
+        deferred = request(
+            "POST",
+            self.credentials.url + "tokens",
+            body={"auth": {
+                  "RAX-KSKEY:apiKeyCredentials": {
+                    "username": self.credentials.username,
+                    "apiKey": self.credentials.password,
+                    "tenantName": self.credentials.project_name}}}
+            )
+        return deferred.addCallback(self._handle_v2_auth)
+
+    def _handle_v2_auth(self, result):
+        access_details = self._json(result, 200, 'access')
+        token_details = access_details["token"]
+        self.token = token_details["id"]
+
+        # TODO: care about token_details["expires"]
+        # Don't need to we're not preserving tokens.
+        services = []
+        log.debug("openstack: authenticated til %r", token_details['expires'])
+        region = self.credentials.region
+        # HP cloud uses both az-1.region-a.geo-1 and region-a.geo-1 forms, not
+        # clear what should be in config or what the correct logic is.
+        if region is not None:
+            base_region = region.split('.', 1)[-1]
+        # GZ: 2012-07-03: Should split extraction of endpoints, add logging,
+        #                 and make more robust.
+        for catalog in access_details["serviceCatalog"]:
+            for endpoint in catalog["endpoints"]:
+                if region is not None and region != endpoint["region"]:
+                    if base_region != endpoint["region"]:
+                        continue
+                services.append((catalog["type"], str(endpoint["publicURL"])))
+                break
+
+        if not services:
+            raise errors.ProviderInteractionError("No suitable endpoints")
+
+        self.services = dict(services)
+
+    def is_authenticated(self):
+        return self.token is not None
+
+    @defer.inlineCallbacks
+    def authed_request(self, method, url, headers=None, body=None):
+        log.debug("openstack: %s %r", method, url)
+        request_headers = [("X-Auth-Token", self.token)]
+        if headers:
+            request_headers += headers
+        response, body = yield request(method, url, request_headers, body)
+        log.debug("openstack: %d %r", response.code, body)
+        defer.returnValue((response, body))
+
+    def _empty(self, result, code):
+        response, body = result
+        if response.code != code:
+            # XXX: This is a deeply unhelpful error, need context from request
+            raise errors.ProviderInteractionError("Unexpected %d: %r" % (
+                response.code, body))
+
+    def _json(self, result, code, root=None):
+        response, body = result
+        if response.code != code:
+            raise errors.ProviderInteractionError("Unexpected %d: %r" % (
+                response.code, body))
+        type_headers = response.headers.getRawHeaders("Content-Type")
+
+        found = False
+        for h in type_headers:
+            if 'application/json' in h:
+                found = True
+        if not found:
+            raise errors.ProviderInteractionError(
+                "Expected json response got %s" % type_headers)
+
+        data = json.loads(body)
+        if root is not None:
+            return data[root]
+        return data
+
+
+class _NovaClient(object):
+
+    def __init__(self, client):
+        self._client = client
+
+    @defer.inlineCallbacks
+    def request(self, method, parts, headers=None, body=None):
+        if not self._client.is_authenticated():
+            yield self._client.authenticate()
+        url = self._client._make_url("compute", parts)
+        result = yield self._client.authed_request(method, url, headers, body)
+        defer.returnValue(result)
+
+    def delete(self, parts, code=202):
+        deferred = self.request("DELETE", parts)
+        return deferred.addCallback(self._client._empty, code)
+
+    def get(self, parts, root, code=200):
+        deferred = self.request("GET", parts)
+        return deferred.addCallback(self._client._json, code, root)
+
+    def post(self, parts, jsonobj, root, code=200):
+        deferred = self.request("POST", parts, None, jsonobj)  # XXX
+        return deferred.addCallback(self._client._json, code, root)
+
+    def post_no_data(self, parts, root, code=200):
+        deferred = self.request("POST", parts, None, "")  # XXX
+        return deferred.addCallback(self._client._json, code, root)
+
+    def post_no_result(self, parts, jsonobj, code=202):
+        deferred = self.request("POST", parts, None, jsonobj)  # XXX
+        return deferred.addCallback(self._client._empty, code)
+
+    def list_flavors(self):
+        return self.get("flavors", "flavors")
+
+    def get_server(self, server_id):
+        return self.get(["servers", server_id], "server")
+
+    # GZ 2012-05-31: Appending detail isn't a known path for some reason.
+    def get_server_detail(self, server_id):
+        return self.get(["servers", server_id, "detail"], "server")
+
+    def list_servers(self):
+        return self.get(["servers"], "servers")
+
+    def list_servers_detail(self):
+        return self.get(["servers", "detail"], "servers")
+
+    def delete_server(self, server_id):
+        return self.delete(["servers", server_id], code=204)
+
+    def run_server(self, image_id, flavor_id, name, security_group_names=None,
+        user_data=None):
+        server = {
+            'name': name,
+            'flavorRef': flavor_id,
+            'imageRef': image_id,
+        }
+        if user_data is not None:
+            server["user_data"] = base64.b64encode(user_data)
+        if security_group_names is not None:
+            server["security_groups"] = [{'name': n}
+                for n in security_group_names]
+        return self.post(["servers"], {'server': server},
+            root="server", code=202)
+
+    def get_server_security_groups(self, server_id):
+        d = self.get(
+            ["servers", server_id, "os-security-groups"],
+            root="security_groups")
+        # 2012-07-12: Workaround lack of this api in HP cloud
+        d.addErrback(
+            lambda f: self.get_server(server_id).addCallback(
+                operator.itemgetter("security_groups")))
+        return d
+
+    def list_security_groups(self):
+        return self.get(["os-security-groups"], "security_groups")
+
+    def create_security_group(self, name, description):
+        return self.post("os-security-groups", {
+            'security_group': {
+                'name': name,
+                'description': description,
+                }
+            },
+            root="security_group")
+
+    def delete_security_group(self, group_id):
+        return self.delete(["os-security-groups", group_id])
+
+    def add_security_group_rule(self, parent_group_id, **kwargs):
+        rule = {'parent_group_id': parent_group_id}
+        using_group = "group_id" in kwargs
+        if using_group:
+            rule['group_id'] = kwargs['group_id']
+        elif "cidr" in kwargs:
+            rule['cidr'] = kwargs['cidr']
+        if not using_group or "ip_protocol" in kwargs:
+            rule['ip_protocol'] = kwargs['ip_protocol']
+            rule['from_port'] = kwargs['from_port']
+            rule['to_port'] = kwargs['to_port']
+        return self.post("os-security-group-rules",
+            {'security_group_rule': rule},
+            root="security_group_rule")
+
+    def delete_security_group_rule(self, rule_id):
+        return self.delete(["os-security-group-rules", rule_id])
+
+    def add_server_security_group(self, server_id, group_name):
+        return self.post_no_result(["servers", server_id, "action"], {
+            "addSecurityGroup": {
+                "name": group_name,
+            }})
+
+    def remove_server_security_group(self, server_id, group_name):
+        return self.post_no_result(["servers", server_id, "action"], {
+            "removeSecurityGroup": {
+                "name": group_name,
+            }})
+
+    def list_floating_ips(self):
+        return self.get(["os-floating-ips"], "floating_ips")
+
+    def get_floating_ip(self, ip_id):
+        return self.get(["os-floating-ips", ip_id], "floating_ip")
+
+    def allocate_floating_ip(self):
+        return self.post_no_data(["os-floating-ips"], "floating_ip")
+
+    def delete_floating_ip(self, ip_id):
+        return self.delete(["os-floating-ips", ip_id])
+
+    def add_floating_ip(self, server_id, addr):
+        return self.post_no_result(["servers", server_id, "action"], {
+            'addFloatingIp': {
+                'address': addr,
+            }})
+
+    def remove_floating_ip(self, server_id, addr):
+        return self.post_no_result(["servers", server_id, "action"], {
+            'removeFloatingIp': {
+                'address': addr,
+            }})
+
+
+class _SwiftClient(object):
+
+    def __init__(self, client):
+        self._client = client
+
+    @defer.inlineCallbacks
+    def request(self, method, parts, headers=None, body=None):
+        if not self._client.is_authenticated():
+            yield self._client.authenticate()
+        url = self._client._make_url("object-store", parts)
+        result = yield self._client.authed_request(method, url, headers, body)
+        defer.returnValue(result)
+
+    def public_object_url(self, container, object_name):
+        if not self._client.is_authenticated():
+            raise ValueError("Need to have authenticated to get object url")
+        return self._client._make_url("object-store", [container, object_name])
+
+    def put_container(self, container_name):
+        # Juju expects there to be a (semi) public url for some objects. This
+        # could probably be more restrictive or placed in a seperate container
+        # with some refactoring, but for now just make everything public.
+        read_acl_header = ("X-Container-Read", ".r:*")
+        return self.request("PUT", [container_name], [read_acl_header], "")
+
+    def delete_container(self, container_name):
+        return self.request("DELETE", [container_name])
+
+    def head_object(self, container, object_name):
+        return self.request("HEAD", [container, object_name])
+
+    def get_object(self, container, object_name):
+        return self.request("GET", [container, object_name])
+
+    def delete_object(self, container, object_name):
+        return self.request("DELETE", [container, object_name])
+
+    def put_object(self, container, object_name, bytestring):
+        return self.request("PUT", [container, object_name], None, bytestring)

=== added file 'juju/providers/openstack/credentials.py'
--- juju/providers/openstack/credentials.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/credentials.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,101 @@
+"""Handling of the credentials needed to authenticate with the OpenStack api
+
+Supports several different sets of credentials different auth modes need:
+* 'legacy' is built into nova and deprecated in favour of using keystone
+* 'keypair' works with the HP public cloud implemention of keystone
+* 'userpass' is the way keystone seems to want to do authentication generally
+"""
+
+import os
+
+
+class OpenStackCredentials(object):
+    """Encapsulation of credentials used to authenticate with OpenStack"""
+
+    _config_vars = {
+            'auth-url': ("OS_AUTH_URL", "NOVA_URL"),
+            'username': ("OS_USERNAME", "NOVA_USERNAME"),
+            'password': ("OS_PASSWORD", "NOVA_PASSWORD"),
+            # HP exposes both a numeric id and a name for tenants, passed back
+            # as tenantId and tenantName. Use the name only for simplicity.
+            'project-name': ("OS_TENANT_NAME", "NOVA_PROJECT_NAME",
+                             "NOVA_PROJECT_ID"),
+            'region': ("OS_REGION_NAME", "NOVA_REGION_NAME", "NOVA_REGION"),
+            # The key variables don't seem to have modern OS_ prefixed aliases
+            'access-key': ("NOVA_API_KEY",),
+            'secret-key': ("EC2_SECRET_KEY", "AWS_SECRET_ACCESS_KEY"),
+            # A usable mode can normally be guessed, but may be configured
+            'auth-mode': (),
+        }
+
+    # Really, legacy auth could pass in the project id and keystone doesn't
+    # require it, but this is what the client expects for now.
+    _modes = {
+        'userpass': ('username', 'password', 'project-name'),
+        'rax': ('username', 'password', 'project-name'),
+        'keypair': ('access-key', 'secret-key', 'project-name'),
+        'legacy': ('username', 'access-key'),
+        }
+
+    _version_to_mode = {
+        "v2.0": 'userpass',
+        "v1.1": 'legacy',
+        "v1.0": 'legacy',
+        }
+
+    def __init__(self, creds_dict):
+        url = creds_dict.get("auth-url")
+        if not url:
+            raise ValueError("Missing config 'auth-url' for OpenStack api")
+        mode = creds_dict.get("auth-mode")
+        if mode is None:
+            mode = self._guess_auth_mode(url)
+        elif mode not in self._modes:
+            # The juju.environment.config layer should raise a pretty error
+            raise ValueError("Unknown 'auth-mode' value %r" % (self.mode,))
+        missing_keys = [key for key in self._modes[mode]
+            if not creds_dict.get(key)]
+        if missing_keys:
+            raise ValueError("Missing config %s required for %s auth" % (
+                ", ".join(map(repr, missing_keys)), mode))
+        self.url = url
+        self.mode = mode
+        for key in self._config_vars:
+            if key not in ("auth-url", "auth-mode"):
+                setattr(self, key.replace("-", "_"), creds_dict.get(key))
+
+    @classmethod
+    def _guess_auth_mode(cls, url):
+        """Pick a mode based on the version at the end of `url` given"""
+        final_part = url.rstrip("/").rsplit("/", 1)[-1]
+        try:
+            return cls._version_to_mode[final_part]
+        except KeyError:
+            raise ValueError(
+                "Missing config 'auth-mode' as unknown version"
+                " in 'auth-url' given: " + url)
+
+    @classmethod
+    def _get(cls, config, key):
+        """Retrieve `key` from `config` if present or in matching envvars"""
+        val = config.get(key)
+        if val is None:
+            for env_key in cls._config_vars[key]:
+
+                val = os.environ.get(env_key)
+                if val:
+                    return val
+        return val
+
+    @classmethod
+    def from_environment(cls, config):
+        """Create credentials from `config` falling back to environment"""
+        return cls(dict((k, cls._get(config, k)) for k in cls._config_vars))
+
+    def set_config_defaults(self, data):
+        """Populate `data` with these credentials where not already set"""
+        for key in self._config_vars:
+            if key not in data:
+                val = getattr(self, key.replace("auth-", "").replace("-", "_"))
+                if val is not None:
+                    data[key] = val

=== added file 'juju/providers/openstack/files.py'
--- juju/providers/openstack/files.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/files.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,84 @@
+"""OpenStack provider file storage on Swift
+
+Basically a limited wrapper around the underlying api calls, with a few added
+quirks. There's some specific handling for 404 responses, raises FileNotFound
+on GET, and on PUT attempts to create the container then retries.
+
+Expects file-like objects for data. This isn't terribly useful as it doesn't
+fit well with the twisted model for chunking http data and most objects are
+small anyway.
+
+The main complication is the get_url method, which requires the generation of
+link that can be used by any http client to fetch a particular object without
+authentication. This is not possible in the general case in swift, however
+there are a few ways around the problem:
+
+* The stub nova-objectstore service doesn't check access anyway, see lp:947374
+* The tempurl swift middleware if enabled can do this with some advance setup
+* A container can have a more permissive ACL applying to all objects within
+
+All of these require touching the network to at least get the swift endpoint
+from the identity service, which as get_url doesn't return a deferred is
+problematic. In practice it's only used after putting an object however, so
+just raising if client has not yet been authenticated is good enough.
+"""
+
+from cStringIO import StringIO
+
+from twisted.internet import defer
+
+from juju import errors
+
+
+class FileStorage(object):
+    """Swift-backed :class:`FileStorage` abstraction"""
+
+    def __init__(self, swift, container):
+        self._swift = swift
+        self._container = container
+
+    def get_url(self, name):
+        return self._swift.public_object_url(self._container, name)
+
+    @defer.inlineCallbacks
+    def get(self, name):
+        """Get a file object from Swift.
+
+        :param unicode name: S3 key for the desired file
+
+        :return: an open file object
+        :rtype: :class:`twisted.internet.defer.Deferred`
+
+        :raises: :exc:`juju.errors.FileNotFound` if the file doesn't exist
+        """
+        response, body = yield self._swift.get_object(self._container, name)
+        if response.code == 404:
+            raise errors.FileNotFound(name)
+        if response.code != 200:
+            raise errors.ProviderInteractionError(
+                "Couldn't fetch object %r %r" % (response.code, body))
+        defer.returnValue(StringIO(body))
+
+    @defer.inlineCallbacks
+    def put(self, remote_path, file_object):
+        """Upload a file to Swift.
+
+        :param unicode remote_path: key on which to store the content
+
+        :param file_object: open file object containing the content
+
+        :rtype: :class:`twisted.internet.defer.Deferred`
+        """
+        data = file_object.read()
+        response, body = yield self._swift.put_object(self._container,
+            remote_path, data)
+        if response.code == 404:
+            response, body = yield self._swift.put_container(self._container)
+            if response.code != 201:
+                raise errors.ProviderInteractionError(
+                    "Couldn't create container %r" % (self._container,))
+            response, body = yield self._swift.put_object(self._container,
+                remote_path, data)
+        if response.code != 201:
+            raise errors.ProviderInteractionError(
+                "Couldn't create object %r %r" % (response.code, remote_path))

=== added file 'juju/providers/openstack/launch.py'
--- juju/providers/openstack/launch.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/launch.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,131 @@
+"""Helpers for creating servers catered to Juju needs with OpenStack
+
+Specific notes:
+* Expects a public address for each machine, as that's what EC2 promises,
+  would be good to weaken this requirement.
+* Creates a per-machine security group in case need to poke ports open later,
+  as EC2 doesn't support changing groups later, but OpenStack does.
+* Needs to tell cloud-init how to get server id, currently in essex metadata
+  service gives i-08x style only, so cheat and use filestorage.
+* Config must specify a flavor, as these vary between Openstack deployments.
+  Working out constraints should resolve this.
+* Config must specify an image id, as there's no standard way of looking up
+  from distro series across clouds yet in essex.
+* Would be really nice to put the service name in the server name, but it's
+  not passed down into LaunchMachine currently.
+
+There are some race issues with the current setup:
+* Storing of server id needs to complete before cloud-init does the lookup.
+* A floating ip may be assigned to another server before the current one
+  finishes launching and can use the available ip itself.
+"""
+
+from cStringIO import StringIO
+
+from twisted.internet import (
+    defer,
+    reactor,
+    )
+
+from juju.errors import ProviderError, ProviderInteractionError
+from juju.lib import twistutils
+from juju.providers.common.launch import LaunchMachine
+
+from .machine import machine_from_instance, get_server_status
+
+from .client import log
+
+
+class NovaLaunchMachine(LaunchMachine):
+    """OpenStack Nova operation for creating a server"""
+
+    _DELAY_FOR_ADDRESSES = 5  # seconds
+
+    @defer.inlineCallbacks
+    def start_machine(self, machine_id, zookeepers):
+        """Actually launch an instance on Nova.
+
+        :param str machine_id: the juju machine ID to assign
+
+        :param zookeepers: the machines currently running zookeeper, to which
+            the new machine will need to connect
+        :type zookeepers: list of
+            :class:`juju.providers.openstack.machine.NovaProviderMachine`
+
+        :return: a singe-entry list containing a
+            :class:`juju.providers.openstack.machine.NovaProviderMachine`
+            representing the newly-launched machine
+        :rtype: :class:`twisted.internet.defer.Deferred`
+        """
+        cloud_init = self._create_cloud_init(machine_id, zookeepers)
+        cloud_init.set_provider_type(self._provider.provider_type)
+        filestorage = self._provider.get_file_storage()
+        # Only the master is required to get its own instance id like this.
+        if self._master:
+            id_name = "juju_master_id"
+            cloud_init.set_instance_id_accessor("$(curl %s)" % (
+                filestorage.get_url(id_name),))
+        user_data = cloud_init.render()
+
+        # For openstack deployments, really need image id configured as there
+        # are no standards to provide a fallback value.
+        image_id = self._provider.config.get("default-image-id")
+        if image_id is None:
+            raise ProviderError("Need to specify a default-image-id")
+
+        security_groups = (
+            yield self._provider.port_manager.ensure_groups(machine_id))
+
+        # Until constraints are implemented, need to a configured instance
+        # type and resolve it to a flavor id here.
+        flavor_name = self._provider.config.get("default-instance-type")
+        if flavor_name is None:
+            raise ProviderError("Need to specify a default-instance-type")
+        flavors = yield self._provider.nova.list_flavors()
+        flavor_map = dict((f['name'], f['id']) for f in flavors)
+        if not flavor_name in flavor_map:
+            ProviderError("Unknown instance type given: %r" % (flavor_name,))
+
+        server = yield self._provider.nova.run_server(
+            name="juju %s instance %s" %
+                (self._provider.environment_name, machine_id,),
+            image_id=image_id,
+            flavor_id=flavor_map[flavor_name],
+            security_group_names=security_groups,
+            user_data=user_data)
+
+        if self._master:
+            yield filestorage.put(id_name, StringIO(str(server['id'])))
+
+        # For private clouds allow an option of attaching public
+        # floating ips to all the machines. None of the extant public
+        # clouds need this.
+        if self._provider.config.get('use-floating-ip'):
+            # Not possible to attach a floating ip to a newly booted
+            # server, must wait for networking to be ready when some
+            # kind of address exists.
+            while not server.get('addresses'):
+                status = get_server_status(server)
+                if status != "pending":
+                    raise ProviderInteractionError(
+                        "Server out of pending status "
+                        "without addresses set: %r" % server)
+                # Bad, bad place to be doing a wait loop directly
+                yield twistutils.sleep(self._DELAY_FOR_ADDRESSES)
+                log.debug("Waited for %d seconds for networking on server %r",
+                          self._DELAY_FOR_ADDRESSES, server['id'])
+                server = yield self._provider.nova.get_server(server['id'])
+            yield _assign_floating_ip(self._provider, server['id'])
+
+        defer.returnValue([machine_from_instance(server)])
+
+
[e-mail]
+def _assign_floating_ip(provider, server_id):
+    floating_ips = yield provider.nova.list_floating_ips()
+    for floating_ip in floating_ips:
+        if floating_ip['instance_id'] is None:
+            break
+    else:
+        floating_ip = yield provider.nova.allocate_floating_ip()
+    yield provider.nova.add_floating_ip(server_id, floating_ip['ip'])

=== added file 'juju/providers/openstack/machine.py'
--- juju/providers/openstack/machine.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/machine.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,68 @@
+"""Helpers for mapping Nova api results to the juju machine abstraction"""
+
+from juju.machine import ProviderMachine
+
+
+_SERVER_STATE_MAP = {
+    None: 'pending',
+    'ACTIVE': 'running',
+    'BUILD': 'pending',
+    'BUILDING': 'pending',
+    'REBUILDING': 'pending',
+    'DELETED': 'terminated',
+    'STOPPED': 'stopped',
+    }
+
+
+class NovaProviderMachine(ProviderMachine):
+    """Nova-specific ProviderMachine implementation"""
+
+
+def get_server_status(server):
+    status = server.get('status')
+    if status is not None and "(" in status:
+        status = status.split("(", 1)[0]
+    return _SERVER_STATE_MAP.get(status, 'unknown')
+
+
+def get_server_addresses(server):
+    private_addr = public_addr = None
+    addresses = server.get("addresses")
+    if addresses is not None:
+        # Issue with some setups, have custom network only, use as private
+        network = ()
+        for name in sorted(addresses):
+            if name not in ("private", "public"):
+                network = addresses[name]
+                if network:
+                    break
+        network = addresses.get("private", network)
+        for address in network:
+            if address.get("version", 0) == 4:
+                private_addr = address['addr']
+                break
+        # Issue with HP cloud, public address is second in private network
+        network = addresses.get("public", network[1:])
+        for address in network:
+            if address.get("version", 0) == 4:
+                public_addr = address['addr']
+    return private_addr, public_addr
+
+
+def machine_from_instance(server):
+    """Create an :class:`NovaProviderMachine` from a server details dict
+
+    :param server: a dictionary of server info as given by the Nova api
+
+    :return: a matching :class:`NovaProviderMachine`
+    """
+    private_addr, public_addr = get_server_addresses(server)
+    # Juju assumes it always needs a public address and loops waiting for one.
+    # In fact a private address is generally fine provided it can be sshed to.
+    if public_addr is None and private_addr is not None:
+        public_addr = private_addr
+    return NovaProviderMachine(
+        server['id'],
+        public_addr,
+        private_addr,
+        get_server_status(server))

=== added file 'juju/providers/openstack/ports.py'
--- juju/providers/openstack/ports.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/ports.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,179 @@
+"""Manage port access to machines using Nova security group extension
+
+The mechanism is based on the existing scheme used by the EC2 provider.
+
+Each machine is launched with two security groups, a juju group that is shared
+across all machines and allows access to 22/tcp for ssh, and a machine group
+just for that server so ports can be opened and closed on an individual level.
+
+There is some mismatch between the port hole poking and security group models:
+* A new security group is created for every machine
+* Rules are not shared between service units but set up again each launch
+* Support for port ranges is not exposed
+
+The Nova security group module follows the EC2 example quite closely, but as
+of Essex it's still under contrib and has a number of quirks:
+* To run a server with, or add or remove groups from a server, 'name' is used
+* To get details, delete, or add or remove rules from a group, 'id' is needed
+
+The only way of getting 'id' if 'name' is known is by listing all groups then
+looking at the details of the one with the matching name.
+"""
+
+from twisted.internet import (
+    defer,
+    )
+
+from juju import errors
+
+from .client import log
+
+
+class NovaPortManager(object):
+    """Mapping of port-based juju interface to Nova security group actions
+
+    There is the potential to record some state on the instance to reduce api
+    round-trips when, for instance, launching multiple machines at once, but
+    for now
+    """
+
+    def __init__(self, nova, environment_name):
+        self.nova = nova
+        self.tag = environment_name
+
+    def _juju_group_name(self):
+        return "juju-%s" % (self.tag,)
+
+    def _machine_group_name(self, machine_id):
+        return "juju-%s-%s" % (self.tag, machine_id)
+
+    @defer.inlineCallbacks
+    def _get_machine_group(self, machine, machine_id):
+        """Get details of the machine specific security group
+
+        As only the name of the group can be derived, this means listing every
+        security group for that server and seeing which has a matching name.
+        """
+        group_name = self._machine_group_name(machine_id)
+        server_id = machine.instance_id
+        groups = yield self.nova.get_server_security_groups(server_id)
+        for group in groups:
+            if group['name'] == group_name:
+                defer.returnValue(group)
+        raise errors.ProviderInteractionError(
+            "Missing security group %r for machine %r" %
+                (group_name, server_id))
+
+    @defer.inlineCallbacks
+    def open_port(self, machine, machine_id, port, protocol="tcp"):
+        """Allow access to a port for the given machine only"""
+        group = yield self._get_machine_group(machine, machine_id)
+        yield self.nova.add_security_group_rule(group['id'],
+            ip_protocol=protocol, from_port=port, to_port=port)
+        log.debug("Opened %s/%s on machine %r",
+            port, protocol, machine.instance_id)
+
+    @defer.inlineCallbacks
+    def close_port(self, machine, machine_id, port, protocol="tcp"):
+        """Revoke access to a port for the given machine only"""
+        group = yield self._get_machine_group(machine, machine_id)
+        for rule in group["rules"]:
+            if (port == rule["from_port"] == rule["to_port"] and
+                rule["ip_protocol"] == protocol):
+                yield self.nova.delete_security_group_rule(rule["id"])
+                log.debug("Closed %s/%s on machine %r",
+                          port, protocol, machine.instance_id)
+                return
+        raise errors.ProviderInteractionError(
+            "Couldn't close unopened %s/%s on machine %r",
+                  port, protocol, machine.instance_id)
+
+    @defer.inlineCallbacks
+    def get_opened_ports(self, machine, machine_id):
+        """Get a set of opened port/protocol pairs for a machine"""
+        group = yield self._get_machine_group(machine, machine_id)
+        opened_ports = set()
+        for rule in group.get("rules", []):
+            if not rule.get("group"):
+                protocol = rule["ip_protocol"]
+                from_port = rule["from_port"]
+                to_port = rule["to_port"]
+                if from_port == to_port:
+                    opened_ports.add((from_port, protocol))
+        defer.returnValue(opened_ports)
+
+    @defer.inlineCallbacks
+    def ensure_groups(self, machine_id):
+        """Get names of the security groups for a machine, creating if needed
+
+        If the juju group already exists, it is assumed to be correctly set up.
+        If the machine group already exists, it is deleted then recreated.
+        """
+        security_groups = yield self.nova.list_security_groups()
+        groups_by_name = dict((sg['name'], sg['id']) for sg in security_groups)
+
+        juju_group = self._juju_group_name()
+        if not juju_group in groups_by_name:
+            log.debug("Creating juju security group %s", juju_group)
+            sg = yield self.nova.create_security_group(juju_group,
+                "juju group for %s" % (self.tag,))
+            # Add external ssh access
+            yield self.nova.add_security_group_rule(sg['id'],
+                ip_protocol="tcp", from_port=22, to_port=22)
+            # Add internal group access
+            yield self.nova.add_security_group_rule(
+                parent_group_id=sg['id'], group_id=sg['id'],
+                ip_protocol="tcp", from_port=1, to_port=65535)
+
+        machine_group = self._machine_group_name(machine_id)
+        if machine_group in groups_by_name:
+            yield self.nova.delete_security_group(
+                groups_by_name[machine_group])
+        log.debug("Creating machine security group %s", machine_group)
+        yield self.nova.create_security_group(machine_group,
+            "juju group for %s machine %s" % (self.tag, machine_id))
+
+        defer.returnValue([juju_group, machine_group])
+
+    @defer.inlineCallbacks
+    def get_machine_groups(self, machine, with_juju_group=False):
+        try:
+            ret = yield self.get_machine_groups_pure(machine, with_juju_group)
+        except errors.ProviderInteractionError, e:
+            # XXX: Need to wire up treatment of 500s properly in client
+            if getattr(e, "kind", None) == "computeError":
+                try:
+                    yield self.nova.get_server(machine.instance_id)
+                except errors.ProviderInteractionError, e:
+                    pass  # just rebinding e
+            if True or getattr(e, "kind", None) == "itemNotFound":
+                defer.returnValue(None)
+            raise
+        defer.returnValue(ret)
+
+    @defer.inlineCallbacks
+    def get_machine_groups_pure(self, machine, with_juju_group=False):
+        server_id = machine.instance_id
+        groups = yield self.nova.get_server_security_groups(server_id)
+        juju_group = self._juju_group_name()
+        groups_by_name = dict((g['name'], g['id']) for g in groups
+            if g['name'].startswith(juju_group))
+        if juju_group not in groups_by_name:
+            # Not a juju machine, shouldn't touch
+            defer.returnValue(None)
+        if not with_juju_group:
+            groups_by_name.pop(juju_group)
+        # else assumption: only one remaining group, is the machine group
+        defer.returnValue(groups_by_name)
+
+    @defer.inlineCallbacks
+    def delete_juju_group(self):
+        security_groups = yield self.nova.list_security_groups()
+        juju_group = self._juju_group_name()
+        for group in security_groups:
+            if group['name'] == juju_group:
+                break
+        else:
+            log.debug("Can't delete missing juju group")
+            return
+        yield self.nova.delete_security_group(group['id'])

=== added file 'juju/providers/openstack/provider.py'
--- juju/providers/openstack/provider.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/provider.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,189 @@
+"""Provider interface implementation for OpenStack backend
+
+Much of the logic is implemented in sibling modules, but the overall model is
+exposed here.
+
+Still in need of work here:
+* Implement constraints using the Nova flavors api. This will always mean an
+  api call rather than hard coding values as is done with EC2. Things like
+  memory and cpu count are broadly equivalent, but there's no guarentee what
+  details are exposed and ranking by price will generally not be an option.
+"""
+
+import logging
+
+from twisted.internet import defer
+
+from juju import errors
+from juju.providers.common.base import MachineProviderBase
+
+from .client import _OpenStackClient, _NovaClient, _SwiftClient
+from . import credentials
+from .files import FileStorage
+from .launch import NovaLaunchMachine
+from .machine import (
+    NovaProviderMachine, get_server_status, machine_from_instance
+    )
+from .ports import NovaPortManager
+
+
+log = logging.getLogger("juju.openstack")
+
+
+class MachineProvider(MachineProviderBase):
+    """MachineProvider for use in an OpenStack environment"""
+
+    Credentials = credentials.OpenStackCredentials
+
+    def __init__(self, environment_name, config):
+        super(MachineProvider, self).__init__(environment_name, config)
+        self.credentials = self.Credentials.from_environment(config)
+        client = _OpenStackClient(self.credentials)
+        self.nova = _NovaClient(client)
+        self.swift = _SwiftClient(client)
+        self.port_manager = NovaPortManager(self.nova, environment_name)
+
+    @property
+    def provider_type(self):
+        return "openstack"
+
+    def get_serialization_data(self):
+        """Get provider configuration suitable for serialization.
+
+        Also fills in credential information that may have earlier been
+        extracted from the environment.
+        """
+        data = super(MachineProvider, self).get_serialization_data()
+        self.credentials.set_config_defaults(data)
+        return data
+
+    def get_file_storage(self):
+        """Retrieve a Swift-backed :class:`FileStorage`."""
+        return FileStorage(self.swift, self.config["control-bucket"])
+
+    def start_machine(self, machine_data, master=False):
+        """Start an OpenStack machine.
+
+        :param dict machine_data: desired characteristics of the new machine;
+            it must include a "machine-id" key, and may include a "constraints"
+            key to specify the underlying OS and hardware.
+
+        :param bool master: if True, machine will initialize the juju admin
+            and run a provisioning agent, in addition to running a machine
+            agent.
+        """
+        return NovaLaunchMachine.launch(self, machine_data, master)
+
+    @defer.inlineCallbacks
+    def get_machines(self, instance_ids=()):
+        """List machines running in the provider.
+
+        :param list instance_ids: ids of instances you want to get. Leave empty
+            to list every
+            :class:`juju.providers.openstack.machine.NovaProviderMachine` owned
+            by this provider.
+
+        :return: a list of
+            :class:`juju.providers.openstack.machine.NovaProviderMachine`
+            instances
+        :rtype: :class:`twisted.internet.defer.Deferred`
+
+        :raises: :exc:`juju.errors.MachinesNotFound`
+        """
+        if len(instance_ids) == 1:
+            try:
+                instances = [(yield self.nova.get_server(instance_ids[0]))]
+            except errors.ProviderInteractionError, e:
+                # XXX: Need to wire up treatment of 404s properly in client
+                if True or getattr(e, "kind", None) == "itemNotFound":
+                    raise errors.MachinesNotFound(set(instance_ids))
+                raise
+            instance_ids = frozenset(instance_ids)
+        else:
+            instances = yield self.nova.list_servers()
+            if instance_ids:
+                instance_ids = frozenset(instance_ids)
+                instances = [instance for instance in instances
+                    if instance['id'] in instance_ids]
+
+        # Only want to deal with servers that were created by juju, checking
+        # the name begins with the prefix launch uses is good enough.
+        name_prefix = "juju %s instance " % (self.environment_name,)
+        machines = []
+        for instance in instances:
+            if (instance['name'].startswith(name_prefix) and
+                    get_server_status(instance) in ("running", "pending")):
+                machines.append(machine_from_instance(instance))
+
+        if instance_ids:
+            # We were asked for a specific list of machines, and if we can't
+            # completely fulfil that request we should blow up.
+            missing = instance_ids.difference(m.instance_id for m in machines)
+            if missing:
+                raise errors.MachinesNotFound(missing)
+
+        defer.returnValue(machines)
+
+    @defer.inlineCallbacks
+    def _delete_machine(self, machine, full=False):
+        server_id = machine.instance_id
+        server = yield self.nova.get_server(server_id)
+        if not server['name'].startswith(
+            "juju %s instance" % self.environment_name):
+            raise errors.MachinesNotFound(set([machine.instance_id]))
+        yield self.nova.delete_server(server_id)
+        defer.returnValue(machine)
+
+    def shutdown_machine(self, machine):
+        if not isinstance(machine, NovaProviderMachine):
+            raise errors.ProviderError(
+                "Need a NovaProviderMachine to shutdown not: %r" % (machine,))
+        # EC2 provider re-gets the machine to see if it's still in existance
+        # and can be shutdown, instead just handle an error? 404-ish?
+        return self._delete_machine(machine)
+
+    @defer.inlineCallbacks
+    def destroy_environment(self):
+        """Terminate all associated machines and security groups.
+
+        The super defintion of this method terminates each machine in
+        the environment; this needs to be augmented here by also
+        removing the security group for the environment.
+
+        :rtype: :class:`twisted.internet.defer.Deferred`
+        """
+        machines = yield self.get_machines()
+        deleted_machines = yield defer.gatherResults(
+            [self._delete_machine(m, True) for m in machines])
+        yield self.save_state({})
+        defer.returnValue(deleted_machines)
+
+    def shutdown_machines(self, machines):
+        """Terminate machines associated with this provider.
+
+        :param machines: machines to shut down
+        :type machines: list of
+            :class:`juju.providers.openstack.machine.NovaProviderMachine`
+
+        :return: list of terminated
+            :class:`juju.providers.openstack.machine.NovaProviderMachine`
+            instances
+        :rtype: :class:`twisted.internet.defer.Deferred`
+        """
+        # XXX: need to actually handle errors as non-terminated machines
+        # and not include them in the resulting list
+        return defer.gatherResults(
+            [self.shutdown_machine(m) for m in machines], consumeErrors=True)
+
+    def open_port(self, machine, machine_id, port, protocol="tcp"):
+        """Authorizes `port` using `protocol` on EC2 for `machine`."""
+        return self.port_manager.open_port(machine, machine_id, port, protocol)
+
+    def close_port(self, machine, machine_id, port, protocol="tcp"):
+        """Revokes `port` using `protocol` on EC2 for `machine`."""
+        return self.port_manager.close_port(
+            machine, machine_id, port, protocol)
+
+    def get_opened_ports(self, machine, machine_id):
+        """Returns a set of open (port, proto) pairs for `machine`."""
+        return self.port_manager.get_opened_ports(machine, machine_id)

=== added directory 'juju/providers/openstack/tests'
=== added file 'juju/providers/openstack/tests/__init__.py'
--- juju/providers/openstack/tests/__init__.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/tests/__init__.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,206 @@
+"""Infrastructure shared by all tests for OpenStack provider"""
+
+import json
+
+from twisted.internet import (
+    defer,
+    )
+from twisted.web import (
+    http,
+    http_headers,
+    )
+
+from juju import errors
+from juju.lib import (
+    mocker,
+    )
+
+from juju.providers.openstack import (
+    client,
+    files,
+    machine,
+    ports,
+    provider,
+    )
+
+
+class FakeResponse(object):
+
+    def __init__(self, code):
+        if code not in http.RESPONSES:
+            raise ValueError("Unknown http code: %r" % (code,))
+        self.code = code
+        self.headers = http_headers.Headers(
+            {"Content-Type": ["application/json"]})
+
+
+class OpenStackTestMixin(object):
+    """Helpers for test classes that want to exercise the OpenStack APIs
+
+    This goes all the way down to the http layer, which is really a little too
+    low for most of what the test cases want to assert. With some cleanups to
+    the client code, tests could instead mock out client methods.
+    """
+
+    environment_name = "testing"
+    api_url = "http://test.invalid/v2.0/"
+
+    def setUp(self):
+        self._mock_request = self.mocker.replace(
+            "juju.providers.openstack.client.request",
+            passthrough=False)
+        self.mocker.order()
+        self.nova_url = self.api_url + "nova"
+        self.swift_url = self.api_url + "swift"
+        self._mock_request("POST", self.api_url + "tokens", body=mocker.ANY)
+        self.mocker.result(defer.succeed((FakeResponse(200),
+            json.dumps({"access": {
+                "token": {"id": "tok", "expires": "2030-01-01T00:00:00"},
+                "serviceCatalog": [
+                    {
+                        "type": "compute",
+                        "endpoints": [{"publicURL": self.nova_url}]
+                    },
+                    {
+                        "type": "object-store",
+                        "endpoints": [{"publicURL": self.swift_url}]
+                    }
+                ]
+            }}))))
+        # Clear the environment so provider won't pick up config from envvars
+        self.change_environment()
+
+    def get_config(self):
+        return {
+            "type": "openstack",
+            "admin-secret": "password",
+            "access-key": "90abcdef",
+            "secret-key": "xxxxxxxx",
+            "auth-mode": "keypair",
+            "auth-url": self.api_url,
+            "project-name": "test_project",
+            "control-bucket": self.environment_name,
+            "default-instance-type": "standard.xsmall",
+            "default-image-id": 42,
+            "use-floating-ip": True,
+            }
+
+    def get_provider(self):
+        """Return the openstack machine provider.
+
+        This should only be invoked after mocker is in replay mode.
+        """
+        return provider.MachineProvider(self.environment_name,
+            self.get_config())
+
+    def make_server(self, server_id, name=None, status="ACTIVE"):
+        if name is None:
+            # Would be machine id rather than server id really but will do.
+            name = "juju testing instance " + str(server_id)
+        return {
+            "id": server_id,
+            "name": name,
+            "status": status,
+            "addresses": {},
+            }
+
+    def assert_not_found(self, deferred, server_ids):
+        self.assertFailure(deferred, errors.MachinesNotFound)
+        return deferred.addCallback(self._check_not_found, server_ids)
+
+    def _check_not_found(self, error, server_ids):
+        self.assertEqual(error.instance_ids, server_ids)
+
+    def _mock_nova(self, method, path, body):
+        self._mock_request(method,
+            "%s/%s" % (self.nova_url, path),
+            [("X-Auth-Token", "tok")],
+            body)
+
+    def _mock_swift(self, method, path, body, extra_headers=None):
+        headers = [("X-Auth-Token", "tok")]
+        if extra_headers is not None:
+            headers += extra_headers
+        url = "%s/%s" % (self.swift_url, path)
+        self._mock_request(method, url, headers, body)
+
+    def expect_nova_get(self, path, code=200, response=""):
+        self._mock_nova("GET", path, None)
+        if not isinstance(response, str):
+            response = json.dumps(response)
+        self.mocker.result(defer.succeed((FakeResponse(code), response)))
+
+    def expect_nova_post(self, path, body, code=200, response=""):
+        self._mock_nova("POST", path, body)
+        if not isinstance(response, str):
+            response = json.dumps(response)
+        self.mocker.result(defer.succeed((FakeResponse(code), response)))
+
+    def expect_nova_delete(self, path, code=202, response=""):
+        self._mock_nova("DELETE", path, None)
+        self.mocker.result(defer.succeed((FakeResponse(code), response)))
+
+    def expect_swift_get(self, path, code=200, response=""):
+        self._mock_swift("GET", path, None)
+        self.mocker.result(defer.succeed((FakeResponse(code), response)))
+
+    def expect_swift_put(self, path, body, code=201, response=""):
+        self._mock_swift("PUT", path, body)
+        self.mocker.result(defer.succeed((FakeResponse(code), response)))
+
+    def expect_swift_put_container(self, container, code=201, response=""):
+        self._mock_swift("PUT", container, "", [('X-Container-Read', '.r:*')])
+        self.mocker.result(defer.succeed((FakeResponse(code), response)))
+
+
+class MockedProvider(object):
+
+    provider_type = "openstack"
+    environment_name = "testing"
+    api_url = "http://testing.invalid/2.0/"
+
+    default_config = {
+        "type": "openstack",
+        "admin-secret": "asecret",
+        "username": "auser",
+        "password": "xxxxxxxx",
+        "auth-url": api_url,
+        "project-name": "aproject",
+        "control-bucket": environment_name,
+        "default-instance-type": "standard.xsmall",
+        "default-image-id": 42,
+        }
+
+    def __init__(self, mocker, config=None):
+        if config is None:
+            config = dict(self.default_config)
+        self.config = config
+        self.nova = mocker.proxy(client._NovaClient(None), passthrough=False)
+        self.swift = mocker.proxy(client._SwiftClient(None), passthrough=False)
+        self.port_manager = mocker.proxy(
+            ports.NovaPortManager(self.nova, self.environment_name))
+        self.provider_actions = mocker.mock()
+        self.mocker = mocker
+
+    def get_file_storage(self):
+        return files.FileStorage(self.swift, self.config['control-bucket'])
+
+    def __getattr__(self, attr):
+        return getattr(self.provider_actions, attr)
+
+    def expect_swift_public_object_url(self, object_name):
+        container_name = self.config['control-bucket']
+        event = self.swift.public_object_url(container_name, object_name)
+        self.mocker.result("%sswift/%s" % (self.api_url, object_name))
+        return event
+
+    def expect_swift_put(self, object_name, body, code=201, response=""):
+        container_name = self.config['control-bucket']
+        event = self.swift.put_object(container_name, object_name, body)
+        self.mocker.result(defer.succeed((FakeResponse(code), response)))
+        return event
+
+    def expect_zookeeper_machines(self, iid):
+        event = self.provider_actions.get_zookeeper_machines()
+        self.mocker.result(defer.succeed([machine.NovaProviderMachine(iid)]))
+        return event

=== added file 'juju/providers/openstack/tests/test_bootstrap.py'
--- juju/providers/openstack/tests/test_bootstrap.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/tests/test_bootstrap.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,213 @@
+"""Tests for bootstrapping juju on openstack
+
+Bootstrap touches a lot of the other parts of the provider, including machine
+launching, security groups and so on. Testing this in an end-to-end fashion as
+is done currently duplicates many checks from other more focussed tests.
+"""
+
+import logging
+import yaml
+
+from juju.lib import (
+    mocker,
+    testing,
+    )
+from juju.machine import constraints
+
+from juju.providers.openstack.machine import NovaProviderMachine
+
+from juju.providers.openstack.tests import OpenStackTestMixin
+
+
+class OpenStackBootstrapTest(OpenStackTestMixin, testing.TestCase):
+
+    def expect_verify(self):
+        self.expect_swift_put("testing/bootstrap-verify",
+            "storage is writable")
+
+    def expect_provider_state_fresh(self):
+        self.expect_swift_get("testing/provider-state")
+        self.expect_verify()
+
+    def expect_create_group(self):
+        self.expect_nova_post("os-security-groups",
+            {'security_group': {
+                'name': 'juju-testing',
+                'description': 'juju group for testing',
+            }},
+            response={'security_group': {
+                'id': 1,
+            }})
+        self.expect_nova_post("os-security-group-rules",
+            {'security_group_rule': {
+                'parent_group_id': 1,
+                'ip_protocol': "tcp",
+                'from_port': 22,
+                'to_port': 22,
+            }},
+            response={'security_group_rule': {
+                'id': 144, 'parent_group_id': 1,
+            }})
+        self.expect_nova_post("os-security-group-rules",
+            {'security_group_rule': {
+                'parent_group_id': 1,
+                'group_id': 1,
+                'ip_protocol': "tcp",
+                'from_port': 1,
+                'to_port': 65535,
+            }},
+            response={'security_group_rule': {
+                'id': 145, 'parent_group_id': 1,
+            }})
+
+    def expect_create_machine_group(self, machine_id):
+        machine = str(machine_id)
+        self.expect_nova_post("os-security-groups",
+            {'security_group': {
+                'name': 'juju-testing-' + machine,
+                'description': 'juju group for testing machine ' + machine,
+            }},
+            response={'security_group': {
+                'id': 2,
+            }})
+
+    def _match_server(self, data):
+        userdata = data['server'].pop('user_data').decode("base64")
+        self.assertEqual("#cloud-config", userdata.split("\n", 1)[0])
+        cloud_init = yaml.load(userdata)
+        # TODO: assertions on cloud-init content
+        self.assertEqual({'server': {
+                'flavorRef': 1,
+                'imageRef': 42,
+                'name': 'juju testing instance 0',
+                'security_groups': [
+                    {'name': 'juju-testing'},
+                    {'name': 'juju-testing-0'},
+                ],
+            }},
+            data)
+        return True
+
+    def expect_launch(self):
+        self.expect_nova_get("flavors",
+            response={'flavors': [{'id': 1, 'name': "standard.xsmall"}]})
+        self.expect_nova_post("servers", mocker.MATCH(self._match_server),
+            code=202, response={'server': {
+                'id': '1000',
+                'status': "PENDING",
+                'addresses': {'private': [{'version': 4, 'addr': "4.4.4.4"}]},
+            }})
+        self.expect_swift_put("testing/juju_master_id", "1000")
+        self.expect_nova_get("os-floating-ips",
+            response={'floating_ips': [
+                {'id': 80, 'instance_id': None, 'ip': "8.8.8.8"}
+            ]})
+        self.expect_nova_post("servers/1000/action",
+            {"addFloatingIp": {"address": "8.8.8.8"}}, code=202)
+        self.expect_swift_put("testing/provider-state",
+            yaml.dump({'zookeeper-instances': ['1000']}))
+
+    def _check_machine(self, machine_list):
+        [machine] = machine_list
+        self.assertTrue(isinstance(machine, NovaProviderMachine))
+        self.assertEqual(machine.instance_id, '1000')
+
+    def bootstrap(self):
+        provider = self.get_provider()
+        constraint_set = constraints.ConstraintSet(provider.provider_type)
+        return provider.bootstrap(constraint_set.load({}))
+
+    def test_bootstrap_clean(self):
+        """Bootstrap from a clean slate makes groups and zookeeper instance"""
+        self.expect_provider_state_fresh()
+        self.expect_nova_get("os-security-groups",
+            response={'security_groups': []})
+        self.expect_create_group()
+        self.expect_create_machine_group(0)
+        self.expect_launch()
+        self.mocker.replay()
+
+        log = self.capture_logging("juju.common", level=logging.DEBUG)
+        deferred = self.bootstrap()
+        deferred.addCallback(self._check_machine)
+        def check_log(_):
+            log_text = log.getvalue()
+            self.assertIn("Launching juju bootstrap instance", log_text)
+            self.assertNotIn("previously bootstrapped", log_text)
+        return deferred.addCallback(check_log)
+
+    def test_bootstrap_existing_group(self):
+        """Bootstrap reuses an exisiting provider security group"""
+        self.expect_provider_state_fresh()
+        self.expect_nova_get("os-security-groups",
+            response={'security_groups': [
+                    {'name': "juju-testing", 'id': 1},
+                ]})
+        self.expect_create_machine_group(0)
+        self.expect_launch()
+        self.mocker.replay()
+        return self.bootstrap().addCallback(self._check_machine)
+
+    def test_bootstrap_existing_machine_group(self):
+        """Bootstrap deletes and remakes an existing machine security group"""
+        self.expect_provider_state_fresh()
+        self.expect_nova_get("os-security-groups",
+            response={'security_groups': [
+                    {'name': "juju-testing-0", 'id': 3},
+                ]})
+        self.expect_create_group()
+        self.expect_nova_delete("os-security-groups/3")
+        self.expect_create_machine_group(0)
+        self.expect_launch()
+        self.mocker.replay()
+        return self.bootstrap().addCallback(self._check_machine)
+
+    def test_existing_machine(self):
+        """A preexisting zookeeper instance is returned if present"""
+        self.expect_swift_get("testing/provider-state",
+            response=yaml.dump({'zookeeper-instances': ['1000']}))
+        self.expect_nova_get("servers/1000",
+            response={'server': {
+                'id': '1000',
+                'name': 'juju testing instance 0',
+                'state': "RUNNING"
+            }})
+        self.mocker.replay()
+
+        log = self.capture_logging("juju.common")
+        self.capture_logging("juju.openstack") # Drop to avoid stderr kipple
+        deferred = self.bootstrap()
+        deferred.addCallback(self._check_machine)
+        def check_log(_):
+            self.assertEqual("juju environment previously bootstrapped.\n",
+                log.getvalue())
+        return deferred.addCallback(check_log)
+
+    def test_existing_machine_missing(self):
+        """Bootstrap overwrites existing zookeeper if instance is present"""
+        self.expect_swift_get("testing/provider-state",
+            response=yaml.dump({'zookeeper-instances': [3000]}))
+        self.expect_nova_get("servers/3000", code=404,
+            response={'itemNotFound':
+                {'message': "The resource could not be found.", 'code': 404}
+            })
+        self.expect_verify()
+        self.expect_nova_get("os-security-groups",
+            response={'security_groups': [
+                    {'name': "juju-testing", 'id': 1},
+                    {'name': "juju-testing-0", 'id': 3},
+                ]})
+        self.expect_nova_delete("os-security-groups/3")
+        self.expect_create_machine_group(0)
+        self.expect_launch()
+        self.mocker.replay()
+
+        log = self.capture_logging("juju.common", level=logging.DEBUG)
+        self.capture_logging("juju.openstack") # Drop to avoid stderr kipple
+        deferred = self.bootstrap()
+        deferred.addCallback(self._check_machine)
+        def check_log(_):
+            log_text = log.getvalue()
+            self.assertIn("Launching juju bootstrap instance", log_text)
+            self.assertNotIn("previously bootstrapped", log_text)
+        return deferred.addCallback(check_log)

=== added file 'juju/providers/openstack/tests/test_client.py'
--- juju/providers/openstack/tests/test_client.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/tests/test_client.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,75 @@
+"""Tests for OpenStack API twisted client"""
+
+from juju.lib import testing
+
+from juju.providers.openstack import client
+
+
+class StubClient(client._OpenStackClient):
+
+    def __init__(self):
+        self.url = "http://testing.invalid"
+
+    make_url = client._OpenStackClient._make_url
+
+
+class TestMakeUrl(testing.TestCase):
+
+    def setUp(self):
+        self.client = StubClient()
+        self.client.services = {
+            "compute": self.client.url + "/nova",
+            "object-store": self.client.url + "/swift",
+            }
+
+    def test_list_str(self):
+        self.assertEqual("http://testing.invalid/nova/servers",
+            self.client.make_url("compute", ["servers"]))
+        self.assertEqual("http://testing.invalid/swift/container/object",
+            self.client.make_url("object-store", ["container", "object"]))
+
+    def test_list_int(self):
+        self.assertEqual("http://testing.invalid/nova/servers/1000",
+            self.client.make_url("compute", ["servers", 1000]))
+        self.assertEqual("http://testing.invalid/nova/servers/1000/detail",
+            self.client.make_url("compute", ["servers", 1000, "detail"]))
+
+    def test_list_unicode(self):
+        url = self.client.make_url("object-store", ["container", u"\xa7"])
+        self.assertIsInstance(url, str)
+        self.assertEqual("http://testing.invalid/swift/container/%C2%A7", url)
+
+    def test_str(self):
+        self.assertEqual("http://testing.invalid/nova/servers",
+            self.client.make_url("compute", "servers"))
+        self.assertEqual("http://testing.invalid/swift/container/object",
+            self.client.make_url("object-store", "container/object"))
+
+    def test_trailing_slash(self):
+        self.client.services["object-store"] += "/"
+        self.assertEqual("http://testing.invalid/nova/container",
+            self.client.make_url("compute", "container"))
+        self.assertEqual("http://testing.invalid/nova/container/object",
+            self.client.make_url("compute", ["container", "object"]))
+
+
+class TestPlan(testing.TestCase):
+    """Ideas for tests needed"""
+
+    # auth request without auth
+    # get bytes, content-length 0, return ""
+    # get bytes, not ResponseDone, raise wrapped in ProviderError (with any bytes?)
+    # get bytes, type json, return bytes
+    # get json, content length 0, raise ProviderError
+    # get json, bad header (several forms), raise ProviderError (with bytes)
+    # get json, not ResponseDone, raise wrapped in ProviderError (with any bytes?)
+    # get json, undecodable, raise wrapped in ProviderError with bytes
+    # get json, mismatching root, raise ProviderError with bytes or json?
+    # wrong code, no json header, raise ProviderError with bytes
+    # wrong code, not ResponseDone, raise ProviderError from code with any bytes
+    # wrong code, undecodable, raise ProviderError from code with bytes
+    # wrong code, has mystery root, raise ProviderError from code with bytes or json?
+    # wrong code, has good root, no message
+    # wrong code, has good root, no code
+    # wrong code, has good root, differing code
+    # wrong code, has good root, message, and matching code

=== added file 'juju/providers/openstack/tests/test_credentials.py'
--- juju/providers/openstack/tests/test_credentials.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/tests/test_credentials.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,216 @@
+"""Tests for handling of OpenStack credentials in config and environment"""
+
+from juju.lib import (
+    testing,
+    )
+from juju.providers.openstack import (
+    credentials,
+    )
+
+
+class OpenStackCredentialsTests(testing.TestCase):
+
+    def test_required_url(self):
+        e = self.assertRaises(ValueError, credentials.OpenStackCredentials, {})
+        self.assertIn("Missing config 'auth-url'", str(e))
+
+    def test_required_mode_if_unguessable(self):
+        e = self.assertRaises(ValueError, credentials.OpenStackCredentials, {
+            'auth-url': "http://example.com",
+            })
+        self.assertIn("Missing config 'auth-mode'", str(e))
+
+    def test_legacy(self):
+        creds = credentials.OpenStackCredentials({
+            'auth-url': "http://example.com",
+            'auth-mode': "legacy",
+            'username': "luser",
+            'access-key': "laccess",
+            })
+        self.assertEqual(creds.url, "http://example.com")
+        self.assertEqual(creds.mode, "legacy")
+        self.assertEqual(creds.username, "luser")
+        self.assertEqual(creds.access_key, "laccess")
+
+    def test_legacy_required_username(self):
+        e = self.assertRaises(ValueError, credentials.OpenStackCredentials, {
+            'auth-url': "http://example.com",
+            'auth-mode': "legacy",
+            'access-key': "laccess",
+            })
+        self.assertIn("Missing config 'username'", str(e))
+
+    def test_legacy_required_access_key(self):
+        e = self.assertRaises(ValueError, credentials.OpenStackCredentials, {
+            'auth-url': "http://example.com",
+            'auth-mode': "legacy",
+            'username': "luser",
+            })
+        self.assertIn("Missing config 'access-key'", str(e))
+
+    # v1.0 auth is gone from the upstream codebase so maybe remove support
+    def test_legacy_guess_v1_0(self):
+        creds = credentials.OpenStackCredentials({
+            'auth-url': "http://example.com/v1.0/",
+            'username': "luser",
+            'access-key': "laccess",
+            })
+        self.assertEqual(creds.mode, "legacy")
+
+    def test_legacy_guess_v1_1(self):
+        creds = credentials.OpenStackCredentials({
+            'auth-url': "http://example.com/v1.1/",
+            'username': "luser",
+            'access-key': "laccess",
+            })
+        self.assertEqual(creds.mode, "legacy")
+
+    def test_userpass(self):
+        creds = credentials.OpenStackCredentials({
+            'auth-url': "http://example.com",
+            'auth-mode': "userpass",
+            'username': "uuser",
+            'password': "upass",
+            'project-name': "uproject",
+            })
+        self.assertEqual(creds.url, "http://example.com")
+        self.assertEqual(creds.mode, "userpass")
+        self.assertEqual(creds.username, "uuser")
+        self.assertEqual(creds.password, "upass")
+        self.assertEqual(creds.project_name, "uproject")
+
+    def test_userpass_guess_v2_0_no_slash(self):
+        creds = credentials.OpenStackCredentials({
+            'auth-url': "http://example.com/v2.0",
+            'username': "uuser",
+            'password': "upass",
+            'project-name': "uproject",
+            })
+        self.assertEqual(creds.mode, "userpass")
+
+    def test_userpass_guess_v2_0_slash(self):
+        creds = credentials.OpenStackCredentials({
+            'auth-url': "http://example.com/v2.0/",
+            'username': "uuser",
+            'password': "upass",
+            'project-name': "uproject",
+            })
+        self.assertEqual(creds.mode, "userpass")
+
+    def test_keypair(self):
+        creds = credentials.OpenStackCredentials({
+            'auth-url': "http://example.com",
+            'auth-mode': "keypair",
+            'access-key': "kaccess",
+            'secret-key': "ksecret",
+            'project-name': "kproject",
+            })
+        self.assertEqual(creds.url, "http://example.com")
+        self.assertEqual(creds.mode, "keypair")
+        self.assertEqual(creds.access_key, "kaccess")
+        self.assertEqual(creds.secret_key, "ksecret")
+        self.assertEqual(creds.project_name, "kproject")
+
+
+class FromEnvironmentTests(testing.TestCase):
+
+    def test_required_url(self):
+        self.change_environment()
+        e = self.assertRaises(ValueError,
+            credentials.OpenStackCredentials.from_environment, {})
+        self.assertIn("Missing config 'auth-url'", str(e))
+
+    def test_required_mode_if_unguessable(self):
+        self.change_environment(**{"NOVA_URL": "http://example.com"})
+        e = self.assertRaises(ValueError,
+            credentials.OpenStackCredentials.from_environment, {})
+        self.assertIn("Missing config 'auth-mode'", str(e))
+
+    def test_legacy(self):
+        self.change_environment(**{
+            "NOVA_URL": "http://example.com/v1.1/",
+            "NOVA_USERNAME": "euser",
+            "NOVA_API_KEY": "ekey",
+            })
+        creds = credentials.OpenStackCredentials.from_environment({})
+        self.assertEqual(creds.mode, "legacy")
+        self.assertEqual(creds.username, "euser")
+        self.assertEqual(creds.access_key, "ekey")
+
+    def test_keypair(self):
+        self.change_environment(**{
+            "NOVA_URL": "http://example.com/v2.0/",
+            "NOVA_API_KEY": "eaccess",
+            "NOVA_PROJECT_NAME": "eproject",
+            "NOVA_PROJECT_ID": "349212",
+            "NOVA_REGION_NAME": "eregion",
+            "EC2_SECRET_KEY": "esecret",
+            })
+        creds = credentials.OpenStackCredentials.from_environment({
+            'auth-mode': "keypair",
+            })
+        self.assertEqual(creds.access_key, "eaccess")
+        self.assertEqual(creds.secret_key, "esecret")
+        self.assertEqual(creds.project_name, "eproject")
+        self.assertEqual(creds.region, "eregion")
+
+    def test_userpass(self):
+        self.change_environment(**{
+            "OS_AUTH_URL": "http://example.com/v2.0/",
+            "OS_USERNAME": "euser",
+            "OS_PASSWORD": "epass",
+            "OS_TENANT_NAME": "eproject",
+            "OS_REGION_NAME": "eregion",
+            })
+        creds = credentials.OpenStackCredentials.from_environment({})
+        self.assertEqual(creds.mode, "userpass")
+        self.assertEqual(creds.username, "euser")
+        self.assertEqual(creds.password, "epass")
+        self.assertEqual(creds.project_name, "eproject")
+        self.assertEqual(creds.region, "eregion")
+
+    def test_prefer_os_auth_url(self):
+        self.change_environment(**{
+            "NOVA_URL": "http://example.com/v1.1/",
+            "NOVA_API_KEY": "eaccess",
+            "OS_AUTH_URL": "http://example.com/v2.0/",
+            "OS_USERNAME": "euser",
+            "OS_PASSWORD": "epass",
+            "OS_TENANT_NAME": "eproject",
+            })
+        creds = credentials.OpenStackCredentials.from_environment({})
+        self.assertEqual(creds.url, "http://example.com/v2.0/")
+        self.assertEqual(creds.mode, "userpass")
+
+
+class SetConfigDefaultsTests(testing.TestCase):
+
+    def test_set_all(self):
+        config = {
+            'auth-url': "http://example.com",
+            'auth-mode': "legacy",
+            'username': "luser",
+            'access-key': "laccess",
+            }
+        creds = credentials.OpenStackCredentials(config)
+        new_config = {}
+        creds.set_config_defaults(new_config)
+        self.assertEqual(config, new_config)
+
+    def test_set_only_missing(self):
+        config = {
+            'auth-url': "http://example.com/v1.1/",
+            'username': "luser",
+            'access-key': "laccess",
+            }
+        creds = credentials.OpenStackCredentials(config)
+        new_config = {
+            'username': "nuser",
+            }
+        creds.set_config_defaults(new_config)
+        self.assertEqual({
+            'auth-url': "http://example.com/v1.1/",
+            'auth-mode': "legacy",
+            'username': "nuser",
+            'access-key': "laccess",
+            }, new_config)

=== added file 'juju/providers/openstack/tests/test_files.py'
--- juju/providers/openstack/tests/test_files.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/tests/test_files.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,103 @@
+"""Tests for file storage backend based on Swift"""
+
+from cStringIO import StringIO
+
+from twisted.internet import (
+    defer,
+    )
+
+from juju import errors
+from juju.lib import (
+    testing,
+    )
+from juju.providers.openstack.tests import OpenStackTestMixin
+
+
+class FileStorageTestCase(OpenStackTestMixin, testing.TestCase):
+
+    def get_storage(self):
+        provider = self.get_provider()
+        storage = provider.get_file_storage()
+        return storage
+
+    def test_put_file(self):
+        """A file can be put in the storage"""
+        content = "some text"
+        self.expect_swift_put("testing/object", content)
+        self.mocker.replay()
+        return self.get_storage().put("object", StringIO(content))
+
+    def test_put_file_unicode(self):
+        """A file with a unicode name is put in UTF-8 url encoded form"""
+        content = "some text"
+        self.expect_swift_put("testing/%C2%A7", content)
+        self.mocker.replay()
+        return self.get_storage().put(u"\xa7", StringIO(content))
+
+    def test_put_file_create_container(self):
+        """The container will be created if it doesn't exist yet"""
+        content = "some text"
+        self.expect_swift_put("testing/object", content, code=404)
+        self.expect_swift_put_container("testing")
+        self.expect_swift_put("testing/object", content)
+        self.mocker.replay()
+        return self.get_storage().put("object", StringIO(content))
+
+    def test_put_file_unknown_error(self):
+        """Unexpected errors from client propogate"""
+        content = "some text"
+        self._mock_swift("PUT", "testing/object", content)
+        self.mocker.result(defer.fail(ValueError("Something unexpected")))
+        self.mocker.replay()
+        deferred = self.get_storage().put("object", StringIO(content))
+        return self.assertFailure(deferred, ValueError)
+
+    @defer.inlineCallbacks
+    def test_get_url(self):
+        """A url can be generated for any stored file."""
+        self.mocker.replay()
+        storage = self.get_storage()
+        yield storage._swift._client.authenticate()
+        url = storage.get_url("object")
+        self.assertEqual(self.swift_url + "/testing/object", url)
+
+    @defer.inlineCallbacks
+    def test_get_url_unicode(self):
+        """A url can be generated for *any* stored file."""
+        self.mocker.replay()
+        storage = self.get_storage()
+        yield storage._swift._client.authenticate()
+        url = storage.get_url(u"\xa7")
+        self.assertEqual(self.swift_url + "/testing/%C2%A7", url)
+
+    @defer.inlineCallbacks
+    def test_get_file(self):
+        """Retrieving a file returns a file-like object with the content"""
+        content = "some text"
+        self.expect_swift_get("testing/object", response=content)
+        self.mocker.replay()
+        result = yield self.get_storage().get("object")
+        self.assertEqual(result.read(), content)
+
+    @defer.inlineCallbacks
+    def test_get_file_unicode(self):
+        """Retrieving a file with a unicode key uses a UTF-8 url"""
+        content = "some text"
+        self.expect_swift_get(u"testing/%C2%A7", response=content)
+        self.mocker.replay()
+        result = yield self.get_storage().get(u"\xa7")
+        self.assertEqual(result.read(), content)
+
+    def test_get_file_nonexistant(self):
+        """Retrieving a nonexistant file raises a file not found error."""
+        self.expect_swift_get(u"testing/missing", code=404)
+        self.mocker.replay()
+        deferred = self.get_storage().get("missing")
+        return self.assertFailure(deferred, errors.FileNotFound)
+
+    def test_get_file_error(self):
+        """An error from the client results is attributed to the provider"""
+        self.expect_swift_get(u"testing/unavailable", code=500)
+        self.mocker.replay()
+        deferred = self.get_storage().get("unavailable")
+        return self.assertFailure(deferred, errors.ProviderInteractionError)

=== added file 'juju/providers/openstack/tests/test_getmachines.py'
--- juju/providers/openstack/tests/test_getmachines.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/tests/test_getmachines.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,115 @@
+"""Tests for OpenStack provider method for listing live juju machines"""
+
+from juju.lib import testing
+from juju.providers.openstack.machine import NovaProviderMachine
+from juju.providers.openstack.tests import OpenStackTestMixin
+
+
+class GetMachinesTests(OpenStackTestMixin, testing.TestCase):
+
+    def check_machines(self, machines, expected):
+        machine_details = []
+        for m in machines:
+            self.assertTrue(isinstance(m, NovaProviderMachine))
+            machine_details.append((m.instance_id, m.state))
+        machine_details.sort()
+        expected.sort()
+        self.assertEqual(expected, machine_details)
+
+    def test_all_none(self):
+        self.expect_nova_get("servers", response={"servers": []})
+        self.mocker.replay()
+        return self.get_provider().get_machines().addCallback(
+            self.check_machines, [])
+
+    def test_all_single(self):
+        self.expect_nova_get("servers", response={"servers": [
+            self.make_server(1000),
+            ]})
+        self.mocker.replay()
+        return self.get_provider().get_machines().addCallback(
+            self.check_machines, [(1000, 'running')])
+
+    def test_all_multiple(self):
+        self.expect_nova_get("servers", response={"servers": [
+            self.make_server(1001),
+            self.make_server(1002, status="BUILDING"),
+            ]})
+        self.mocker.replay()
+        return self.get_provider().get_machines().addCallback(
+            self.check_machines, [(1001, 'running'), (1002, 'pending')])
+
+    def test_all_some_dead(self):
+        self.expect_nova_get("servers", response={"servers": [
+            self.make_server(1001, status="BUILDING"),
+            self.make_server(1002, status="DELETED"),
+            ]})
+        self.mocker.replay()
+        return self.get_provider().get_machines().addCallback(
+            self.check_machines, [(1001, 'pending')])
+
+    def test_all_some_other(self):
+        self.expect_nova_get("servers", response={"servers": [
+            self.make_server(1001, name="nova started server"),
+            self.make_server(1002),
+            ]})
+        self.mocker.replay()
+        return self.get_provider().get_machines().addCallback(
+            self.check_machines, [(1002, 'running')])
+
+    def test_two_none(self):
+        self.expect_nova_get("servers", response={"servers": []})
+        self.mocker.replay()
+        deferred = self.get_provider().get_machines([1001, 1002])
+        return self.assert_not_found(deferred, [1001, 1002])
+
+    def test_two_some_dead(self):
+        self.expect_nova_get("servers", response={"servers": [
+            self.make_server(1001, status="BUILDING"),
+            self.make_server(1002, status="DELETED"),
+            ]})
+        self.mocker.replay()
+        deferred = self.get_provider().get_machines([1001, 1002])
+        return self.assert_not_found(deferred, [1002])
+
+    def test_two_some_other(self):
+        self.expect_nova_get("servers", response={"servers": [
+            self.make_server(1001, name="nova started server"),
+            self.make_server(1002),
+            ]})
+        self.mocker.replay()
+        deferred = self.get_provider().get_machines([1001, 1002])
+        return self.assert_not_found(deferred, [1001])
+
+    def test_one_running(self):
+        self.expect_nova_get("servers/1000",
+            response={"server": self.make_server(1000)})
+        self.mocker.replay()
+        deferred = self.get_provider().get_machines([1000])
+        return deferred.addCallback(self.check_machines, [(1000, 'running')])
+
+    def test_one_dead(self):
+        self.expect_nova_get("servers/1000",
+            response={"server": self.make_server(1000, status="DELETED")})
+        self.mocker.replay()
+        deferred = self.get_provider().get_machines([1000])
+        return self.assert_not_found(deferred, [1000])
+
+    def test_one_other(self):
+        self.expect_nova_get("servers/1000",
+            response={"server": self.make_server(1000, name="testing")})
+        self.mocker.replay()
+        deferred = self.get_provider().get_machines([1000])
+        return self.assert_not_found(deferred, [1000])
+
+    def test_one_missing(self):
+        self.expect_nova_get("servers/1000",
+            code=404, response={"itemNotFound":
+                {"message": "The resource could not be found.", "code": 404}})
+        self.mocker.replay()
+        deferred = self.get_provider().get_machines([1000])
+        return self.assert_not_found(deferred, [1000])
+    # XXX: Need to do error handling properly in client still
+    test_one_missing.skip = True
+
+    # XXX: EC2 also has some tests for wrapping unexpected errors from backend

=== added file 'juju/providers/openstack/tests/test_launch.py'
--- juju/providers/openstack/tests/test_launch.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/tests/test_launch.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,113 @@
+"""Tests for launching a new server customised for juju using Nova"""
+
+import logging
+
+from twisted.internet import (
+    defer,
+    )
+
+from juju import errors
+from juju.lib import (
+    mocker,
+    testing,
+    )
+from juju.machine import (
+    constraints,
+    )
+
+from juju.providers.openstack import (
+    launch,
+    tests,
+    )
+
+class MockedLaunchProvider(tests.MockedProvider):
+
+    def launch(self, machine_id, master=False):
+        constraint_set = constraints.ConstraintSet(self.provider_type)
+        details = {
+            'machine-id': machine_id,
+            'constraints': constraint_set.load({}),
+            }
+        return launch.NovaLaunchMachine.launch(self, details, master)
+
+    def expect_launch_setup(self, machine_id):
+        self.port_manager.ensure_groups(machine_id)
+        self.mocker.result(defer.succeed(["juju-x", "juju-y"]))
+        self.nova.list_flavors()
+        self.mocker.result(defer.succeed([
+                {'id': 1, 'name': "standard.xsmall"},
+            ]))
+
+    def expect_run_server(self, machine_id, response):
+        self.nova.run_server(
+            name="juju testing instance " + machine_id,
+            image_id=42,
+            flavor_id=1,
+            security_group_names=["juju-x", "juju-y"],
+            user_data=mocker.ANY,
+            )
+        self.mocker.result(defer.succeed(response))
+
+    def expect_available_floating_ip(self, server_id):
+        self.nova.list_floating_ips()
+        self.mocker.result(defer.succeed([
+            {'instance_id': None, 'ip': "198.162.1.0"},
+            ]))
+        self.nova.add_floating_ip(server_id, "198.162.1.0")
+        self.mocker.result(defer.succeed(None))
+
+
+class NovaLaunchMachineTests(testing.TestCase):
+
+    def test_launch_requires_default_image_id(self):
+        config = dict(MockedLaunchProvider.default_config)
+        del config['default-image-id']
+        provider = MockedLaunchProvider(self.mocker, config)
+        provider.expect_zookeeper_machines(1000)
+        self.mocker.replay()
+        deferred = provider.launch("1")
+        return self.assertFailure(deferred, errors.ProviderError)
+
+    def test_start_machine(self):
+        provider = MockedLaunchProvider(self.mocker)
+        provider.expect_zookeeper_machines(1000)
+        provider.expect_launch_setup("1")
+        provider.expect_run_server("1", response={
+            'id': 1001,
+            'addresses': {'public': []},
+            })
+        self.mocker.replay()
+        return provider.launch("1")
+
+    def test_start_machine_delay(self):
+        provider = MockedLaunchProvider(self.mocker)
+        provider.config["use-floating-ip"] = True
+        provider.expect_zookeeper_machines(1000)
+        provider.expect_launch_setup("1")
+        provider.expect_run_server("1", response={
+            'id': 1001,
+            })
+        provider.nova.get_server(1001)
+        self.mocker.result(defer.succeed({
+            'id': 1001,
+            'addresses': {'public': []},
+            }))
+        provider.expect_available_floating_ip(1001)
+        self.mocker.result(defer.succeed(None))
+        self.mocker.replay()
+        self.patch(launch.NovaLaunchMachine, "_DELAY_FOR_ADDRESSES", 0)
+        return provider.launch("1")
+
+    def test_start_machine_master(self):
+        provider = MockedLaunchProvider(self.mocker)
+        provider.expect_swift_public_object_url("juju_master_id")
+        provider.expect_launch_setup("0")
+        provider.expect_run_server("0", response={
+            'id': 1000,
+            'addresses': {'public': []},
+            })
+        provider.expect_swift_put("juju_master_id", "1000")
+        provider.provider_actions.save_state({'zookeeper-instances': [1000]})
+        self.mocker.result(defer.succeed(None))
+        self.mocker.replay()
+        return provider.launch("0", master=True)

=== added file 'juju/providers/openstack/tests/test_machine.py'
--- juju/providers/openstack/tests/test_machine.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/tests/test_machine.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,110 @@
+"""Tests for server wrapper and helper functions"""
+
+from juju.providers.openstack.machine import (
+    get_server_addresses,
+    get_server_status,
+    )
+
+from juju.lib.testing import TestCase
+
+
+class GetServerStatusTest(TestCase):
+    """Tests for mapping of Nova status names to EC2 style names"""
+
+    def test_build_schedualing(self):
+        self.assertEqual("pending",
+            get_server_status({u'status': u'BUILD(scheduling)'}))
+
+    def test_build_spawning(self):
+        self.assertEqual("pending",
+            get_server_status({u'status': u'BUILD(spawning)'}))
+
+    def test_active(self):
+        self.assertEqual("running",
+            get_server_status({u'status': u'ACTIVE'}))
+
+    def test_no_status(self):
+        self.assertEqual("pending",
+            get_server_status({}))
+
+    def test_mystery_status(self):
+        self.assertEqual("unknown",
+            get_server_status({u'status': u'NEVER_BEFORE_SEEN_MYSTERY'}))
+
+
+class GetServerAddressesTest(TestCase):
+    """Tests for deriving a public and private address from Nova dict"""
+
+    def test_missing(self):
+        self.assertEqual((None, None), get_server_addresses({}))
+
+    def test_empty(self):
+        self.assertEqual((None, None),
+            get_server_addresses({u'addresses': {}}))
+
+    def test_private_only(self):
+        self.assertEqual(("127.0.0.4", None),
+            get_server_addresses({u'addresses': {
+                "private": [{"addr": "127.0.0.4", "version": 4}],
+            }}))
+
+    def test_private_plus(self):
+        self.assertEqual(("127.0.0.4", "8.8.4.4"),
+            get_server_addresses({u'addresses': {
+                "private": [
+                    {"addr": "127.0.0.4", "version": 4},
+                    {"addr": "8.8.4.4", "version": 4},
+                ],
+            }}))
+
+    def test_public_only(self):
+        self.assertEqual((None, "8.8.8.8"),
+            get_server_addresses({u'addresses': {
+                "public": [{"addr": "8.8.8.8", "version": 4}],
+            }}))
+
+    def test_public_and_private(self):
+        self.assertEqual(("127.0.0.4", "8.8.8.8"),
+            get_server_addresses({u'addresses': {
+                "public": [{"addr": "8.8.8.8", "version": 4}],
+                "private": [{"addr": "127.0.0.4", "version": 4}],
+            }}))
+
+    def test_public_and_private_plus(self):
+        self.assertEqual(("127.0.0.4", "8.8.8.8"),
+            get_server_addresses({u'addresses': {
+                "public": [{"addr": "8.8.8.8", "version": 4}],
+                "private": [
+                    {"addr": "127.0.0.4", "version": 4},
+                    {"addr": "8.8.4.4", "version": 4},
+                ],
+            }}))
+
+    def test_custom_only(self):
+        self.assertEqual(("127.0.0.2", None),
+            get_server_addresses({u'addresses': {
+                "special": [{"addr": "127.0.0.2", "version": 4}],
+            }}))
+
+    def test_custom_plus(self):
+        self.assertEqual(("127.0.0.2", "8.8.4.4"),
+            get_server_addresses({u'addresses': {
+                "special": [
+                    {"addr": "127.0.0.2", "version": 4},
+                    {"addr": "8.8.4.4", "version": 4},
+                ],
+            }}))
+
+    def test_custom_and_private(self):
+        self.assertEqual(("127.0.0.4", None),
+            get_server_addresses({u'addresses': {
+                "special": [{"addr": "127.0.0.2", "version": 4}],
+                "private": [{"addr": "127.0.0.4", "version": 4}],
+            }}))
+
+    def test_custom_and_public(self):
+        self.assertEqual(("127.0.0.2", "8.8.8.8"),
+            get_server_addresses({u'addresses': {
+                "special": [{"addr": "127.0.0.2", "version": 4}],
+                "public": [{"addr": "8.8.8.8", "version": 4}],
+            }}))

=== added file 'juju/providers/openstack/tests/test_ports.py'
--- juju/providers/openstack/tests/test_ports.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/tests/test_ports.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,396 @@
+"""Tests for emulating port management with security groups"""
+
+import logging
+
+from juju import errors
+from juju.lib.testing import TestCase
+from juju.providers.openstack.machine import NovaProviderMachine
+from juju.providers.openstack.ports import NovaPortManager
+from juju.providers.openstack.tests import OpenStackTestMixin
+
+
+class ProviderPortMgmtTests(OpenStackTestMixin, TestCase):
+    """Tests for provider exposed port management methods"""
+
+    def expect_create_rule(self, group_id, proto, port):
+        self.expect_nova_post("os-security-group-rules",
+            {'security_group_rule': {
+                'parent_group_id': group_id,
+                'ip_protocol': proto,
+                'from_port': port,
+                'to_port': port,
+            }},
+            response={'security_group_rule': {
+                'id': 144, 'parent_group_id': group_id,
+            }})
+
+    def expect_existing_rule(self, rule_id, proto, port):
+        self.expect_nova_get("servers/1000/os-security-groups",
+            response={'security_groups': [
+                    {'name': "juju-testing-1", 'id': 1, 'rules': [{
+                        'id': rule_id,
+                        'parent_group_id': 1,
+                        'ip_protocol': proto,
+                        'from_port': port,
+                        'to_port': port,
+                        }]
+                    },
+                ]})
+
+    def test_open_port(self):
+        """Opening a port adds the rule to the appropriate security group"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            response={'security_groups': [
+                    {'name': "juju-testing-1", 'id': 1},
+                ]})
+        self.expect_create_rule(1, "tcp", 80)
+        self.mocker.replay()
+
+        log = self.capture_logging("juju.openstack", level=logging.DEBUG)
+        machine = NovaProviderMachine('1000', "server1000.testing.invalid")
+        deferred = self.get_provider().open_port(machine, "1", 80)
+        def _check_log(_):
+            self.assertIn("Opened 80/tcp on machine '1000'",
+                log.getvalue())
+        return deferred.addCallback(_check_log)
+
+    def test_open_port_missing_group(self):
+        """Missing security group raises an error on deleting port"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            response={'security_groups': []})
+        self.mocker.replay()
+
+        machine = NovaProviderMachine('1000', "server1000.testing.invalid")
+        deferred = self.get_provider().open_port(machine, "1", 80)
+        return self.assertFailure(deferred, errors.ProviderInteractionError)
+
+    def test_close_port(self):
+        """Closing a port removes the matching rule from the security group"""
+        self.expect_existing_rule(12, "tcp", 80)
+        self.expect_nova_delete("os-security-group-rules/12")
+        self.mocker.replay()
+
+        log = self.capture_logging("juju.openstack", level=logging.DEBUG)
+        machine = NovaProviderMachine('1000', "server1000.testing.invalid")
+        deferred = self.get_provider().close_port(machine, "1", 80)
+        def _check_log(_):
+            self.assertIn("Closed 80/tcp on machine '1000'",
+                log.getvalue())
+        return deferred.addCallback(_check_log)
+
+    def test_close_port_missing_group(self):
+        """Missing security group raises an error on closing port"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            response={'security_groups': []})
+        self.mocker.replay()
+
+        machine = NovaProviderMachine('1000', "server1000.testing.invalid")
+        deferred = self.get_provider().close_port(machine, "1", 80)
+        return self.assertFailure(deferred, errors.ProviderInteractionError)
+
+    def test_close_port_missing_rule(self):
+        """Missing security group rule raises an error on closing port"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            response={'security_groups': [{
+                'name': "juju-testing-1", 'id': 1, "rules": [],
+            }]})
+        self.mocker.replay()
+
+        machine = NovaProviderMachine('1000', "server1000.testing.invalid")
+        deferred = self.get_provider().close_port(machine, "1", 80)
+        return self.assertFailure(deferred, errors.ProviderInteractionError)
+
+    def test_close_port_mismatching_rule(self):
+        """Rule with different port raises an error on closing port"""
+        self.expect_existing_rule(12, "tcp", 8080)
+        self.mocker.replay()
+
+        machine = NovaProviderMachine('1000', "server1000.testing.invalid")
+        deferred = self.get_provider().close_port(machine, "1", 80)
+        return self.assertFailure(deferred, errors.ProviderInteractionError)
+
+    def test_get_opened_ports_none(self):
+        """No opened ports are listed when there are no rules"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            response={'security_groups': [{
+                'name': "juju-testing-1", 'id': 1, "rules": [],
+            }]})
+        self.mocker.replay()
+
+        machine = NovaProviderMachine('1000', "server1000.testing.invalid")
+        deferred = self.get_provider().get_opened_ports(machine, "1")
+        return deferred.addCallback(self.assertEqual, set())
+
+    def test_get_opened_ports_one(self):
+        """Opened port is listed when there is a matching rule"""
+        self.expect_existing_rule(12, "tcp", 80)
+        self.mocker.replay()
+
+        machine = NovaProviderMachine('1000', "server1000.testing.invalid")
+        deferred = self.get_provider().get_opened_ports(machine, "1")
+        return deferred.addCallback(self.assertEqual, set([(80, "tcp")]))
+
+    def test_get_opened_ports_group_ignored(self):
+        """Opened ports exclude rules delegating to other security groups"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            response={'security_groups': [{
+                'name': "juju-testing-1", 'id': 1, "rules": [{
+                        'id': 12,
+                        'parent_group_id': 1,
+                        'ip_protocol': None,
+                        'from_port': None,
+                        'to_port': None,
+                        'group': {'name': "juju-testing"},
+                        }],
+            }]})
+        self.mocker.replay()
+
+        machine = NovaProviderMachine('1000', "server1000.testing.invalid")
+        deferred = self.get_provider().get_opened_ports(machine, "1")
+        return deferred.addCallback(self.assertEqual, set())
+
+    def test_get_opened_ports_multiport_ignored(self):
+        """Opened ports exclude rules spanning multiple ports"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            response={'security_groups': [{
+                'name': "juju-testing-1", 'id': 1, "rules": [{
+                        'id': 12,
+                        'parent_group_id': 1,
+                        'ip_protocol': "tcp",
+                        'from_port': 8080,
+                        'to_port': 8081,
+                        }],
+            }]})
+        self.mocker.replay()
+
+        machine = NovaProviderMachine('1000', "server1000.testing.invalid")
+        deferred = self.get_provider().get_opened_ports(machine, "1")
+        return deferred.addCallback(self.assertEqual, set())
+
+
+class PortManagerTestMixin(OpenStackTestMixin):
+
+    def get_port_manager(self):
+        provider = self.get_provider()
+        return NovaPortManager(provider.nova, provider.environment_name)
+
+
+class EnsureGroupsTests(PortManagerTestMixin, TestCase):
+    """Tests for ensure_groups method used when launching machines"""
+
+    def expect_create_juju_group(self):
+        self.expect_nova_post("os-security-groups",
+            {'security_group': {
+                'name': 'juju-testing',
+                'description': 'juju group for testing',
+            }},
+            response={'security_group': {
+                'id': 1,
+            }})
+        self.expect_nova_post("os-security-group-rules",
+            {'security_group_rule': {
+                'parent_group_id': 1,
+                'ip_protocol': "tcp",
+                'from_port': 22,
+                'to_port': 22,
+            }},
+            response={'security_group_rule': {
+                'id': 144, 'parent_group_id': 1,
+            }})
+        self.expect_nova_post("os-security-group-rules",
+            {'security_group_rule': {
+                'parent_group_id': 1,
+                'group_id': 1,
+                'ip_protocol': "tcp",
+                'from_port': 1,
+                'to_port': 65535,
+            }},
+            response={'security_group_rule': {
+                'id': 145, 'parent_group_id': 1,
+            }})
+
+    def expect_create_machine_group(self, machine_id):
+        machine = str(machine_id)
+        self.expect_nova_post("os-security-groups",
+            {'security_group': {
+                'name': 'juju-testing-' + machine,
+                'description': 'juju group for testing machine ' + machine,
+            }},
+            response={'security_group': {
+                'id': 2,
+            }})
+
+    def check_group_names(self, result, machine_id):
+        self.assertEqual(["juju-testing", "juju-testing-" + str(machine_id)],
+            result)
+
+    def test_none_existing(self):
+        """When no groups exist juju and machine security groups are created"""
+        self.expect_nova_get("os-security-groups",
+            response={'security_groups': []})
+        self.expect_create_juju_group()
+        self.expect_create_machine_group(0)
+        self.mocker.replay()
+        deferred = self.get_port_manager().ensure_groups(0)
+        return deferred.addCallback(self.check_group_names, 0)
+
+    def test_other_existing(self):
+        """Existing groups in a different environment are not affected"""
+        self.expect_nova_get("os-security-groups",
+            response={'security_groups': [
+                    {'name': "juju-testingish", 'id': 7},
+                    {'name': "juju-testingish-0", 'id': 8},
+                ]})
+        self.expect_create_juju_group()
+        self.expect_create_machine_group(0)
+        self.mocker.replay()
+        deferred = self.get_port_manager().ensure_groups(0)
+        return deferred.addCallback(self.check_group_names, 0)
+
+    def test_existing_juju_group(self):
+        """An exisiting juju security group is reused"""
+        self.expect_nova_get("os-security-groups",
+            response={'security_groups': [
+                    {'name': "juju-testing", 'id': 1},
+                ]})
+        self.expect_create_machine_group(0)
+        self.mocker.replay()
+        deferred = self.get_port_manager().ensure_groups(0)
+        return deferred.addCallback(self.check_group_names, 0)
+
+    def test_existing_machine_group(self):
+        """An existing machine security group is deleted and remade"""
+        self.expect_nova_get("os-security-groups",
+            response={'security_groups': [
+                    {'name': "juju-testing-6", 'id': 3},
+                ]})
+        self.expect_create_juju_group()
+        self.expect_nova_delete("os-security-groups/3")
+        self.expect_create_machine_group(6)
+        self.mocker.replay()
+        deferred = self.get_port_manager().ensure_groups(6)
+        return deferred.addCallback(self.check_group_names, 6)
+
+
+class GetMachineGroupsTests(PortManagerTestMixin, TestCase):
+    """Tests for get_machine_groups method needed for machine shutdown"""
+
+    def test_normal(self):
+        """A standard juju machine returns the machine group name and id"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            response={'security_groups': [
+                    {'id': 7, 'name': "juju-testing"},
+                    {'id': 8, 'name': "juju-testing-0"},
+                ]})
+        self.mocker.replay()
+        machine = NovaProviderMachine(1000)
+        deferred = self.get_port_manager().get_machine_groups(machine)
+        return deferred.addCallback(self.assertEqual, {"juju-testing-0": 8})
+
+    def test_normal_include_juju(self):
+        """If param with_juju_group=True the juju group is also returned"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            response={'security_groups': [
+                    {'id': 7, 'name': "juju-testing"},
+                    {'id': 8, 'name': "juju-testing-0"},
+                ]})
+        self.mocker.replay()
+        machine = NovaProviderMachine(1000)
+        deferred = self.get_port_manager().get_machine_groups(machine, True)
+        return deferred.addCallback(self.assertEqual,
+            {"juju-testing": 7, "juju-testing-0": 8})
+
+    def test_extra_group(self):
+        """Additional groups not in the juju namespace are ignored"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            response={'security_groups': [
+                    {'id': 1, 'name': "default"},
+                    {'id': 7, 'name': "juju-testing"},
+                    {'id': 8, 'name': "juju-testing-0"},
+                ]})
+        self.mocker.replay()
+        machine = NovaProviderMachine(1000)
+        deferred = self.get_port_manager().get_machine_groups(machine)
+        return deferred.addCallback(self.assertEqual, {"juju-testing-0": 8})
+
+    def test_other_group(self):
+        """A server not managed by juju returns nothing"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            response={'security_groups': [
+                    {'id': 1, 'name': "default"},
+                ]})
+        self.mocker.replay()
+        machine = NovaProviderMachine(1000)
+        deferred = self.get_port_manager().get_machine_groups(machine)
+        return deferred.addCallback(self.assertEqual, None)
+
+    def test_missing_groups(self):
+        """A server with no groups returns nothing"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            response={'security_groups': []})
+        self.mocker.replay()
+        machine = NovaProviderMachine(1000)
+        deferred = self.get_port_manager().get_machine_groups(machine)
+        return deferred.addCallback(self.assertEqual, None)
+
+    def test_error_missing_server(self):
+        """A server that doesn't exist or has been deleted returns nothing"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            code=404, response={"itemNotFound": {
+                "message": "Instance 1000 could not be found.",
+                "code": 404,
+            }})
+        self.mocker.replay()
+        machine = NovaProviderMachine(1000)
+        deferred = self.get_port_manager().get_machine_groups(machine)
+        return deferred.addCallback(self.assertEqual, None)
+    # XXX: Broken by workaround for HP not supporting this api
+    test_error_missing_server.skip = True
+
+    def test_error_missing_page(self):
+        """Unexpected errors from the client are propogated"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            code=404, response="404 Not Found\n\n"
+                "The resource could not be found.\n\n   ")
+        self.mocker.replay()
+        machine = NovaProviderMachine(1000)
+        deferred = self.get_port_manager().get_machine_groups(machine)
+        return self.assertFailure(deferred, errors.ProviderInteractionError)
+    # XXX: Need implemention of fancy error to exception mapping
+    test_error_missing_page.skip = True
+
+    def test_error_missing_server_fault(self):
+        "A bogus compute fault due to lp:1010486 returns nothing"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            code=500, response={"computeFault": {
+                "message": "The server has either erred or is incapable of"
+                    " performing the requested operation.",
+                "code": 500,
+            }})
+        self.expect_nova_get("servers/1000",
+            code=404, response={"itemNotFound": {
+                "message": "The resource could not be found.",
+                "code": 404,
+            }})
+        self.mocker.replay()
+        machine = NovaProviderMachine(1000)
+        deferred = self.get_port_manager().get_machine_groups(machine)
+        return deferred.addCallback(self.assertEqual, None)
+    # XXX: Need implemention of fancy error to exception mapping
+    test_error_missing_server_fault.skip = True
+
+    def test_error_really_fault(self):
+        """A real compute fault is propogated"""
+        self.expect_nova_get("servers/1000/os-security-groups",
+            code=500, response={"computeFault": {
+                "message": "The server has either erred or is incapable of"
+                    " performing the requested operation.",
+                "code": 500,
+            }})
+        self.expect_nova_get("servers/1000", response={"server": {"id": 1000}})
+        self.mocker.replay()
+        machine = NovaProviderMachine(1000)
+        deferred = self.get_port_manager().get_machine_groups(machine)
+        return self.assertFailure(deferred, errors.ProviderInteractionError)
+    # XXX: Need implemention of fancy error to exception mapping
+    test_error_really_fault.skip = True

=== added file 'juju/providers/openstack/tests/test_provider.py'
--- juju/providers/openstack/tests/test_provider.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/tests/test_provider.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,116 @@
+"""Testing for the OpenStack provider interface"""
+
+from juju.lib.testing import TestCase
+from juju.environment.errors import EnvironmentsConfigError
+
+from juju.providers.openstack.files import FileStorage
+from juju.providers.openstack.provider import MachineProvider
+
+
+class ProviderTestCase(TestCase):
+
+    environment_name = "testing"
+
+    _test_environ = {
+        "NOVA_URL": "http://environ.invalid",
+        "NOVA_API_KEY": "env-key",
+        "EC2_SECRET_KEY": "env-xxxx",
+        "NOVA_PROJECT_ID": "env-project",
+        }
+
+    def get_config(self):
+        return {
+            "type": "openstack",
+            "auth-mode": "keypair",
+            "access-key": "key",
+            "secret-key": "xxxxxxxx",
+            "auth-url": "http://testing.invalid",
+            "project-name": "project",
+            "control-bucket": self.environment_name,
+            }
+
+    def test_empty_config_raises(self):
+        """Passing no config raises an exception about lacking credentials"""
+        self.change_environment()
+        # XXX: Should this raise EnvironmentsConfigError instead?
+        self.assertRaises(ValueError,
+            MachineProvider, self.environment_name, {})
+
+    def test_client_params(self):
+        """Config details get passed through to OpenStack client correctly"""
+        config = self.get_config()
+        provider = MachineProvider(self.environment_name, config)
+        creds = provider.credentials
+        self.assertEquals("key", creds.access_key)
+        self.assertEquals("xxxxxxxx", creds.secret_key)
+        self.assertEquals("http://testing.invalid", creds.url)
+        self.assertEquals("project", creds.project_name)
+        self.assertIs(creds, provider.nova._client.credentials)
+        self.assertIs(creds, provider.swift._client.credentials)
+
+    def test_provider_attributes(self):
+        """
+        The provider environment name and config should be available as
+        parameters in the provider.
+        """
+        provider = MachineProvider(self.environment_name, self.get_config())
+        self.assertEqual(provider.environment_name, self.environment_name)
+        self.assertEqual(provider.config.get("type"), "openstack")
+        self.assertEqual(provider.provider_type, "openstack")
+
+    def test_get_file_storage(self):
+        """The file storage is accessible via the machine provider."""
+        provider = MachineProvider(self.environment_name, self.get_config())
+        storage = provider.get_file_storage()
+        self.assertTrue(isinstance(storage, FileStorage))
+
+    def test_config_serialization(self):
+        """
+        The provider configuration can be serialized to yaml.
+        """
+        self.change_environment()
+        config = self.get_config()
+        expected = config.copy()
+        config["authorized-keys-path"] = self.makeFile("key contents")
+        expected["authorized-keys"] = "key contents"
+        provider = MachineProvider(self.environment_name, config)
+        self.assertEqual(expected, provider.get_serialization_data())
+
+    def test_config_environment_extraction(self):
+        """
+        The provider serialization loads keys as needed from the environment.
+
+        Variables from the configuration take precendence over those from
+        the environment, when serializing.
+        """
+        self.change_environment(**self._test_environ)
+        provider = MachineProvider(self.environment_name, {
+            "auth-mode": "keypair",
+            "project-name": "other-project",
+            "authorized-keys": "key-data",
+            })
+        serialized = provider.get_serialization_data()
+        expected = {
+            "auth-mode": "keypair",
+            "access-key": "env-key",
+            "secret-key": "env-xxxx",
+            "auth-url": "http://environ.invalid",
+            "project-name": "other-project",
+            "authorized-keys": "key-data",
+            }
+        self.assertEqual(expected, serialized)
+
+    def test_conflicting_authorized_keys_options(self):
+        """
+        We can't handle two different authorized keys options, so deny
+        constructing an environment that way.
+        """
+        config = self.get_config()
+        config["authorized-keys"] = "File content"
+        config["authorized-keys-path"] = "File path"
+        error = self.assertRaises(EnvironmentsConfigError,
+            MachineProvider, self.environment_name, config)
+        self.assertEquals(
+            str(error),
+            "Environment config cannot define both authorized-keys and "
+            "authorized-keys-path. Pick one!")

=== added file 'juju/providers/openstack/tests/test_shutdown.py'
--- juju/providers/openstack/tests/test_shutdown.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/tests/test_shutdown.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,130 @@
+"""Tests for terminating machines and cleaning up the environment"""
+
+from juju import errors
+from juju.lib import testing
+from juju.machine import ProviderMachine
+from juju.providers.openstack.machine import NovaProviderMachine
+from juju.providers.openstack.tests import OpenStackTestMixin
+
+
+class ShutdownMachineTests(OpenStackTestMixin, testing.TestCase):
+
+    def test_shutdown_single(self):
+        self.expect_nova_get("servers/1000",
+            response={"server": {
+                'name': "juju testing instance 0",
+            }})
+        self.expect_nova_delete("servers/1000", code=204)
+        self.mocker.replay()
+        machine = NovaProviderMachine(1000)
+        deferred = self.get_provider().shutdown_machine(machine)
+        deferred.addCallback(self.assertIs, machine)
+
+    def test_shutdown_single_other(self):
+        self.expect_nova_get("servers/1000",
+            response={"server": {
+                'name': "some other instance",
+            }})
+        self.mocker.replay()
+        machine = NovaProviderMachine(1000)
+        deferred = self.get_provider().shutdown_machine(machine)
+        return self.assert_not_found(deferred, [1000])
+
+    def test_shutdown_single_wrong_machine(self):
+        self.mocker.reset()
+        machine = ProviderMachine("i-000003E8")
+        e = self.assertRaises(errors.ProviderError,
+            self.get_provider().shutdown_machine, machine)
+        self.assertIn("Need a NovaProviderMachine to shutdown", str(e))
+
+    def test_shutdown_multi_none(self):
+        self.mocker.reset()
+        deferred = self.get_provider().shutdown_machines([])
+        return deferred.addCallback(self.assertEqual, [])
+
+    def test_shutdown_multi_some_invalid(self):
+        """No machines are shutdown if some are invalid"""
+        self.mocker.unorder()
+        self.expect_nova_get("servers/1001",
+            response={"server": {
+                'name': "juju testing instance 1",
+            }})
+        self.expect_nova_get("servers/1002",
+            response={"server": {
+                'name': "some other instance",
+            }})
+        self.mocker.replay()
+        machines = [NovaProviderMachine(1001), NovaProviderMachine(1002)]
+        deferred = self.get_provider().shutdown_machines(machines)
+        return self.assert_not_found(deferred, [1002])
+    # XXX: dumb requirement to keep all running if some invalid, drop this
+    test_shutdown_multi_some_invalid.skip = True
+
+    # GZ 2012-06-11: Corner case difference, EC2 rechecks machine statuses and
+    #                group membership on shutdown.
+
+    def test_shutdown_multi_success(self):
+        """Machines are shutdown and groups except juju group are deleted"""
+        self.mocker.unorder()
+        self.expect_nova_get("servers/1001",
+            response={"server": {
+                'name': "juju testing instance 1",
+            }})
+        self.expect_nova_get("servers/1002",
+            response={"server": {
+                'name': "juju testing instance 2",
+            }})
+        self.expect_nova_delete("servers/1001", code=204)
+        self.expect_nova_delete("servers/1002", code=204)
+        self.mocker.replay()
+        machines = [NovaProviderMachine(1001), NovaProviderMachine(1002)]
+        deferred = self.get_provider().shutdown_machines(machines)
+        return deferred
+
+
+class DestroyEnvironmentTests(OpenStackTestMixin, testing.TestCase):
+
+    def check_machine_ids(self, machines, server_ids):
+        self.assertEqual(set(m.instance_id for m in machines), set(server_ids))
+
+    def test_destroy_environment(self):
+        self.mocker.unorder()
+        self.expect_swift_put("testing/provider-state", "{}\n")
+        self.expect_nova_get("servers", response={"servers": [
+            self.make_server(1001),
+            self.make_server(1002),
+            ]})
+        self.expect_nova_get("servers/1001",
+            response={"server": {
+                'name': "juju testing instance 1",
+            }})
+        self.expect_nova_get("servers/1002",
+            response={"server": {
+                'name': "juju testing instance 2",
+            }})
+        self.expect_nova_delete("servers/1001", code=204)
+        self.expect_nova_delete("servers/1002", code=204)
+        self.mocker.replay()
+        deferred = self.get_provider().destroy_environment()
+        return deferred.addCallback(self.check_machine_ids, [1001, 1002])
+
+    def test_s3_failure(self):
+        self.mocker.unorder()
+        self.expect_swift_put("testing/provider-state", "{}\n",
+            code=500, response="Server unavailable or something")
+        # XXX: normal server shutdown should be expected here
+        self.mocker.replay()
+        deferred = self.get_provider().destroy_environment()
+        return deferred.addCallback(self.assertEqual, [])
+    # XXX: Need to bolster swift robustness in response to api errors
+    test_s3_failure.skip = True
+
+    # GZ 2012-06-15: Always tries removing juju group unlike EC2 currently
+    def test_shutdown_no_instances(self):
+        """With no instances no shutdowns are attempted"""
+        self.mocker.unorder()
+        self.expect_swift_put("testing/provider-state", "{}\n")
+        self.expect_nova_get("servers", response={"servers": []})
+        self.mocker.replay()
+        deferred = self.get_provider().destroy_environment()
+        return deferred.addCallback(self.assertEqual, [])

=== added file 'juju/providers/openstack/tests/test_state.py'
--- juju/providers/openstack/tests/test_state.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack/tests/test_state.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,58 @@
+"""Tests for the common state interface over the the Openstack provider
+
+The testcases here largely duplicate those in openstack.tests.test_files as
+state handling is a pretty thin layer over the file storage.
+"""
+
+import yaml
+
+from juju.lib.testing import TestCase
+from juju.providers.openstack.tests import OpenStackTestMixin
+
+
+class OpenStackStateTest(OpenStackTestMixin, TestCase):
+
+    def test_save(self):
+        """Saving a dict puts yaml serialized bytes in provider-state"""
+        state = {"zookeeper-instances": [
+             [1000, "x1.example.com"],
+             ]}
+        self.expect_swift_put("testing/provider-state", yaml.dump(state))
+        self.mocker.replay()
+        return self.get_provider().save_state(state)
+
+    def test_save_missing_container(self):
+        """Saving will create the container when it does not exist already"""
+        state = {"zookeeper-instances": [
+             [1000, "x1.example.com"],
+             ]}
+        state_bytes = yaml.dump(state)
+        self.expect_swift_put("testing/provider-state", state_bytes, code=404)
+        self.expect_swift_put_container("testing")
+        self.expect_swift_put("testing/provider-state", state_bytes)
+        self.mocker.replay()
+        return self.get_provider().save_state(state)
+
+    def test_load(self):
+        """Loading deserializes yaml from provider-state to a python dict"""
+        state = {"zookeeper-instances": []}
+        self.expect_swift_get("testing/provider-state",
+            response=yaml.dump(state))
+        self.mocker.replay()
+        deferred = self.get_provider().load_state()
+        return deferred.addCallback(self.assertEqual, state)
+
+    def test_load_missing(self):
+        """Loading returns False if provider-state does not exist"""
+        self.expect_swift_get("testing/provider-state", code=404,
+            response={})
+        self.mocker.replay()
+        deferred = self.get_provider().load_state()
+        return deferred.addCallback(self.assertIs, False)
+
+    def test_load_no_content(self):
+        """Loading returns False if provider-state is empty"""
+        self.expect_swift_get("testing/provider-state", response="")
+        self.mocker.replay()
+        deferred = self.get_provider().load_state()
+        return deferred.addCallback(self.assertIs, False)

=== added directory 'juju/providers/openstack_s3'
=== added file 'juju/providers/openstack_s3/__init__.py'
--- juju/providers/openstack_s3/__init__.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack_s3/__init__.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,48 @@
+"""Provider interface implementation for Openstack with S3 storage"""
+
+import os
+
+from txaws.service import AWSServiceRegion
+
+from juju.providers.openstack import provider as os_provider
+from juju.providers.openstack import credentials
+from juju.providers.ec2 import files as s3_files
+
+
+class HybridCredentials(credentials.OpenStackCredentials):
+    """Encapsulation of credentials with S3 required values included"""
+
+    _config_vars = {
+        'combined-key':  ("EC2_ACCESS_KEY", "AWS_ACCESS_KEY_ID"),
+        's3-uri': ("S3_URL",),
+    }
+    _config_vars.update(credentials.OpenStackCredentials._config_vars)
+
+
+class MachineProvider(os_provider.MachineProvider):
+    """MachineProvider for use in Openstack environment but with S3 API"""
+
+    Credentials = HybridCredentials
+
+    def __init__(self, environment_name, config):
+        super(MachineProvider, self).__init__(environment_name, config)
+
+        del self.swift
+
+        # If access or secret keys are still blank, inside txaws environment
+        # a ValueError will be raised after rechecking the environment.
+        self._aws_service = AWSServiceRegion(
+            access_key=self.credentials.combined_key,
+            secret_key=self.credentials.secret_key,
+            ec2_uri="", # The EC2 client will not be used
+            s3_uri=self.credentials.s3_uri)
+
+        self.s3 = self._aws_service.get_s3_client()
+
+    @property
+    def provider_type(self):
+        return "openstack_s3"
+
+    def get_file_storage(self):
+        """Retrieve a S3 API compatible backend FileStorage class"""
+        return s3_files.FileStorage(self.s3, self.config["control-bucket"])

=== added directory 'juju/providers/openstack_s3/tests'
=== added file 'juju/providers/openstack_s3/tests/__init__.py'
=== added file 'juju/providers/openstack_s3/tests/test_provider.py'
--- juju/providers/openstack_s3/tests/test_provider.py	1970-01-01 00:00:00 +0000
+++ juju/providers/openstack_s3/tests/test_provider.py	2012-07-18 19:47:20 +0000
@@ -0,0 +1,122 @@
+"""Testing for the OpenStack provider interface"""
+
+from juju.lib.testing import TestCase
+from juju.environment.errors import EnvironmentsConfigError
+
+from juju.providers.ec2.files import FileStorage
+from juju.providers.openstack_s3 import MachineProvider
+
+
+class ProviderTestCase(TestCase):
+
+    environment_name = "testing"
+
+    _test_environ = {
+        "NOVA_URL": "http://environ.invalid",
+        "NOVA_API_KEY": "env-key",
+        "EC2_ACCESS_KEY": "env-key:env-project",
+        "EC2_SECRET_KEY": "env-xxxx",
+        "NOVA_PROJECT_ID": "env-project",
+        "S3_URL": "http://environ.invalid:3333",
+        }
+
+    def get_config(self):
+        return {
+            "type": "openstack_s3",
+            "auth-mode": "keypair",
+            "access-key": "key",
+            "secret-key": "xxxxxxxx",
+            "auth-url": "http://testing.invalid",
+            "project-name": "project",
+            "control-bucket": self.environment_name,
+            "combined-key": "key:project",
+            "s3-uri": "http://testing.invalid:3333",
+            }
+
+    def test_client_params(self):
+        """Config details get passed through to OpenStack client correctly"""
+        config = self.get_config()
+        provider = MachineProvider(self.environment_name, config)
+        creds = provider.credentials
+        self.assertEquals("key", creds.access_key)
+        self.assertEquals("xxxxxxxx", creds.secret_key)
+        self.assertEquals("http://testing.invalid", creds.url)
+        self.assertEquals("project", creds.project_name)
+        self.assertIs(creds, provider.nova._client.credentials)
+
+    def test_s3_params(self):
+        """Config details get passed through to txaws S3 client correctly"""
+        config = self.get_config()
+        s3 = MachineProvider(self.environment_name, config).s3
+        self.assertEquals("http://testing.invalid:3333/", s3.endpoint.get_uri())
+        self.assertEquals("key:project", s3.creds.access_key)
+        self.assertEquals("xxxxxxxx", s3.creds.secret_key)
+
+    def test_provider_attributes(self):
+        """
+        The provider environment name and config should be available as
+        parameters in the provider.
+        """
+        provider = MachineProvider(self.environment_name, self.get_config())
+        self.assertEqual(provider.environment_name, self.environment_name)
+        self.assertEqual(provider.config.get("type"), "openstack_s3")
+        self.assertEqual(provider.provider_type, "openstack_s3")
+
+    def test_get_file_storage(self):
+        """The file storage is accessible via the machine provider."""
+        provider = MachineProvider(self.environment_name, self.get_config())
+        storage = provider.get_file_storage()
+        self.assertTrue(isinstance(storage, FileStorage))
+
+    def test_config_serialization(self):
+        """
+        The provider configuration can be serialized to yaml.
+        """
+        self.change_environment()
+        config = self.get_config()
+        expected = config.copy()
+        config["authorized-keys-path"] = self.makeFile("key contents")
+        expected["authorized-keys"] = "key contents"
+        provider = MachineProvider(self.environment_name, config)
+        self.assertEqual(expected, provider.get_serialization_data())
+
+    def test_config_environment_extraction(self):
+        """
+        The provider serialization loads keys as needed from the environment.
+
+        Variables from the configuration take precendence over those from
+        the environment, when serializing.
+        """
+        self.change_environment(**self._test_environ)
+        provider = MachineProvider(self.environment_name, {
+            "auth-mode": "keypair",
+            "project-name": "other-project",
+            "authorized-keys": "key-data",
+            })
+        serialized = provider.get_serialization_data()
+        expected = {
+            "auth-mode": "keypair",
+            "access-key": "env-key",
+            "secret-key": "env-xxxx",
+            "auth-url": "http://environ.invalid",
+            "project-name": "other-project",
+            "authorized-keys": "key-data",
+            "combined-key": "env-key:env-project",
+            "s3-uri": "http://environ.invalid:3333",
+            }
+        self.assertEqual(expected, serialized)
+
+    def test_conflicting_authorized_keys_options(self):
+        """
+        We can't handle two different authorized keys options, so deny
+        constructing an environment that way.
+        """
+        config = self.get_config()
+        config["authorized-keys"] = "File content"
+        config["authorized-keys-path"] = "File path"
+        error = self.assertRaises(EnvironmentsConfigError,
+            MachineProvider, self.environment_name, config)
+        self.assertEquals(
+            str(error),
+            "Environment config cannot define both authorized-keys and "
+            "authorized-keys-path. Pick one!")

=== modified file 'juju/state/initialize.py'
--- juju/state/initialize.py	2012-03-29 01:37:57 +0000
+++ juju/state/initialize.py	2012-07-18 19:47:20 +0000
@@ -30,7 +30,10 @@
         """
         self.client = client
         self.admin_identity = admin_identity
+        if instance_id.isdigit():
+            instance_id = int(instance_id)
         self.instance_id = instance_id
+
         self.constraints_data = constraints_data
         self.provider_type = provider_type
 
@@ -57,6 +60,7 @@
         # Poke constraints data into a machine state to represent this machine.
         manager = MachineStateManager(self.client)
         machine_state = yield manager.add_machine_state(constraints)
+
         yield machine_state.set_instance_id(self.instance_id)
 
         # Set up environment constraints similarly.

=== modified file 'juju/unit/address.py'
--- juju/unit/address.py	2012-03-22 09:08:12 +0000
+++ juju/unit/address.py	2012-07-18 19:47:20 +0000
@@ -16,6 +16,8 @@
     provider_type = yield settings.get_provider_type()
     if provider_type == "ec2":
         returnValue(EC2UnitAddress())
+    if provider_type in ("openstack", "openstack_s3"):
+        returnValue(OpenStackUnitAddress())
     elif provider_type == "local":
         returnValue(LocalUnitAddress())
     elif provider_type == "orchestra":
@@ -24,7 +26,6 @@
         returnValue(DummyUnitAddress())
     elif provider_type == "maas":
         returnValue(MAASUnitAddress())
-
     raise JujuError(
         "Unknown provider type: %r, unit addresses unknown." % provider_type)
 
@@ -32,10 +33,10 @@
 class UnitAddress(object):
 
     def get_private_address(self):
-        raise NotImplemented()
+        raise NotImplementedError(self.get_private_address)
 
     def get_public_address(self):
-        raise NotImplemented()
+        raise NotImplementedError(self.get_public_address)
 
 
 class DummyUnitAddress(UnitAddress):
@@ -62,6 +63,27 @@
         returnValue(content.strip())
 
 
+class OpenStackUnitAddress(UnitAddress):
+    """Address determination of a service unit on an OpenStack server
+
+    Unlike EC2 there are no promises that an instance will have a resolvable
+    hostname, or for that matter a public ip address.
+    """
+
+    def _get_metadata_string(self, key):
+        return client.getPage("http://169.254.169.254/1.0/meta-data/" + key)
+
+    def get_private_address(self):
+        return self._get_metadata_string("local-ipv4")
+
+    @inlineCallbacks
+    def get_public_address(self):
+        address = yield self._get_metadata_string("public-ipv4")
+        if not address:
+            address = yield self.get_private_address()
+        returnValue(address)
+
+
 class LocalUnitAddress(UnitAddress):
 
     def get_private_address(self):
@@ -87,6 +109,7 @@
         output = subprocess.check_output(["hostname", "-f"])
         return output.strip()
 
+
 class MAASUnitAddress(UnitAddress):
 
     def get_private_address(self):

=== modified file 'juju/unit/tests/test_address.py'
--- juju/unit/tests/test_address.py	2012-03-22 09:08:12 +0000
+++ juju/unit/tests/test_address.py	2012-07-18 19:47:20 +0000
@@ -8,7 +8,7 @@
 from juju.lib.testing import TestCase
 from juju.unit.address import (
     EC2UnitAddress, LocalUnitAddress, OrchestraUnitAddress, DummyUnitAddress,
-    MAASUnitAddress, get_unit_address)
+    MAASUnitAddress, OpenStackUnitAddress, UnitAddress, get_unit_address)
 from juju.state.environment import GlobalSettingsStateManager
 
 
@@ -32,6 +32,16 @@
         self.assertTrue(isinstance(address, EC2UnitAddress))
 
     @inlineCallbacks
+    def test_get_openstack_address(self):
+        address = yield self.get_address_for("openstack")
+        self.assertTrue(isinstance(address, OpenStackUnitAddress))
+
+    @inlineCallbacks
+    def test_get_openstack_s3_address(self):
+        address = yield self.get_address_for("openstack_s3")
+        self.assertTrue(isinstance(address, OpenStackUnitAddress))
+
+    @inlineCallbacks
     def test_get_local_address(self):
         address = yield self.get_address_for("local")
         self.assertTrue(isinstance(address, LocalUnitAddress))
@@ -55,6 +65,22 @@
         return self.assertFailure(self.get_address_for("foobar"), JujuError)
 
 
+class SubclassAddressTest(TestCase):
+
+    class TestingAddress(UnitAddress):
+        """An address class that neglects to implement the required methods"""
+
+    def test_get_public_address(self):
+        err = self.assertRaises(NotImplementedError,
+            self.TestingAddress().get_public_address)
+        self.assertIn("TestingAddress.get_public_address", str(err))
+
+    def test_get_private_address(self):
+        err = self.assertRaises(NotImplementedError,
+            self.TestingAddress().get_private_address)
+        self.assertIn("TestingAddress.get_private_address", str(err))
+
+
 class DummyAddressTest(TestCase):
 
     def setUp(self):
@@ -93,6 +119,35 @@
             (yield self.address.get_public_address()), "foobar")
 
 
+class OpenStackAddressTest(TestCase):
+
+    def setUp(self):
+        self.address = OpenStackUnitAddress()
+        self.patch(client, "getPage", self._fetch_metadata)
+
+    def _fetch_metadata(self, url):
+        head, tail = url.rsplit("/", 1)
+        self.assertEqual("http://169.254.169.254/1.0/meta-data", head)
+        return succeed(self.meta.pop(tail))
+
+    @inlineCallbacks
+    def test_get_private_address(self):
+        self.meta = {"local-ipv4": "192.168.0.2"}
+        self.assertEqual("192.168.0.2",
+            (yield self.address.get_private_address()))
+
+    @inlineCallbacks
+    def test_get_public_address_present(self):
+        self.meta = {"public-ipv4": "8.8.8.8"}
+        self.assertEqual("8.8.8.8", (yield self.address.get_public_address()))
+
+    @inlineCallbacks
+    def test_get_public_address_missing(self):
+        self.meta = {"public-ipv4": "", "local-ipv4": "192.168.0.2"}
+        self.assertEqual("192.168.0.2",
+            (yield self.address.get_public_address()))
+
+
 class LocalAddressTest(TestCase):
 
     def setUp(self):