Merge lp:~sandy-walsh/nova/zones into lp:~hudson-openstack/nova/trunk

[Sandy Walsh]

Added http://mynova/v1.0/zones/ api options for add/remove/update/delete zones. child_zones table added to database and migration. Changed novarc vars from CLOUD_SERVERS_* to NOVA_* to work with novatools. See python-novatools on github for help testing this.

This is the first phase of the multi-zone clustering. Right now it's just the API/DB stuff for adding/listing/deleting zones.

Zones are just URL + username/password credentials of child zones.

python-novatools (on github) was also updated to support these changes.

Next we'll get these zones talking to each other.

[Todd Willey (xtoddx)]

was the bin/nova-combined change accidental?

[Eric Day (eday)]

I would s/ChildZone/Zone/ for consistent naming. Having the API deal with 'zones' and the DB deal with 'child zones' may make someone think they are not the same.

review: Needs Fixing

[Sandy Walsh (sandy-walsh)]

Thanks for the catch todd. Missed that on my diff.

Eric, yeah, I debated that one. I agree & will change. Thanks.

[lp:~sandy-walsh/nova/zones]

646. By Sandy Walsh on 2011-02-15

fixed nova-combined debug hack and renamed ChildZone to Zone

647. By Sandy Walsh on 2011-02-15

better filtering

648. By Sandy Walsh on 2011-02-15

trunk merge

[Jay Pipes (jaypipes)]

Hi! Great work so far, Sandy.

A couple notes:

1) 223 === added file 'nova/db/sqlalchemy/migrate_repo/versions/003_cactus.py'

I think we wanted to go with >1 migrate scripts per release, with a migration script for each patch that makes changes to the underlying database schema.

So, I'd recommend s/003_cactus.py/003_add_zone_tables.py/.

2) Looks like you are still calling things child zones in the migration script and in the nova.db.api docstrings...

Other than those little things, lgtm.

review: Needs Fixing

[lp:~sandy-walsh/nova/zones]

649. By Sandy Walsh on 2011-02-15

fixed / renamed migration scripts

[Eric Day (eday)]

lgtm

review: Approve

[lp:~sandy-walsh/nova/zones]

650. By Sandy Walsh on 2011-02-17

changed from 003-004 migration

[Matt Dietz (cerberus)]

lgtm.

review: Approve

[Matt Dietz (cerberus)]

Whoops, missed the NASA copyright in the migration file. Should be the Openstack one

review: Needs Fixing

[lp:~sandy-walsh/nova/zones]

651. By Sandy Walsh on 2011-02-17

copyright notice

[Matt Dietz (cerberus)]

again.

review: Approve

=== modified file 'nova/api/openstack/__init__.py'
--- nova/api/openstack/__init__.py	2011-02-16 20:12:54 +0000
+++ nova/api/openstack/__init__.py	2011-02-17 21:54:13 +0000
@@ -34,6 +34,7 @@
 from nova.api.openstack import images
 from nova.api.openstack import servers
 from nova.api.openstack import shared_ip_groups
+from nova.api.openstack import zones
 
 
 LOG = logging.getLogger('nova.api.openstack')
@@ -81,6 +82,9 @@
             server_members['resume'] = 'POST'
             server_members['reset_network'] = 'POST'
 
+            mapper.resource("zone", "zones", controller=zones.Controller(),
+                        collection={'detail': 'GET'})
+
         mapper.resource("server", "servers", controller=servers.Controller(),
                         collection={'detail': 'GET'},
                         member=server_members)
 
=== modified file 'nova/api/openstack/auth.py'
--- nova/api/openstack/auth.py	2011-01-06 18:57:48 +0000
+++ nova/api/openstack/auth.py	2011-02-17 21:54:13 +0000
@@ -19,6 +19,7 @@
 import hashlib
 import json
 import time
+import logging
 
 import webob.exc
 import webob.dec
 
=== modified file 'nova/api/openstack/servers.py'
--- nova/api/openstack/servers.py	2011-02-17 19:38:11 +0000
+++ nova/api/openstack/servers.py	2011-02-17 21:54:13 +0000
@@ -1,5 +1,3 @@
-# vim: tabstop=4 shiftwidth=4 softtabstop=4
-
 # Copyright 2010 OpenStack LLC.
 # All Rights Reserved.
 #
 
=== added file 'nova/api/openstack/zones.py'
--- nova/api/openstack/zones.py	1970-01-01 00:00:00 +0000
+++ nova/api/openstack/zones.py	2011-02-17 21:54:13 +0000
@@ -0,0 +1,80 @@
+# Copyright 2010 OpenStack LLC.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import common
+import logging
+
+from nova import flags
+from nova import wsgi
+from nova import db
+
+
+FLAGS = flags.FLAGS
+
+
+def _filter_keys(item, keys):
+    """
+    Filters all model attributes except for keys
+    item is a dict
+
+    """
+    return dict((k, v) for k, v in item.iteritems() if k in keys)
+
+
+def _scrub_zone(zone):
+    return _filter_keys(zone, ('id', 'api_url'))
+
+
+class Controller(wsgi.Controller):
+
+    _serialization_metadata = {
+        'application/xml': {
+            "attributes": {
+                "zone": ["id", "api_url"]}}}
+
+    def index(self, req):
+        """Return all zones in brief"""
+        items = db.zone_get_all(req.environ['nova.context'])
+        items = common.limited(items, req)
+        items = [_scrub_zone(item) for item in items]
+        return dict(zones=items)
+
+    def detail(self, req):
+        """Return all zones in detail"""
+        return self.index(req)
+
+    def show(self, req, id):
+        """Return data about the given zone id"""
+        zone_id = int(id)
+        zone = db.zone_get(req.environ['nova.context'], zone_id)
+        return dict(zone=_scrub_zone(zone))
+
+    def delete(self, req, id):
+        zone_id = int(id)
+        db.zone_delete(req.environ['nova.context'], zone_id)
+        return {}
+
+    def create(self, req):
+        context = req.environ['nova.context']
+        env = self._deserialize(req.body, req)
+        zone = db.zone_create(context, env["zone"])
+        return dict(zone=_scrub_zone(zone))
+
+    def update(self, req, id):
+        context = req.environ['nova.context']
+        env = self._deserialize(req.body, req)
+        zone_id = int(id)
+        zone = db.zone_update(context, zone_id, env["zone"])
+        return dict(zone=_scrub_zone(zone))
 
=== modified file 'nova/auth/novarc.template'
--- nova/auth/novarc.template	2011-01-04 01:06:50 +0000
+++ nova/auth/novarc.template	2011-02-17 21:54:13 +0000
@@ -10,7 +10,6 @@
 export EUCALYPTUS_CERT=${NOVA_CERT} # euca-bundle-image seems to require this set
 alias ec2-bundle-image="ec2-bundle-image --cert ${EC2_CERT} --privatekey ${EC2_PRIVATE_KEY} --user 42 --ec2cert ${NOVA_CERT}"
 alias ec2-upload-bundle="ec2-upload-bundle -a ${EC2_ACCESS_KEY} -s ${EC2_SECRET_KEY} --url ${S3_URL} --ec2cert ${NOVA_CERT}"
-export CLOUD_SERVERS_API_KEY="%(access)s"
-export CLOUD_SERVERS_USERNAME="%(user)s"
-export CLOUD_SERVERS_URL="%(os)s"
-
+export NOVA_API_KEY="%(access)s"
+export NOVA_USERNAME="%(user)s"
+export NOVA_URL="%(os)s"
 
=== modified file 'nova/db/api.py'
--- nova/db/api.py	2011-02-17 19:32:30 +0000
+++ nova/db/api.py	2011-02-17 21:54:13 +0000
@@ -1000,3 +1000,31 @@
 def console_get(context, console_id, instance_id=None):
     """Get a specific console (possibly on a given instance)."""
     return IMPL.console_get(context, console_id, instance_id)
+
+
+####################
+
+
+def zone_create(context, values):
+    """Create a new child Zone entry."""
+    return IMPL.zone_create(context, values)
+
+
+def zone_update(context, zone_id, values):
+    """Update a child Zone entry."""
+    return IMPL.zone_update(context, values)
+
+
+def zone_delete(context, zone_id):
+    """Delete a child Zone."""
+    return IMPL.zone_delete(context, zone_id)
+
+
+def zone_get(context, zone_id):
+    """Get a specific child Zone."""
+    return IMPL.zone_get(context, zone_id)
+
+
+def zone_get_all(context):
+    """Get all child Zones."""
+    return IMPL.zone_get_all(context)
 
=== modified file 'nova/db/sqlalchemy/api.py'
--- nova/db/sqlalchemy/api.py	2011-02-17 19:32:30 +0000
+++ nova/db/sqlalchemy/api.py	2011-02-17 21:54:13 +0000
@@ -2058,3 +2058,47 @@
         raise exception.NotFound(_("No console with id %(console_id)s"
                                    " %(idesc)s") % locals())
     return result
+
+
+####################
+
+
+@require_admin_context
+def zone_create(context, values):
+    zone = models.Zone()
+    zone.update(values)
+    zone.save()
+    return zone
+
+
+@require_admin_context
+def zone_update(context, zone_id, values):
+    zone = session.query(models.Zone).filter_by(id=zone_id).first()
+    if not zone:
+        raise exception.NotFound(_("No zone with id %(zone_id)s") % locals())
+    zone.update(values)
+    zone.save()
+    return zone
+
+
+@require_admin_context
+def zone_delete(context, zone_id):
+    session = get_session()
+    with session.begin():
+        session.execute('delete from zones '
+                        'where id=:id', {'id': zone_id})
+
+
+@require_admin_context
+def zone_get(context, zone_id):
+    session = get_session()
+    result = session.query(models.Zone).filter_by(id=zone_id).first()
+    if not result:
+        raise exception.NotFound(_("No zone with id %(zone_id)s") % locals())
+    return result
+
+
+@require_admin_context
+def zone_get_all(context):
+    session = get_session()
+    return session.query(models.Zone).all()
 
=== added file 'nova/db/sqlalchemy/migrate_repo/versions/004_add_zone_tables.py'
--- nova/db/sqlalchemy/migrate_repo/versions/004_add_zone_tables.py	1970-01-01 00:00:00 +0000
+++ nova/db/sqlalchemy/migrate_repo/versions/004_add_zone_tables.py	2011-02-17 21:54:13 +0000
@@ -0,0 +1,61 @@
+# Copyright 2010 OpenStack LLC.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from sqlalchemy import *
+from migrate import *
+
+from nova import log as logging
+
+
+meta = MetaData()
+
+
+#
+# New Tables
+#
+zones = Table('zones', meta,
+        Column('created_at', DateTime(timezone=False)),
+        Column('updated_at', DateTime(timezone=False)),
+        Column('deleted_at', DateTime(timezone=False)),
+        Column('deleted', Boolean(create_constraint=True, name=None)),
+        Column('id', Integer(),  primary_key=True, nullable=False),
+        Column('api_url',
+               String(length=255, convert_unicode=False, assert_unicode=None,
+                      unicode_error=None, _warn_on_bytestring=False)),
+        Column('username',
+               String(length=255, convert_unicode=False, assert_unicode=None,
+                      unicode_error=None, _warn_on_bytestring=False)),
+        Column('password',
+               String(length=255, convert_unicode=False, assert_unicode=None,
+                      unicode_error=None, _warn_on_bytestring=False)),
+        )
+
+
+#
+# Tables to alter
+#
+
+# (none currently)
+
+
+def upgrade(migrate_engine):
+    # Upgrade operations go here. Don't create your own engine;
+    # bind migrate_engine to your metadata
+    meta.bind = migrate_engine
+    for table in (zones, ):
+        try:
+            table.create()
+        except Exception:
+            logging.info(repr(table))
 
=== modified file 'nova/db/sqlalchemy/migration.py'
--- nova/db/sqlalchemy/migration.py	2011-02-16 18:24:46 +0000
+++ nova/db/sqlalchemy/migration.py	2011-02-17 21:54:13 +0000
@@ -55,8 +55,8 @@
         engine = sqlalchemy.create_engine(FLAGS.sql_connection, echo=False)
         meta.reflect(bind=engine)
         try:
-            for table in ('auth_tokens', 'export_devices', 'fixed_ips',
-                          'floating_ips', 'instances',
+            for table in ('auth_tokens', 'zones', 'export_devices',
+                          'fixed_ips', 'floating_ips', 'instances',
                           'key_pairs', 'networks', 'projects', 'quotas',
                           'security_group_instance_association',
                           'security_group_rules', 'security_groups',
 
=== modified file 'nova/db/sqlalchemy/models.py'
--- nova/db/sqlalchemy/models.py	2011-02-14 23:18:59 +0000
+++ nova/db/sqlalchemy/models.py	2011-02-17 21:54:13 +0000
@@ -536,6 +536,15 @@
     pool = relationship(ConsolePool, backref=backref('consoles'))
 
 
+class Zone(BASE, NovaBase):
+    """Represents a child zone of this zone."""
+    __tablename__ = 'zones'
+    id = Column(Integer, primary_key=True)
+    api_url = Column(String(255))
+    username = Column(String(255))
+    password = Column(String(255))
+
+
 def register_models():
     """Register Models and create metadata.
 
@@ -548,7 +557,7 @@
               Volume, ExportDevice, IscsiTarget, FixedIp, FloatingIp,
               Network, SecurityGroup, SecurityGroupIngressRule,
               SecurityGroupInstanceAssociation, AuthToken, User,
-              Project, Certificate, ConsolePool, Console)  # , Image, Host
+              Project, Certificate, ConsolePool, Console, Zone)
     engine = create_engine(FLAGS.sql_connection, echo=False)
     for model in models:
         model.metadata.create_all(engine)
 
=== added file 'nova/tests/api/openstack/test_zones.py'
--- nova/tests/api/openstack/test_zones.py	1970-01-01 00:00:00 +0000
+++ nova/tests/api/openstack/test_zones.py	2011-02-17 21:54:13 +0000
@@ -0,0 +1,141 @@
+# Copyright 2010 OpenStack LLC.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+import unittest
+
+import stubout
+import webob
+import json
+
+import nova.db
+from nova import context
+from nova import flags
+from nova.api.openstack import zones
+from nova.tests.api.openstack import fakes
+
+
+FLAGS = flags.FLAGS
+FLAGS.verbose = True
+
+
+def zone_get(context, zone_id):
+    return dict(id=1, api_url='http://foo.com', username='bob',
+                password='xxx')
+
+
+def zone_create(context, values):
+    zone = dict(id=1)
+    zone.update(values)
+    return zone
+
+
+def zone_update(context, zone_id, values):
+    zone = dict(id=zone_id, api_url='http://foo.com', username='bob',
+                password='xxx')
+    zone.update(values)
+    return zone
+
+
+def zone_delete(context, zone_id):
+    pass
+
+
+def zone_get_all(context):
+    return [
+        dict(id=1, api_url='http://foo.com', username='bob',
+                 password='xxx'),
+        dict(id=2, api_url='http://blah.com', username='alice',
+                 password='qwerty')
+    ]
+
+
+class ZonesTest(unittest.TestCase):
+    def setUp(self):
+        self.stubs = stubout.StubOutForTesting()
+        fakes.FakeAuthManager.auth_data = {}
+        fakes.FakeAuthDatabase.data = {}
+        fakes.stub_out_networking(self.stubs)
+        fakes.stub_out_rate_limiting(self.stubs)
+        fakes.stub_out_auth(self.stubs)
+
+        self.allow_admin = FLAGS.allow_admin_api
+        FLAGS.allow_admin_api = True
+
+        self.stubs.Set(nova.db, 'zone_get', zone_get)
+        self.stubs.Set(nova.db, 'zone_get_all', zone_get_all)
+        self.stubs.Set(nova.db, 'zone_update', zone_update)
+        self.stubs.Set(nova.db, 'zone_create', zone_create)
+        self.stubs.Set(nova.db, 'zone_delete', zone_delete)
+
+    def tearDown(self):
+        self.stubs.UnsetAll()
+        FLAGS.allow_admin_api = self.allow_admin
+
+    def test_get_zone_list(self):
+        req = webob.Request.blank('/v1.0/zones')
+        res = req.get_response(fakes.wsgi_app())
+        res_dict = json.loads(res.body)
+
+        self.assertEqual(res.status_int, 200)
+        self.assertEqual(len(res_dict['zones']), 2)
+
+    def test_get_zone_by_id(self):
+        req = webob.Request.blank('/v1.0/zones/1')
+        res = req.get_response(fakes.wsgi_app())
+        res_dict = json.loads(res.body)
+
+        self.assertEqual(res_dict['zone']['id'], 1)
+        self.assertEqual(res_dict['zone']['api_url'], 'http://foo.com')
+        self.assertFalse('password' in res_dict['zone'])
+        self.assertEqual(res.status_int, 200)
+
+    def test_zone_delete(self):
+        req = webob.Request.blank('/v1.0/zones/1')
+        res = req.get_response(fakes.wsgi_app())
+
+        self.assertEqual(res.status_int, 200)
+
+    def test_zone_create(self):
+        body = dict(zone=dict(api_url='http://blah.zoo', username='fred',
+                        password='fubar'))
+        req = webob.Request.blank('/v1.0/zones')
+        req.method = 'POST'
+        req.body = json.dumps(body)
+
+        res = req.get_response(fakes.wsgi_app())
+        res_dict = json.loads(res.body)
+
+        self.assertEqual(res.status_int, 200)
+        self.assertEqual(res_dict['zone']['id'], 1)
+        self.assertEqual(res_dict['zone']['api_url'], 'http://blah.zoo')
+        self.assertFalse('username' in res_dict['zone'])
+
+    def test_zone_update(self):
+        body = dict(zone=dict(username='zeb', password='sneaky'))
+        req = webob.Request.blank('/v1.0/zones/1')
+        req.method = 'PUT'
+        req.body = json.dumps(body)
+
+        res = req.get_response(fakes.wsgi_app())
+        res_dict = json.loads(res.body)
+
+        self.assertEqual(res.status_int, 200)
+        self.assertEqual(res_dict['zone']['id'], 1)
+        self.assertEqual(res_dict['zone']['api_url'], 'http://foo.com')
+        self.assertFalse('username' in res_dict['zone'])
+
+
+if __name__ == '__main__':
+    unittest.main()