Merge lp:~mdragon/nova/xs-console into lp:~hudson-openstack/nova/trunk

[Monsyne Dragon]

Implementation of xs-console blueprint (adds support for console proxies like xvp)

If you spin up the nova-console service, you should be able to see the xvp.conf being edited, and the xvp daemon started/stopped if you exercise the openstack console api (consoles sub-resource on servers)

[lp:~mdragon/nova/xs-console]

505. By Monsyne Dragon on 2011-01-06

pep8 fix

[Cory Wright (corywright)]

lgtm. Maybe update this at some point with some docstrings for the ConsoleAPI methods. otherwise everything looks good.

review: Approve

[Sandy Walsh (sandy-walsh)]

This is a just a source code review. I haven't tried to run the branch tests yet:

Overall, awesome work! Very impressive first commit!

nova/console/driver.py -> ConsoleProxy? Perhaps nova/console/proxy.py (since the class is truly a proxy and not a driver)

574 ... should there be debug code in here? Can't this be stubbed in the tests via stubout?

I think ConsolePool should be ConsoleProxyPool. It wasn't clear to me what a "console" was (since I always just viewed a console as an endpoint). A ConsoleProxy, however, makes more sense that it's a running entity.

I must be missing something, but I don't see where the ConsoleProxyManager is used? Other than via the tests. Yet, this is the only way to get a ConsoleProxy instantiated, afaik. Do you have any examples of the curl or python-cloudserver calls to add/remove a console?

I assume the proxies run on the console application server?

I have not reviewed the breadth of the tests yet. I'm not marking it Approved or Needs Fixing until you've had a chance to give feedback.

[Monsyne Dragon (mdragon)]

nova/console/driver.py ConsoleProxy class is the base class for the drivers (such as XVPConsoleProxy)

(I think it got that name because I was wavering between console and consoleproxy as the name for the service)

re: diff line 574: this is code used for unittests to eliminate the rpc call. THis is that same as how compute.manager uses the stub_network flag for tests.

Really, the 'console' is an entry in the config for the running xvp console proxy daemon.
the 'ConsolePool' represents a group of consoles that all live in the save (in this case xenserver) host.

the xvp daemon is the actual proxy (for multiple consoles)

ConsoleProxyManager is the main manager for the nova-console worker.
Ultimately the calls to add/remove consoles are made through the openstack api. (console is a sub-resource of server)

[Sandy Walsh (sandy-walsh)]

Change based on conversation with vishy: I assume the proxies run on the api servers?

[Monsyne Dragon (mdragon)]

> Change based on conversation with vishy: I assume the proxies run on the api
> servers?

No, the proxy runs on a console proxy server

[Sandy Walsh (sandy-walsh)]

Based on discussion with Dragon in IRC (thx again D):

1. The xcp daemon runs on the nova-console server and it assumed to have two NIC's: 1 public and 1 private. Clients obviously come in on the public one.

2. I think the flags for testing can be removed and stubout used. Dragon will have a look, but this works currently.

3. The Service fires up the Manager. API doesn't need to specify which manager.

4. The common pattern in Nova is:
 * Managers are Factories of Drivers.
 * Drivers are abstract Facades on something.
So calling driver.py proxy.py might add more confusion.

Approved code wise. Haven't had it running yet.

review: Approve

[Vish Ishaya (vishvananda)]

This should be a quick change, but the standard way we're doing apis in the other components is to name the class API and then import it in __init__.py. Your way is fine too, I just think that we should be consistent across components.

review: Needs Fixing

[lp:~mdragon/nova/xs-console]

506. By Monsyne Dragon on 2011-01-08

change API classname to match the way other API's are done.

[Monsyne Dragon (mdragon)]

> This should be a quick change, but the standard way we're doing apis in the
> other components is to name the class API and then import it in __init__.py.
> Your way is fine too, I just think that we should be consistent across
> components.

Ok, makes sense. I have pushed this change to the branch.

[lp:~mdragon/nova/xs-console]

507. By Monsyne Dragon on 2011-01-10

re-merged in trunk to correct conflict

[Matt Dietz (cerberus)]

I'm not sure I follow the utility of having the driver.ConsoleProxy class. It looks to be an abstract class of sorts, except in every subclassing you re-implement every method. I would understand the utility if it seemed a lot of the functionality could be passed down, but as is it seems to be an unnecessary contract. Am I missing something?

Otherwise seems ok to me.

review: Needs Fixing

[lp:~mdragon/nova/xs-console]

508. By Monsyne Dragon on 2011-01-10

remove uneeded superclass

509. By Monsyne Dragon on 2011-01-10

whups, fix accidental change to nova-combined

[Monsyne Dragon (mdragon)]

> I'm not sure I follow the utility of having the driver.ConsoleProxy class. It
> looks to be an abstract class of sorts, except in every subclassing you re-
> implement every method. I would understand the utility if it seemed a lot of
> the functionality could be passed down, but as is it seems to be an
> unnecessary contract. Am I missing something?

Nope. I was thinking there would be more common code, but not so. I have removed the driver.ConsoleProxy class. (leave that to future refactorings if need be, once we actually have more than 1 real console driver class.)

[Matt Dietz (cerberus)]

lgtm.

review: Approve

[Trey Morris (tr3buchet)]

long read! Looks good to me, except for a few places you used '\' for line breaks where you didn't have to from lines 833 to 908. However if it "needs fixing" is up to you.

review: Approve

=== modified file 'Authors'
--- Authors	2011-01-09 19:13:19 +0000
+++ Authors	2011-01-10 21:04:17 +0000
@@ -26,6 +26,7 @@
 Ken Pepple <[e-mail]>
 Matt Dietz <[e-mail]>
 Michael Gundlach <[e-mail]>
+Monsyne Dragon <[e-mail]>
 Monty Taylor <[e-mail]>
 Paul Voccio <[e-mail]>
 Rick Clark <[e-mail]>

=== added file 'bin/nova-console'
--- bin/nova-console	1970-01-01 00:00:00 +0000
+++ bin/nova-console	2011-01-10 21:04:17 +0000
@@ -0,0 +1,44 @@
+#!/usr/bin/env python
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright (c) 2010 Openstack, LLC.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""Starter script for Nova Console Proxy."""
+
+import eventlet
+eventlet.monkey_patch()
+
+import gettext
+import os
+import sys
+
+# If ../nova/__init__.py exists, add ../ to Python search path, so that
+# it will override what happens to be installed in /usr/(local/)lib/python...
+possible_topdir = os.path.normpath(os.path.join(os.path.abspath(sys.argv[0]),
+                                   os.pardir,
+                                   os.pardir))
+if os.path.exists(os.path.join(possible_topdir, 'nova', '__init__.py')):
+    sys.path.insert(0, possible_topdir)
+
+gettext.install('nova', unicode=1)
+
+from nova import service
+from nova import utils
+
+if __name__ == '__main__':
+    utils.default_flagfile()
+    service.serve()
+    service.wait()

=== modified file 'nova/api/openstack/__init__.py'
--- nova/api/openstack/__init__.py	2011-01-07 14:46:17 +0000
+++ nova/api/openstack/__init__.py	2011-01-10 21:04:17 +0000
@@ -31,6 +31,7 @@
 from nova import wsgi
 from nova.api.openstack import faults
 from nova.api.openstack import backup_schedules
+from nova.api.openstack import consoles
 from nova.api.openstack import flavors
 from nova.api.openstack import images
 from nova.api.openstack import servers
@@ -100,6 +101,11 @@
                         parent_resource=dict(member_name='server',
                         collection_name='servers'))
 
+        mapper.resource("console", "consoles",
+                        controller=consoles.Controller(),
+                        parent_resource=dict(member_name='server',
+                        collection_name='servers'))
+
         mapper.resource("image", "images", controller=images.Controller(),
                         collection={'detail': 'GET'})
         mapper.resource("flavor", "flavors", controller=flavors.Controller(),

=== added file 'nova/api/openstack/consoles.py'
--- nova/api/openstack/consoles.py	1970-01-01 00:00:00 +0000
+++ nova/api/openstack/consoles.py	2011-01-10 21:04:17 +0000
@@ -0,0 +1,96 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright 2010 OpenStack LLC.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+from webob import exc
+
+from nova import console
+from nova import exception
+from nova import wsgi
+from nova.api.openstack import faults
+
+
+def _translate_keys(cons):
+    """Coerces a console instance into proper dictionary format """
+    pool = cons['pool']
+    info = {'id': cons['id'],
+            'console_type': pool['console_type']}
+    return dict(console=info)
+
+
+def _translate_detail_keys(cons):
+    """Coerces a console instance into proper dictionary format with
+    correctly mapped attributes """
+    pool = cons['pool']
+    info = {'id': cons['id'],
+            'console_type': pool['console_type'],
+            'password': cons['password'],
+            'port': cons['port'],
+            'host': pool['public_hostname']}
+    return dict(console=info)
+
+
+class Controller(wsgi.Controller):
+    """The Consoles Controller for the Openstack API"""
+
+    _serialization_metadata = {
+        'application/xml': {
+            'attributes': {
+                'console': []}}}
+
+    def __init__(self):
+        self.console_api = console.API()
+        super(Controller, self).__init__()
+
+    def index(self, req, server_id):
+        """Returns a list of consoles for this instance"""
+        consoles = self.console_api.get_consoles(
+                                    req.environ['nova.context'],
+                                    int(server_id))
+        return dict(consoles=[_translate_keys(console)
+                              for console in consoles])
+
+    def create(self, req, server_id):
+        """Creates a new console"""
+        #info = self._deserialize(req.body, req)
+        self.console_api.create_console(
+                                req.environ['nova.context'],
+                                int(server_id))
+
+    def show(self, req, server_id, id):
+        """Shows in-depth information on a specific console"""
+        try:
+            console = self.console_api.get_console(
+                                        req.environ['nova.context'],
+                                        int(server_id),
+                                        int(id))
+        except exception.NotFound:
+            return faults.Fault(exc.HTTPNotFound())
+        return _translate_detail_keys(console)
+
+    def update(self, req, server_id, id):
+        """You can't update a console"""
+        raise faults.Fault(exc.HTTPNotImplemented())
+
+    def delete(self, req, server_id, id):
+        """Deletes a console"""
+        try:
+            self.console_api.delete_console(req.environ['nova.context'],
+                                            int(server_id),
+                                            int(id))
+        except exception.NotFound:
+            return faults.Fault(exc.HTTPNotFound())
+        return exc.HTTPAccepted()

=== modified file 'nova/compute/manager.py'
--- nova/compute/manager.py	2011-01-08 14:35:50 +0000
+++ nova/compute/manager.py	2011-01-10 21:04:17 +0000
@@ -35,6 +35,8 @@
 """
 
 import datetime
+import logging
+import socket
 import functools
 
 from nova import exception
@@ -52,6 +54,9 @@
                     'Driver to use for controlling virtualization')
 flags.DEFINE_string('stub_network', False,
                     'Stub network related code')
+flags.DEFINE_string('console_host', socket.gethostname(),
+                    'Console proxy host to use to connect to instances on'
+                    'this host.')
 
 LOG = logging.getLogger('nova.compute.manager')
 
@@ -122,6 +127,15 @@
             state = power_state.NOSTATE
         self.db.instance_set_state(context, instance_id, state)
 
+    def get_console_topic(self, context, **_kwargs):
+        """Retrieves the console host for a project on this host
+           Currently this is just set in the flags for each compute
+           host."""
+        #TODO(mdragon): perhaps make this variable by console_type?
+        return self.db.queue_get_for(context,
+                                     FLAGS.console_topic,
+                                     FLAGS.console_host)
+
     def get_network_topic(self, context, **_kwargs):
         """Retrieves the network host for a project on this host"""
         # TODO(vish): This method should be memoized. This will make
@@ -136,6 +150,9 @@
                                      FLAGS.network_topic,
                                      host)
 
+    def get_console_pool_info(self, context, console_type):
+        return self.driver.get_console_pool_info(console_type)
+
     @exception.wrap_exception
     def refresh_security_group_rules(self, context,
                                      security_group_id, **_kwargs):

=== added directory 'nova/console'
=== added file 'nova/console/__init__.py'
--- nova/console/__init__.py	1970-01-01 00:00:00 +0000
+++ nova/console/__init__.py	2011-01-10 21:04:17 +0000
@@ -0,0 +1,13 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+"""
+:mod:`nova.console` -- Console Prxy to set up VM console access (i.e. with xvp)
+=====================================================
+
+.. automodule:: nova.console
+   :platform: Unix
+   :synopsis: Wrapper around console proxies such as xvp to set up
+              multitenant VM console access
+.. moduleauthor:: Monsyne Dragon <[e-mail]>
+"""
+from nova.console.api import API

=== added file 'nova/console/api.py'
--- nova/console/api.py	1970-01-01 00:00:00 +0000
+++ nova/console/api.py	2011-01-10 21:04:17 +0000
@@ -0,0 +1,75 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright (c) 2010 Openstack, LLC.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+Handles ConsoleProxy API requests
+"""
+
+from nova import exception
+from nova.db import base
+
+
+from nova import flags
+from nova import rpc
+
+
+FLAGS = flags.FLAGS
+
+
+class API(base.Base):
+    """API for spining up or down console proxy connections"""
+
+    def __init__(self, **kwargs):
+        super(API, self).__init__(**kwargs)
+
+    def get_consoles(self, context, instance_id):
+        return self.db.console_get_all_by_instance(context, instance_id)
+
+    def get_console(self, context, instance_id, console_id):
+        return self.db.console_get(context, console_id, instance_id)
+
+    def delete_console(self, context, instance_id, console_id):
+        console = self.db.console_get(context,
+                                      console_id,
+                                      instance_id)
+        pool = console['pool']
+        rpc.cast(context,
+                 self.db.queue_get_for(context,
+                                       FLAGS.console_topic,
+                                       pool['host']),
+                 {"method": "remove_console",
+                  "args": {"console_id": console['id']}})
+
+    def create_console(self, context, instance_id):
+        instance = self.db.instance_get(context, instance_id)
+        #NOTE(mdragon): If we wanted to return this the console info
+        #               here, as we would need to do a call.
+        #               They can just do an index later to fetch
+        #               console info. I am not sure which is better
+        #               here.
+        rpc.cast(context,
+                 self._get_console_topic(context, instance['host']),
+                 {"method": "add_console",
+                  "args": {"instance_id": instance_id}})
+
+    def _get_console_topic(self, context, instance_host):
+        topic = self.db.queue_get_for(context,
+                                      FLAGS.compute_topic,
+                                      instance_host)
+        return rpc.call(context,
+                        topic,
+                        {"method": "get_console_topic", "args": {'fake': 1}})

=== added file 'nova/console/fake.py'
--- nova/console/fake.py	1970-01-01 00:00:00 +0000
+++ nova/console/fake.py	2011-01-10 21:04:17 +0000
@@ -0,0 +1,58 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright (c) 2010 Openstack, LLC.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+Fake ConsoleProxy driver for tests.
+"""
+
+from nova import exception
+
+
+class FakeConsoleProxy(object):
+    """Fake ConsoleProxy driver."""
+
+    @property
+    def console_type(self):
+        return "fake"
+
+    def setup_console(self, context, console):
+        """Sets up actual proxies"""
+        pass
+
+    def teardown_console(self, context, console):
+        """Tears down actual proxies"""
+        pass
+
+    def init_host(self):
+        """Start up any config'ed consoles on start"""
+        pass
+
+    def generate_password(self, length=8):
+        """Returns random console password"""
+        return "fakepass"
+
+    def get_port(self, context):
+        """get available port for consoles that need one"""
+        return 5999
+
+    def fix_pool_password(self, password):
+        """Trim password to length, and any other massaging"""
+        return password
+
+    def fix_console_password(self, password):
+        """Trim password to length, and any other massaging"""
+        return password

=== added file 'nova/console/manager.py'
--- nova/console/manager.py	1970-01-01 00:00:00 +0000
+++ nova/console/manager.py	2011-01-10 21:04:17 +0000
@@ -0,0 +1,127 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright (c) 2010 Openstack, LLC.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+Console Proxy Service
+"""
+
+import functools
+import logging
+import socket
+
+from nova import exception
+from nova import flags
+from nova import manager
+from nova import rpc
+from nova import utils
+
+FLAGS = flags.FLAGS
+flags.DEFINE_string('console_driver',
+                    'nova.console.xvp.XVPConsoleProxy',
+                    'Driver to use for the console proxy')
+flags.DEFINE_boolean('stub_compute', False,
+                     'Stub calls to compute worker for tests')
+flags.DEFINE_string('console_public_hostname',
+                    socket.gethostname(),
+                    'Publicly visable name for this console host')
+
+
+class ConsoleProxyManager(manager.Manager):
+
+    """ Sets up and tears down any proxy connections needed for accessing
+        instance consoles securely"""
+
+    def __init__(self, console_driver=None, *args, **kwargs):
+        if not console_driver:
+            console_driver = FLAGS.console_driver
+        self.driver = utils.import_object(console_driver)
+        super(ConsoleProxyManager, self).__init__(*args, **kwargs)
+        self.driver.host = self.host
+
+    def init_host(self):
+        self.driver.init_host()
+
+    @exception.wrap_exception
+    def add_console(self, context, instance_id, password=None,
+                    port=None, **kwargs):
+        instance = self.db.instance_get(context, instance_id)
+        host = instance['host']
+        name = instance['name']
+        pool = self.get_pool_for_instance_host(context, host)
+        try:
+            console = self.db.console_get_by_pool_instance(context,
+                                                      pool['id'],
+                                                      instance_id)
+        except exception.NotFound:
+            logging.debug("Adding console")
+            if not password:
+                password = self.driver.generate_password()
+            if not port:
+                port = self.driver.get_port(context)
+            console_data = {'instance_name': name,
+                            'instance_id': instance_id,
+                            'password': password,
+                            'pool_id': pool['id']}
+            if port:
+                console_data['port'] = port
+            console = self.db.console_create(context, console_data)
+            self.driver.setup_console(context, console)
+        return console['id']
+
+    @exception.wrap_exception
+    def remove_console(self, context, console_id, **_kwargs):
+        try:
+            console = self.db.console_get(context, console_id)
+        except exception.NotFound:
+            logging.debug(_('Tried to remove non-existant console '
+                            '%(console_id)s.') %
+                            {'console_id': console_id})
+            return
+        self.db.console_delete(context, console_id)
+        self.driver.teardown_console(context, console)
+
+    def get_pool_for_instance_host(self, context, instance_host):
+        context = context.elevated()
+        console_type = self.driver.console_type
+        try:
+            pool = self.db.console_pool_get_by_host_type(context,
+                                                         instance_host,
+                                                         self.host,
+                                                         console_type)
+        except exception.NotFound:
+            #NOTE(mdragon): Right now, the only place this info exists is the
+            #               compute worker's flagfile, at least for
+            #               xenserver. Thus we ned to ask.
+            if FLAGS.stub_compute:
+                pool_info = {'address': '127.0.0.1',
+                             'username': 'test',
+                             'password': '1234pass'}
+            else:
+                pool_info = rpc.call(context,
+                                 self.db.queue_get_for(context,
+                                                   FLAGS.compute_topic,
+                                                   instance_host),
+                       {"method": "get_console_pool_info",
+                        "args": {"console_type": console_type}})
+            pool_info['password'] = self.driver.fix_pool_password(
+                                                    pool_info['password'])
+            pool_info['host'] = self.host
+            pool_info['public_hostname'] = FLAGS.console_public_hostname
+            pool_info['console_type'] = self.driver.console_type
+            pool_info['compute_host'] = instance_host
+            pool = self.db.console_pool_create(context, pool_info)
+        return pool

=== added file 'nova/console/xvp.conf.template'
--- nova/console/xvp.conf.template	1970-01-01 00:00:00 +0000
+++ nova/console/xvp.conf.template	2011-01-10 21:04:17 +0000
@@ -0,0 +1,16 @@
+# One time password use with time window
+OTP ALLOW IPCHECK HTTP 60
+#if $multiplex_port
+MULTIPLEX $multiplex_port 
+#end if
+
+#for $pool in $pools
+POOL $pool.address
+  DOMAIN $pool.address
+  MANAGER root $pool.password 
+  HOST $pool.address
+    VM - dummy 0123456789ABCDEF
+  #for $console in $pool.consoles
+    VM #if $multiplex_port then '-' else $console.port # $console.instance_name $pass_encode($console.password)
+  #end for 
+#end for 

=== added file 'nova/console/xvp.py'
--- nova/console/xvp.py	1970-01-01 00:00:00 +0000
+++ nova/console/xvp.py	2011-01-10 21:04:17 +0000
@@ -0,0 +1,194 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright (c) 2010 Openstack, LLC.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+XVP (Xenserver VNC Proxy) driver.
+"""
+
+import fcntl
+import logging
+import os
+import signal
+import subprocess
+
+from Cheetah.Template import Template
+
+from nova import context
+from nova import db
+from nova import exception
+from nova import flags
+from nova import utils
+
+flags.DEFINE_string('console_xvp_conf_template',
+                    utils.abspath('console/xvp.conf.template'),
+                    'XVP conf template')
+flags.DEFINE_string('console_xvp_conf',
+                    '/etc/xvp.conf',
+                    'generated XVP conf file')
+flags.DEFINE_string('console_xvp_pid',
+                    '/var/run/xvp.pid',
+                    'XVP master process pid file')
+flags.DEFINE_string('console_xvp_log',
+                    '/var/log/xvp.log',
+                    'XVP log file')
+flags.DEFINE_integer('console_xvp_multiplex_port',
+                     5900,
+                     "port for XVP to multiplex VNC connections on")
+FLAGS = flags.FLAGS
+
+
+class XVPConsoleProxy(object):
+    """Sets up XVP config, and manages xvp daemon"""
+
+    def __init__(self):
+        self.xvpconf_template = open(FLAGS.console_xvp_conf_template).read()
+        self.host = FLAGS.host  # default, set by manager.
+        super(XVPConsoleProxy, self).__init__()
+
+    @property
+    def console_type(self):
+        return "vnc+xvp"
+
+    def get_port(self, context):
+        """get available port for consoles that need one"""
+        #TODO(mdragon): implement port selection for non multiplex ports,
+        #               we are not using that, but someone else may want
+        #               it.
+        return FLAGS.console_xvp_multiplex_port
+
+    def setup_console(self, context, console):
+        """Sets up actual proxies"""
+        self._rebuild_xvp_conf(context.elevated())
+
+    def teardown_console(self, context, console):
+        """Tears down actual proxies"""
+        self._rebuild_xvp_conf(context.elevated())
+
+    def init_host(self):
+        """Start up any config'ed consoles on start"""
+        ctxt = context.get_admin_context()
+        self._rebuild_xvp_conf(ctxt)
+
+    def fix_pool_password(self, password):
+        """Trim password to length, and encode"""
+        return self._xvp_encrypt(password, is_pool_password=True)
+
+    def fix_console_password(self, password):
+        """Trim password to length, and encode"""
+        return self._xvp_encrypt(password)
+
+    def generate_password(self, length=8):
+        """Returns random console password"""
+        return os.urandom(length * 2).encode('base64')[:length]
+
+    def _rebuild_xvp_conf(self, context):
+        logging.debug("Rebuilding xvp conf")
+        pools = [pool for pool in
+                 db.console_pool_get_all_by_host_type(context, self.host,
+                                                       self.console_type)
+                  if pool['consoles']]
+        if not pools:
+            logging.debug("No console pools!")
+            self._xvp_stop()
+            return
+        conf_data = {'multiplex_port': FLAGS.console_xvp_multiplex_port,
+                     'pools': pools,
+                     'pass_encode': self.fix_console_password}
+        config = str(Template(self.xvpconf_template, searchList=[conf_data]))
+        self._write_conf(config)
+        self._xvp_restart()
+
+    def _write_conf(self, config):
+        logging.debug('Re-wrote %s' % FLAGS.console_xvp_conf)
+        with open(FLAGS.console_xvp_conf, 'w') as cfile:
+            cfile.write(config)
+
+    def _xvp_stop(self):
+        logging.debug("Stopping xvp")
+        pid = self._xvp_pid()
+        if not pid:
+            return
+        try:
+            os.kill(pid, signal.SIGTERM)
+        except OSError:
+            #if it's already not running, no problem.
+            pass
+
+    def _xvp_start(self):
+        if self._xvp_check_running():
+            return
+        logging.debug("Starting xvp")
+        try:
+            utils.execute('xvp -p %s -c %s -l %s' %
+                          (FLAGS.console_xvp_pid,
+                           FLAGS.console_xvp_conf,
+                           FLAGS.console_xvp_log))
+        except exception.ProcessExecutionError, err:
+            logging.error("Error starting xvp: %s" % err)
+
+    def _xvp_restart(self):
+        logging.debug("Restarting xvp")
+        if not self._xvp_check_running():
+            logging.debug("xvp not running...")
+            self._xvp_start()
+        else:
+            pid = self._xvp_pid()
+            os.kill(pid, signal.SIGUSR1)
+
+    def _xvp_pid(self):
+        try:
+            with open(FLAGS.console_xvp_pid, 'r') as pidfile:
+                pid = int(pidfile.read())
+        except IOError:
+            return None
+        except ValueError:
+            return None
+        return pid
+
+    def _xvp_check_running(self):
+        pid = self._xvp_pid()
+        if not pid:
+            return False
+        try:
+            os.kill(pid, 0)
+        except OSError:
+            return False
+        return True
+
+    def _xvp_encrypt(self, password, is_pool_password=False):
+        """Call xvp to obfuscate passwords for config file.
+
+        Args:
+            - password: the password to encode, max 8 char for vm passwords,
+                        and 16 chars for pool passwords. passwords will
+                        be trimmed to max len before encoding.
+            - is_pool_password: True if this this is the XenServer api password
+                                False if it's a VM console password
+              (xvp uses different keys and max lengths for pool passwords)
+
+        Note that xvp's obfuscation should not be considered 'real' encryption.
+        It simply DES encrypts the passwords with static keys plainly viewable
+        in the xvp source code."""
+        maxlen = 8
+        flag = '-e'
+        if is_pool_password:
+            maxlen = 16
+            flag = '-x'
+        #xvp will blow up on passwords that are too long (mdragon)
+        password = password[:maxlen]
+        out, err = utils.execute('xvp %s' % flag, process_input=password)
+        return out.strip()

=== modified file 'nova/db/api.py'
--- nova/db/api.py	2011-01-10 15:57:13 +0000
+++ nova/db/api.py	2011-01-10 21:04:17 +0000
@@ -906,3 +906,57 @@
 
     """
     return IMPL.host_get_networks(context, host)
+
+
+##################
+
+
+def console_pool_create(context, values):
+    """Create console pool."""
+    return IMPL.console_pool_create(context, values)
+
+
+def console_pool_get(context, pool_id):
+    """Get a console pool."""
+    return IMPL.console_pool_get(context, pool_id)
+
+
+def console_pool_get_by_host_type(context, compute_host, proxy_host,
+                                  console_type):
+    """Fetch a console pool for a given proxy host, compute host, and type."""
+    return IMPL.console_pool_get_by_host_type(context,
+                                              compute_host,
+                                              proxy_host,
+                                              console_type)
+
+
+def console_pool_get_all_by_host_type(context, host, console_type):
+    """Fetch all pools for given proxy host and type."""
+    return IMPL.console_pool_get_all_by_host_type(context,
+                                                  host,
+                                                  console_type)
+
+
+def console_create(context, values):
+    """Create a console."""
+    return IMPL.console_create(context, values)
+
+
+def console_delete(context, console_id):
+    """Delete a console."""
+    return IMPL.console_delete(context, console_id)
+
+
+def console_get_by_pool_instance(context, pool_id, instance_id):
+    """Get console entry for a given instance and pool."""
+    return IMPL.console_get_by_pool_instance(context, pool_id, instance_id)
+
+
+def console_get_all_by_instance(context, instance_id):
+    """Get consoles for a given instance."""
+    return IMPL.console_get_all_by_instance(context, instance_id)
+
+
+def console_get(context, console_id, instance_id=None):
+    """Get a specific console (possibly on a given instance)."""
+    return IMPL.console_get(context, console_id, instance_id)

=== modified file 'nova/db/sqlalchemy/api.py'
--- nova/db/sqlalchemy/api.py	2011-01-10 15:57:13 +0000
+++ nova/db/sqlalchemy/api.py	2011-01-10 21:04:17 +0000
@@ -1863,3 +1863,111 @@
                        filter_by(deleted=False).\
                        filter_by(host=host).\
                        all()
+
+
+##################
+
+
+def console_pool_create(context, values):
+    pool = models.ConsolePool()
+    pool.update(values)
+    pool.save()
+    return pool
+
+
+def console_pool_get(context, pool_id):
+    session = get_session()
+    result = session.query(models.ConsolePool).\
+                     filter_by(deleted=False).\
+                     filter_by(id=pool_id).\
+                     first()
+    if not result:
+        raise exception.NotFound(_("No console pool with id %(pool_id)s") %
+                                 {'pool_id': pool_id})
+
+    return result
+
+
+def console_pool_get_by_host_type(context, compute_host, host,
+                                  console_type):
+    session = get_session()
+    result = session.query(models.ConsolePool).\
+                   filter_by(host=host).\
+                   filter_by(console_type=console_type).\
+                   filter_by(compute_host=compute_host).\
+                   filter_by(deleted=False).\
+                   options(joinedload('consoles')).\
+                   first()
+    if not result:
+        raise exception.NotFound(_('No console pool of type %(type)s '
+                                   'for compute host %(compute_host)s '
+                                   'on proxy host %(host)s') %
+                                   {'type': console_type,
+                                    'compute_host': compute_host,
+                                    'host': host})
+    return result
+
+
+def console_pool_get_all_by_host_type(context, host, console_type):
+    session = get_session()
+    return session.query(models.ConsolePool).\
+                   filter_by(host=host).\
+                   filter_by(console_type=console_type).\
+                   filter_by(deleted=False).\
+                   options(joinedload('consoles')).\
+                   all()
+
+
+def console_create(context, values):
+    console = models.Console()
+    console.update(values)
+    console.save()
+    return console
+
+
+def console_delete(context, console_id):
+    session = get_session()
+    with session.begin():
+        # consoles are meant to be transient. (mdragon)
+        session.execute('delete from consoles '
+                        'where id=:id', {'id': console_id})
+
+
+def console_get_by_pool_instance(context, pool_id, instance_id):
+    session = get_session()
+    result = session.query(models.Console).\
+                   filter_by(pool_id=pool_id).\
+                   filter_by(instance_id=instance_id).\
+                   options(joinedload('pool')).\
+                   first()
+    if not result:
+        raise exception.NotFound(_('No console for instance %(instance_id)s '
+                                 'in pool %(pool_id)s') %
+                                 {'instance_id': instance_id,
+                                  'pool_id': pool_id})
+    return result
+
+
+def console_get_all_by_instance(context, instance_id):
+    session = get_session()
+    results = session.query(models.Console).\
+                   filter_by(instance_id=instance_id).\
+                   options(joinedload('pool')).\
+                   all()
+    return results
+
+
+def console_get(context, console_id, instance_id=None):
+    session = get_session()
+    query = session.query(models.Console).\
+                    filter_by(id=console_id)
+    if instance_id:
+        query = query.filter_by(instance_id=instance_id)
+    result = query.options(joinedload('pool')).first()
+    if not result:
+        idesc = (_("on instance %s") % instance_id)  if instance_id else ""
+        raise exception.NotFound(_("No console with id %(console_id)s"
+                                   " %(instance)s") %
+                                  {'instance': idesc,
+                                  'console_id': console_id})
+    return result

=== modified file 'nova/db/sqlalchemy/models.py'
--- nova/db/sqlalchemy/models.py	2011-01-06 22:35:48 +0000
+++ nova/db/sqlalchemy/models.py	2011-01-10 21:04:17 +0000
@@ -540,6 +540,31 @@
     host = Column(String(255))  # , ForeignKey('hosts.id'))
 
 
+class ConsolePool(BASE, NovaBase):
+    """Represents pool of consoles on the same physical node."""
+    __tablename__ = 'console_pools'
+    id = Column(Integer, primary_key=True)
+    address = Column(String(255))
+    username = Column(String(255))
+    password = Column(String(255))
+    console_type = Column(String(255))
+    public_hostname = Column(String(255))
+    host = Column(String(255))
+    compute_host = Column(String(255))
+
+
+class Console(BASE, NovaBase):
+    """Represents a console session for an instance."""
+    __tablename__ = 'consoles'
+    id = Column(Integer, primary_key=True)
+    instance_name = Column(String(255))
+    instance_id = Column(Integer)
+    password = Column(String(255))
+    port = Column(Integer, nullable=True)
+    pool_id = Column(Integer, ForeignKey('console_pools.id'))
+    pool = relationship(ConsolePool, backref=backref('consoles'))
+
+
 def register_models():
     """Register Models and create metadata.
 
@@ -552,7 +577,7 @@
               Volume, ExportDevice, IscsiTarget, FixedIp, FloatingIp,
               Network, SecurityGroup, SecurityGroupIngressRule,
               SecurityGroupInstanceAssociation, AuthToken, User,
-              Project, Certificate)  # , Image, Host
+              Project, Certificate, ConsolePool, Console)  # , Image, Host
     engine = create_engine(FLAGS.sql_connection, echo=False)
     for model in models:
         model.metadata.create_all(engine)

=== modified file 'nova/flags.py'
--- nova/flags.py	2011-01-04 05:23:35 +0000
+++ nova/flags.py	2011-01-10 21:04:17 +0000
@@ -216,6 +216,8 @@
 DEFINE_string('s3_host', '127.0.0.1', 's3 host (for infrastructure)')
 DEFINE_string('s3_dmz', '127.0.0.1', 's3 dmz ip (for instances)')
 DEFINE_string('compute_topic', 'compute', 'the topic compute nodes listen on')
+DEFINE_string('console_topic', 'console',
+              'the topic console proxy nodes listen on')
 DEFINE_string('scheduler_topic', 'scheduler',
               'the topic scheduler nodes listen on')
 DEFINE_string('volume_topic', 'volume', 'the topic volume nodes listen on')
@@ -269,6 +271,8 @@
 
 DEFINE_string('compute_manager', 'nova.compute.manager.ComputeManager',
               'Manager for compute')
+DEFINE_string('console_manager', 'nova.console.manager.ConsoleProxyManager',
+              'Manager for console proxy')
 DEFINE_string('network_manager', 'nova.network.manager.VlanManager',
               'Manager for network')
 DEFINE_string('volume_manager', 'nova.volume.manager.VolumeManager',

=== added file 'nova/tests/test_console.py'
--- nova/tests/test_console.py	1970-01-01 00:00:00 +0000
+++ nova/tests/test_console.py	2011-01-10 21:04:17 +0000
@@ -0,0 +1,129 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright (c) 2010 Openstack, LLC.
+# Administrator of the National Aeronautics and Space Administration.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+Tests For Console proxy.
+"""
+
+import datetime
+import logging
+
+from nova import context
+from nova import db
+from nova import exception
+from nova import flags
+from nova import test
+from nova import utils
+from nova.auth import manager
+from nova.console import manager as console_manager
+
+FLAGS = flags.FLAGS
+
+
+class ConsoleTestCase(test.TestCase):
+    """Test case for console proxy"""
+    def setUp(self):
+        logging.getLogger().setLevel(logging.DEBUG)
+        super(ConsoleTestCase, self).setUp()
+        self.flags(console_driver='nova.console.fake.FakeConsoleProxy',
+                   stub_compute=True)
+        self.console = utils.import_object(FLAGS.console_manager)
+        self.manager = manager.AuthManager()
+        self.user = self.manager.create_user('fake', 'fake', 'fake')
+        self.project = self.manager.create_project('fake', 'fake', 'fake')
+        self.context = context.get_admin_context()
+        self.host = 'test_compute_host'
+
+    def tearDown(self):
+        self.manager.delete_user(self.user)
+        self.manager.delete_project(self.project)
+        super(ConsoleTestCase, self).tearDown()
+
+    def _create_instance(self):
+        """Create a test instance"""
+        inst = {}
+        #inst['host'] = self.host
+        #inst['name'] = 'instance-1234'
+        inst['image_id'] = 'ami-test'
+        inst['reservation_id'] = 'r-fakeres'
+        inst['launch_time'] = '10'
+        inst['user_id'] = self.user.id
+        inst['project_id'] = self.project.id
+        inst['instance_type'] = 'm1.tiny'
+        inst['mac_address'] = utils.generate_mac()
+        inst['ami_launch_index'] = 0
+        return db.instance_create(self.context, inst)['id']
+
+    def test_get_pool_for_instance_host(self):
+        pool = self.console.get_pool_for_instance_host(self.context, self.host)
+        self.assertEqual(pool['compute_host'], self.host)
+
+    def test_get_pool_creates_new_pool_if_needed(self):
+        self.assertRaises(exception.NotFound,
+                          db.console_pool_get_by_host_type,
+                          self.context,
+                          self.host,
+                          self.console.host,
+                          self.console.driver.console_type)
+        pool = self.console.get_pool_for_instance_host(self.context,
+                                                       self.host)
+        pool2 = db.console_pool_get_by_host_type(self.context,
+                              self.host,
+                              self.console.host,
+                              self.console.driver.console_type)
+        self.assertEqual(pool['id'], pool2['id'])
+
+    def test_get_pool_does_not_create_new_pool_if_exists(self):
+        pool_info = {'address': '127.0.0.1',
+                     'username': 'test',
+                     'password': '1234pass',
+                     'host': self.console.host,
+                     'console_type': self.console.driver.console_type,
+                     'compute_host': 'sometesthostname'}
+        new_pool = db.console_pool_create(self.context, pool_info)
+        pool = self.console.get_pool_for_instance_host(self.context,
+                                                       'sometesthostname')
+        self.assertEqual(pool['id'], new_pool['id'])
+
+    def test_add_console(self):
+        instance_id = self._create_instance()
+        self.console.add_console(self.context, instance_id)
+        instance = db.instance_get(self.context, instance_id)
+        pool = db.console_pool_get_by_host_type(self.context,
+                                                instance['host'],
+                                                self.console.host,
+                                            self.console.driver.console_type)
+
+        console_instances = [con['instance_id'] for con in pool.consoles]
+        self.assert_(instance_id in console_instances)
+
+    def test_add_console_does_not_duplicate(self):
+        instance_id = self._create_instance()
+        cons1 = self.console.add_console(self.context, instance_id)
+        cons2 = self.console.add_console(self.context, instance_id)
+        self.assertEqual(cons1, cons2)
+
+    def test_remove_console(self):
+        instance_id = self._create_instance()
+        console_id = self.console.add_console(self.context, instance_id)
+        self.console.remove_console(self.context, console_id)
+
+        self.assertRaises(exception.NotFound,
+                          db.console_get,
+                          self.context,
+                          console_id)

=== modified file 'nova/tests/test_virt.py'
--- nova/tests/test_virt.py	2011-01-10 10:32:17 +0000
+++ nova/tests/test_virt.py	2011-01-10 21:04:17 +0000
@@ -249,7 +249,7 @@
       '-A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable ',
       '-A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable ',
       'COMMIT',
-      '# Completed on Mon Dec  6 11:54:13 2010'
+      '# Completed on Mon Dec  6 11:54:13 2010',
     ]
 
     def test_static_filters(self):

=== modified file 'nova/virt/fake.py'
--- nova/virt/fake.py	2010-12-30 21:23:14 +0000
+++ nova/virt/fake.py	2011-01-10 21:04:17 +0000
@@ -289,6 +289,11 @@
     def get_console_output(self, instance):
         return 'FAKE CONSOLE OUTPUT'
 
+    def get_console_pool_info(self, console_type):
+        return  {'address': '127.0.0.1',
+                 'username': 'fakeuser',
+                 'password': 'fakepassword'}
+
 
 class FakeInstance(object):
 

=== modified file 'nova/virt/hyperv.py'
--- nova/virt/hyperv.py	2011-01-07 14:46:17 +0000
+++ nova/virt/hyperv.py	2011-01-10 21:04:17 +0000
@@ -92,7 +92,7 @@
     'Reboot': 10,
     'Reset': 11,
     'Paused': 32768,
-    'Suspended': 32769
+    'Suspended': 32769,
 }
 
 

=== modified file 'nova/virt/libvirt_conn.py'
--- nova/virt/libvirt_conn.py	2011-01-08 14:35:50 +0000
+++ nova/virt/libvirt_conn.py	2011-01-10 21:04:17 +0000
@@ -707,6 +707,14 @@
         domain = self._conn.lookupByName(instance_name)
         return domain.interfaceStats(interface)
 
+    def get_console_pool_info(self, console_type):
+        #TODO(mdragon): console proxy should be implemented for libvirt,
+        #               in case someone wants to use it with kvm or
+        #               such. For now return fake data.
+        return  {'address': '127.0.0.1',
+                 'username': 'fakeuser',
+                 'password': 'fakepassword'}
+
     def refresh_security_group_rules(self, security_group_id):
         self.firewall_driver.refresh_security_group_rules(security_group_id)
 

=== modified file 'nova/virt/xenapi_conn.py'
--- nova/virt/xenapi_conn.py	2011-01-07 14:46:17 +0000
+++ nova/virt/xenapi_conn.py	2011-01-10 21:04:17 +0000
@@ -52,6 +52,7 @@
 """
 
 import sys
+import urlparse
 import xmlrpclib
 
 from eventlet import event
@@ -190,6 +191,12 @@
         """Detach volume storage to VM instance"""
         return self._volumeops.detach_volume(instance_name, mountpoint)
 
+    def get_console_pool_info(self, console_type):
+        xs_url = urlparse.urlparse(FLAGS.xenapi_connection_url)
+        return  {'address': xs_url.netloc,
+                 'username': FLAGS.xenapi_connection_username,
+                 'password': FLAGS.xenapi_connection_password}
+
 
 class XenAPISession(object):
     """The session to invoke XenAPI SDK calls"""