Merge lp:~nii/nova/nova-deployment-tool into lp:~hudson-openstack/nova/trunk

[Shigetoshi Yokoyama]

First Merge proposal to trunk. I add a directory in the contrib directory, named nova-deployment-tool2011. That is all I would like to contribute.

Tentative Manual will be uploaded within a week.

[Todd Willey (xtoddx)]

The files that have a copyright header probably also need the standard apache license header used throughout the project.

[Soren Hansen (soren)]

By accepting this into trunk, it's implied that it will be kept up-to-date, given that people checking out trunk will actually be able to find it as part of the checkout. I know that *I* will not be maintaining it. Unless you can find a nova-core person that will volunteer to keep this up-to-date (or at the very least pull it out once it becomes out-dated (which won't be long at the pace this project moves)), I can't approve it. Sorry.

Just to be clear, this has nothing to do with the quality of your work (I've only glanced at it). This is about ensuring the future coherence of trunk.

Disapproving for now, but if you can find a nova-core person that will promise to keep it up-to-date, I'm very willing to reconsider.

review: Disapprove

[Shigetoshi Yokoyama (yoko)]

> By accepting this into trunk, it's implied that it will be kept up-to-date,
> given that people checking out trunk will actually be able to find it as part
> of the checkout. I know that *I* will not be maintaining it. Unless you can
> find a nova-core person that will volunteer to keep this up-to-date (or at the
> very least pull it out once it becomes out-dated (which won't be long at the
> pace this project moves)), I can't approve it. Sorry.
>
> Just to be clear, this has nothing to do with the quality of your work (I've
> only glanced at it). This is about ensuring the future coherence of trunk.
>
> Disapproving for now, but if you can find a nova-core person that will promise
> to keep it up-to-date, I'm very willing to reconsider.

I appreciate that you navigate me, informatively. I think our contribution is
close to the puppet tool which is put under the contrib directory by some of
NASA members.
I have been talking to Jesse and Todd on this deployment tool issues since my
blueprint proposal. I suppose I should ask them to be the volunteer.

[Vish Ishaya (vishvananda)]

I don't really see a problem with it going into contrib. We don't
really have a good other location for items like this.

On Wednesday, January 12, 2011, Shigetoshi Yokoyama <email address hidden> wrote:
>> By accepting this into trunk, it's implied that it will be kept up-to-date,
>> given that people checking out trunk will actually be able to find it as part
>> of the checkout. I know that *I* will not be maintaining it. Unless you can
>> find a nova-core person that will volunteer to keep this up-to-date (or at the
>> very least pull it out once it becomes out-dated (which won't be long at the
>> pace this project moves)), I can't approve it. Sorry.
>>
>> Just to be clear, this has nothing to do with the quality of your work (I've
>> only glanced at it). This is about ensuring the future coherence of trunk.
>>
>> Disapproving for now, but if you can find a nova-core person that will promise
>> to keep it up-to-date, I'm very willing to reconsider.
>
> I appreciate that you navigate me, informatively. I think our contribution is
> close to the puppet tool which is put under the contrib directory by some of
> NASA members.
> I have been talking to Jesse and Todd on this deployment tool issues since my
> blueprint proposal. I suppose I should ask them to be the volunteer.
>
>
> --
> https://code.launchpad.net/~nii/nova/nova-deployment-tool/+merge/45335
> You are subscribed to branch lp:nova.
>

[Shigetoshi Yokoyama (yoko)]

> The files that have a copyright header probably also need the standard apache
> license header used throughout the project.

I thank you for the comment. I will follow it when I update the code, around 24th of Jan.

BTW, this merge proposal was 'temporally' disapproved by Soren until I find a volunteer
in nova-core team. Of course, I have a commitment to make the code update but I need
your help to be the volunteer in oder to get an approval. Could you kindly become the
person?

I hope this version is close to what you wanted to have, using puppet external node
functionality. We are now writing a manual for this tool and doing user test with it.
I will update the code and put this manual in wiki, next Monday. -- yoko

[Soren Hansen (soren)]

> I don't really see a problem with it going into contrib. We don't
> really have a good other location for items like this.

I believe we do. It's called "outside".

If people can check out trunk and find 7 different sets of deployment tools, none of which work, because none of them are kept up-to-date, everybody loses. The user wanting use install and run Nova doesn't get what they want, and we potentially lose a new user. If people want to get stuff into our trunk, they have to commit to keeping it up-to-date. Getting a nova-core person to vouch for that commitment guarantees that someone who actively follows what goes on in trunk every day looks after it.

A quick glance at this tool reveals that it's already out of date. It's still referencing a bunch of libvirt templates that no longer exist, and the one that remains isn't installed where this tool expects it to. I don't blame the author, because this project moves so fast, it's really, really hard to keep this stuff up-to-date.

[Shigetoshi Yokoyama (yoko)]

> > I don't really see a problem with it going into contrib. We don't
> > really have a good other location for items like this.
>
> I believe we do. It's called "outside".
>
> If people can check out trunk and find 7 different sets of deployment tools,
> none of which work, because none of them are kept up-to-date, everybody loses.
> The user wanting use install and run Nova doesn't get what they want, and we
> potentially lose a new user. If people want to get stuff into our trunk, they
> have to commit to keeping it up-to-date. Getting a nova-core person to vouch
> for that commitment guarantees that someone who actively follows what goes on
> in trunk every day looks after it.
>
> A quick glance at this tool reveals that it's already out of date. It's still
> referencing a bunch of libvirt templates that no longer exist, and the one
> that remains isn't installed where this tool expects it to. I don't blame the
> author, because this project moves so fast, it's really, really hard to keep
> this stuff up-to-date.

I have a commitment to make the code up-to-date when we release for Bexar. So,
I am expecting to be very tough days just before 3rd of Mar. As Soren says, I
think it's really, really hard to keep this stuff up-to-date anytime, even after
the release.

However, I believe that if it is up-to-date to the release version,
tools 'inside' the release makes sense in terms of user benefits. We should
inform aim of tool by which users can know which is fit to their needs.

I did not know any other ways than to follow the procedure, which I was taught,
in order to contribute to the community.

1) Write a Blueprint
2) Present it at the developer conference
3) Get an Approval to it
4) Upload codes
5) Propose merge to trunk by 6th of Jan.
6) Get an Approval to it

I am now asking Todd to become a volunteer following Sore's advice. Please
help me to find a way to contribute the code for openstack users' value.

I was thinking that the tool would make installation easier and help everybody
wins. -- yoko

[Soren Hansen (soren)]

>>> I don't really see a problem with it going into contrib. We don't
>>> really have a good other location for items like this.
>> I believe we do. It's called "outside".

I think to a great extent this is the problem we need to address. My primary
concern is related to having this in trunk. Trunk is very much a moving target.
If this stuff is only going to be updated ocasionally (perhaps in sync with
various freezes or whatnot), that means it'll be out-of-date most of the
time. To me, this means it belongs somewhere else. Not necessarily outside of
OpenStack, but outside of Nova's trunk.

If this is the one tool we're going to use, and the one set of puppet recipes
we're going to encourage people to use, I have no problem "advertising" it on
the front page of the wiki, on the Launchpad page for Nova, etc. etc.

> I have a commitment to make the code up-to-date when we release for
> Bexar. So, I am expecting to be very tough days just before 3rd of
> Mar. As Soren says, I think it's really, really hard to keep this
> stuff up-to-date anytime, even after the release.

That's another good reason why you may want to keep this separate from
trunk. If you expect to keep adjusting it after release, you can't do
that if it's in trunk. We're not going to release new tarballs of Nova
itself to publish new puppet recipes.

> We should inform aim of tool by which users can know which is fit to
> their needs.

Yes. See below.

> I did not know any other ways than to follow the procedure, which I
> was taught, in order to contribute to the community.

You've done nothing wrong :) I really do appreciate the contribution
and I'm not meaning to reject your work. I'm just don't believe the
specific location for it (Nova's trunk) is what is best for neither your
work, nor Nova. Imagine this scenario: You make sure the scripts and
the recipes are in perfect order by the time of Bexar's release. The
next day, we make some changes on trunk that are incompatible with your
scripts. People checking out trunk will still find your recipes, try
them, and find that they do not give them a functional cloud. They
either blame you, because they can see that you provided the recipes, or
they blame the entire OpenStack project. That doesn't help anyone at
all.

In summary, if this is in Nova's trunk, it needs to be up-to-date. If
it's kept somewhere else (maybe in a separate "nova-deployment" project
on Launchpad under the OpenStack umbrella) it's easier to have it say
"This tool is for deploying OpenStack Bexar" so that people know exactly
what to expect.

> 1) Write a Blueprint
> 2) Present it at the developer conference
> 3) Get an Approval to it
> 4) Upload codes
> 5) Propose merge to trunk by 6th of Jan.
> 6) Get an Approval to it

Yup, this looks correct to me. Again, you did nothing wrong. :)

> I am now asking Todd to become a volunteer following Sore's advice.
> Please help me to find a way to contribute the code for openstack
> users' value.
>
> I was thinking that the tool would make installation easier and help everybody
> wins. -- yoko

I completely agree. I think this could also serve well as a tool for
automated full-scale testing. It's good stuff! My onl...

[Soren Hansen (soren)]

As an interesting data point, the packaging code for Ubuntu isn't in Nova's trunk either. It's not even kept under the nova project on Launchpad.

[Thierry Carrez (ttx)]

I agree it is probably better for everyone to have this very interesting OpenStack contribution land into its own OpenStack project, rather than in nova core trunk.

You will have to adjust the deployment tool until the very last minute before release, which is difficult to do with the freezes we impose on the core code. Your tool is, like the Ubuntu packaging rules, the Nova web UI or the iOS clients, a consumer of a stable Nova core code. As such it makes much more sense from a release perspective to have its own project, rather than artificially impose the core release freezes onto your own development.

If you agree with that, we can help you setting up the separate launchpad project under the Openstack umbrella. As far as discoverability goes, if the documentation is updated to reference the tool, I'm pretty sure people will be able to easily find it.

[Vish Ishaya (vishvananda)]

Soren made some excellent points. I am convinced that it is probably best to put this into its own launchpad project.

Vish

On Jan 13, 2011, at 5:11 AM, Thierry Carrez wrote:

> I agree it is probably better for everyone to have this very interesting OpenStack contribution land into its own OpenStack project, rather than in nova core trunk.
>
> You will have to adjust the deployment tool until the very last minute before release, which is difficult to do with the freezes we impose on the core code. Your tool is, like the Ubuntu packaging rules, the Nova web UI or the iOS clients, a consumer of a stable Nova core code. As such it makes much more sense from a release perspective to have its own project, rather than artificially impose the core release freezes onto your own development.
>
> If you agree with that, we can help you setting up the separate launchpad project under the Openstack umbrella. As far as discoverability goes, if the documentation is updated to reference the tool, I'm pretty sure people will be able to easily find it.
> --
> https://code.launchpad.net/~nii/nova/nova-deployment-tool/+merge/45335
> You are subscribed to branch lp:nova.

[Shigetoshi Yokoyama (yoko)]

> >>> I don't really see a problem with it going into contrib. We don't
> >>> really have a good other location for items like this.
> >> I believe we do. It's called "outside".
>
> I think to a great extent this is the problem we need to address. My primary
> concern is related to having this in trunk. Trunk is very much a moving
> target.
> If this stuff is only going to be updated ocasionally (perhaps in sync with
> various freezes or whatnot), that means it'll be out-of-date most of the
> time. To me, this means it belongs somewhere else. Not necessarily outside of
> OpenStack, but outside of Nova's trunk.
>
> If this is the one tool we're going to use, and the one set of puppet recipes
> we're going to encourage people to use, I have no problem "advertising" it on
> the front page of the wiki, on the Launchpad page for Nova, etc. etc.
>
> > I have a commitment to make the code up-to-date when we release for
> > Bexar. So, I am expecting to be very tough days just before 3rd of
> > Mar. As Soren says, I think it's really, really hard to keep this
> > stuff up-to-date anytime, even after the release.
>
> That's another good reason why you may want to keep this separate from
> trunk. If you expect to keep adjusting it after release, you can't do
> that if it's in trunk. We're not going to release new tarballs of Nova
> itself to publish new puppet recipes.
>
> > We should inform aim of tool by which users can know which is fit to
> > their needs.
>
> Yes. See below.
>
> > I did not know any other ways than to follow the procedure, which I
> > was taught, in order to contribute to the community.
>
> You've done nothing wrong :) I really do appreciate the contribution
> and I'm not meaning to reject your work. I'm just don't believe the
> specific location for it (Nova's trunk) is what is best for neither your
> work, nor Nova. Imagine this scenario: You make sure the scripts and
> the recipes are in perfect order by the time of Bexar's release. The
> next day, we make some changes on trunk that are incompatible with your
> scripts. People checking out trunk will still find your recipes, try
> them, and find that they do not give them a functional cloud. They
> either blame you, because they can see that you provided the recipes, or
> they blame the entire OpenStack project. That doesn't help anyone at
> all.
>
> In summary, if this is in Nova's trunk, it needs to be up-to-date. If
> it's kept somewhere else (maybe in a separate "nova-deployment" project
> on Launchpad under the OpenStack umbrella) it's easier to have it say
> "This tool is for deploying OpenStack Bexar" so that people know exactly
> what to expect.
>
> > 1) Write a Blueprint
> > 2) Present it at the developer conference
> > 3) Get an Approval to it
> > 4) Upload codes
> > 5) Propose merge to trunk by 6th of Jan.
> > 6) Get an Approval to it
>
> Yup, this looks correct to me. Again, you did nothing wrong. :)
>
> > I am now asking Todd to become a volunteer following Sore's advice.
> > Please help me to find a way to contribute the code for openstack
> > users' value.
> >
> > I was thinking that the tool would make installation easier and help
> everybody
> > win...

[Shigetoshi Yokoyama (yoko)]

> I agree it is probably better for everyone to have this very interesting
> OpenStack contribution land into its own OpenStack project, rather than in
> nova core trunk.
>
> You will have to adjust the deployment tool until the very last minute before
> release, which is difficult to do with the freezes we impose on the core code.
> Your tool is, like the Ubuntu packaging rules, the Nova web UI or the iOS
> clients, a consumer of a stable Nova core code. As such it makes much more
> sense from a release perspective to have its own project, rather than
> artificially impose the core release freezes onto your own development.
>
> If you agree with that, we can help you setting up the separate launchpad
> project under the Openstack umbrella. As far as discoverability goes, if the
> documentation is updated to reference the tool, I'm pretty sure people will be
> able to easily find it.

I agree that it is better for everyone to put our code into a separate launchpad
project under the Openstack umbrella. I appreciate that you said you could help
it. Actually, Anne has been helping us to put our documentation as well and it
is now in http://wiki.openstack.org/NovaInstall/NovaDeploymentTool. Thanks. -- yoko

[Shigetoshi Yokoyama (yoko)]

I thank your feedback. It made me be able to move forward. -- yoko

> Soren made some excellent points. I am convinced that it is probably best to
> put this into its own launchpad project.
>
> Vish
>

[Soren Hansen (soren)]

Alright, it seems we're in agreement. This will be merged in a separate project, so I'm rejecting this merge proposal in Nova.

Unmerged revisions

471. By Shigetoshi Yokoyama on 2011-01-06

Initial import

470. By Shigetoshi Yokoyama on 2010-12-22

Initial import

=== added directory 'contrib/nova-deployment-tool2011'
=== added file 'contrib/nova-deployment-tool2011/Readme'
--- contrib/nova-deployment-tool2011/Readme	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/Readme	2011-01-06 09:41:56 +0000
@@ -0,0 +1,24 @@
+1. Before installation
+
+You should do the following steps before installation.
+Attention: replace $USER_NAME with the real user name.
+
+1.1 Append the following line to the end of sudoers via visudo.
+$USER_NAME ALL=(ALL) NOPASSWD:ALL
+
+1.2 Create authentication keys. Leave key file as default value, and leave passphrase as empty.
+ssh-keygen -t rsa
+
+1.3 Add authentication key.
+cat /home/$USER_NAME/.ssh/id_rsa.pub >> /home/$USER_NAME/.ssh/authorized_keys
+
+1.4 Confirm that no password is needed when executing command "sudo -i".
+
+1.5 Confirm that no password is needed when executing command "ssh $USER_NAME@127.0.0.1".
+
+
+2. Installation
+
+2.1 Execute deploy.py.
+
+

=== added directory 'contrib/nova-deployment-tool2011/conf'
=== added file 'contrib/nova-deployment-tool2011/conf/deploy.conf'
--- contrib/nova-deployment-tool2011/conf/deploy.conf	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/conf/deploy.conf	2011-01-06 09:41:56 +0000
@@ -0,0 +1,12 @@
+nova_api=ubuntu1
+nova_objectstore=ubuntu1
+nova_compute=ubuntu3
+nova_scheduler=ubuntu4
+nova_network=ubuntu5
+nova_volume=ubuntu6
+euca2ools=ubuntu1
+
+mysql=ubuntu1
+puppet_server=ubuntu7
+
+ssh_user=nii

=== added directory 'contrib/nova-deployment-tool2011/conf/etc-default'
=== added file 'contrib/nova-deployment-tool2011/conf/etc-default/puppetmaster'
--- contrib/nova-deployment-tool2011/conf/etc-default/puppetmaster	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/conf/etc-default/puppetmaster	2011-01-06 09:41:56 +0000
@@ -0,0 +1,38 @@
+# Defaults for puppetmaster - sourced by /etc/init.d/puppetmaster
+
+# Start puppetmaster on boot? If you are using passenger, you should
+# have this set to "no"
+START=yes
+
+# Startup options
+DAEMON_OPTS=""
+
+# What server type to run 
+# Options: 
+# 	   webrick: default, cannot handle more than ~30 nodes
+#	   mongrel: scales better than webrick because you can run
+#	    	    multiple processes if you are getting
+#	    	    connection-reset or End-of-file errors, switch to
+#	    	    mongrel. Requires front-end web-proxy such as
+#	    	    apache, nginx, or pound, more information:
+#	            http://reductivelabs.com/trac/puppet/wiki/UsingMongrel
+#          passenger: no need to set this, be sure to set START=no above
+SERVERTYPE=webrick
+
+# How many puppetmaster instances to start? Its pointless to set this
+# higher than 1 if you are not using mongrel. 
+PUPPETMASTERS=1
+
+# What port should the puppetmaster listen on (default: 8140). If
+# PUPPETMASTERS is set to a number greater than 1, then the port for
+# the first puppetmaster will be set to the port listed below, and
+# further instances will be incremented by one 
+#
+# NOTE: if you are using mongrel, then you will need to have a
+# front-end web-proxy (such as apache, nginx, pound) that takes
+# incoming requests on the port your clients are connecting to
+# (default is: 8140), and then passes them off to the mongrel
+# processes.  In this case it is recommended to run your web-proxy on
+# port 8140 and change the below number to something else, such as
+# 18140.
+PORT=8140

=== added file 'contrib/nova-deployment-tool2011/conf/flags.conf'
--- contrib/nova-deployment-tool2011/conf/flags.conf	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/conf/flags.conf	2011-01-06 09:41:56 +0000
@@ -0,0 +1,3 @@
+--volume_group=ubuntu3
+--fixed_range=192.168.2.64/26
+--network_size=64

=== added file 'contrib/nova-deployment-tool2011/conf/logging.conf'
--- contrib/nova-deployment-tool2011/conf/logging.conf	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/conf/logging.conf	2011-01-06 09:41:56 +0000
@@ -0,0 +1,34 @@
+[loggers]
+keys=root,deploy
+
+[handlers]
+keys=consoleHandler,fileHandler
+
+[formatters]
+keys=simpleFormatter
+
+[logger_root]
+handlers=
+
+[logger_deploy]
+level=DEBUG
+handlers=consoleHandler,fileHandler
+qualName=deploy
+
+[handler_consoleHandler]
+class=StreamHandler
+level=DEBUG
+formatter=simpleFormatter
+args=(sys.stdout,)
+
+[handler_fileHandler]
+class=FileHandler
+level=DEBUG
+formatter=simpleFormatter
+args=('log/deploy.log', 'w')
+filename=deploy.log
+mode=w
+
+[formatter_simpleFormatter]
+format=%(asctime)s - %(levelname)s - %(message)s
+datefmt=

=== added directory 'contrib/nova-deployment-tool2011/conf/puppet-server'
=== added file 'contrib/nova-deployment-tool2011/conf/puppet-server/autosign.conf'
--- contrib/nova-deployment-tool2011/conf/puppet-server/autosign.conf	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/conf/puppet-server/autosign.conf	2011-01-06 09:41:56 +0000
@@ -0,0 +1,1 @@
+*

=== added file 'contrib/nova-deployment-tool2011/conf/puppet-server/fileserver.conf'
--- contrib/nova-deployment-tool2011/conf/puppet-server/fileserver.conf	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/conf/puppet-server/fileserver.conf	2011-01-06 09:41:56 +0000
@@ -0,0 +1,18 @@
+# This file consists of arbitrarily named sections/modules
+# defining where files are served from and to whom
+
+# Define a section 'files'
+# Adapt the allow/deny settings to your needs. Order
+# for allow/deny does not matter, allow always takes precedence
+# over deny
+[files]
+  path /etc/puppet/files
+  allow *
+#  allow *.example.com
+#  deny *.evil.example.com
+#  allow 192.168.0.0/24
+
+[plugins]
+#  allow *.example.com
+#  deny *.evil.example.com
+#  allow 192.168.0.0/24

=== added file 'contrib/nova-deployment-tool2011/conf/puppet-server/puppet.conf'
--- contrib/nova-deployment-tool2011/conf/puppet-server/puppet.conf	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/conf/puppet-server/puppet.conf	2011-01-06 09:41:56 +0000
@@ -0,0 +1,19 @@
+[main]
+logdir=/var/log/puppet
+vardir=/var/lib/puppet
+ssldir=/var/lib/puppet/ssl
+rundir=/var/run/puppet
+factpath=$vardir/lib/facter
+templatedir=$confdir/templates
+#prerun_command=/etc/puppet/etckeeper-commit-pre
+#postrun_command=/etc/puppet/etckeeper-commit-post
+
+external_nodes = /etc/puppet/files/external_nodes.py
+node_terminus = exec
+
+[master]
+# These are needed when the puppetmaster is run by passenger
+# and can safely be removed if webrick is used.
+ssl_client_header = SSL_CLIENT_S_DN 
+ssl_client_verify_header = SSL_CLIENT_VERIFY
+

=== added file 'contrib/nova-deployment-tool2011/deploy.py'
--- contrib/nova-deployment-tool2011/deploy.py	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/deploy.py	2011-01-06 09:41:56 +0000
@@ -0,0 +1,367 @@
+#! /usr/bin/python
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+#
+# Copyright (c) National Institute of Informatics 2010 All rights reserved.
+#
+
+
+"""
+Install nova into single machine or multiple machines
+"""
+
+import logging
+import logging.config
+import os
+import commands
+import time
+import sys
+import traceback
+
+TOOL_HOME = os.path.abspath(os.path.dirname(__file__))
+COMPONENTS = ["mysql", "nova_api", "nova_objectstore", "nova_compute", 
+              "nova_scheduler", "nova_network", "nova_volume", "euca2ools"]
+CONF = {}
+
+os.chdir(TOOL_HOME)
+
+#initialize LOGGER
+logging.config.fileConfig("conf/logging.conf")
+LOGGER = logging.getLogger("deploy")
+
+class CmdError(Exception):
+    """
+    Error when the execution of shell command failed.
+    """
+
+    def __init__(self, code, message, cmd):
+        Exception.__init__(self)
+        self.code = code
+        self.message = message
+        self.cmd = cmd
+   
+class ServiceFailedError(Exception):
+    """
+    Error when servcie cannot be started.
+    """
+
+    def __init__(self, service):
+        Exception.__init__(self)
+        self.service = service
+
+def pretty_print(msg, is_ended = False):
+    """
+    Print msg in pretty format.
+
+    Keyword arguments:
+    msg -- the message to be printed
+    is_ended -- whether it is the end part of the block or not (default False)
+
+    """
+
+    LOGGER.info("---------------------------------------------------")
+    LOGGER.info("      %s" % msg)
+    LOGGER.info("---------------------------------------------------")
+    if is_ended:
+        LOGGER.info("")
+        LOGGER.info("")
+        LOGGER.info("")
+
+def print_file(path):
+    """
+    Print the content of the file specified by path.
+    
+    Keyword arguments:
+    path -- the path of the file to be printed.
+
+    """
+
+    LOGGER.info("*****The content of file '%s' is from here:*****" % path)
+
+    with open(path, "r") as lines:
+        for line in lines:
+            LOGGER.info(line.rstrip())    
+
+    LOGGER.info("*****The content of file '%s' is to here.  *****" % path)
+    LOGGER.info("")
+    LOGGER.info("")
+
+def exec_cmd(cmd):
+    """
+    Execute the command.
+    
+    Keyword arguments:
+    cmd -- the command to be executed.
+
+    """
+
+    LOGGER.info("Execute shell command: %s" % cmd)
+    LOGGER.debug("*****The output of the shell command is from here:*****")
+    result = commands.getstatusoutput(cmd)
+    for line in result[1].split("\n"):
+        LOGGER.debug(line)
+    LOGGER.debug("*****The output of the shell command is to here.  *****")
+    LOGGER.info("")
+    LOGGER.info("")
+
+    #When running puppetd, there was no error in messages, 
+    #but status 512 was returned, so I add the condition 
+    #(cmd.find("puppetd") != -1 and result[0] == 512)
+    if result[0] != 0 and not (cmd.find("puppetd") != -1 and result[0] == 512):
+        raise CmdError(result[0], result[1], cmd)
+
+    return result
+
+def exec_cmd_with_sudo(cmd):
+    """
+    Execute command as root.
+    
+    Keyword arguments:
+    cmd -- the command to be executed
+
+    """
+
+    return exec_cmd("sudo " + cmd)
+
+def exec_ssh_cmd(target, cmd, user = "root"):
+    """
+    Execute remote command with ssh.
+
+    Keyword arguments:
+    target -- the target computer where the command will be executed
+    cmd -- the command to be executed
+    user -- the user to execute command (default: root)
+
+    """
+
+    return exec_cmd("%s@%s 'sudo -u %s sh -c \"%s\"'" 
+        % (CONF["ssh_cmd_prefix"], target, user, cmd))
+
+def copy_to_remote(target, src_path, dest_path):
+    """
+    Copy files to remote computer.
+
+    Keyword arguments:
+    target -- the remote computer to which files will be copied.
+    src_path -- the local path of the files to be copied.
+    dest_path -- the remote path whick the files will be copied to.
+
+    """
+
+    tmp_folder = str(int(time.time() * 1000))
+
+    #Check if the dest_path exists
+    try:
+        exec_ssh_cmd(target, "ls %s > /dev/null" % dest_path)
+    except CmdError:
+        exec_ssh_cmd(target, "mkdir -p %s" % dest_path)
+
+    exec_ssh_cmd(target, "mkdir /home/%s/%s" 
+        % (CONF["ssh_user"], tmp_folder), CONF["ssh_user"])
+    exec_cmd("scp -r %s %s@%s:~/%s/"
+        % (src_path, CONF["ssh_user"], target, tmp_folder))
+    exec_ssh_cmd(target, "cp -rp /home/%s/%s/* %s" 
+        % (CONF["ssh_user"], tmp_folder, dest_path))
+    exec_ssh_cmd(target, "rm -rf /home/%s/%s" 
+        % (CONF["ssh_user"], tmp_folder))
+
+def append_to_remote_file(target, src_file_path, dest_file_path):
+    """
+    Append a file contents to the remote file.
+
+    Keyword arguments:
+    target -- the remote computer in which the destination file will be.
+    src_file_path -- the file contents will be copied.
+    dest_file_path -- the remote file will be appended to.
+    """
+    
+    tmp_folder = str(int(time.time() * 1000))
+    exec_ssh_cmd(target, "mkdir /home/%s/%s"
+        % (CONF["ssh_user"], tmp_folder), CONF["ssh_user"])
+    exec_cmd("scp -r %s %s@%s:~/%s/"
+        % (src_file_path, CONF["ssh_user"], target, tmp_folder))
+    exec_ssh_cmd(target, "cat ~/%s/* >> %s" % (tmp_folder, dest_file_path))
+    exec_ssh_cmd(target, "rm -rf /home/%s/%s"
+        % (CONF["ssh_user"], tmp_folder))    
+
+
+def load_conf(path):
+    """
+    Load configuration from the file specified by path.
+
+    Keyword arguments:
+    path -- the path of the file where configuration will be loaded from
+
+    """
+
+    pretty_print("Loading configuration begins")
+    
+    LOGGER.info("Load configuration from file %s" % path)
+    
+    print_file(path)
+
+    with open(path, "r") as lines:
+        for line in lines:
+            line = line.strip()
+            if line == "":
+                continue
+
+            pair = line.split("=")
+            CONF[pair[0].rstrip()] = pair[1].lstrip()
+    
+    pretty_print("Loading configuration ends", True)
+
+def install_puppet_server():
+    """
+    Install puppet server.
+    """
+
+    pretty_print("Installing puppet server begins")
+
+    target = CONF["puppet_server"]
+    copy_to_remote(target, "%s/conf/puppet-server/*" % TOOL_HOME, 
+        "/etc/puppet/")
+    copy_to_remote(target, "%s/conf/etc-default/puppetmaster" % TOOL_HOME,
+        "/etc/default/") 
+
+    exec_ssh_cmd(target, "apt-get -y --force-yes install puppetmaster")
+
+    copy_to_remote(target, "%s/manifests/*" % TOOL_HOME,
+        "/etc/puppet/manifests")
+    copy_to_remote(target, "%s/files/*" % TOOL_HOME, "/etc/puppet/files/")
+    copy_to_remote(target, "%s/templates/*" % TOOL_HOME, 
+        "/etc/puppet/templates/")
+    copy_to_remote(target, "%s/conf/deploy.conf" % TOOL_HOME,
+        "/etc/puppet/files") 
+    append_to_remote_file(target, "%s/conf/flags.conf" % TOOL_HOME, 
+        "/etc/puppet/templates/nova.conf.erb")
+ 
+    exec_ssh_cmd(target, "service puppetmaster restart")
+
+    pretty_print("Installing puppet server ends", True)
+
+def install_puppet_client():
+    """
+    Install puppet client.
+    """
+
+    pretty_print("Installing puppet client begins")
+
+    targets = []
+    for component in COMPONENTS:
+        targets += CONF[component].split(",")
+
+    for target in set(targets):
+        exec_ssh_cmd(target, "apt-get -y --force-yes install puppet")
+        
+    pretty_print("Installing puppet client ends", True)
+
+def exec_puppet_client():
+    """
+    Execute puppet client on each servers to install nova components.
+    """
+
+    pretty_print("Executing puppetclient begins.")
+    
+    targets = []
+    for component in COMPONENTS:
+        targets += CONF[component].split(",")
+
+    for target in set(targets):
+        try:
+            exec_ssh_cmd(target, 
+                "puppetd --verbose --test --server %s"
+                % CONF["puppet_server"])
+        except CmdError:
+            pass
+
+    for component in COMPONENTS:
+        targets = CONF[component].split(",")
+        service = component.replace("_", "-")
+        for target in targets:
+            if component == "euca2ools":
+                continue
+
+            try:
+                exec_ssh_cmd(target, "service %s restart; sleep 1"
+                    % service)
+            except CmdError:
+                pass
+
+    copy_to_remote(CONF["mysql"], TOOL_HOME + "/setup-network.sh", "/tmp/")
+    
+    try:
+        exec_ssh_cmd(CONF["mysql"], "/tmp/setup-network.sh 2>&1") 
+    except CmdError:
+        pass    
+
+    pretty_print("Executing puppetclient ends.", True)
+
+def wait_until_services_started():
+    """
+    Wait until all the service started.
+    """
+
+    pretty_print("Waiting until all services are started begins.")
+
+    for component in COMPONENTS:
+        if component == "euca2ools":
+            continue
+
+        targets = CONF[component].split(",")
+        service = component.replace("_", "-")
+        for target in targets:
+            result = exec_ssh_cmd(target, "service %s status" % service)
+            while result[1].find("failed") == -1 and \
+                result[1].find("running") == -1:
+            
+                time.sleep(10)
+                result = exec_ssh_cmd(target, "service %s status" % service)
+
+            if result[1].find("failed") != -1:
+                raise ServiceFailedError(service)
+
+    pretty_print("Waiting until all services are started ends.", True)
+
+def do_test():
+    """
+    Do test.
+    """
+
+    pretty_print("Test begins.")
+
+    target = CONF["euca2ools"]
+    copy_to_remote(target, TOOL_HOME + "/test.sh", "/tmp/")
+    copy_to_remote(target, TOOL_HOME + "/images.tgz", "/tmp/")
+    exec_ssh_cmd(target, "/tmp/test.sh 2>&1")
+
+    pretty_print("Test ends.", True)
+
+def main():
+    """
+    Main function.
+    """
+
+    load_conf("conf/deploy.conf")
+    CONF["ssh_cmd_prefix"] = ("ssh -o StrictHostKeyChecking=no -t "
+        + CONF["ssh_user"])
+
+    try: 
+        install_puppet_server()
+        install_puppet_client()
+        exec_puppet_client()
+        wait_until_services_started()
+        do_test()
+    except CmdError as cmd_error:
+        LOGGER.error("Error occured when executing the shell command: " 
+           + cmd_error.cmd)
+        LOGGER.error("Error code: " + str(cmd_error.code))
+    except ServiceFailedError as service_failed_error:
+        LOGGER.error("Error occured when staring the service(%s)." 
+            % service_failed_error.service)
+    except:
+        LOGGER.error("Unexpected error:" + str(sys.exc_info()[0]))
+        traceback.print_exc(file=sys.stdout)
+
+if __name__ == '__main__':
+    main()

=== added directory 'contrib/nova-deployment-tool2011/files'
=== added file 'contrib/nova-deployment-tool2011/files/external_nodes.py'
--- contrib/nova-deployment-tool2011/files/external_nodes.py	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/files/external_nodes.py	2011-01-06 09:41:56 +0000
@@ -0,0 +1,44 @@
+#!/usr/bin/python
+
+import os
+import sys
+import re
+import time
+import yaml
+
+hostname = sys.argv[1]
+
+os.chdir(os.path.abspath(os.path.dirname(__file__)))
+
+parameters = {}
+classes = []
+with open("deploy.conf", "r") as lines:
+    for line in lines:
+        line = line.strip()
+        if line == "":
+            continue
+
+        pair = line.split("=")
+        component = pair[0].strip()
+        hostnames = pair[1].strip().split(",")
+        if component == "puppet_server":
+            parameters["puppet_server"] = hostnames[0]
+            continue
+
+        if component == "mysql":
+            parameters["mysql_addr"] = hostnames[0]
+
+        if component == "nova_api":
+            parameters["cc_addr"] = hostnames[0]
+
+        if component == "nova_objectstore":
+            parameters["s3_addr"] = hostnames[0]
+
+        if hostname in hostnames:
+            classes.append(component)
+
+yamldoc = {"classes": classes, "parameters": parameters}
+print yaml.dump(yamldoc, explicit_start=True,
+    default_flow_style=False)
+
+exit(0)

=== added file 'contrib/nova-deployment-tool2011/files/mysql-init.sh'
--- contrib/nova-deployment-tool2011/files/mysql-init.sh	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/files/mysql-init.sh	2011-01-06 09:41:56 +0000
@@ -0,0 +1,14 @@
+#!/bin/bash
+
+#
+# Copyright (c) National Institute of Informatics 2010 All rights reserved.
+#
+
+sudo sed -i "s/127.0.0.1/0.0.0.0/g" /etc/mysql/my.cnf
+
+mysql -uroot -pnova -e "CREATE DATABASE nova;"
+mysql -uroot -pnova -e "GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION;"
+mysql -uroot -pnova -e "SET PASSWORD FOR 'root'@'%' = PASSWORD('nova');"
+
+#sudo nova-manage network create 10.0.0.0/8 3 16
+#mysql -uroot -pnova -e "UPDATE networks SET BRIDGE = 'br100'" nova

=== added file 'contrib/nova-deployment-tool2011/files/mysql-preseed.sh'
--- contrib/nova-deployment-tool2011/files/mysql-preseed.sh	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/files/mysql-preseed.sh	2011-01-06 09:41:56 +0000
@@ -0,0 +1,13 @@
+#!/bin/bash
+
+#
+# Copyright (c) National Institute of Informatics 2010 All rights reserved.
+#
+
+MYSQL_PASS="nova"
+
+cat <<MYSQL_PRESEED | debconf-set-selections
+mysql-server-5.1 mysql-server/root_password password $MYSQL_PASS
+mysql-server-5.1 mysql-server/root_password_again password $MYSQL_PASS
+mysql-server-5.1 mysql-server/start_on_boot boolean true
+MYSQL_PRESEED

=== added directory 'contrib/nova-deployment-tool2011/log'
=== added directory 'contrib/nova-deployment-tool2011/manifests'
=== added file 'contrib/nova-deployment-tool2011/manifests/site.pp'
--- contrib/nova-deployment-tool2011/manifests/site.pp	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/manifests/site.pp	2011-01-06 09:41:56 +0000
@@ -0,0 +1,114 @@
+/*
+ * Copyright (c) National Institute of Informatics 2010 All rights reserved.
+ *
+ */
+
+$work_dir = "/tmp"
+$nova_conf_tpl = "nova.conf.erb"
+
+Exec {
+    path => "/usr/bin:/bin:/usr/sbin:/sbin:$work_dir",
+    cwd => "$work_dir"
+}
+
+File {
+    owner => root,
+    group => root,
+    mode => 777
+}
+
+define nova_component() {
+    package {
+        "$title":
+            ensure => present,
+            require => Package["python-nova"]
+    }
+}
+
+package {
+    "python-mysqldb":
+        ensure => present;
+    "python-nova":
+        ensure => present;
+    "nova-common":
+        ensure => present;
+}
+
+file {
+    "/etc/nova/nova.conf":
+        content => template("/etc/puppet/templates/$nova_conf_tpl"),
+        mode => 666;
+}
+
+class nova_api {
+    nova_component { 
+        ["rabbitmq-server", "nova-api", "python-libvirt"]:
+    }
+}
+
+class nova_objectstore {
+    nova_component { 
+        "nova-objectstore": 
+    }
+}
+
+class nova_volume {
+    nova_component {
+        "nova-volume":
+    }
+}
+
+class nova_compute {
+    nova_component { 
+        "nova-compute":
+    }
+}
+
+class nova_scheduler {
+    nova_component { 
+        "nova-scheduler": 
+    }
+}
+
+class nova_network {
+    nova_component { 
+        "nova-network":
+    }
+}
+
+class mysql {
+    file { 
+        "$work_dir/mysql-preseed.sh":
+            alias => "mysql-preseed.sh",
+            source => "puppet://$puppet_server/files/mysql-preseed.sh";
+        "$work_dir/mysql-init.sh":
+            alias => "mysql-init.sh",
+            source => "puppet://$puppet_server/files/mysql-init.sh";
+    }
+    exec { 
+        "mysql-preseed.sh > mysql-preseed.log 2>&1":
+            alias => "mysql-preseed.sh",
+            require => File["mysql-preseed.sh"];
+        "mysql-init.sh > mysql-init.log 2>&1":
+            alias => "mysql-init.sh",
+            require => [Package["mysql-server", "nova-common"], File["mysql-init.sh"], Service["mysql"]],
+    }
+    package { 
+        "mysql-server":
+            ensure => present,
+            require => Exec["mysql-preseed.sh"],
+            notify => Service["mysql"]
+    }
+    service {
+        "mysql":
+            ensure => running
+    }
+}
+
+class euca2ools {
+    package { 
+        ["euca2ools", "unzip"]:
+            ensure => present
+    }
+}
+

=== added file 'contrib/nova-deployment-tool2011/setup-network.sh'
--- contrib/nova-deployment-tool2011/setup-network.sh	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/setup-network.sh	2011-01-06 09:41:56 +0000
@@ -0,0 +1,8 @@
+#!/bin/bash
+
+#
+# Copyright (c) National Institute of Informatics 2010 All rights reserved.
+#
+
+sudo nova-manage network create 10.0.0.0/8 3 16
+mysql -uroot -pnova -e "UPDATE networks SET BRIDGE = 'br100'" nova

=== added directory 'contrib/nova-deployment-tool2011/templates'
=== added file 'contrib/nova-deployment-tool2011/templates/nova.conf.erb'
--- contrib/nova-deployment-tool2011/templates/nova.conf.erb	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/templates/nova.conf.erb	2011-01-06 09:41:56 +0000
@@ -0,0 +1,32 @@
+--sql_connection=mysql://root:nova@<%= mysql_addr  %>/nova
+--s3_host=<%= s3_addr %>
+--rabbit_host=<%= cc_addr %>
+--cc_host=<%= cc_addr %>
+--ec2_url=http://<%= cc_addr %>:8773/services/Cloud
+--network_manager=nova.network.manager.FlatManager
+
+--daemonize=1
+
+--dhcpbridge_flagfile=/etc/nova/nova.conf
+--dhcpbridge=/usr/bin/nova-dhcpbridge
+
+--FAKE_subdomain=ec2
+
+--ca_path=/var/lib/nova/CA
+--keys_path=/var/lib/nova/keys
+--networks_path=/var/lib/nova/networks
+--instances_path=/var/lib/nova/instances
+--images_path=/var/lib/nova/images
+--buckets_path=/var/lib/nova/buckets
+
+
+--injected_network_template=/usr/share/nova/interfaces.template
+--libvirt_rescue_uml_xml_template=/usr/share/nova/libvirt.rescue.uml.xml.template
+--libvirt_rescue_xml_template=/usr/share/nova/libvirt.rescue.qemu.xml.template
+--libvirt_uml_xml_template=/usr/share/nova/libvirt.uml.xml.template
+--libvirt_xml_template=/usr/share/nova/libvirt.qemu.xml.template
+--vpn_client_template=/usr/share/nova/client.ovpn.template
+--credentials_template=/usr/share/nova/novarc.template
+
+--logdir=/var/log/nova
+--verbose

=== added file 'contrib/nova-deployment-tool2011/test.sh'
--- contrib/nova-deployment-tool2011/test.sh	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/test.sh	2011-01-06 09:41:56 +0000
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+
+#
+# Copyright (c) National Institute of Informatics 2010 All rights reserved.
+#
+
+apt-get -y install gawk
+
+cd /tmp/
+
+tar xvzf images.tgz
+
+sudo nova-manage user admin anne
+sudo nova-manage project create IRT anne
+
+rm nova.zip
+sudo nova-manage project zipfile IRT anne
+
+rm -rf env
+unzip -d env/ nova.zip
+. env/novarc
+
+euca-bundle-image -i images/aki-lucid/image -p kernel --kernel true
+euca-bundle-image -i images/ari-lucid/image -p ramdisk --ramdisk true
+euca-upload-bundle -m /tmp/kernel.manifest.xml -b mybucket
+euca-upload-bundle -m /tmp/ramdisk.manifest.xml -b mybucket
+kernel=`euca-register mybucket/kernel.manifest.xml | gawk '{print $2}'`
+echo "kernel: [$kernel]"
+ramdisk=`euca-register mybucket/ramdisk.manifest.xml | gawk '{print $2}'`
+echo "ramdisk: [$ramdisk]"
+
+euca-bundle-image -i images/ami-tiny/image -p machine --kernel $kernel --ramdisk $ramdisk
+euca-upload-bundle -m /tmp/machine.manifest.xml -b mybucket
+image=`euca-register mybucket/machine.manifest.xml | gawk '{print $2}'`
+echo "image: [$image]"
+
+euca-add-keypair mykey > mykey.priv
+chmod 400 mykey.priv
+
+#Wait for image processing for a while. Sometimes there will be error while executing command,
+#such as "mkdir -p ..." and "resize2fs ..."
+sleep 10
+euca-run-instances $image --kernel $kernel --ramdisk $ramdisk -k mykey

=== added file 'contrib/nova-deployment-tool2011/uninstall.sh'
--- contrib/nova-deployment-tool2011/uninstall.sh	1970-01-01 00:00:00 +0000
+++ contrib/nova-deployment-tool2011/uninstall.sh	2011-01-06 09:41:56 +0000
@@ -0,0 +1,23 @@
+#!/bin/bash
+
+sudo apt-get -y remove puppet puppetmaster
+
+sudo apt-get -y remove nova-common nova-doc python-nova nova-api nova-network nova-volume nova-objectstore nova-scheduler
+
+sudo service mysql stop
+sudo apt-get -y remove rabbitmq-server mysql-server
+#sudo rm /var/log/nova/nova-*
+
+#sudo apt-get -y autoremove
+
+sudo rm -rf /etc/puppet
+sudo rm -rf /var/lib/puppet
+sudo rm -rf /var/log/puppet
+sudo rm -rf /var/run/puppet
+sudo rm -rf /var/cache/apt/archives/puppet*
+sudo rm -rf /var/lib/update-rc.d/puppet*
+sudo rm -rf /etc/logrotate.d/puppet
+sudo rm -rf /etc/logcheck/ignore.d.server/puppet*
+
+
+sudo rm -rf /etc/default/puppet*