Merge lp:~termie/nova/eventlet_objectstore into lp:~hudson-openstack/nova/trunk

[termie]

Ports the Tornado version of an S3 server to eventlet and wsgi, first step in deprecating the twistd-based objectstore.

This is a trivial implementation, never meant for production, it exists to provide an s3-look-alike objectstore for use when developing/testing things related to the amazon APIs (eucatools, etc), any production deployment would be expected to use Swift + an S3 interface.

In later patches I expect to be able to remove the old objectstore code entirely.

[lp:~termie/nova/eventlet_objectstore]

760. By Rick Harris on 2011-03-04

Use %s for instance-delete logging in case instance_id comes through as a string.

[Vish Ishaya (vishvananda)]

Code looks awesome. Unfortunately this breaks register_image until my patch here is merged:
lp:~vishvananda/nova/kill-objectstore is merged. To avoid having trunk temporarily broken, I suggest that you merge my patch and repropose the merge with that one as a prereq.

review: Needs Fixing

[lp:~termie/nova/eventlet_objectstore]

761. By Todd Willey on 2011-03-05

Fix renaming of instance fields using update_instance api method.

[Soren Hansen (soren)]

In nova/test.py, you import eventlet.greenpool, yet you don't use it. Is that a mistake or does importing the greenpool module make eventlet do something differently behind the scenes? If so, it would probably be good to add a comment to that effect.

I wonder what the rationale behind not removing the twisted based handler is?

You're removing a bunch of tests for the objectstore (the ones that do not directly pertain to the S3 API). Why is that?

review: Needs Information

[lp:~termie/nova/eventlet_objectstore]

762. By Matt Dietz on 2011-03-07

Implementation for XenServer migrations. There are several places for optimization but I based the current implementation on the chance scheduler just to be safe. Beyond that, a few features are missing, such as ensuring the IP address is transferred along with the migrated instance. This will be added in a subsequent patch. Finally, everything is implemented through the Openstack API resize hooks, but actual resizing of the instance RAM and hard drive space is not yet implemented.

763. By Kevin L. Mitchell on 2011-03-07

Addresses bugs 704985 and 705453 by:

1) Adding new nova-api flag --paste_config, the value of which is used in place of the hard-coded "nova-api.conf"

2) Defaulting the new --paste_config to "api-paste.ini"

764. By Kevin L. Mitchell on 2011-03-07

Inhibit inclusion of stack traces in the logs UNLESS --verbose has been specified. This should help keep the logs compact, helping admins find the messages they're interested in (e.g., "Can't connect to MySQL server on '127.0.0.1' (111)") without having to sort through the stack traces, while still allowing developers to see those traces at will.

765. By Masanori Itoh on 2011-03-08

This fix is an updated version of Todd's lp720157. Adds SignatureVersion checking for Amazon EC2 API requests, and resolves bug #720157.

766. By Trey Morris on 2011-03-08

refactored up nova/virt/xenapi/vmops _get_vm_opaque_ref()
no longer inspects the param to check to see if it is an opaque ref
works better for unittests

767. By justinsb on 2011-03-08

Fix the bug where fakerabbit is doing a sort of prefix matching on the AMQP routing key

768. By Rick Harris on 2011-03-08

Use disk_format and container_format in place of image type.

769. By Matt Dietz on 2011-03-08

Fixes lp730960 - mangled instance creation in virt drivers due to improper merge conflict resolution

770. By justinsb on 2011-03-09

Refactoring nova-api to be a service, so that we can reuse it in unit tests

771. By Christian Berendt on 2011-03-09

replaced ConnectionFailed with Exception in tools/euca-get-ajax-console
was not working for me with euca2tools 1.2 (version 2007-10-10, release 31337)

772. By Naveed Massjouni on 2011-03-09

Fixes bug #729400. Invalid values for offset and limit params in http requests now return a 400 response with a useful message in the body. Also added and updated tests.

773. By Soren Hansen on 2011-03-09

Add a decorator that lets you synchronise actions across multiple binaries. Like, say, ensuring that only one worker manipulates iptables at a time.

774. By Dan Prince on 2011-03-09

Update the create server call in the Openstack API so that it generates an 'adminPass' and calls set_admin_password in the compute API. This gets us closer to parity with the Cloud Servers v1.0 spec.

775. By Ricardo Carrillo Cruz on 2011-03-09

Hi guys

This branch fixes lp722982 (ability to delete networks with nova-manage) and lp715427 (nova-manage does not check if a network exists before creating it) .

Regards

776. By Erica Windisch on 2011-03-09

Fixes bug 726359. Passes unit tests.

Changes parameters of utils.execute to utils.execute(*cmd, **kwargs). The kwargs themselves have not changed, other than changing check_exit_code to default to 0. The exit code of the process is now checked against this variable, unless None.

777. By Brian Waldon on 2011-03-09

- Content-Type and Accept headers handled properly
- Content-Type added to responses
- Query extensions no long cause computeFaults
- adding wsgi.Request object
- removing request-specific code from wsgi.Serializer

778. By Erica Windisch on 2011-03-09

Fixes doc build after execvp patch.

779. By Vish Ishaya on 2011-03-09

Fixes a race condition where multiple greenthreads were attempting to resize a file at the same time. Adds tests to verify that the image caching call will run concurrently for different files, but will block other greenthreads trying to cache the same file.

780. By Vish Ishaya on 2011-03-10

Fixes nova.sh to run properly the first time. We have to get the zip file after nova-api is running.

781. By Vish Ishaya on 2011-03-10

Modifies S3ImageService to wrap LocalImageService or GlanceImageService. It now pulls the parts out of s3, decrypts them locally, and sends them to the underlying service. It includes various fixes for image/glance.py, image/local.py and the tests.

I also uncovered a bug in glance so for the glance backend to work properly, it requires the patch to glance here lp:~vishvananda/glance/fix-update or Glance's Cactus trunk r80.

782. By Sandy Walsh on 2011-03-10

Introduces the ZoneManager to the Scheduler which polls the child zones and caches their availability and capabilities.

783. By Soren Hansen on 2011-03-10

Fix a few things that were either missed in the execvp conversion or stuff that was merged after it, but wasn't updated accordingly.

784. By Soren Hansen on 2011-03-10

Another little bit of fallout from the execvp branch.

785. By Ken Pepple on 2011-03-10

initializing instance power state on launch to 0 (fixes EC2 API bug)

786. By Soren Hansen on 2011-03-10

Add a new IptablesManager that takes care of all uses of iptables.

Port all uses of iptables (in linux_net and libvirt_conn) over to use this new manager.

It wraps all uses of iptables so that each component can maintain its own set of rules without interfering with other components and/or existing system rules.

iptables-restore is an atomic interface to netfilter in the kernel. This means we can make a bunch of changes at a time, minimising the number of calls to iptables.

787. By Dan Prince on 2011-03-11

Make linux_net ensure_bridge commands that add and remove ip addr's from
devices/bridges work with with the latest utils.execute method (execvp).

788. By Monsyne Dragon on 2011-03-11

Adds in multi-tenant support to openstack api.
Allows for multiple accounts (projects) with admin api for creating accounts & users.

789. By Ryan Lane on 2011-03-11

Changes the output of status in describe_volumes from showing the user as the owner of the volume to showing the project as the owner.

790. By Erica Windisch on 2011-03-11

Passes net variable as value of keyword argument process_input. Prior to the execvp patch, this was passed positionally.

791. By Cory Wright on 2011-03-11

This change adds the ability to boot Windows and Linux instances in XenServer using different sets of vm-params.

Images in glance should have a property set with a key of `os_type` and a value of either `windows` or `linux`. Images without `os_type` set default to `linux`.

Linux images boot para-virtualized, while Windows images boot via HVM.

792. By Soren Hansen on 2011-03-11

Remove race condition when refreshing security groups and destroying instances at the same time.

793. By Kevin L. Mitchell on 2011-03-11

Replace raw SQL calls through session.execute() with SQLAlchemy code.

794. By Josh Kearney on 2011-03-12

Use a consistent naming scheme for XenAPI variables.

795. By Soren Hansen on 2011-03-13

Fix instructions for setting up the initial database.

796. By Erica Windisch on 2011-03-14

fixes: 733137

797. By Soren Hansen on 2011-03-14

Make nova-dhcpbridge output lease information in dnsmasq's leasesfile format.

798. By Soren Hansen on 2011-03-14

Only include kernel and ramdisk ID in meta-data output if they are actually set.

799. By Kei Masumoto on 2011-03-14

NTT's live-migration branch, merged with trunk, conflicts resolved, and migrate file renamed.

800. By Soren Hansen on 2011-03-14

Include cpuinfo.xml.template in tarball.

801. By justinsb on 2011-03-14

Fix capitalization of ApiError (it was mistakenly called APIError)

802. By justinsb on 2011-03-15

Implements basic OpenStack API client, ready to support API tests

803. By Josh Kearney on 2011-03-15

PEP8 0.5.0 cleanup.

804. By justinsb on 2011-03-15

Removed duplicated tests.

One was a straight copy-and-paste, one was different, but the masked test didn't pass.

805. By Soren Hansen on 2011-03-15

Make Authors check account for tests being run with different os.getcwd() depending on how they're run. Add missing people to Authors.

806. By Ken Pepple on 2011-03-15

just fixing a small typo in nova-manage vm live-migration

807. By Brian Lamar on 2011-03-15

Fixed lp732866 by catching relevant `exception.NotFound` exception. Tests did not uncover this vulnerability due to "incorrect" FakeAuthManager. I say "incorrect" because potentially different implementations (LDAP or Database driven) of AuthManager might return different errors from `get_user_from_access_key`.

Also, removed all references to 'bacon', 'ham', 'herp', and 'derp' and replaced them with hopefully more helpful terms.

Long story short it addresses the immediate issue while throughly ignoring the larger issue, which is correctly testing all implementations of Auth. I find this acceptable as currently the future of auth is in flux.

808. By Christian Berendt on 2011-03-16

added new class Instances for managaging instances
added new method list in class Instances:

# nova-manage instance list
instance node type state launched image kernel ramdisk project user zone index
i-00000547 XXXXXXX m1.small running 2011-02-18 08:36:37 ami-a03ndz0q ami-0isqekvw testing berendt None 0
i-00000548 XXXXXXX m1.small running 2011-02-18 08:37:17 ami-a03ndz0q ami-0isqekvw testing berendt None 1
i-00000549 XXXXXXX m1.small running 2011-02-18 08:37:52 ami-a03ndz0q ami-0isqekvw testing berendt None 2

# nova-manage instance list ares
instance node type state launched image kernel ramdisk project user zone index
i-00000c1c ares m1.tiny running 2011-02-26 22:51:40 ami-pus9dj84 ami-zhcv0yyx ami-av96fu30 testing berendt None 1

extended the method list in class FixedIps to lookup ip addresses assigned to a specified node:

# nova-manage fixed list ares
network IP address MAC address hostname host
192.168.3.0/24 192.168.3.6 02:16:3e:75:d7:9a i-00000c1c ares

809. By Thierry Carrez on 2011-03-16

Fixes euca-get-ajax-console returning Unknown Error, by using the correct
exception in get_open_port() logic. Patch from Tushar Patil.

810. By justinsb on 2011-03-16

Don't generate insecure passwords where it's easy to use urandom instead

811. By Soren Hansen on 2011-03-16

Add missing fallback chain for ipv6.

812. By Jay Pipes on 2011-03-16

Keypairs are not required in the OpenStack API; don't require them!

This cleans up some weirdness from the original branch (https://code.launchpad.net/~justin-fathomdb/nova/bug732204/+merge/53359) with justin's LP username in r804.

813. By Christian Berendt on 2011-03-16

fixes bug 735298: start of nova-compute not possible because of wrong xml paths to the //host/cpu section in "virsh capabilities", used in nova/virt/libvirt_conn.py

814. By Mark Washenberger on 2011-03-16

Fix lp727225 by adding support for personality files to the openstack api.

Description:

This merge adds support for personality files to the openstack api. It leverages previous work which added this functionality to the compute api, compute manager, and xen api.

815. By Josh Kearney on 2011-03-16

Log the use of utils.synchronized.

816. By Soren Hansen on 2011-03-17

Remove unconditional raise, probably left over from debugging.

817. By Soren Hansen on 2011-03-17

Comparisons to None should not use == or !=.

Stop converting sets to lists before comparing them. They might be in different order after being list()ified.

818. By Matt Dietz on 2011-03-17

Fixes lp736343 - Incorrect mapping of instance type id to flavor id in Openstack API

819. By Monsyne Dragon on 2011-03-17

Add support for network QoS (ratelimiting) for XenServer.
Rate is pulled from the flavor (instance_type) when constructing a vm.

820. By justinsb on 2011-03-17

Backfix of bugfix of issue blocking creating servers with metadata

821. By Naveed Massjouni on 2011-03-17

- general approach for openstack api versioning
- openstack api version now preserved in request context
- added view builder classes to handle os api responses
- added imageRef and flavorRef to os api v1.1 servers
- modified addresses container structure in os api v1.1 servers

822. By Soren Hansen on 2011-03-17

Make utils.execute not overwrite std{in,out,err} args to Popen on retries.
Make utils.execute reject unknown kwargs.

Add a couple of unit tests for utils.execute.

823. By Soren Hansen on 2011-03-17

Fix a number of place in the volume driver where the argv hadn't been fully split

824. By Soren Hansen on 2011-03-17

Fix a couple of things that assume that libvirt == kvm/qemu.

825. By Dan Prince on 2011-03-17

Update the Openstack API to handle case where personality is set but null in the request to create a server.

826. By Todd Willey on 2011-03-18

Make the rpc cast/call debug calls show what topic they are sending to. This aides in debuugging.

827. By Todd Willey on 2011-03-18

When changing the project manager, if the new manager is not yet a project member, be sure to make them be a project member.

828. By Anthony Young on 2011-03-18

Make "ApiError" the default error code for ApiError instances, rather than "Unknown."

Note Dashoard's error handling code:

        except boto.exception.BotoServerError, e:
            if e.status == 400 and e.error_code == 'ApiError':
                raise NovaApiError(e)
            elif e.status == 401:
                raise NovaUnauthorizedError()
            elif e.status == 503:
                raise NovaUnavailableError(e)
            raise NovaServerError(e)

So an error_code of 'Unknown' will raise an ugly exception.

This change also makes the euca errors a bit prettier:

> euca-attach-volume -i i-1 vol-2 -d /dev/vdb
> ApiError: ApiError: Volume status must be available

Rather than:

> euca-attach-volume -i i-1 vol-2 -d /dev/vdb
> Unknown: Unknown: Volume status must be available

Though the reiteration of the error code is still a bit ugly.

829. By justinsb on 2011-03-18

Mark instance metadata as deleted when we delete the instance

830. By Brian Lamar on 2011-03-18

Re-implementation (or just implementation in many cases) of Limits in the OpenStack API. Limits is now available through /limits and the concept of a limit has been extended to include arbitrary regex / http verb combinations along with correct XML/JSON serialization. Tests included.

831. By Dan Prince on 2011-03-18

Change cloud.id_to_ec2_id to ec2utils.id_to_ec2_id. Fixes EC2 API error handling when invalid instances and volume names are specified.

832. By Brian Lamar on 2011-03-18

Replaced all pylint "disable-msg=" with "disable=" and "enable-msg=" with "enable=".

833. By justinsb on 2011-03-18

Cleanup of FakeAuthManager

834. By Tushar Patil on 2011-03-19

Added run_instances method to the connection.py of the contrib/boto_v6/ec2 which would return ReservationV6 object instead of Reservation in order to access attribute dns_name_v6 of an instance.

835. By Soren Hansen on 2011-03-20

Make smoketests' exit code reveal whether they were succesful.

Adjust volume tests to check the exact size of the block device, instead of a rounded-off size of the resulting filesystem.

Make proxy.sh work with both variants of netcat.

836. By justinsb on 2011-03-20

Tell PyLint not to complain about the "_" function

837. By Ken Pepple on 2011-03-21

fixes nova-manage instance_type compatibility with postgres db

838. By Josh Kleinpeter on 2011-03-21

Changed fixed_range (CIDR) to be required in the nova-manage command; changed default num_networks to 1.

839. By Brian Waldon on 2011-03-21

Openstack api 1.0 flavors resource now implemented to match the spec

840. By Josh Kearney on 2011-03-21

Provide more useful exception messages when unable to load the virtual driver.

841. By Ken Pepple on 2011-03-21

wrap and log errors getting image ids from local image store

842. By Soren Hansen on 2011-03-21

Wrap update_dhcp in utils.synchronized.

843. By Trey Morris on 2011-03-22

xenapi support for multi_nic. This is a phase of multi_nic which allows xenapi to work as is and with multi_nic. The other virt driver(s) need to be updated with the same support.

844. By Todd Willey on 2011-03-22

Fix a typo in the ec2 admin api.

845. By Tushar Patil on 2011-03-22

Fix for LP Bug #704300

846. By Todd Willey on 2011-03-23

Fix the describe_vpns admin api call.

Firstly, use the correct mechanism for mapping ec2 ids from instance ids.

Secondly, if a vpn doesn't have an ip/port assignment from the project, don't try utils.vpn_ping, as it will raise an exception.

847. By justinsb on 2011-03-23

Test the login behavior of the OpenStack API. Uncovered bug732866

848. By Vish Ishaya on 2011-03-23

Correctly imports greenthread in libvirt_conn.py. It is used by live_migrate().

849. By Anne Gentle on 2011-03-23

Adding a talk bubble to the nova.openstack.org site that points readers to the 2011.1 site and the docs.openstack.org site - similar to the swift.openstack.org site. I believe it helps people see more sites are available, plus they can get to the Bexar site if they want to. Going forward it'll be nice to use this talk bubble to point people to the trunk site from released sites.

850. By Tushar Patil on 2011-03-23

Enable flat manager support for ipv6.

851. By Anthony Young on 2011-03-23

Fix for #740742 - format describe_instance_output correctly to prevent errors in dashboard.

Without this change, it is not possible to properly select instance types when launching instances with the OS dashboard.

852. By Hisaharu Ishii on 2011-03-23

We update update_ra method to synchronize, in order to prevent crash when we request multiple instance at once.

853. By Salvatore Orlando on 2011-03-23

This branch contains the fix for bug #740929
It makes sure cidr_v6 is not null before building the 'ip6s' key in the network info dictionary.
This way utils.to_global_ipv6 does not fail because of cidr==None.

854. By Mark Washenberger on 2011-03-23

Fix lp735636 by standardizing the format of image timestamp properties as datetime objects.

So far, only glance supports returning image create, update, and delete timestamps. And only the openstack api reports those dates. Thus, the fixes were confined to those two modules.

855. By Matt Dietz on 2011-03-23

Updates the previously merged xs_migration functionality to allow upsizing of the RAM and disk quotas for a XenServer instance.

856. By Soren Hansen on 2011-03-24

Move all types of locking into utils.synchronize decorator.

Convert all uses of semaphores to use this decorator.

857. By Tushar Patil on 2011-03-24

boto_v6 module is imported if the flag "use_ipv6" is set to True

858. By Josh Kearney on 2011-03-24

Offers the ability to run a periodic_task that sweeps through rescued instances older than 24 hours and forcibly unrescues them.

Flag added: rescue_timeout (default is 0 - disabled)

859. By Todd Willey on 2011-03-24

Fix issues with certificate updating & whitespace removal

860. By justinsb on 2011-03-24

better logging of exceptions

861. By Naveed Massjouni on 2011-03-24

Added a mechanism for versioned controllers for openstack api versions 1.0/1.1.
Create servers in the 1.1 api now supports imageRef/flavorRef instead of imageId/flavorId.

862. By justinsb on 2011-03-24

Poll instance states periodically, so that we can detect when something changes 'behind the scenes'.

Beginnings of work on Bug #661214 and Bug #661260.

863. By Soren Hansen on 2011-03-24

Pass a fake timing source to live_migration_pre in every test that expectes it to fail, shaving off a whole minute of test run time.

864. By Matt Dietz on 2011-03-24

Fixes lp740322: cannot run test_localization in isolation

865. By Sandy Walsh on 2011-03-24

Aggregates capabilities from Compute, Network, Volume to the ZoneManager in Scheduler.

866. By Muneyuki Noguchi on 2011-03-24

Fix lp741514 by declaring libvirt_type in nova-manage.

867. By justinsb on 2011-03-24

Detect if user is running the default Lucid version of libvirt, and give a nicer error message

868. By justinsb on 2011-03-24

Don't try to parse the empty string as a datetime

869. By Josh Kleinpeter on 2011-03-24

Made service_get_all()'s disabled parameter default to None. Pass False for enabled services; True for disabled services. Calls to this method have been updated to remain consistent.

870. By Brian Lamar on 2011-03-24

Pylint 'Undefined variable' E0602 error fixes.

871. By Todd Willey on 2011-03-24

Fix api logging to show proper path and controller:action.

872. By justinsb on 2011-03-24

Fix some errors that pylint found in nova/api/openstack/servers.py

This was meant more as a test that pylint is actually being helpful now (it is), but these are real errors.

873. By termie on 2011-03-24

Fixes a bug that was causing tests to fail on OS X by ensuring that greenthread sleep is called during retry loops.

874. By Sandy Walsh on 2011-03-24

In this branch we are forwarding incoming requests to child zones when the requested resource is not found in the current zone.

For example: If 'nova pause 123' is issued against Zone 1, but instance 123 does not live in Zone 1, the call will be forwarded to all child zones hoping someone can deal with it.

NOTE: This currently only works with OpenStack API requests and routing checks are only being done against Compute/instance_id checks.
Specifically:
* servers.get/pause/unpause/diagnostics/suspend/resume/rescue/unrescue/delete
* servers.create is pending for distributed scheduler
* servers.get_all will get added early in Diablo.

What I've been doing for testing:
1. Set up a Nova deployment in a VM (Zone0)
2. Clone the VM and set --zone_name=zone1 (and change all the IP addresses to the new address in nova.conf, glance.conf and novarc)
3. Set --enable_zone_routing=true on all zones
4. use the --connection_type=fake driver for compute to keep things easy
5. Add Zone1 as a child of Zone0 (nova zone-add)

(make sure the instance id's are different in each zone)

Example of calls being sent to child zones:
http://paste.openstack.org/show/964/

875. By termie on 2011-03-24

Additions to the Direct API:

 * Add an example of a versioned api
 * Add some more docs to direct.py
 * Add Limited, an API limiting/versioning wrapper
 * Improve the formatting of the stack tool
 * Add support for volume and network services to the direct api

876. By Ilya Alekseyev on 2011-03-24

libvirt driver multi_nic support. In this phase libvirt can work with and without multi_nic support, as in multi_nic support for xenapi: https://code.launchpad.net/~tr3buchet/nova/xs_multi_nic/+merge/53458

877. By Josh Kearney on 2011-03-24

Adds unit test coverage for XenAPI Rescue & Unrescue.

878. By Dan Prince on 2011-03-24

Implement API extensions for the Openstack API. Based on the Openstack 1.1 API the following types of extensions are supported:

-Top level resources (extension)
-Action extensions (add an extra action to a core nova controller)
-Response extensions (inject data into response from core nova controllers)

To add an extension simply drop an extension file into the configured osapi_extensions_path (which defaults to /var/lib/nova/extensions).

See nova/tests/api/openstack/extensions/foxinsocks.py for an example Extension.

879. By Salvatore Orlando on 2011-03-24

Fix for bug #740947
Executing parted with sudo in _write_partition (vm_utils.py)

880. By Rick Harris on 2011-03-24

Adds serverId to OpenStack API image detail per related_image blueprint

881. By Sateesh on 2011-03-24

Implementation of blueprint hypervisor-vmware-vsphere-support. (Link to blueprint: https://blueprints.launchpad.net/nova/+spec/hypervisor-vmware-vsphere-support)

Adds support for hypervisor vmware ESX/ESXi server in OpenStack (Nova).

Key features included are,
1) Support for FLAT and VLAN networking model
2) Support for Guest console access through VMware vmrc
3) Integrated with Glance service for image storage and retrival

Documentation: A readme file at "doc/source/vmwareapi_readme.rst" encapsulates configuration/installation instructions required to use this module/feature.

882. By termie on 2011-03-24

add s3server, pre-modifications

883. By termie on 2011-03-24

port s3server to eventlet/wsgi

884. By termie on 2011-03-24

rename objectstore tests

885. By termie on 2011-03-24

update test base class to monkey patch wsgi

886. By termie on 2011-03-24

port the objectstore tests to the new tests

887. By termie on 2011-03-24

remove twisted objectstore

888. By termie on 2011-03-24

don't require integrated tests to recycle connections

889. By termie on 2011-03-24

add descriptive docstring

Unmerged revisions

=== modified file 'bin/nova-objectstore'
--- bin/nova-objectstore	2010-12-14 23:22:03 +0000
+++ bin/nova-objectstore	2011-03-04 05:34:14 +0000
@@ -36,9 +36,10 @@
 gettext.install('nova', unicode=1)
 
 from nova import flags
+from nova import log as logging
 from nova import utils
-from nova import twistd
-from nova.objectstore import handler
+from nova import wsgi
+from nova.objectstore import s3server
 
 
 FLAGS = flags.FLAGS
@@ -46,7 +47,9 @@
 
 if __name__ == '__main__':
     utils.default_flagfile()
-    twistd.serve(__file__)
-
-if __name__ == '__builtin__':
-    application = handler.get_application()  # pylint: disable-msg=C0103
+    FLAGS(sys.argv)
+    logging.setup()
+    router = s3server.S3Application(FLAGS.buckets_path)
+    server = wsgi.Server()
+    server.start(router, FLAGS.s3_port, host=FLAGS.s3_host)
+    server.wait()

=== modified file 'nova/objectstore/bucket.py'
--- nova/objectstore/bucket.py	2011-02-19 06:49:13 +0000
+++ nova/objectstore/bucket.py	2011-03-04 05:34:14 +0000
@@ -33,8 +33,7 @@
 
 
 FLAGS = flags.FLAGS
-flags.DEFINE_string('buckets_path', '$state_path/buckets',
-                    'path to s3 buckets')
+flags.DECLARE('buckets_path', 'nova.objectstore.s3server')
 
 
 class Bucket(object):

=== added file 'nova/objectstore/s3server.py'
--- nova/objectstore/s3server.py	1970-01-01 00:00:00 +0000
+++ nova/objectstore/s3server.py	2011-03-04 05:34:14 +0000
@@ -0,0 +1,303 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+#
+# Copyright 2010 United States Government as represented by the
+# Administrator of the National Aeronautics and Space Administration.
+# Copyright 2010 OpenStack LLC.
+# Copyright 2009 Facebook
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may
+# not use this file except in compliance with the License. You may obtain
+# a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+# License for the specific language governing permissions and limitations
+# under the License.
+
+"""Implementation of an S3-like storage server based on local files.
+
+Useful to test features that will eventually run on S3, or if you want to
+run something locally that was once running on S3.
+
+We don't support all the features of S3, but it does work with the
+standard S3 client for the most basic semantics. To use the standard
+S3 client with this module:
+
+    c = S3.AWSAuthConnection("", "", server="localhost", port=8888,
+                             is_secure=False)
+    c.create_bucket("mybucket")
+    c.put("mybucket", "mykey", "a value")
+    print c.get("mybucket", "mykey").body
+
+"""
+
+import bisect
+import datetime
+import hashlib
+import os
+import os.path
+import urllib
+
+import routes
+import webob
+
+from nova import flags
+from nova import log as logging
+from nova import utils
+from nova import wsgi
+
+
+FLAGS = flags.FLAGS
+flags.DEFINE_string('buckets_path', '$state_path/buckets',
+                    'path to s3 buckets')
+
+
+class S3Application(wsgi.Router):
+    """Implementation of an S3-like storage server based on local files.
+
+    If bucket depth is given, we break files up into multiple directories
+    to prevent hitting file system limits for number of files in each
+    directories. 1 means one level of directories, 2 means 2, etc.
+
+    """
+
+    def __init__(self, root_directory, bucket_depth=0, mapper=None):
+        if mapper is None:
+            mapper = routes.Mapper()
+
+        mapper.connect('/',
+                controller=lambda *a, **kw: RootHandler(self)(*a, **kw))
+        mapper.connect('/{bucket}/{object_name}',
+                controller=lambda *a, **kw: ObjectHandler(self)(*a, **kw))
+        mapper.connect('/{bucket_name}/',
+                controller=lambda *a, **kw: BucketHandler(self)(*a, **kw))
+        self.directory = os.path.abspath(root_directory)
+        if not os.path.exists(self.directory):
+            os.makedirs(self.directory)
+        self.bucket_depth = bucket_depth
+        super(S3Application, self).__init__(mapper)
+
+
+class BaseRequestHandler(wsgi.Controller):
+    def __init__(self, application):
+        self.application = application
+
+    @webob.dec.wsgify
+    def __call__(self, request):
+        method = request.method.lower()
+        f = getattr(self, method, self.invalid)
+        self.request = request
+        self.response = webob.Response()
+        params = request.environ['wsgiorg.routing_args'][1]
+        del params['controller']
+        f(**params)
+        return self.response
+
+    def get_argument(self, arg, default):
+        return self.request.str_params.get(arg, default)
+
+    def set_header(self, header, value):
+        self.response.headers[header] = value
+
+    def set_status(self, status_code):
+        self.response.status = status_code
+
+    def finish(self, body=''):
+        self.response.body = utils.utf8(body)
+
+    def invalid(self, **kwargs):
+        pass
+
+    def render_xml(self, value):
+        assert isinstance(value, dict) and len(value) == 1
+        self.set_header("Content-Type", "application/xml; charset=UTF-8")
+        name = value.keys()[0]
+        parts = []
+        parts.append('<' + utils.utf8(name) +
+                     ' xmlns="http://doc.s3.amazonaws.com/2006-03-01">')
+        self._render_parts(value.values()[0], parts)
+        parts.append('</' + utils.utf8(name) + '>')
+        self.finish('<?xml version="1.0" encoding="UTF-8"?>\n' +
+                    ''.join(parts))
+
+    def _render_parts(self, value, parts=[]):
+        if isinstance(value, basestring):
+            parts.append(utils.xhtml_escape(value))
+        elif isinstance(value, int) or isinstance(value, long):
+            parts.append(str(value))
+        elif isinstance(value, datetime.datetime):
+            parts.append(value.strftime("%Y-%m-%dT%H:%M:%S.000Z"))
+        elif isinstance(value, dict):
+            for name, subvalue in value.iteritems():
+                if not isinstance(subvalue, list):
+                    subvalue = [subvalue]
+                for subsubvalue in subvalue:
+                    parts.append('<' + utils.utf8(name) + '>')
+                    self._render_parts(subsubvalue, parts)
+                    parts.append('</' + utils.utf8(name) + '>')
+        else:
+            raise Exception("Unknown S3 value type %r", value)
+
+    def _object_path(self, bucket, object_name):
+        if self.application.bucket_depth < 1:
+            return os.path.abspath(os.path.join(
+                self.application.directory, bucket, object_name))
+        hash = hashlib.md5(object_name).hexdigest()
+        path = os.path.abspath(os.path.join(
+            self.application.directory, bucket))
+        for i in range(self.application.bucket_depth):
+            path = os.path.join(path, hash[:2 * (i + 1)])
+        return os.path.join(path, object_name)
+
+
+class RootHandler(BaseRequestHandler):
+    def get(self):
+        names = os.listdir(self.application.directory)
+        buckets = []
+        for name in names:
+            path = os.path.join(self.application.directory, name)
+            info = os.stat(path)
+            buckets.append({
+                "Name": name,
+                "CreationDate": datetime.datetime.utcfromtimestamp(
+                    info.st_ctime),
+            })
+        self.render_xml({"ListAllMyBucketsResult": {
+            "Buckets": {"Bucket": buckets},
+        }})
+
+
+class BucketHandler(BaseRequestHandler):
+    def get(self, bucket_name):
+        prefix = self.get_argument("prefix", u"")
+        marker = self.get_argument("marker", u"")
+        max_keys = int(self.get_argument("max-keys", 50000))
+        path = os.path.abspath(os.path.join(self.application.directory,
+                                            bucket_name))
+        terse = int(self.get_argument("terse", 0))
+        if not path.startswith(self.application.directory) or \
+           not os.path.isdir(path):
+            self.set_status(404)
+            return
+        object_names = []
+        for root, dirs, files in os.walk(path):
+            for file_name in files:
+                object_names.append(os.path.join(root, file_name))
+        skip = len(path) + 1
+        for i in range(self.application.bucket_depth):
+            skip += 2 * (i + 1) + 1
+        object_names = [n[skip:] for n in object_names]
+        object_names.sort()
+        contents = []
+
+        start_pos = 0
+        if marker:
+            start_pos = bisect.bisect_right(object_names, marker, start_pos)
+        if prefix:
+            start_pos = bisect.bisect_left(object_names, prefix, start_pos)
+
+        truncated = False
+        for object_name in object_names[start_pos:]:
+            if not object_name.startswith(prefix):
+                break
+            if len(contents) >= max_keys:
+                truncated = True
+                break
+            object_path = self._object_path(bucket_name, object_name)
+            c = {"Key": object_name}
+            if not terse:
+                info = os.stat(object_path)
+                c.update({
+                    "LastModified": datetime.datetime.utcfromtimestamp(
+                        info.st_mtime),
+                    "Size": info.st_size,
+                })
+            contents.append(c)
+            marker = object_name
+        self.render_xml({"ListBucketResult": {
+            "Name": bucket_name,
+            "Prefix": prefix,
+            "Marker": marker,
+            "MaxKeys": max_keys,
+            "IsTruncated": truncated,
+            "Contents": contents,
+        }})
+
+    def put(self, bucket_name):
+        path = os.path.abspath(os.path.join(
+            self.application.directory, bucket_name))
+        if not path.startswith(self.application.directory) or \
+           os.path.exists(path):
+            self.set_status(403)
+            return
+        os.makedirs(path)
+        self.finish()
+
+    def delete(self, bucket_name):
+        path = os.path.abspath(os.path.join(
+            self.application.directory, bucket_name))
+        if not path.startswith(self.application.directory) or \
+           not os.path.isdir(path):
+            self.set_status(404)
+            return
+        if len(os.listdir(path)) > 0:
+            self.set_status(403)
+            return
+        os.rmdir(path)
+        self.set_status(204)
+        self.finish()
+
+
+class ObjectHandler(BaseRequestHandler):
+    def get(self, bucket, object_name):
+        object_name = urllib.unquote(object_name)
+        path = self._object_path(bucket, object_name)
+        if not path.startswith(self.application.directory) or \
+           not os.path.isfile(path):
+            self.set_status(404)
+            return
+        info = os.stat(path)
+        self.set_header("Content-Type", "application/unknown")
+        self.set_header("Last-Modified", datetime.datetime.utcfromtimestamp(
+            info.st_mtime))
+        object_file = open(path, "r")
+        try:
+            self.finish(object_file.read())
+        finally:
+            object_file.close()
+
+    def put(self, bucket, object_name):
+        object_name = urllib.unquote(object_name)
+        bucket_dir = os.path.abspath(os.path.join(
+            self.application.directory, bucket))
+        if not bucket_dir.startswith(self.application.directory) or \
+           not os.path.isdir(bucket_dir):
+            self.set_status(404)
+            return
+        path = self._object_path(bucket, object_name)
+        if not path.startswith(bucket_dir) or os.path.isdir(path):
+            self.set_status(403)
+            return
+        directory = os.path.dirname(path)
+        if not os.path.exists(directory):
+            os.makedirs(directory)
+        object_file = open(path, "w")
+        object_file.write(self.request.body)
+        object_file.close()
+        self.set_header('ETag',
+                        '"%s"' % hashlib.md5(self.request.body).hexdigest())
+        self.finish()
+
+    def delete(self, bucket, object_name):
+        object_name = urllib.unquote(object_name)
+        path = self._object_path(bucket, object_name)
+        if not path.startswith(self.application.directory) or \
+           not os.path.isfile(path):
+            self.set_status(404)
+            return
+        os.unlink(path)
+        self.set_status(204)
+        self.finish()

=== modified file 'nova/test.py'
--- nova/test.py	2011-03-01 00:28:46 +0000
+++ nova/test.py	2011-03-04 05:34:14 +0000
@@ -24,6 +24,7 @@
 
 
 import datetime
+import functools
 import os
 import shutil
 import uuid
@@ -32,6 +33,8 @@
 import mox
 import shutil
 import stubout
+from eventlet import greenpool
+from eventlet import greenthread
 
 from nova import context
 from nova import db
@@ -39,6 +42,7 @@
 from nova import flags
 from nova import rpc
 from nova import service
+from nova import wsgi
 
 
 FLAGS = flags.FLAGS
@@ -79,6 +83,7 @@
         self.injected = []
         self._services = []
         self._monkey_patch_attach()
+        self._monkey_patch_wsgi()
         self._original_flags = FLAGS.FlagValuesDict()
 
     def tearDown(self):
@@ -99,7 +104,8 @@
             self.reset_flags()
 
             # Reset our monkey-patches
-            rpc.Consumer.attach_to_eventlet = self.originalAttach
+            rpc.Consumer.attach_to_eventlet = self.original_attach
+            wsgi.Server.start = self.original_start
 
             # Stop any timers
             for x in self.injected:
@@ -141,12 +147,33 @@
         return svc
 
     def _monkey_patch_attach(self):
-        self.originalAttach = rpc.Consumer.attach_to_eventlet
+        self.original_attach = rpc.Consumer.attach_to_eventlet
 
-        def _wrapped(innerSelf):
-            rv = self.originalAttach(innerSelf)
+        def _wrapped(inner_self):
+            rv = self.original_attach(inner_self)
             self.injected.append(rv)
             return rv
 
-        _wrapped.func_name = self.originalAttach.func_name
+        _wrapped.func_name = self.original_attach.func_name
         rpc.Consumer.attach_to_eventlet = _wrapped
+
+    def _monkey_patch_wsgi(self):
+        """Allow us to kill servers spawned by wsgi.Server."""
+        # TODO(termie): change these patterns to use functools
+        self.original_start = wsgi.Server.start
+
+        @functools.wraps(self.original_start)
+        def _wrapped_start(inner_self, *args, **kwargs):
+            original_spawn_n = inner_self.pool.spawn_n
+
+            @functools.wraps(original_spawn_n)
+            def _wrapped_spawn_n(*args, **kwargs):
+                rv = greenthread.spawn(*args, **kwargs)
+                self._services.append(rv)
+
+            inner_self.pool.spawn_n = _wrapped_spawn_n
+            self.original_start(inner_self, *args, **kwargs)
+            inner_self.pool.spawn_n = original_spawn_n
+
+        _wrapped_start.func_name = self.original_start.func_name
+        wsgi.Server.start = _wrapped_start

=== removed file 'nova/tests/objectstore_unittest.py'
--- nova/tests/objectstore_unittest.py	2011-02-23 19:56:37 +0000
+++ nova/tests/objectstore_unittest.py	1970-01-01 00:00:00 +0000
@@ -1,315 +0,0 @@
-# vim: tabstop=4 shiftwidth=4 softtabstop=4
-
-# Copyright 2010 United States Government as represented by the
-# Administrator of the National Aeronautics and Space Administration.
-# All Rights Reserved.
-#
-#    Licensed under the Apache License, Version 2.0 (the "License"); you may
-#    not use this file except in compliance with the License. You may obtain
-#    a copy of the License at
-#
-#         http://www.apache.org/licenses/LICENSE-2.0
-#
-#    Unless required by applicable law or agreed to in writing, software
-#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
-#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
-#    License for the specific language governing permissions and limitations
-#    under the License.
-
-"""
-Unittets for S3 objectstore clone.
-"""
-
-import boto
-import glob
-import hashlib
-import os
-import shutil
-import tempfile
-
-from boto.s3.connection import S3Connection, OrdinaryCallingFormat
-from twisted.internet import reactor, threads, defer
-from twisted.web import http, server
-
-from nova import context
-from nova import flags
-from nova import objectstore
-from nova import test
-from nova.auth import manager
-from nova.exception import NotEmpty, NotFound
-from nova.objectstore import image
-from nova.objectstore.handler import S3
-
-
-FLAGS = flags.FLAGS
-
-# Create a unique temporary directory. We don't delete after test to
-# allow checking the contents after running tests. Users and/or tools
-# running the tests need to remove the tests directories.
-OSS_TEMPDIR = tempfile.mkdtemp(prefix='test_oss-')
-
-# Create bucket/images path
-os.makedirs(os.path.join(OSS_TEMPDIR, 'images'))
-os.makedirs(os.path.join(OSS_TEMPDIR, 'buckets'))
-
-
-class ObjectStoreTestCase(test.TestCase):
-    """Test objectstore API directly."""
-
-    def setUp(self):
-        """Setup users and projects."""
-        super(ObjectStoreTestCase, self).setUp()
-        self.flags(buckets_path=os.path.join(OSS_TEMPDIR, 'buckets'),
-                   images_path=os.path.join(OSS_TEMPDIR, 'images'),
-                   ca_path=os.path.join(os.path.dirname(__file__), 'CA'))
-
-        self.auth_manager = manager.AuthManager()
-        self.auth_manager.create_user('user1')
-        self.auth_manager.create_user('user2')
-        self.auth_manager.create_user('admin_user', admin=True)
-        self.auth_manager.create_project('proj1', 'user1', 'a proj', ['user1'])
-        self.auth_manager.create_project('proj2', 'user2', 'a proj', ['user2'])
-        self.context = context.RequestContext('user1', 'proj1')
-
-    def tearDown(self):
-        """Tear down users and projects."""
-        self.auth_manager.delete_project('proj1')
-        self.auth_manager.delete_project('proj2')
-        self.auth_manager.delete_user('user1')
-        self.auth_manager.delete_user('user2')
-        self.auth_manager.delete_user('admin_user')
-        super(ObjectStoreTestCase, self).tearDown()
-
-    def test_buckets(self):
-        """Test the bucket API."""
-        objectstore.bucket.Bucket.create('new_bucket', self.context)
-        bucket = objectstore.bucket.Bucket('new_bucket')
-
-        # creator is authorized to use bucket
-        self.assert_(bucket.is_authorized(self.context))
-
-        # another user is not authorized
-        context2 = context.RequestContext('user2', 'proj2')
-        self.assertFalse(bucket.is_authorized(context2))
-
-        # admin is authorized to use bucket
-        admin_context = context.RequestContext('admin_user', None)
-        self.assertTrue(bucket.is_authorized(admin_context))
-
-        # new buckets are empty
-        self.assertTrue(bucket.list_keys()['Contents'] == [])
-
-        # storing keys works
-        bucket['foo'] = "bar"
-
-        self.assertEquals(len(bucket.list_keys()['Contents']), 1)
-
-        self.assertEquals(bucket['foo'].read(), 'bar')
-
-        # md5 of key works
-        self.assertEquals(bucket['foo'].md5, hashlib.md5('bar').hexdigest())
-
-        # deleting non-empty bucket should throw a NotEmpty exception
-        self.assertRaises(NotEmpty, bucket.delete)
-
-        # deleting key
-        del bucket['foo']
-
-        # deleting empty bucket
-        bucket.delete()
-
-        # accessing deleted bucket throws exception
-        self.assertRaises(NotFound, objectstore.bucket.Bucket, 'new_bucket')
-
-    def test_images(self):
-        self.do_test_images('1mb.manifest.xml', True,
-                            'image_bucket1', 'i-testing1')
-
-    def test_images_no_kernel_or_ramdisk(self):
-        self.do_test_images('1mb.no_kernel_or_ramdisk.manifest.xml',
-                            False, 'image_bucket2', 'i-testing2')
-
-    def do_test_images(self, manifest_file, expect_kernel_and_ramdisk,
-                             image_bucket, image_name):
-        "Test the image API."
-
-        # create a bucket for our bundle
-        objectstore.bucket.Bucket.create(image_bucket, self.context)
-        bucket = objectstore.bucket.Bucket(image_bucket)
-
-        # upload an image manifest/parts
-        bundle_path = os.path.join(os.path.dirname(__file__), 'bundle')
-        for path in glob.glob(bundle_path + '/*'):
-            bucket[os.path.basename(path)] = open(path, 'rb').read()
-
-        # register an image
-        image.Image.register_aws_image(image_name,
-                                       '%s/%s' % (image_bucket, manifest_file),
-                                       self.context)
-
-        # verify image
-        my_img = image.Image(image_name)
-        result_image_file = os.path.join(my_img.path, 'image')
-        self.assertEqual(os.stat(result_image_file).st_size, 1048576)
-
-        sha = hashlib.sha1(open(result_image_file).read()).hexdigest()
-        self.assertEqual(sha, '3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3')
-
-        if expect_kernel_and_ramdisk:
-            # Verify the default kernel and ramdisk are set
-            self.assertEqual(my_img.metadata['kernelId'], 'aki-test')
-            self.assertEqual(my_img.metadata['ramdiskId'], 'ari-test')
-        else:
-            # Verify that the default kernel and ramdisk (the one from FLAGS)
-            # doesn't get embedded in the metadata
-            self.assertFalse('kernelId' in my_img.metadata)
-            self.assertFalse('ramdiskId' in my_img.metadata)
-
-        # verify image permissions
-        context2 = context.RequestContext('user2', 'proj2')
-        self.assertFalse(my_img.is_authorized(context2))
-
-        # change user-editable fields
-        my_img.update_user_editable_fields({'display_name': 'my cool image'})
-        self.assertEqual('my cool image', my_img.metadata['displayName'])
-        my_img.update_user_editable_fields({'display_name': ''})
-        self.assert_(not my_img.metadata['displayName'])
-
-
-class TestHTTPChannel(http.HTTPChannel):
-    """Dummy site required for twisted.web"""
-
-    def checkPersistence(self, _, __):  # pylint: disable-msg=C0103
-        """Otherwise we end up with an unclean reactor."""
-        return False
-
-
-class TestSite(server.Site):
-    """Dummy site required for twisted.web"""
-    protocol = TestHTTPChannel
-
-
-class S3APITestCase(test.TestCase):
-    """Test objectstore through S3 API."""
-
-    def setUp(self):
-        """Setup users, projects, and start a test server."""
-        super(S3APITestCase, self).setUp()
-
-        FLAGS.auth_driver = 'nova.auth.ldapdriver.FakeLdapDriver'
-        FLAGS.buckets_path = os.path.join(OSS_TEMPDIR, 'buckets')
-
-        self.auth_manager = manager.AuthManager()
-        self.admin_user = self.auth_manager.create_user('admin', admin=True)
-        self.admin_project = self.auth_manager.create_project('admin',
-                                                              self.admin_user)
-
-        shutil.rmtree(FLAGS.buckets_path)
-        os.mkdir(FLAGS.buckets_path)
-
-        root = S3()
-        self.site = TestSite(root)
-        # pylint: disable-msg=E1101
-        self.listening_port = reactor.listenTCP(0, self.site,
-                                                interface='127.0.0.1')
-        # pylint: enable-msg=E1101
-        self.tcp_port = self.listening_port.getHost().port
-
-        if not boto.config.has_section('Boto'):
-            boto.config.add_section('Boto')
-        boto.config.set('Boto', 'num_retries', '0')
-        self.conn = S3Connection(aws_access_key_id=self.admin_user.access,
-                                 aws_secret_access_key=self.admin_user.secret,
-                                 host='127.0.0.1',
-                                 port=self.tcp_port,
-                                 is_secure=False,
-                                 calling_format=OrdinaryCallingFormat())
-
-        def get_http_connection(host, is_secure):
-            """Get a new S3 connection, don't attempt to reuse connections."""
-            return self.conn.new_http_connection(host, is_secure)
-
-        self.conn.get_http_connection = get_http_connection
-
-    def _ensure_no_buckets(self, buckets):  # pylint: disable-msg=C0111
-        self.assertEquals(len(buckets), 0, "Bucket list was not empty")
-        return True
-
-    def _ensure_one_bucket(self, buckets, name):  # pylint: disable-msg=C0111
-        self.assertEquals(len(buckets), 1,
-                          "Bucket list didn't have exactly one element in it")
-        self.assertEquals(buckets[0].name, name, "Wrong name")
-        return True
-
-    def test_000_list_buckets(self):
-        """Make sure we are starting with no buckets."""
-        deferred = threads.deferToThread(self.conn.get_all_buckets)
-        deferred.addCallback(self._ensure_no_buckets)
-        return deferred
-
-    def test_001_create_and_delete_bucket(self):
-        """Test bucket creation and deletion."""
-        bucket_name = 'testbucket'
-
-        deferred = threads.deferToThread(self.conn.create_bucket, bucket_name)
-        deferred.addCallback(lambda _:
-                             threads.deferToThread(self.conn.get_all_buckets))
-
-        deferred.addCallback(self._ensure_one_bucket, bucket_name)
-
-        deferred.addCallback(lambda _:
-                             threads.deferToThread(self.conn.delete_bucket,
-                                                   bucket_name))
-        deferred.addCallback(lambda _:
-                             threads.deferToThread(self.conn.get_all_buckets))
-        deferred.addCallback(self._ensure_no_buckets)
-        return deferred
-
-    def test_002_create_bucket_and_key_and_delete_key_again(self):
-        """Test key operations on buckets."""
-        bucket_name = 'testbucket'
-        key_name = 'somekey'
-        key_contents = 'somekey'
-
-        deferred = threads.deferToThread(self.conn.create_bucket, bucket_name)
-        deferred.addCallback(lambda b:
-                             threads.deferToThread(b.new_key, key_name))
-        deferred.addCallback(lambda k:
-                             threads.deferToThread(k.set_contents_from_string,
-                                                   key_contents))
-
-        def ensure_key_contents(bucket_name, key_name, contents):
-            """Verify contents for a key in the given bucket."""
-            bucket = self.conn.get_bucket(bucket_name)
-            key = bucket.get_key(key_name)
-            self.assertEquals(key.get_contents_as_string(), contents,
-                              "Bad contents")
-
-        deferred.addCallback(lambda _:
-                             threads.deferToThread(ensure_key_contents,
-                                                   bucket_name, key_name,
-                                                   key_contents))
-
-        def delete_key(bucket_name, key_name):
-            """Delete a key for the given bucket."""
-            bucket = self.conn.get_bucket(bucket_name)
-            key = bucket.get_key(key_name)
-            key.delete()
-
-        deferred.addCallback(lambda _:
-                             threads.deferToThread(delete_key, bucket_name,
-                                                   key_name))
-        deferred.addCallback(lambda _:
-                             threads.deferToThread(self.conn.get_bucket,
-                                                   bucket_name))
-        deferred.addCallback(lambda b: threads.deferToThread(b.get_all_keys))
-        deferred.addCallback(self._ensure_no_buckets)
-        return deferred
-
-    def tearDown(self):
-        """Tear down auth and test server."""
-        self.auth_manager.delete_user('admin')
-        self.auth_manager.delete_project('admin')
-        stop_listening = defer.maybeDeferred(self.listening_port.stopListening)
-        super(S3APITestCase, self).tearDown()
-        return defer.DeferredList([stop_listening])

=== added file 'nova/tests/test_objectstore.py'
--- nova/tests/test_objectstore.py	1970-01-01 00:00:00 +0000
+++ nova/tests/test_objectstore.py	2011-03-04 05:34:14 +0000
@@ -0,0 +1,150 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright 2010 United States Government as represented by the
+# Administrator of the National Aeronautics and Space Administration.
+# All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+"""
+Unittets for S3 objectstore clone.
+"""
+
+import boto
+import glob
+import hashlib
+import os
+import shutil
+import tempfile
+
+from boto import exception as boto_exception
+from boto.s3 import connection as s3
+
+from nova import context
+from nova import exception
+from nova import flags
+from nova import objectstore
+from nova import wsgi
+from nova import test
+from nova.auth import manager
+#from nova.exception import NotEmpty, NotFound
+from nova.objectstore import s3server
+
+
+FLAGS = flags.FLAGS
+
+# Create a unique temporary directory. We don't delete after test to
+# allow checking the contents after running tests. Users and/or tools
+# running the tests need to remove the tests directories.
+OSS_TEMPDIR = tempfile.mkdtemp(prefix='test_oss-')
+
+# Create bucket/images path
+os.makedirs(os.path.join(OSS_TEMPDIR, 'images'))
+os.makedirs(os.path.join(OSS_TEMPDIR, 'buckets'))
+
+
+class S3APITestCase(test.TestCase):
+    """Test objectstore through S3 API."""
+
+    def setUp(self):
+        """Setup users, projects, and start a test server."""
+        super(S3APITestCase, self).setUp()
+        self.flags(auth_driver='nova.auth.ldapdriver.FakeLdapDriver',
+                   buckets_path=os.path.join(OSS_TEMPDIR, 'buckets'),
+                   s3_host='127.0.0.1')
+
+        self.auth_manager = manager.AuthManager()
+        self.admin_user = self.auth_manager.create_user('admin', admin=True)
+        self.admin_project = self.auth_manager.create_project('admin',
+                                                              self.admin_user)
+
+        shutil.rmtree(FLAGS.buckets_path)
+        os.mkdir(FLAGS.buckets_path)
+
+        router = s3server.S3Application(FLAGS.buckets_path)
+        server = wsgi.Server()
+        server.start(router, FLAGS.s3_port, host=FLAGS.s3_host)
+
+        if not boto.config.has_section('Boto'):
+            boto.config.add_section('Boto')
+        boto.config.set('Boto', 'num_retries', '0')
+        conn = s3.S3Connection(aws_access_key_id=self.admin_user.access,
+                               aws_secret_access_key=self.admin_user.secret,
+                               host=FLAGS.s3_host,
+                               port=FLAGS.s3_port,
+                               is_secure=False,
+                               calling_format=s3.OrdinaryCallingFormat())
+        self.conn = conn
+
+        def get_http_connection(host, is_secure):
+            """Get a new S3 connection, don't attempt to reuse connections."""
+            return self.conn.new_http_connection(host, is_secure)
+
+        self.conn.get_http_connection = get_http_connection
+
+    def _ensure_no_buckets(self, buckets):  # pylint: disable-msg=C0111
+        self.assertEquals(len(buckets), 0, "Bucket list was not empty")
+        return True
+
+    def _ensure_one_bucket(self, buckets, name):  # pylint: disable-msg=C0111
+        self.assertEquals(len(buckets), 1,
+                          "Bucket list didn't have exactly one element in it")
+        self.assertEquals(buckets[0].name, name, "Wrong name")
+        return True
+
+    def test_000_list_buckets(self):
+        """Make sure we are starting with no buckets."""
+        self._ensure_no_buckets(self.conn.get_all_buckets())
+
+    def test_001_create_and_delete_bucket(self):
+        """Test bucket creation and deletion."""
+        bucket_name = 'testbucket'
+
+        self.conn.create_bucket(bucket_name)
+        self._ensure_one_bucket(self.conn.get_all_buckets(), bucket_name)
+        self.conn.delete_bucket(bucket_name)
+        self._ensure_no_buckets(self.conn.get_all_buckets())
+
+    def test_002_create_bucket_and_key_and_delete_key_again(self):
+        """Test key operations on buckets."""
+        bucket_name = 'testbucket'
+        key_name = 'somekey'
+        key_contents = 'somekey'
+
+        b = self.conn.create_bucket(bucket_name)
+        k = b.new_key(key_name)
+        k.set_contents_from_string(key_contents)
+
+        bucket = self.conn.get_bucket(bucket_name)
+
+        # make sure the contents are correct
+        key = bucket.get_key(key_name)
+        self.assertEquals(key.get_contents_as_string(), key_contents,
+                          "Bad contents")
+
+        # delete the key
+        key.delete()
+
+        self._ensure_no_buckets(bucket.get_all_keys())
+
+    def test_unknown_bucket(self):
+        bucket_name = 'falalala'
+        self.assertRaises(boto_exception.S3ResponseError,
+                          self.conn.get_bucket,
+                          bucket_name)
+
+    def tearDown(self):
+        """Tear down auth and test server."""
+        self.auth_manager.delete_user('admin')
+        self.auth_manager.delete_project('admin')
+        super(S3APITestCase, self).tearDown()