Merge lp:~niemeyer/pyjuju/user-oriented-docs into lp:pyjuju

[Gustavo Niemeyer]

This cleans up the documentation so that it is user oriented and up-to-date
(no non-existing features looking like they are in place).

For your viewing pleasure, a compiled version of the tip of this
branch is at:

    http://j.mp/ensemble-docs

[lp:~niemeyer/pyjuju/user-oriented-docs]

187. By Gustavo Niemeyer on 2011-03-18

getting-started: Lowercase title.

[Ahmed Kamal (kim0)]

FWIW, I've read the changes, and they look good. Got a couple of notes
- Can we explain what bootstrapping means somewhere
- Are relation interfaces of enum'eratable types? or free form

[Benjamin Saller (bcsaller)]

This patch represent a much needed improvement to the documentation
situation (which I think is now better than many project at this phase
of its lifecycle).

More detailed comments below.

> === added file 'docs/source/about.rst'
> --- docs/source/about.rst       1970-01-01 00:00:00 +0000
> +++ docs/source/about.rst       2011-03-17 08:38:25 +0000
> @@ -0,0 +1,32 @@
> +About Ensemble
> +==============
> +
> +Since long ago, Linux server deployments have been moving towards the
> +collaboration of multiple physical machines. In some cases, different servers
> +run each a different set of applications, bringing organization, isolation,
> +reserved resources, and other desirable characteristics to the composed
> +assembly. In other situations, servers are set up with very similar
> +configurations, so that the system becomes more scalable by having load
> +distributed among the several instances, and so that the overall system becomes
> +more reliable when the failure of any individual machine does not affect the
> +assembly as a whole. In this reality, server administrators become invaluable
> +maestros which orchestrate the placement and connectivity of services within
> +the assembly of servers.
> +
> +Given that scenario, it's surprising that most of the efforts towards advancing
> +the management of software configuration is still bound to individual machines.
> +Package managers, and softwares like dbus and gconf are examples of this. Other
> +efforts do look at the problem of managing multiple machines as a unit, but
> +interestingly, they are still a mechanism for scaling up the management of
> +services individually. In other words, they empower the administrator with the
> +ability to tweak the individual configuration of multiple services at once,
> +but they do not collaborate towards offering services themselves and other tools
> +the understanding of the composed ensemble. This distinction looks subtle in
> +principle, but it may be a key factor in enabling all the parties (system
> +administrators, software developers, vendors, and integrators) to collaborate
> +in deploying, maintaining, and enriching distributed software configurations.
> +
> +This is the challenge which motivates the research happening through the
> +Ensemble project at Canonical. Ensemble aims to be a service deployment and
> +orchestration tool which enables the same kind of collaboration which today is
> +seen around package management to happen on a higher level, around services.
>

[1] Understanding that machines is not central is important but I
think for end users the 'apt for cloud' idea should be emphasized a
bit more here. While it is the conclusion it might still make some
sense to talk about the desire of users to have a service running in
the cloud with much less through to how its deployed. The SaaS story.

As a nit-pick I think it should be 'software like dbus ...'

> === removed file 'docs/source/cli.rst'
> --- docs/source/cli.rst 2010-11-11 20:13:38 +0000
> +++ docs/source/cli.rst 1970-01-01 00:00:00 +0000
> -Service Deployment
> -------------------
> -
> -The ``ensemble deploy`` command lets you add a service to an
> -environment::
> -
> -  $...

[lp:~niemeyer/pyjuju/user-oriented-docs]

188. By Gustavo Niemeyer on 2011-03-22

Fixes after review from Ben.

[Gustavo Niemeyer (niemeyer)]

> This patch represent a much needed improvement to the documentation
> situation (which I think is now better than many project at this phase
> of its lifecycle).

Thanks Ben, and thanks for the great review as well.

> [1] Understanding that machines is not central is important but I
> think for end users the 'apt for cloud' idea should be emphasized a
> bit more here. While it is the conclusion it might still make some
> sense to talk about the desire of users to have a service running in
> the cloud with much less through to how its deployed. The SaaS story.

Good point. I've changed the last paragraph and attempted to drive it
a towards that side. Please note that we're walking a thin line
in those terms. We can't emphasize too much the "no thinking" aspect,
because it's both untrue (people will still need a good level of
knowledge at this point) and discouraging for an important part of our
user base, in the sense that it may drive the *real* admins away
("What!? No thinking!? Nevermind.").

> As a nit-pick I think it should be 'software like dbus ...'

I certainly appreciate that kind of nitpick, but I'm not entirely sure
I get it. Is it the case that "software" has no plural form?

> [2] I'm fine with this file being removed but we should have a
> sections that at minimum dumps the logical output of
>
> for subcommand in ensemble:
> ensemble subcommand --help

This sounds like a nice idea, but we have to automate that, otherwise
it very quickly becomes obsolete and then is worse than having nothing
(after all, people can already type --help and get those details).

We can introduce that automated generation in a separate branch,
if desired.

> I think as it stands now there is no usage for things like deploy in the
> docs.

We need to improve the getting-started document. I'm hoping to take
the input from Clint and append at the bottom of it in a separate
branch.

> [3] While these types of notes (the not yet supported one) are helpful
> they are hard to track down and keep up to date. I suppose the author
> of a change adding support for this feature would be able to grep for
> all usage of 'peer' in the docs and find things like this, but I worry
> that it will not always be clear.

Agreed. I'm also not sure about how to handle those details properly.
Either we review every once in a while, or we comment out that
information entirely.

What I'm trying to achieve with those notes is making the documentation
realistic. Having a full body of documentation which mixes fantasy
and working pieces generates lack of trust in what is written. To
avoid that, we have to ensure that whatever we ask people to read should
be the status quo. This also means that those new specifications we
are writing should either sit within the Drafts section, or should have
a top-level ".. :note:" pointing the status out.

> [4] I'll have to merge in the other common elements when the
> config/settings stuff lands, just noting it here so we both keep it in
> mind later.

Super, thanks.

You can also document it right away, and leave it commented out for
the moment, or add the note saying it's in progress.

> [5] Very nice to have these additio...

[lp:~niemeyer/pyjuju/user-oriented-docs]

189. By Gustavo Niemeyer on 2011-03-22

s/softwares/software/, thanks Ben.

[Benjamin Saller (bcsaller)]

After the last minor rounds change this looks good. Thanks +1

review: Approve

=== modified file 'README'
--- README	2011-03-03 07:17:37 +0000
+++ README	2011-03-22 18:29:09 +0000
@@ -9,5 +9,5 @@
 
 The Ensemble bug tracker is at https://bugs.launchpad.net/ensemble/.
 
-doc/sources/getting_started.rst explains how to actually get Ensemble up and
+doc/sources/getting-started.rst explains how to actually get Ensemble up and
 running.

=== added file 'docs/source/about.rst'
--- docs/source/about.rst	1970-01-01 00:00:00 +0000
+++ docs/source/about.rst	2011-03-22 18:29:09 +0000
@@ -0,0 +1,38 @@
+About Ensemble
+==============
+
+Since long ago, Linux server deployments have been moving towards the
+collaboration of multiple physical machines. In some cases, different servers
+run each a different set of applications, bringing organization, isolation,
+reserved resources, and other desirable characteristics to the composed
+assembly. In other situations, servers are set up with very similar
+configurations, so that the system becomes more scalable by having load
+distributed among the several instances, and so that the overall system becomes
+more reliable when the failure of any individual machine does not affect the
+assembly as a whole. In this reality, server administrators become invaluable
+maestros which orchestrate the placement and connectivity of services within
+the assembly of servers.
+
+Given that scenario, it's surprising that most of the efforts towards advancing
+the management of software configuration are still bound to individual machines.
+Package managers, and software like dbus and gconf are examples of this. Other
+efforts do look at the problem of managing multiple machines as a unit, but
+interestingly, they are still a mechanism for scaling up the management of
+services individually. In other words, they empower the administrator with the
+ability to tweak the individual configuration of multiple services at once,
+but they do not collaborate towards offering services themselves and other tools
+the understanding of the composed ensemble. This distinction looks subtle in
+principle, but it may be a key factor in enabling all the parties (system
+administrators, software developers, vendors, and integrators) to collaborate
+in deploying, maintaining, and enriching distributed software configurations.
+
+This is the challenge which motivates the research happening through the
+Ensemble project at Canonical. Ensemble aims to be a service deployment and
+orchestration tool which enables the same kind of collaboration and ease of
+use which today is seen around package management to happen on a higher
+level, around services.  With Ensemble, different authors are able to create
+services independently, and make those services communicate through a simple
+configuration protocol.  Then, users can take the product of both authors
+and very comfortably deploy those services in an environment, in way
+resembling how people are able to install a network of packages with a single
+command via APT.

=== removed file 'docs/source/cli.rst'
--- docs/source/cli.rst	2010-11-11 20:13:38 +0000
+++ docs/source/cli.rst	1970-01-01 00:00:00 +0000
@@ -1,178 +0,0 @@
-.. _cli:
-
-Command-line Interface
-======================
-
-:term:`Ensemble` offers a command-line utility to manage your deployments.
-
-The use of the command-line interface requires the configuration of
-one or more :ref:`environments <environment>` and one
-or more :ref:`formula repositories <formula-repository>` (see the
-:ref:`getting started <getting-started>` section for a general
-introduction).
-
-
-Formula Search
---------------
-
-The ``ensemble search`` command lets you query one or more formula
-repositories to list matching formulas::
-
-  $ ensemble search [OPTION] ...
-
-Formulas for which the provided query string occur in their name are
-listed in alphabetical order.
-
-
-Service Deployment
-------------------
-
-The ``ensemble deploy`` command lets you add a service to an
-environment::
-
-  $ ensemble deploy [OPTION] ...
-
-Since there can be multiple distinct services of the same kind running
-in a single deployment, we identify each service with a string
-identifier.
-
-Each service must be listed as ``<service>:<name>``. For instance, we
-could add a blogging engine::
-
-  $ ensemble deploy wordpress:myblog
-
-This assumes an environment configured with the name ``"default"``. By
-providing the ``--environment`` option, an alternative environment may
-be used.
-
-::
-
-  $ ensemble deploy --environment mycloud wordpress:myblog
-
-
-Managing Relations
-------------------
-
-Services are linked by relations. In order for a service to talk to
-another service, you need to first add that relation in your Ensemble
-environment (synonym, establish a relation).  Likewise, you can also
-remove a given relation between services (synonym, break a relation).
-
-.. note:: Service interchangeability
-  
-  Relation management is fundamental to Ensemble. Services do not
-  define relations, only how they may be related to other services.
-  This concept then enables an interchangeability of services. For
-  example, if your service uses the MySQL database service, you can
-  replace it with a logically equivalent MySQL service, perhaps with
-  better scalability because it uses sharding. To perform this
-  migration, use the following process:
-
-    1. Remove the relation to the old service.
-
-    2. Add a relation to the desired new service in its place.
-
-  This section documents the necessary commands to add and remove
-  relations.
-
-To add a relation between two services, use the ``ensemble
-add-relation`` command::
-
-  $ ensemble add-relation <service name 1>[:<relation name>] <service name 2>[:<relation name>]
-
-The order of services in this command (``<service name 1>``, ``<service
-name 2>``) is not important. Instead, the services themselves
-define whether they are a provider or consumer on a given ``<relation
-name>``. The ``<relation name>`` is only necessary to disambiguate
-situations where a service consumes or produces multiple relations.
-
-For example, you might have a service provider in your formula::
-
-  ensemble: formula
-  name: mysql
-  revision: 1
-  summary: "A pretty popular database"
-  provides:
-    server: mysql
-
-And you also have a formula for a corresponding service consumer::
-
-  ensemble: formula
-  name: wordpress
-  revision: 3
-  summary: "A pretty popular blog engine"
-  provides:
-    url: http
-  requires:
-    db: mysql
-
-Let's work through how to read these formulas. ``mysql`` is the common
-relation type for these two services. In the ``mysql`` service (note
-it's the same name as the relation type!), the relation name is
-``server``, whereas in ``wordpress``, the relation name is ``db``.
-Assume you have deployed your services, as in the previous
-section. Now add the required relation::
-
-  $ ensemble add-relation wordpress mysql     
-  Adding wordpress:db(5) ---mysql--> mysql:server(2)...
-  Added mysql relation to all service units.
-
-In the above formulas, there are no ambiguities, so the relation name
-can be omitted. For the sake of this example, imagine you currently
-have running 2 MySQL service units for the backing database and 5
-WordPress service units for the frontend under Ensemble. This command
-output then indicates that the relation was successfully added to all
-of them.
-
-.. note::
-
-  Was the underlying physical linkage pairwise for all the service
-  units, or a subset?  It's up to your formula :ref:`hooks <hooks>` to
-  decide the actual interconnectivity of the service units implied by
-  an established relation.
-
-Additional verbosity is supported via a ``--verbose`` flag. You might
-use this flag to help debug a hook.
-
-The following commands are also valid and entirely equivalent to the
-preceding command, given the above formula definitions::
-
-  $ ensemble add-relation mysql wordpress:db 
-
-  $ ensemble add-relation mysql:server wordpress:db
-
-  $ ensemble add-relation wordpress mysql:server
-
-In any of these invocations, during its execution the ``ensemble
-add-relation`` command will in turn propagate through the Ensemble
-platform to all running service units on both sides of the ``mysql``
-relation type. This means that for the MySQL service, Ensemble will
-run the hook ``server-relation-changed`` script defined by the
-corresponding formula in the corresponding service units; for
-WordPress, the ``db-relation-changed`` hook is run.
-
-You can also remove relations. You might need to take this action
-because you're replacing a service. Or you might have multiple
-interfaces for a relation. Maybe you support both `http` and `https`,
-and now you would like to stop supporting `http`. In any of these
-cases, use the ``ensemble remove-relation`` command::
-
-  $ ensemble remove-relation <service name 1>[:<relation name>] <service name 2>[:<relation name>]
-
-(This command is also symmetrical with respect to ``<service name>``.)
-
-To remove the ``mysql`` relation in the previous example::
-
-  $ ensemble remove-relation mysql wordpress
-  Removing wordpress:db(5) ---mysql--> mysql:server(2)...
-  Removed mysql relation from all service units.
-
-During this removal, the ``<relation name>-relation-changed`` hooks
-are called. The command completes upon successful removal of the
-relation.
-
-.. note::
-
-  TODO Add sequence diagrams with `sdedit
-  <http://www.shibu.jp/sdeditext/>`_ to show add/remove in conjunction
-  with deploy

=== added directory 'docs/source/drafts'
=== renamed file 'docs/source/specifications/formula-namespace.rst' => 'docs/source/drafts/formula-namespaces.rst'
--- docs/source/specifications/formula-namespace.rst	2010-07-30 16:17:04 +0000
+++ docs/source/drafts/formula-namespaces.rst	2011-03-22 18:29:09 +0000
@@ -1,6 +1,9 @@
 
 Formula Namespaces
-------------------
+==================
+
+Introduction
+------------
   
 Ensemble supports deployment of formulas from multiple sources.
 

=== added file 'docs/source/ensemble-drafts.rst'
--- docs/source/ensemble-drafts.rst	1970-01-01 00:00:00 +0000
+++ docs/source/ensemble-drafts.rst	2011-03-22 18:29:09 +0000
@@ -0,0 +1,10 @@
+Drafts
+======
+
+This section contains documents which may be unreviewed, incomplete,
+incorrect, out-of-date, or all of those.
+
+.. toctree::
+   :glob:
+
+   drafts/*

=== added file 'docs/source/ensemble-internals.rst'
--- docs/source/ensemble-internals.rst	1970-01-01 00:00:00 +0000
+++ docs/source/ensemble-internals.rst	2011-03-22 18:29:09 +0000
@@ -0,0 +1,11 @@
+Implementation details
+======================
+
+This section details topics which are generally not very useful
+for running Ensemble, but may be interesting if you want to hack it.
+
+.. toctree::
+   :glob:
+
+   internals/*
+

=== renamed file 'docs/source/specifications/formula.rst' => 'docs/source/formula.rst'
--- docs/source/specifications/formula.rst	2010-10-29 13:17:33 +0000
+++ docs/source/formula.rst	2011-03-22 18:29:09 +0000
@@ -1,6 +1,9 @@
 
-Formula Specification
-=====================
+Formulas
+========
+
+Introduction
+------------
 
 In Ensemble, formulas are the description of how services should
 behave when deployed and run.  A single service definition may be run
@@ -15,7 +18,8 @@
 The metadata file
 -----------------
 
-The metadata file contains details about the Formula.
+The file named `metadata.yaml`, at the root of the formula, contains
+details about the formula.
 
 The following fields are supported:
 
@@ -62,11 +66,11 @@
       * **limit:** - The maximum number of relations of this kind which
         may be established to other service units.  Defaults to 1 for
         `requires` relations, and to "none" (no limit) for `provides`
-        and `peers` relations.
+        and `peers` relations (note: that's not yet supported).
 
       * **optional:** - Whether this relation is required for the
         service unit to function or not.  Defaults to `false`, meaning
-        that the relation is required.
+        that the relation is required (note: that's not yet supported).
 
       As a shortcut, if these properties are not defined, and instead
       a single string value is provided next to the relation name,
@@ -132,8 +136,18 @@
     3. A remote service unit which was part of the relation has
        had its relation broken.
 
-    The following environment variables are defined when this hook is
-    run:
+
+Hook environment
+----------------
+
+Hooks can expect to be invoked with a standard environment and
+context. The following be included:
+
+    * `$ENSEMBLE_UNIT_NAME` - The name of the local unit executing,
+      in the form `<service name>/<unit sequence>`. E.g. myblog/3.
+
+Hooks called for relation changes will have the follow additional
+environment variables available to them.
 
     * `$ENSEMBLE_RELATION` - The relation name this hook is running
       for.  It's redundant with the hook name, but is necessary for
@@ -167,7 +181,7 @@
         relation-get port wordpress/3
 
     Get the IP address in a relation which can only be established
-    with a single unit::
+    with a single remote unit::
 
         relation-get ip
 
@@ -176,7 +190,7 @@
         relation-get - wordpress/3
 
     Get all the settings in a relation which can only be established
-    with a single unit::
+    with a single remote unit::
 
         relation-get
 
@@ -206,21 +220,8 @@
         MEMBERS=$(relation-list)
 
 
-Persistence
------------
-
-Right now Ensemble doesn't support any kind of persistence management,
-so any service units destroyed are gone for good, so don't destroy a
-service unit before the data has been properly backed up (manually).
-For now, manual backups can be achieved via snapshotting the instance
-as all formulas are run on EBS backed instances.
-
-In the near future, the formula will be able to define a persistent
-directory which will be automatically managed by Ensemble.
-
-
-Formula metadata examples
--------------------------
+Sample metadata.yaml files
+--------------------------
 
 Below are presented some sample metadata files.
 

=== modified file 'docs/source/getting-started.rst'
--- docs/source/getting-started.rst	2011-03-03 07:17:37 +0000
+++ docs/source/getting-started.rst	2011-03-22 18:29:09 +0000
@@ -1,13 +1,16 @@
 .. _getting-started:
 
-Getting Started
+Getting started
 ===============
 
-This tutorial gets you started with :term:`Ensemble`. A prerequisite
-is the access credentials to a dedicated computing environment such as
-what is offered by a virtualized cloud hosting environment.
-
-The software has been designed for environments which can provide a
+Introduction
+------------
+
+This tutorial gets you started with Ensemble. A prerequisite is the
+access credentials to a dedicated computing environment such as what
+is offered by a virtualized cloud hosting environment.
+
+Ensemble has been designed for environments which can provide a
 new machine with an Ubuntu cloud operating system image
 on-demand. This includes services such as `Amazon EC2
 <http://aws.amazon.com/ec2/>`_ or `RackSpace
@@ -16,27 +19,11 @@
 It's also required that the environment provides a permanent storage
 facility such as `Amazon S3 <https://s3.amazonaws.com/>`_.
 
-Installing the management software
-----------------------------------
-
-:term:`Ensemble` deployments are managed using a command-line
-interface on a machine with network access to the machine provider.
-
-To install the :term:`Ensemble` management client software on your
-Ubuntu desktop system, use your package manager or enter the following
-into a terminal::
-
-  $ sudo apt-get install ensemble-client
-
-This package sets up a default :ref:`formula repository
-<formula-repository>` and installs the :ref:`command line utility
-<cli>`.
-
-You do not normally need to install the server software manually; this
-is automatically taken care of during machine bootstrapping.
+For the moment, though, the only environment supported is EC2.
+
 
 Running from trunk
-~~~~~~~~~~~~~~~~~~
+------------------
 
 To run Ensemble, you will need the following dependencies installed:
 
@@ -46,16 +33,15 @@
 
 You will also need Python 2.6 or better.
 
-To get Ensemble, run::
-
- $ bzr branch lp:ensemble
-
-And to run Ensemble from trunk, do::
-
-  $ PYTHONPATH=.:$PYTHONPATH ./bin/ensemble
-
-Note that you must configure your environment before Ensemble will be of any
-use.
+To get Ensemble and run it from trunk, do::
+
+  $ bzr branch lp:ensemble
+  $ cd ensemble
+  $ export PYTHONPATH=$PWD:$PYTHONPATH
+  $ bin/ensemble
+
+Note that you must configure your environment before Ensemble will
+be of any use.  See below.
 
 
 Configuring your environment
@@ -64,7 +50,7 @@
 Run the command-line utility with no arguments to create a sample
 environment::
 
-  $ ensemble
+  $ bin/ensemble
 
 This will create the file ``~/.ensemble/environments.yaml``, which will look
 something like this::
@@ -77,13 +63,13 @@
       control-bucket: ensemble-faefb490d69a41f0a3616a4808e0766b
       admin-secret: 81a1e7429e6847c4941fda7591246594
 
-Which is a sample :term:`AWS` environment configured to run with :term:`EC2`
-machines and :term:`S3` permanent storage.  To make this environment actually
-useful, you will need to tell Ensemble about an AWS access key and secret key.
-To do this, you can either set the ``AWS_ACCESS_KEY_ID`` and
-``AWS_SECRET_ACCESS_KEY`` environment variables or you can add an
-``access-key`` and a ``secret-key`` to your ``environments.yaml``.  For
-example::
+Which is a sample environment configured to run with EC2 machines and S3
+permanent storage.  To make this environment actually useful, you will need
+to tell Ensemble about an AWS access key and secret key.  To do this, you
+can either set the ``AWS_ACCESS_KEY_ID`` and ``AWS_SECRET_ACCESS_KEY``
+environment variables (as usual for other EC2 tools) or you can add
+``access-key`` and ``secret-key`` options to your ``environments.yaml``.
+For example::
 
   ensemble: environments
 
@@ -95,11 +81,11 @@
       control-bucket: ensemble-faefb490d69a41f0a3616a4808e0766b
       admin-secret: 81a1e7429e6847c4941fda7591246594
 
-The :term:`S3` bucket does not need to exist already; the default name
-is ``"ensemble"``.
+The S3 bucket does not need to exist already.
 
-If you already have an AWS account, you can determine your access key by
-visiting http://aws.amazon.com/account, clicking "Security Credentials" and
-then clicking "Access Credentials".  You'll be taken to a table that lists
-your access keys and has a "show" link for each access key that will reveal
-the associated secret key.
+.. note::
+        If you already have an AWS account, you can determine your access key by
+        visiting http://aws.amazon.com/account, clicking "Security Credentials" and
+        then clicking "Access Credentials".  You'll be taken to a table that lists
+        your access keys and has a "show" link for each access key that will reveal
+        the associated secret key.

=== modified file 'docs/source/glossary.rst'
--- docs/source/glossary.rst	2010-08-02 19:33:23 +0000
+++ docs/source/glossary.rst	2011-03-22 18:29:09 +0000
@@ -4,14 +4,90 @@
 ========
 
 .. glossary::
-   :sorted:
+
+   Bootstrap
+        To boostrap an environment means initializing it so that Services may be
+        deployed on it.
 
    Ensemble
-     ...
+        The whole software here documented.
 
    Environment
-     ...
-
-**XXX** This is all available in the architecture spec.  To avoid
-duplication and conflicts, we must either move from there, or remove this
-glossary and make the spec available instead.
+        An Environment is a configured location where Services can be deployed
+        onto.  An Environment typically has a name, which can usually be omitted
+        when there's a single Environment configured, or when a default is explicitly
+        defined.  Depending on the type of Environment, it may have to be
+        bootstrapped before interactions with it may take place (e.g. EC2).
+        The local environment configuration is defined in the ~/.ensemble/environments.yaml
+        file.
+
+   Formula
+        A Formula provides the definition of the service, including its metadata,
+        dependencies to other services, packages necessary, as well as the logic
+        for management of the application. It is the layer that integrates an
+        external application component like Postgres or Wordpress into Ensemble.
+        An Ensemble Service may generally be seen as the composition of its Ensemble
+        Formula and the upstream application (traditionally made available through
+        its package).
+
+   Repository
+        A location where multiple formulas are stored.  Repositories may be as simple
+        as a directory in the local disk, or as complex as a rich smart server
+        supporting remote searching and so on.
+
+   Relation
+        Relations are the way in which Ensemble enables Services to communicate
+        to each other, and the way in which the topology of Services is assembled.
+        The Formula defines which Relations a given Service may establish, and
+        what kind of interface these Relations require.  In many cases, the
+        establishment of a Relation will result into an actual TCP connection being
+        created between the Service Units, but that's not necessarily the case.
+        Relations may also be established to inform Services of configuration
+        parameters, to request monitoring information, or any other details which
+        the Formula author has chosen to make available.
+
+   Service
+        Ensemble operates in terms of services.
+
+        A service is any application (or set of applications) that is
+        integrated into the framework as an individual component which should
+        generally be joined with other components to perform a more complex
+        goal.
+
+        As an example, Wordpress could be deployed as a service and, to perform
+        its tasks properly, might communicate with a database service
+        and a load balancer service.
+
+   Service Unit
+        A running instance of a given Ensemble Service.  Simple Services may
+        be deployed with a single Service Unit, but it is possible for an
+        individual Service to have multiple Service Units running in independent
+        machines.  All Service Units for a given Service will share the same
+        Formula, the same relations, and the same user-provided configuration.
+        
+        For instance, one may deploy a single MongoDB Service, and specify that
+        it should run 3 Units, so that the replica set is resilient to failures.
+        Internally, even though the replica set shares the same user-provided
+        configuration, each Unit may be performing different roles within th
+        replica set, as defined by the Formula.
+
+   Service Configuration
+        There are many different settings in an Ensemble deployment, but
+        the term Service Configuration refers to the settings which a user can
+        define to customize the behavior of a Service.
+
+        The behavior of a Service when its Service Configuration changes is
+        entirely defined by its Formula.
+
+   Provisioning Agent
+        Software responsible for automatically allocating and terminating
+        machines in an Environment, as necessary for the requested configuration.
+
+   Machine Agent
+        Software which runs inside each machine that is part of an Environment,
+        and is able to handle the needs of deploying and managing Service Units
+        in this machine.
+
+   Service Unit Agent
+        Software which manages all the lifecycle of a single Service Unit.
+

=== renamed file 'docs/source/specifications/debug-hook.rst' => 'docs/source/hook-debugging.rst'
--- docs/source/specifications/debug-hook.rst	2011-02-27 01:52:06 +0000
+++ docs/source/hook-debugging.rst	2011-03-22 18:29:09 +0000
@@ -1,6 +1,9 @@
-Hook Debugging
+Hook debugging
 ==============
 
+Introduction
+------------
+
 An important facility in any distributed system is the ability to
 introspect the running system, and to debug it. Within ensemble the
 actions performed by the system are executing formula defined
@@ -15,7 +18,7 @@
 or debug a hook.
 
 How it works
-============
+------------
 
 When the ensemble user utilizes the hook debug command like so::
 
@@ -61,7 +64,7 @@
 
 
 Internals
-=========
+---------
 
 Internally the ``debug-hooks`` cli begins by verifying its arguments,
 namely the unit exists, and the named hook is valid for the formula.

=== modified file 'docs/source/index.rst'
--- docs/source/index.rst	2010-08-02 21:59:18 +0000
+++ docs/source/index.rst	2011-03-22 18:29:09 +0000
@@ -1,25 +1,20 @@
-Ensemble
-========
-
-With the advent of technologies such as virtualization and cloud
-computing, Linux server deployment is quickly moving towards a
-landscape of multiple machines.
-
-To orchestrate the configuration and deployment of more than a handful
-machines, tools are necessary. :term:`Ensemble` is one such tool that
-brings together a community to collaborate on best practices in the
-huge problem space of system administration.
-
-Core Documentation
-==================
+Ensemble Documentation
+======================
+
+.. note:: Ensemble is still in a stage of fast development, and is not yet
+        ready for prime time.  The current software is being made available as
+        an early technology preview, and while it can be experimented with, it
+        should not be used in real deployments just yet.
 
 .. toctree::
-   :maxdepth: 1
+   :maxdepth: 2
 
-   overview
-   getting_started
-   cli
-   specs
+   about
+   getting-started
+   formula
+   hook-debugging
+   ensemble-internals
+   ensemble-drafts
 
 Index and Glossary
 ==================

=== added directory 'docs/source/internals'
=== renamed file 'docs/source/specifications/agent-state-presence-settings.rst' => 'docs/source/internals/agent-presence.rst'
--- docs/source/specifications/agent-state-presence-settings.rst	2010-11-10 16:52:41 +0000
+++ docs/source/internals/agent-presence.rst	2011-03-22 18:29:09 +0000
@@ -1,5 +1,5 @@
-Agent Presence and Settings
----------------------------
+Agent presence and settings
+===========================
 
 Agents are a set of distributed processes within the ensemble
 framework tasked individually with various ensemble roles. Each agent
@@ -88,7 +88,7 @@
 that it matches the runtime state.
 
 
-Startup and Recovery
+Startup and recovery
 -------------------- 
 
 On startup, an agent will attempt to create its presence node. For
@@ -105,7 +105,7 @@
 process should terminate with an error message.
 
 
-Agent State API
+Agent state API
 ---------------
 
 

=== renamed file 'docs/source/specifications/unit-agent-hooks.rst' => 'docs/source/internals/unit-agent-hooks.rst'
--- docs/source/specifications/unit-agent-hooks.rst	2010-12-03 17:01:20 +0000
+++ docs/source/internals/unit-agent-hooks.rst	2011-03-22 18:29:09 +0000
@@ -1,6 +1,8 @@
+Unit Agent hooks
+================
 
-Unit Agent Process Specification
-================================
+Introduction
+------------
 
 The Unit Agent (**UA**) in Ensemble is responsible for managing and
 maintaining the per-machine service units. By calling life-cycle and
@@ -15,7 +17,7 @@
 Hooks_ are defined in another document. This specification only
 captures how they are invoked and managed, not why.
 
-.. _Hooks: ./formula.html#hooks
+.. _Hooks: ../formula.html#hooks
 
 When the Machine Agent (**MA**) spawns a UA it does so in order to
 manage the smallest managed unit of service deployment and
@@ -28,8 +30,9 @@
 semantics of it are described in this document.
 
 
-Hooks access to Settings
------------------------------
+Hooks access to settings
+------------------------
+
 Hooks have access to two kinds of settings. The first is the
 *"service settings"*, which cover configuration details for the
 all units of the given service.  These are usually provided
@@ -77,7 +80,7 @@
 is updated with the current value.
 
 
-Service Unit Name
+Service Unit name
 -----------------
 
 A service unit name in ensemble is formed by including both the name
@@ -123,9 +126,7 @@
 being returned to the client.
 
 
-
-
-Hook Invocation and Communication
+Hook invocation and communication
 ---------------------------------
 
 Twisted (which is used to handle networking and asynchronous
@@ -182,9 +183,7 @@
     * **wait(client_id, keyname)** - reserved
 
 
-
-
-Unit Agent Internal State
+Unit Agent internal state
 -------------------------
 
 This is a list of internal state which the UA maintains for the proper
@@ -198,7 +197,7 @@
       particular hook invocations.
 
 
-Command Line Interface
+Command line interface
 ----------------------
 
 While the command line utilities provided use the underlying AMP
@@ -253,8 +252,9 @@
         provided to communicate more complex logging messages to the
         UA and help them be made available to the user.
 
-Standard Calling Environment
-----------------------------
+
+Calling environment
+-------------------
 
 Hooks can expect to be invoked with a standard environment and
 context. The following be included:
@@ -285,8 +285,7 @@
     * `$ENSEMBLE_CHANGE` - Set to `joined`, `departed` or `changed`.
 
 
-
-Open Issues
+Open issues
 -----------
 
 There are still a number of open issues with this specification. There

=== renamed file 'docs/source/specifications/unit-agent.rst' => 'docs/source/internals/unit-agent-startup.rst'
--- docs/source/specifications/unit-agent.rst	2010-11-16 22:48:23 +0000
+++ docs/source/internals/unit-agent-startup.rst	2011-03-22 18:29:09 +0000
@@ -1,13 +1,8 @@
-Unit Agent
-----------
-
-A unit is a running machine resource/instance of a service. A unit
-agent is responsible for interacting with zookeeper to coordinate
-state on the units behalf. The agent interacts with the unit via
-invocation of hooks as defined in the formula specification.
-
-Agent State and Startup
-=======================
+Unit Agent startup
+==================
+
+Introduction
+------------
 
 The unit agent manages a state machine workflow for the unit. For each
 transition the agent records the current state of the unit and stores
@@ -58,11 +53,8 @@
 disconnected mode.
 
 
-Relations
-=========
-
-Unit Agent on Start
-+++++++++++++++++++
+Startup sequence
+----------------
 
 The following outlines the set of steps a unit agent executes when
 starting up on a machine resource.
@@ -103,8 +95,9 @@
  
  - Establish watches as outlined below.
  
-Unit Relation Observation
-+++++++++++++++++++++++++
+
+Unit relation observation
+-------------------------
 
 Based on the relation type and the unit's service role, the unit agent
 will establish will retrieve and establish watches on the other units
@@ -140,8 +133,8 @@
    relation-changed hook (type joined).
  
  
-Watch Behavior
-++++++++++++++
+Watch behavior
+--------------
  
  - (w-1) if the service-role child watch fires with a delete event,
    reestablish the watch, and execute the relation-changed hook (type

=== renamed file 'docs/source/specifications/zookeeper-filesystem.rst' => 'docs/source/internals/zookeeper.rst'
--- docs/source/specifications/zookeeper-filesystem.rst	2010-08-27 19:41:44 +0000
+++ docs/source/internals/zookeeper.rst	2011-03-22 18:29:09 +0000
@@ -1,5 +1,5 @@
-ZooKeeper Filesystem
-====================
+ZooKeeper
+=========
 
 This document describes the reasoning behind Ensemble's use of ZooKeeper,
 and also the structure and semantics used by Ensemble in the ZooKeeper

=== removed file 'docs/source/notes.rst'
--- docs/source/notes.rst	2010-08-03 20:14:13 +0000
+++ docs/source/notes.rst	1970-01-01 00:00:00 +0000
@@ -1,19 +0,0 @@
-
-
-Refactoring Bootstrap. We need to initialize the zookeeper hierarchy with the 
-machine structure and appropriate acls prior to usage, which means switching
-bootstrap to a connected model. In order to minimize bootstrap time while
-connected to the console, we'll utilize customized EBS instances. To generate 
-an admin cli credentials for setting up the initial layout acls, we need to 
-modify the default config generation such that it generates a secret key
-directly in the config.. if a config file  already exists sans an admin key, 
-see if we can add a generated secret while preserving formatting.
-
-Provisioning Agent requires machine provider credentials. For the initial 
-provider support this requires that we transport EC2 Credentials to the 
-zookeeper structure. The AWS credentials are transferred along with the 
-machine provider config on bootstrap over the encrypted zookeper connection.
-The AWS credentials can be spec'd in the config file or extracted directly
-from the environment.
-
- 

=== removed file 'docs/source/overview.rst'
--- docs/source/overview.rst	2010-07-23 12:19:42 +0000
+++ docs/source/overview.rst	1970-01-01 00:00:00 +0000
@@ -1,43 +0,0 @@
-Overview
-========
-
-This section provides a brief overview of the system. Without going
-too much into detail, it outlines the most important aspects of the
-system architecture.
-
-Managing Services
------------------
-
-Whether services are stacked, internal or external, there are certain
-characteristics that should apply:
-
-1. Services are simple to configure;
-2. Services can be moved to any machine;
-3. Services can be scaled both up or down;
-4. Services can persist data in an external location;
-5. Services can co-exist on the same machine;
-6. Similar services are interchangeable;
-
-The implementation of those characteristics are the responsibility of
-the formula that manages the service.
-
-Connecting Services
--------------------
-
-Rather than focusing on the configuration of the individual service,
-:term:`Ensemble` focuses on making services operate together. In the
-general case, connecting two services simply requires pointing service
-A to service B.
-
-If we don't care about the details of this :term:`service link`, the
-system will decide a sensible implementation (e.g. TCP/IP on some port
-with a suitable means of authentication). This is possible in
-particular because of the virtualization technology which provides a
-clean space for services to operate in. Risk of error is reduced and
-security aspects can be addressed in a general way.
-
-By defining your system logically in terms of services and the links
-between them, other components can attach themselves and add
-value. For instance, we can ask the system to monitor all service
-links for activity.
-

=== removed directory 'docs/source/specifications'
=== removed file 'docs/source/specifications/environment.rst'
--- docs/source/specifications/environment.rst	2010-08-02 21:59:18 +0000
+++ docs/source/specifications/environment.rst	1970-01-01 00:00:00 +0000
@@ -1,7 +0,0 @@
-.. _environment:
-
-Environment
-===========
-
-This document specifies the environment which is implicitly given by
-the configuration of a machine provider and permanent storage.

=== removed file 'docs/source/specifications/repository.rst'
--- docs/source/specifications/repository.rst	2010-08-02 21:59:18 +0000
+++ docs/source/specifications/repository.rst	1970-01-01 00:00:00 +0000
@@ -1,7 +0,0 @@
-.. _formula-repository:
-
-Formula Repository
-==================
-
-A named collection of formulas; the name is used as a namespace to
-keep track of the formula source.

=== removed file 'docs/source/specifications/template.rst'
--- docs/source/specifications/template.rst	2010-07-27 19:14:59 +0000
+++ docs/source/specifications/template.rst	1970-01-01 00:00:00 +0000
@@ -1,5 +0,0 @@
-Template
-========
-
-This is a specification template. It should not be included in release
-documentation.

=== removed file 'docs/source/specs.rst'
--- docs/source/specs.rst	2010-07-27 19:14:59 +0000
+++ docs/source/specs.rst	1970-01-01 00:00:00 +0000
@@ -1,9 +0,0 @@
-Specifications
-==============
-
-This section details current specifications.
-
-.. toctree::
-   :glob:
-
-   specifications/*