Merge lp:~jimbaker/pyjuju/new-hook-semantics-5-docs into lp:pyjuju

[Jim Baker]

This branch updates the description of the hooks so as to update them with respect to the new hook semantic changes. It also copy edits this formula section as a whole. In particular, the copy editing focus was on clarifying how hooks actually execute, and under what conditions. This is to avoid the confusion that prompted the new-hook-semantics- branches in the first place.

[Ahmed Kamal (kim0)]

Typo line-259 says "with with respect"

[Gustavo Niemeyer (niemeyer)]

Very good changes overall, thanks Jim.

We'll need another round:

[1]

17 +A formula is deployed as a service, potentially multiple times. Each

This is very open and perhaps even misleading. Please take this specific
sentence off. The rest of the paragraph should give better background.

[2]

+The `metadata.yaml` file, at the root of the formula definition,

s/definition/directory/

[3]

- and `peers` relations (note: that's not yet supported).
(...)
- that the relation is required (note: that's not yet supported).

Please add back both unsupported notes. All documentation which is not
under drafts should reflect what is implemented and may be trusted upon,
or declare explicitly what is not yet in place.

130 + At this time, Ensemble does not enforce the `limit` and
131 + `optional` constraints, but Ensemble will do so in the future.

The note really has to be closer to the explanation. Someone reading about
the field to tweak a formula will not read the whole text to see what is
supported or not.

[4]

188 +changes on its underlying machine, and change its own relation
189 +settings. Changes to relation settings then trigger further events
190 +that hooks can respond to. The following section on hook tools

s/its own relation settings/the relation settings/ (the subject of the
sentence is "the hook").

Also, please drop the sentence starting at "Changes to relation". You're
describing what happens from the perspective of an individual unit. For
an individual unit, changing relation settings will not trigger its own
hooks to execute again.

[5]

194 +the desired hook name under the ``hooks/`` directory of the formula
195 +bundle or directory

s/bundle or directory/directory/

Conceptually, we use the bundle term to reference to a file, so it can't
have directories.

[6]

195 +bundle or directory. Ensemble then executes the hook without command
196 +line arguments when the corresponding event occurs.

Please use this in place of the latter sentence:

  Ensemble will execute the hook based on its file name when the corresponding
  event occurs.

Note that in general we don't have to document the lack of something. If it's
not documented to exist, it's fine to not exist.

[7]

198 +Each hook is optional. Not including a corresponding executable in the

s/Each hook is/All hooks are/

[8]

213 + * **stop** - Runs when the service unit is stopped. This hook will run
214 + after any established relations are broken.

Let's take the chance to fix the pre-existing ambiguity in the latter sentence:

  If relations exist, they will be broken and the respective hooks called before
  this hook is called.

[9]

+ 1. A remote service unit joins the relation.

Let's complement:

    1. A remote service unit joins the relation, right after the
       relation-joined hook was called.

[10]

229 + This hook is the most common hook for a formula to implement. It
230 + enables service units to handshake and otherwise negotiate their
231 + interaction, especially with respect to scaling up and down
232 + service deployments.

This doesn't feel very helpful. It's taking assumptions about how the user
sh...

review: Needs Fixing

[Jim Baker (jimbaker)]

Thanks, I forgot to run my double-word-typo script that I have just for this
case! (Need to integrate that into my push process or maybe the emacs fly
mode support.)

On Thu, Apr 14, 2011 at 8:06 AM, Ahmed Kamal <email address hidden>wrote:

> Typo line-259 says "with with respect"
> --
>
> https://code.launchpad.net/~jimbaker/ensemble/new-hook-semantics-5-docs/+merge/57536<https://code.launchpad.net/%7Ejimbaker/ensemble/new-hook-semantics-5-docs/+merge/57536>
> You are the owner of lp:~jimbaker/ensemble/new-hook-semantics-5-docs.
>

[Jim Baker (jimbaker)]

On Thu, Apr 14, 2011 at 8:59 AM, Gustavo Niemeyer <email address hidden>wrote:

> Review: Needs Fixing
> Very good changes overall, thanks Jim.
>
> We'll need another round:
>
> [1]
>
> 17 +A formula is deployed as a service, potentially multiple times.
> Each
>
> This is very open and perhaps even misleading. Please take this specific
> sentence off. The rest of the paragraph should give better background.
>
>
Reworked intro

>
> [2]
>
> +The `metadata.yaml` file, at the root of the formula definition,
>
> s/definition/directory/
>

Changed

>
>
> [3]
>
> - and `peers` relations (note: that's not yet supported).
> (...)
> - that the relation is required (note: that's not yet supported).
>
> Please add back both unsupported notes. All documentation which is not
> under drafts should reflect what is implemented and may be trusted upon,
> or declare explicitly what is not yet in place.
>
> 130 + At this time, Ensemble does not enforce the `limit` and
> 131 + `optional` constraints, but Ensemble will do so in the
> future.
>
> The note really has to be closer to the explanation. Someone reading about
> the field to tweak a formula will not read the whole text to see what is
> supported or not.
>

Moved description to being closer as suggested

>
>
> [4]
>
> 188 +changes on its underlying machine, and change its own relation
> 189 +settings. Changes to relation settings then trigger further events
> 190 +that hooks can respond to. The following section on hook tools
>
> s/its own relation settings/the relation settings/ (the subject of the
> sentence is "the hook").
>
> Also, please drop the sentence starting at "Changes to relation". You're
> describing what happens from the perspective of an individual unit. For
> an individual unit, changing relation settings will not trigger its own
> hooks to execute again.
>

Fixed

>
>
> [5]
>
> 194 +the desired hook name under the ``hooks/`` directory of the
> formula
> 195 +bundle or directory
>
> s/bundle or directory/directory/
>
> Conceptually, we use the bundle term to reference to a file, so it can't
> have directories.
>

This was from the old text in trunk, so thanks for spotting this so it
doesn't get perpetuated

>
>
> [6]
>
> 195 +bundle or directory. Ensemble then executes the hook without
> command
> 196 +line arguments when the corresponding event occurs.
>
> Please use this in place of the latter sentence:
>
> Ensemble will execute the hook based on its file name when the
> corresponding
> event occurs.
>

Much better

>
> Note that in general we don't have to document the lack of something. If
> it's
> not documented to exist, it's fine to not exist.
>

Ack

>
>
> [7]
>
> 198 +Each hook is optional. Not including a corresponding executable in
> the
>
> s/Each hook is/All hooks are/
>

Changed

>
>
> [8]
>
>
> 213 + * **stop** - Runs when the service unit is stopped. This hook
> will run
> 214 + after any established relations are broken.
>
> Let's take the chance to fix the pre-existing ambiguity in the latter
> sentence:
>
> If relations exist, they will be broken and the respective hooks ca...

[lp:~jimbaker/pyjuju/new-hook-semantics-5-docs]

203. By Jim Baker on 2011-04-19

Removed intro for incorporation into a future branch

[Gustavo Niemeyer (niemeyer)]

Thanks, +1 with a small fix for that:

[15]

  Its purpose is to allow the formula to
255 + clean up established state
256 +
257 + The service unit can then clean up any established state.

review: Approve

=== modified file 'docs/source/formula.rst'
--- docs/source/formula.rst	2011-04-08 17:37:35 +0000
+++ docs/source/formula.rst	2011-04-19 15:20:54 +0000
@@ -1,80 +1,86 @@
-
 Formulas
 ========
 
 Introduction
 ------------
 
-In Ensemble, formulas are the description of how services should
-behave when deployed and run.  A single service definition may be run
-multiple times with the same formula and configuration, so we call a
-deployed and running service as a service unit.  Each of those units
-will have an independent copy of the same formula guiding its behavior.
+Formulas define how services integrate and how their service units
+react to events in the distributed environment, as orchestrated by
+Ensemble.
 
-This specification describes how formulas are built and how they
-react once integrated in an Ensemble environment.
+This specification describes how formulas are defined, including their
+metadata and hooks. It also describes the resources available to hooks
+in working with the Ensemble environment.
 
 
 The metadata file
 -----------------
 
-The file named `metadata.yaml`, at the root of the formula, contains
-details about the formula.
-
-The following fields are supported:
+The `metadata.yaml` file, at the root of the formula directory,
+describes the formula. The following fields are supported:
 
   * **name:** - The formula name itself.
 
-  * **revision:** - A monothonically increasing integer which defines the
-    formula version.
-
-  * **summary:** - A one line description of the formula
-
-  * **description:** - Long explanation about the formula features
-
-  * **provides:** - The deployed service unit must have the given relations
-    established with another service unit whose formula requires them for
-    the service to work properly. See below for how to define a relation.
-
-  * **requires:** - The deployed service unit must have the given relations
-    established with another service unit whose formula provides them for
-    the service to work properly. See below for how to define a relation.
-
-  * **peers:** - Relations which are established not in a provides/requires
-    (or client/server) style, but with P2P semantics.  When the formula
-    is deployed as a service unit, all the units from the given service
-    will automatically be made part of the relation.  See below for how to
-    define a relation.
+  * **revision:** - A monotonically increasing integer which defines
+    the formula version. This number is used to determine if the
+    formula needs to be updated in the Ensemble environment. It is
+    also displayed as output for the ``ensemble status`` command, so
+    you can ensure you have the right formula deployed.
+
+  * **summary:** - A one-line description of the formula.
+
+  * **description:** - Long explanation of the formula and its
+    features.
+
+  * **provides:** - The deployed service unit must have the given
+    relations established with another service unit whose formula
+    requires them for the service to work properly. See below for how
+    to define a relation.
+
+  * **requires:** - The deployed service unit must have the given
+    relations established with another service unit whose formula
+    provides them for the service to work properly. See below for how
+    to define a relation.
+
+  * **peers:** - Relations that are established with P2P semantics
+    instead of a provides/requires (or client/server) style.  When the
+    formula is deployed as a service unit, all the units from the
+    given service will automatically be made part of the relation.
+    See below for how to define a relation.
 
 
 Relations available in `provides`, `requires`, and `peers` are defined
-the following way:
+as follows:
 
   * **provides|requires|peers:**
 
-    * **<relation name>:** - This is a user-provided value which identifies
-      this relation uniquely within the given formula.  Examples might be
-      "database", or "cache", or "proxy", or "appserver".
+    * **<relation name>:** - This name is a user-provided value which
+      identifies the relation uniquely within the given formula.
+      Examples include "database", "cache", "proxy", and "appserver".
 
       Each relation may have the following fields defined:
 
-      * **interface:** - This defines the type of this relation.  This
-        relation will only be established with service units which
-        define a compatible relation with the same interface. Examples
-        here might be "http", or "mysql", or "backup-schedule".
+      * **interface:** - This field defines the type of the
+        relation. The relation will only be established with service
+        units that define a compatible relation with the same
+        interface. Examples include "http", "mysql", and
+        "backup-schedule".
 
-      * **limit:** - The maximum number of relations of this kind which
-        may be established to other service units.  Defaults to 1 for
-        `requires` relations, and to "none" (no limit) for `provides`
-        and `peers` relations (note: that's not yet supported).
+      * **limit:** - The maximum number of relations of this kind
+        which may be established to other service units.  Defaults to
+        1 for `requires` relations, and to "none" (no limit) for
+        `provides` and `peers` relations. While you may define it,
+        this field is not yet enforced by Ensemble.
 
       * **optional:** - Whether this relation is required for the
-        service unit to function or not.  Defaults to `false`, meaning
-        that the relation is required (note: that's not yet supported).
+        service unit to function or not.  Defaults to `false`, which
+        means the relation is required. While you may define it, this
+        field is not yet enforced by Ensemble.
 
       As a shortcut, if these properties are not defined, and instead
-      a single string value is provided next to the relation name,
-      the string is taken as the interface value.  For example::
+      a single string value is provided next to the relation name, the
+      string is taken as the interface value, as seen in this
+      example::
 
           requires:
             db: mysql
@@ -86,88 +92,129 @@
 Hooks
 -----
 
-Hooks are the mechanism through which service units are notified of
-events happening around them.  Each hook is an executable within the
-`hooks/` directory of the formula bundle or directory.
-
-Hooks are able to modify the relation settings via command line tools
-offered by Ensemble.  These changes are only perceived by its peers
-when the hook has finished running successfully.  If a hook exits with
-a non-zero error code state changes are not committed.
-
-Some hooks are named after the formula relation which they want to be
-notified about.
-
-The following hooks are currently defined:
-
-  * **install** - Run just once during the life time of a service unit.
-
-  * **start** - Run when the formula is started.  This happens before
-    any relation hooks are called. The purpose of this hook is to get
-    the service ready for relations to be established.
-
-  * **stop** - Run when the formula is stopped.  This will run after any
-    established relations are broken.
-
-  * **<relation name>-relation-established** - This hook is run when a
-    defined relation is established with one or more peers.  Any
-    actions related to this relation being established should be
-    performed.
-
-    An example might be providing via `relation-set` the TCP port of
-    an offered XML-RPC service, or perhaps notifying the started web
-    application that the MySQL database is now available.
-
-  * **<relation name>-relation-broken** - This hook is run when a
-    relation which had at least one other relation hook run for it
-    (successfully or not) is now unavailable.  Its purpose is to
-    allow the formula to clean up established state.
-
-    An example might be cleaning up the configuration changes which
-    were performed when HAProxy was asked to load-balance for another
+Ensemble uses hooks to notify a service unit about changes happening
+in its lifecycle or the larger distributed environment. A hook running
+for a service unit can query this environment, make any desired local
+changes on its underlying machine, and change the relation
+settings.
+
+Each hook for a formula is implemented by placing an executable with
+the desired hook name under the ``hooks/`` directory of the formula
+directory.  Ensemble will execute the hook based on its file name when
+the corresponding event occurs.
+
+All hooks are optional. Not including a corresponding executable in
+the formula is treated by Ensemble as if the hook executed and then
+exited with an exit code of 0.
+
+The following hooks are with respect to the lifecycle of a service unit:
+
+  * **install** - Runs just once during the life time of a service
+    unit. Currently this hook is the right place to ensure any package
+    dependencies are met. However, in the future Ensemble will use the
+    formula metadata to perform this role instead.
+
+  * **start** - Runs when the service unit is started. This happens
+    before any relation hooks are called. The purpose of this hook is
+    to get the service unit ready for relations to be established.
+
+  * **stop** - Runs when the service unit is stopped. If relations
+    exist, they will be broken and the respective hooks called before
+    this hook is called.
+
+The following hooks are called on each service unit as the membership
+of an established relation changes:
+
+  * **<relation name>-relation-joined** - Runs upon each time a remote
+    service unit joins the relation.
+
+  * **<relation name>-relation-changed** - Runs upon each time the
+    following events occur:
+
+    1. A remote service unit joins the relation, right after the
+       **<relation name>-relation-joined** hook was called.
+
+    2. A remote service unit changes its relation settings.
+
+    This hook enables the formula to modify the service unit state
+    (configuration, running processes, or anything else) to adapt to
+    the relation settings of remote units.
+
+    An example usage is that HAProxy needs to be aware of web servers
+    as they become available, including details like its IP
+    address. Web server service units can publish their availability
+    by making the appropriate relation settings in the hook that makes
+    the most sense. Assume the HAProxy uses the relation name of
+    ``server``. Then upon that happening, the HAProxy in its
+    ``server-relation-changed hook`` can then change its own
+    configuration as to what is available to be proxied.
+
+  * **<relation name>-relation-departed** - Runs upon each time a
+    remote service unit leaves a relation. This could happen because
+    the service unit has been removed, its service has been destroyed,
+    or the relation between this service and the remote service has
+    been removed.
+
+    An example usage is that HAProxy needs to be aware of web servers
+    when they are no longer available. It can remove each web server
+    its configuration as the corresponding service unit departs the
+    relation.
+
+This relation hook is with respect to the relation itself:
+
+  * **<relation name>-relation-broken** - Runs when a relation which
+    had at least one other relation hook run for it (successfully or
+    not) is now unavailable.  Its purpose is to allow the formula to
+    clean up established state
+
+    The service unit can then clean up any established state.  An
+    example might be cleaning up the configuration changes which were
+    performed when HAProxy was asked to load-balance for another
     service unit.
 
-  * **<name>-relation-changed** - This hook is called when:
-
-    1. A new remote service unit joins the relation.
-
-    2. A remote service unit changes its relation settings.
-
-    3. A remote service unit which was part of the relation has
-       had its relation broken.
+Note that the coupling between formulas is defined by which settings
+are required and made available to them through the relation hooks and
+how these settings are used. Those conventions then define what the
+relation interface really is, and the **interface** name in the
+`metadata.yaml` file is simply a way to refer to them and avoid the
+attempting of incompatible conversations.  Keep that in mind when
+designing your formulas and relations, since it is a good idea to
+allow the implementation of the formula to change and be replaced with
+alternative versions without changing the relation conventions in a
+backwards incompatible way.
 
 
 Hook environment
 ----------------
 
 Hooks can expect to be invoked with a standard environment and
-context. The following be included:
+context. The following environment variables are set:
 
-    * `$ENSEMBLE_UNIT_NAME` - The name of the local unit executing,
-      in the form `<service name>/<unit sequence>`. E.g. myblog/3.
+  * **$ENSEMBLE_UNIT_NAME** - The name of the local unit executing,
+    in the form ``<service name>/<unit sequence>``. E.g. ``myblog/3``.
 
 Hooks called for relation changes will have the follow additional
-environment variables available to them.
-
-    * `$ENSEMBLE_RELATION` - The relation name this hook is running
-      for.  It's redundant with the hook name, but is necessary for
-      the command line tools to know the current context.
-
-    * `$ENSEMBLE_REMOTE_UNIT` - The unit name of the remote unit
-      which has triggered the hook execution.
+environment variables set:
+
+  * **$ENSEMBLE_RELATION** - The relation name this hook is running
+    for.  It's redundant with the hook name, but is necessary for
+    the command line tools to know the current context.
+
+  * **$ENSEMBLE_REMOTE_UNIT** - The unit name of the remote unit
+    which has triggered the hook execution.
 
 
 Hook tools
 ----------
 
-To implement their functionality, hooks may leaverage a set of
-utilities provided by Ensemble.  These utilities enable the hook to
-collaborate about configuration state, and to inquire about the peers
-the service unit has relations with.
+In implementing their functionality, hooks can leverage a set of
+command tools provided by Ensemble.  These utilities enable the hook
+to collaborate on their relation settings, and to inquire about the
+peers the service unit has relations with.
 
 The following command line tools are made available:
 
- * **relation-get** - Queries a setting from an established relation
+  * **relation-get** - Queries a setting from an established relation
     with one or more service units.  This command will read some
     context information from environment variables (e.g.
     $ENSEMBLE_RELATION_NAME).
@@ -210,13 +257,19 @@
 
   * **relation-list** - List all service units participating in the
     established relation.  This list excludes the local service unit
-    which is executing the command. For provides and requires
-    relation, this will always return a single service unit.
+    which is executing the command. For `provides` and `requires`
+    relations, this command will always return a single service unit.
 
     Example::
 
         MEMBERS=$(relation-list)
 
+Changes to relation settings are only committed if the hook exits with
+a non-zero status code. Such changes will then trigger further hook
+execution in the remote unit(s), through the **<relation
+name>-relation-changed** hook. This mechanism enables a general
+communication mechanism for service units to coordinate.
+
 
 Sample metadata.yaml files
 --------------------------