Merge lp:~hazmat/pyjuju/formula-upgrades-spec into lp:pyjuju

[Kapil Thangavelu]

A spec describing some of the upgrade use cases, and focusing on the implementation of simple formula upgrades and the new hook they propose.

[Gustavo Niemeyer (niemeyer)]

[1]

+Formula Upgrades
+----------------

Our sections have been using mixed casing in the documentation.
In user-oriented-docs I did a full round to clean that up and
make them "Title case".

Please change this to "Formula upgrades" and the other sections
accordingly.

[2]

+A common task when doing formula development, is iterating over
(...)
+In some cases a new formula version, will also reference newer

Please drop the comma in both sentences. In both cases it's
separating the subject from the predicate.

I promise not to bother too much about that kind of issue. It's just
that this specific misuse feels specially weird when reading because
it puts a sentence "pause" in an awkward location.

[3]

+In some cases a new formula version, will also reference newer
+software/package versions or new package versions.

package versions or package versions..

I guess you intended to say new packages or new package versions?

[4]

+formula upgrades as an immediately useful and required use case for
+the purpose.

Please drop or complement "for the purpose" (for the purpose of ...).

[5]

This document should really be split in two: one which talks about
the way in which Ensemble sees upgrades in several areas, and another
which talks about formula upgrades more specifically.

Put the following sections on the generic one:

+Upgrades (the global header + comment)
(...)
+Service Upgrades
(...)
+Formula Upgrades
(...)
+Ensemble Upgrades

Then, put the following one in the formula-specific document:

+Upgrading A Formula

Split this latter document into two parts:

1) User-oriented management of formula upgrading through the UI
2) Writing a formula which is ready for being upgraded

The second one may actually have a better home as a section of the
existent formula documentation in the future, but let's have it
as an external document just for the sake of debating about it in
isolation.

[6]

+ $ ensemble upgrade-formula <service-name>

Can we name this as simply "upgrade"? We don't have deploy-formula,
and it's even more confusing given that the argument after upgrade is
a service name, not a formula name.

[7]

+Which will examine the named service, determine its formula, check
the
+formula's originating repository for a newer version of the formula.
+If a newer formula version is found, it will be uploaded to the
ensemble
+environment, and downloaded to all the running units of the service.

This single sentence is pretty much the whole *actual* specification. :-)

This sentence feels good for a high-level introductory description,
but the specification needs to go into more detail about these phases,
with some content probably moved into an "Internals" section.

E.g. How's a newer version of a formula defined like? What if the
versions conflict (the case brought up by Clint)? What does it mean
to "upload to the environment", and who does it? How will the unit
agent know it has to download a new formula? How does the replacement
take place? What if there are established relations? What if there
was previously existent data in the directory? And so on.

review: Needs Fixing

[Gustavo Niemeyer (niemeyer)]

Some additional comments, and a change re. [6]:

[6]

Actually, changing from the previous comments on [6], nevermind
regarding the command name suggestion. upgrade-formula sounds fine
as a way to disambiguate from "upgrade the software/packages".

But then:

[8]

+After the newer formula is in place and before any hooks are executed
+from it, the unit agent will execute the ``upgrade`` hook on the unit
+if present.

Let's name the hook as "upgrade-formula", to match the command name,
and let's reserve the "upgrade" name for package/software upgrades.

[9]

+This allows the formula to process any upgrade concerns it may have
+with regard to upgrading databases, software, etc before its new
+version of hooks are invoked.

We'll need a way to do pre-post logic. Maybe we need a
upgraded-formula hook too, which is executed *after* the new formula
is in place, and then the "upgrade-formula" one is executed *before*
the old one is removed.

What do you think?

[Kapil Thangavelu (hazmat)]

[9] Followup from IRC.

<kapil> niemeyer, do you have an example of where we would need pre-post upgrade hooks?
<kapil> afaics most of them should be encompassed in a single after hook, since it can process all changes... the formula as distinct from the running software service.
<niemeyer> kapil: The kind of process I had in mind was mostly those where we want to export some data from the old running formula, and reimport it later.
<niemeyer> kapil: I don't yet have a concrete use case though, to be honest.
<kapil> niemeyer, yeah.. i'm just not seeing a case where that couldn't be done in the after hook
<kapil> any data setting is outside of the formula itself, in the zk state or the unit managed program
<niemeyer> kapil: Yeah, I guess you're right, given that hooks are always unstateful, and that the following formula will always have consciousness of its history

[Kapil Thangavelu (hazmat)]

[1-9] addressed. thanks. the branch is ready for another look.

[Gustavo Niemeyer (niemeyer)]

Thanks for the changes Kapil. Looking great now.

+1, taking into account the following points:

[10]

Please merge trunk into the branch and move both documents into the drafts/
section, so that people don't think this is already working.

[11]

50 +The new hook-cli api would be called ``list-relations`` and would list
51 +all the relations a unit is a member of.

Having relation-list and list-relations will be confusing. The internal implementation
of relation-list even uses list_relations() methods.

Then, it's not clear why we need a method with these semantics. The formula necessarily
knows which relations it may be a member of, because it's a fixed set specified in the
formula metadata.

What we need is being able to list the units for other relations, so we might just have
relation-list itself accepting a [name] parameter:

    relation-list => Current behavior
    relation-list db => Same behavior, but for the db relation

[12]

+After the ``upgrade-formula`` hook is, new hooks of the formula will

s/is/is executed|run/

[13]

+The cli command is responsible

s/responsible/responsible for/, given the wording in the items.

[14]

70 + - Determining if a newer version of the formula exists in the
71 + origin repository. Strict linear numbering is used.

There are some details about the namespace here, but I guess that as long as
we check the same place the formula was originally obtained from, it should
be good for now.

[15]

75 + - Marking the unit state as needing an upgrade, with a pointer
76 + to the provider stored formula.

Interesting. I wonder how we should approach this part of the problem properly.

The formula right now is associated to the service, and we have no hints about
which formula was deployed in an individual unit. Also, ideally the mechanism
we implement can optionally upgrade a single unit at a time, out of a set of
multiple.

What about:

- Whenever a unit is deployed, copy the current formula reference from the service
  and into the unit metadata in zookeeper.

- When an upgrade is requested, change the formula reference in the service only.

- Then, for each individual unit which should be upgraded, just flag it as "upgrade=true"
  in its metadata.

- The unit then is responsible for evaluating the distinction between its formula and the
  one currently in its service and perform an action, if necessary.

What do you think?

Can you please integrate something along those lines which you agree with into the document?

[16]

86 + - Transition the unit and its relations to the stopped state (or equivalent).
87 +
88 + - Downloading the formula, and extracting the formula into the unit container.

As a small nit, it shouldn't stop the state before the formula is downloaded and ready to
be unpacked.

review: Approve

[lp:~hazmat/pyjuju/formula-upgrades-spec]

180. By Kapil Thangavelu on 2011-03-24

merge trunk

181. By Kapil Thangavelu on 2011-03-24

post conflict resolve against trunk merge, remove empty specifications directory.

[Benjamin Saller (bcsaller)]

Looks good overall, thanks.

[1]
+ The ``upgrade-formula`` hook will likely need access to a new cli-api
+to access all relations of the unit, in addition to the standard hook
+api commands like ``relation-list``, ``relation-get``,
+``relation-list``.

relation-list appears twice

[2]

Maybe link to the forumla spec in reference to defining version.

[3]

There are still lots of open areas pointed at by the docs that ate my other questions by defining them as out of scope.

[lp:~hazmat/pyjuju/formula-upgrades-spec]

182. By Kapil Thangavelu on 2011-03-29

merge trunk

183. By Kapil Thangavelu on 2011-04-01

address review comments, add note regarding unit state carrying references to the current deployed formula.

[Kapil Thangavelu (hazmat)]

10-16 minus 11 addressed. As we discussed in a stand up meeting, the case i'm considering here is where a formula has multiple optional or effectively anonymous relations. Ie. in the case of memcached and mediawiki, the relation is optional, so its not clear to the mediawiki formula if such a relation exists. And on the reverse side for mysql or memcached, they may have multiple relations under a single name, and they need to be able to iterate over them in turn to be able to operate on the individual relations instead of the aggregate. The cli name could perhaps be better named as 'query-relations' to better distinguish it.

[lp:~hazmat/pyjuju/formula-upgrades-spec]

184. By Kapil Thangavelu on 2011-04-01

address bcsaller's review comments, additional formula spec link and remove extraneous relation-list reference

[Kapil Thangavelu (hazmat)]

Thanks for the review ben, 1-2 addressed. Re 3, there are lots of additional contextually relevant items for upgrades which will need deeper delving in the future.

> Looks good overall, thanks.
>
> [1]
> + The ``upgrade-formula`` hook will likely need access to a new cli-api
> +to access all relations of the unit, in addition to the standard hook
> +api commands like ``relation-list``, ``relation-get``,
> +``relation-list``.
>
> relation-list appears twice
>
>
> [2]
>
> Maybe link to the forumla spec in reference to defining version.
>
>
> [3]
>
> There are still lots of open areas pointed at by the docs that ate my other
> questions by defining them as out of scope.

[Gustavo Niemeyer (niemeyer)]

Even considering the valid points you've made, [11] continues to be a problem. The semantics of such a tool are not clear, and it should certainly not be named list-relations since it conflicts with existing nomenclature.

[Jim Baker (jimbaker)]

+1, looks great to me

(not certain what the specific name should be for list-relations, but something like this seems required)

review: Approve

[lp:~hazmat/pyjuju/formula-upgrades-spec]

185. By Kapil Thangavelu on 2011-04-08

remove per unit relation upgrade discussion and hook-api into new futures section

=== added file 'docs/source/drafts/formula-upgrades.rst'
--- docs/source/drafts/formula-upgrades.rst	1970-01-01 00:00:00 +0000
+++ docs/source/drafts/formula-upgrades.rst	2011-04-08 06:02:24 +0000
@@ -0,0 +1,121 @@
+Formula Upgrades
+================
+
+
+Upgrading a formula
+-------------------
+
+A formula_ can be upgraded via the command line using the following
+syntax::
+
+  $ ensemble upgrade-formula <service-name>
+
+In the case of a local formula the sytax would be::
+
+  $ ensemble upgrade-formula --repository=principia <service-name>
+
+This will examine the named service, determine its formula, and check the
+formula's originating repository for a newer version of the formula.
+If a newer formula version is found, it will be uploaded to the ensemble
+environment, and downloaded to all the running units of the service.
+The unit agent will switch over to executing hooks from the new formula,
+after executing the `upgrade-formula` hook.
+
+.. _formula: ../formula.html
+
+
+Formula upgrade support
+-----------------------
+
+A formula author can add formula specific support for upgrades by 
+providing an additional hook that can customize its upgrade behavior.
+
+The hook ``upgrade-formula`` is executed with the new formula version
+in place on the unit. This hook will be the first executed hook of the
+newly upgraded formula.
+
+
+Currently the standard hook relation api assumes a non-qualified
+usage, in that they assume a default relation context has been
+provided. Within context of an upgrade-formula hook the relation name
+would be a required argument or environment for utilizing the standard
+hook relation api.
+
+The hook is intended to allow the formula to process any upgrade
+concerns it may have with regard to upgrading databases, software, etc
+before its new version of hooks are executed.
+
+After the ``upgrade-formula`` hook is executed, new hooks of the
+formula will be utilized to respond to any system changes.
+
+
+Futures
+-------
+
+The ``upgrade-formula`` hook will likely need access to a new cli-api
+to access all relations of the unit, in addition to the standard hook
+api commands like ``relation-list``, ``relation-get``,
+``relation-set``, to perform per unit relation upgrades. 
+
+The new hook-cli api name is open, but possible suggestions are
+``unit-relations`` or  ``query-relations`` and would list
+all the relations a unit is a member of.
+
+Most `server` services have multiple instances of a named relation.
+Else name iteration of the formula defined relations would suffice. 
+Its an open question on how these effectively anonymous instances 
+of a named relation would be addressed.
+
+The existing relation-* cli would also need to be extended to take
+a relation parameter, or documented usage of environment variables
+when doing relation iteration during upgrades.
+
+Internals
+---------
+
+One state change required for upgrade support is marking each of the 
+units with the formula their running via copying this reference to
+from the service state to the unit state. This will allow for 
+noting/tracking the individual unit upgrades across a service.
+
+The upgrade cli command is responsible for
+
+ - Finding the named service.
+
+ - Determining its formula.
+
+ - Determining if a newer version of the formula exists in the
+   origin repository. Strict linear numbering is used.
+
+ - Uploading the new version of the formula to provider storage.
+
+ - Updating the service state with a reference to the new formula.
+
+ - Marking the associated unit states as needing an upgrade.
+
+As far as determining newer versions, the cli will assume the same formula
+name with the max version number that is greater than the installed to
+be an upgrade.
+
+The unit agent is responsible for
+
+ - Watching the unit state for upgrade changes.
+
+ - Downloading the new formula version
+
+ - Transition the unit and its relations to the stopped state (or
+   equivalent).
+
+ - Extracting the formula into the unit container.
+
+ - Executing the upgrade-formula hook.
+
+ - Clearing the upgrade setting on the unit state, and updating its
+   formula reference.
+
+ - Transitioning the unit and its relation to the running state (or
+   semantic equivalent).
+
+Only the formula directory within a unit container/directory is
+replaced on upgrade, any existing peristent data within the unit
+container is maintained.

=== added file 'docs/source/drafts/upgrades.rst'
--- docs/source/drafts/upgrades.rst	1970-01-01 00:00:00 +0000
+++ docs/source/drafts/upgrades.rst	2011-04-08 06:02:24 +0000
@@ -0,0 +1,54 @@
+Upgrades
+========
+
+A core functionality of any configuration management system is
+handling the full lifecycle of service and configuration
+upgrades. 
+
+Formula upgrades
+----------------
+
+A common task when doing formula development is iterating over
+formula versions via upgrading formulas on the running service to
+verify modified hooks in a live environment.
+
+The use case also extends generically to a user upgrading a deployed
+service's formula with a newer version from an upsteam formula
+repository.
+
+In some cases a new formula version will also reference newer
+software/package versions or new packages.
+
+More details in the `formula upgrades documentation`_
+
+.. _`formula upgrades documentation`: ./formula-upgrades.html
+
+
+Service upgrades
+----------------
+
+There's an interesting set of upgrade use cases which embody lots of
+real world usage, which have are left for future work.
+
+One case is where an application is deployed across multiple service
+units, and the code needs to be upgraded in lock step across all of
+them (either due to software incompatability or data changes in
+related services).
+
+Additionally the practices of a rolling uprade, and cloning a service
+as an upgrade mechanism are also interesting problems, which are left
+for future work.
+
+Ensemble upgrades
+-----------------
+
+One last upgrade scenario, is upgrading of the ensemble software
+itself.
+
+At the moment ensemble is deployed from revision control, although its
+being packaged for the future. Currently all of the ensemble agents
+maintain persistent connections to zookeeper, the failure of which may
+be grounds for the system to take corrective action. As a simple
+notion of performing system wide ensemble upgrades, the software would
+be updated on the existing systems, and then the agents restarted but
+instructed to keep their existing zookeeper session ids.