Merge lp:~hazmat/pyjuju/auto-magic-dependency-spec into lp:pyjuju

[Kapil Thangavelu]

A spec describing the resolution of dependencies and the basic interaction with remote repositories.

[lp:~hazmat/pyjuju/auto-magic-dependency-spec]

178. By Kapil Thangavelu on 2011-03-17

start documenting implementation approach to dependencies.

179. By Kapil Thangavelu on 2011-03-21

go into some details into internals of dependency resolution and deployment.

[Gustavo Niemeyer (niemeyer)]

Kapil, here is some feedback on the spec. Let's freeze the work on automatic
dependencies for the moment, and focus on upgrades, so that we get one thing
done well at a time. We have two specs but neither of them feels close to being
ready, and you won't be able to code these two things at once either way.

[1]

97 +implementation details of a repository. The multi dict ordering is
98 +manipulated after any given result to allow for the origin repository
99 +of the most recent result to be in the front of the lookup order. The
100 +default order per user configuration is stored separately and
101 +resumable as needed.

This whole sentence is extremely dense, and mixes algorithms with storage
format with server implementation with a number of other things, making it
very hard to follow.

We need something which is significantly simpler and better structured.

E.g. What are the specific new concepts being introduced into the code base,
and how do they relate with what's already in place. What are the specific
queries which a repository must answer, and what is the specific information
being sent back in the response. How will the namespacing issues we've
talked about influence the way in which dependencies are automatically
selected. What happens when there are existing services deployed. How to
pick specific formulas/service names for solving certain dependencies.
And so on.

But for now, let's freeze that spec, and focus on upgrades. Pushing these
two big concepts at the same time is not working.

review: Needs Fixing

=== added file 'docs/source/specifications/auto-dependency.rst'
--- docs/source/specifications/auto-dependency.rst	1970-01-01 00:00:00 +0000
+++ docs/source/specifications/auto-dependency.rst	2011-03-21 16:29:52 +0000
@@ -0,0 +1,112 @@
+Installing Dependencies
+=======================
+
+One of the key features for ease of use when utilizing in ensemble is
+the ability to have it fetch and deploy formulas from remote
+repositories with dependency resolution. Ie. invoking::
+
+ ensemble deploy wordpress
+
+Will lookup a formula named wordpress in the configured remote formula
+repositories, and should pick up any required interfaces of wordpress
+and satisfy those dependencies within the environment if they don't
+already exist.
+
+This last part is key, as it embodies ensemble moniker as a network
+enabled apt, where dependency resolution of installed packages is
+satisifed by services available within the local network/environment.
+
+Ensemble allows for multiple formula repositories to be utilized when
+searching for formulas, unlike apt where multiple repositories are
+overlayed into a single global namespace, ensemble provides for
+each repository as an independent namespace. ie. Once a package is 
+installed from a repository, it will not be upgraded automatically
+by a newer version in a different repository. Updates will only
+be detected against its origin repository, unless explicitly modified
+by the user.
+
+Dependencies are not looked up directly by formula name, but by
+interface names, which is also how dependencies are expressed. In the
+example above wordpress requires something providing the mysql
+interface. Ensemble will first attempt to satisfy this dependency by
+examining the services already deployed within the environment to
+determine if any of them provide the required interface.
+
+If the interface dependency cannot be satisified from the local
+environment, the repository of the formula which specified the
+dependency will be utilized.  So in the case of deploying a wordpress
+formula, the repository that contains the wordpress formula would be
+searched next for a formula that provides the required dependency, 
+mysql, if one was not already present in the environment. 
+
+If the required dependencies are not found in this repository,
+ensemble fallbacks to its repository lookup algorithm, searching each
+repository in turn for the dependency. The repository lookup algorithm
+can be configured by the user in their environment configuration. It
+defaults to searching any command line specified formula directories
+or repositories, and then the central ubuntu formula repositories.
+The user may specify additional private or public repositories in
+their environment configuration.
+
+
+Repository Interaction
+======================
+
+Remote repository communication is modeled as a collection of REST
+resources exposed over HTTP. All resources respond with proper cache
+headers (If-Modified-Since and E-tag) to allow for efficient
+detection of changes.  The formulation as a collection of REST
+resources should provide for both the use of a smart formula
+repository server, as well as just a directory exposed over http.
+
+The path to a formula in a repository::
+ 
+ <repo_url>/formulas/<formula_id>
+
+which returns the zipped formula archive.
+
+Formulas can be queried by the interface they provide via examining the 
+interface location.
+
+ <repo_url>/provides/<interface_name>
+
+The returned resource contains urls to a formula one per line. The
+return of fully qualified urls to formulas is to assist in the
+implementation of a smart server that might serve as an aggregate over
+multiple formula repositories (ie. community contrib/ppa).
+
+Additional command line utilities are provided to turn a formula of
+directories into a formula repository.
+
+
+Internals
+=========
+
+An ordered multi dict variation is utilized for dependency resolution,
+with each layer of the multi dict representing a named
+repository. Keys into the multidict are the provide interface
+names. Results returned are a sequence of repository name, and the
+formula urn for eaching matching result. Internally each repository
+presents a retrieval interface to return formulas by interface. The
+internal mechanics of querying this resource, caching values, and
+applying any ordering when multiple results are found are left as
+implementation details of a repository. The multi dict ordering is
+manipulated after any given result to allow for the origin repository
+of the most recent result to be in the front of the lookup order. The
+default order per user configuration is stored separately and
+resumable as needed.
+
+The dependency resolution process is started by examining the required
+interfaces of the formula to be deployed. These populate a deploy dependency
+dictionary, with key of provides to depend list values. The dependency
+values are iteratively resolved via the repository ordered multi dict
+via topological sorts with population of formula names into a deploy 
+formula dict.
+
+After fully solving all required dependencies, the system goes about
+downloading the formulas into the environment, and deploying them
+each in turn with relations.
+
+
+
+