Merge lp:~kim0/pyjuju/user-tutorial-and-FAQ into lp:pyjuju

[Ahmed Kamal]

Added user level tutorial
Started a FAQ document

[Gustavo Niemeyer (niemeyer)]

Wow, this is *awesome* Ahmed. Thanks a lot!

I have several minor nits only due to the doc size, but it's indeed very
good overall.

Would you mind to check these out and let me know what you think?

[1]

+User Tutorial

Can you please lowercase the "Tutorial"? We had casing mixed up, and since the last doc
polishing I'm trying to have all documents standardizing around "Title case", so that
we have some consistency there.

[2]

10 + The name Ensemble is rooted in group communication systems and paxos, where

The Ensemble term actually comes from the meaning of the "ensemble" word itself:

"an assemblage of parts or details (as in a work of art) considered as forming a whole"

[3]

+ servers. Another way to look at it, is that Ensemble helps you manage an

s/,//

[4]

+ make those services communicate through a simple protocol. Users can then

s/communicate/coordinate their communication/

[5]

+ ready to be used in production. However adventerous users are encouraged to

s/However /However, /

s/adventerous/adventurous/

[6]

+ Ensemble itself is developed using python. However writing formulas for

s/However /However, /

[7]

37 + Ensemble can be done in any language. All Ensemble cares about is finding a
38 + set of executable files, which it will trigger appropriately. Hooks can be
39 + written in a variety of languages such as bash, python, perl, c++ and
40 + others

Might be better to end the paragraph in "appropriately.". It already said it
can be written in *any* language, so there's no need to give examples.

[8]

109 +Note the following, machine "0" has been started. This is usually the
110 +bootstrapping node and the first node to be started. The dns-name for the node

s/usually//, both affirmations are necessarily true.

[9]

+is printed. Also the ec2 instance-id is printed. Since no services are yet

s/ec2/EC2/

[10]

+This will connect to the environment, and start tailing logs

s/logs/logs./

[11]

+Deploying Service Units

s/S/s/ and s/U/u/, same as [1].

[12]

+We can see a new ec2 instance has now been spun up for mysql. Information for

s/ec2/EC2/

[13]

171 +We can see a new ec2 instance has now been spun up for mysql. Information for
172 +this instance is displayed as machine number 1. Mysql is now listed under
173 +services, it is apparent the mysql service unit has no relations, since it

Minor sentence refactoring:

  We can see a new ec2 instance has now been spun up for mysql. Information for
  this instance is displayed as machine number 1 and mysql is now listed under
  services. It is apparent that the mysql service unit has no relations, since it

[14]

174 +has not been connected to wordpress yet. Since this is the first mysql service
175 +unit, it is being referred to as mysql/0, subsequent service units would be

It would be nice to mention here, even if very briefly, the distinction between a
service and a service unit.

[15]

183 +Waiting a minute for all services to complete their configuration cycle and
184 +get properly started, then issuing a status command::

s/Waiting a/Let's wait for a/
s/then issuing/then issue/

[16]

+Adding Relation

=> "Adding a relation"

[17]...

review: Approve

[lp:~kim0/pyjuju/user-tutorial-and-FAQ]

207. By Ahmed Kamal on 2011-05-03

Applying merge 58861 comments, text cleanups

[Ahmed Kamal (kim0)]

Thanks! I've applied your comments and pushed

[lp:~kim0/pyjuju/user-tutorial-and-FAQ]

208. By Ahmed Kamal on 2011-05-05

Adding project resources and contact info

209. By Ahmed Kamal on 2011-05-05

Aligning at two leading spaces

210. By Ahmed Kamal on 2011-05-05

Fixing typo, title underline too short

=== added file 'docs/source/faq.rst'
--- docs/source/faq.rst	1970-01-01 00:00:00 +0000
+++ docs/source/faq.rst	2011-05-05 13:46:31 +0000
@@ -0,0 +1,49 @@
+Frequently Asked Questions
+==========================
+
+Where does the name Ensemble come from?
+
+  The word ensemble could be defined as "an assemblage of parts or details
+  (as in a work of art) considered as forming a whole". Ensemble assembles
+  machines and software components and manages them as a whole, providing
+  service deployment and management, hence the name.
+
+Why is Ensemble useful?
+
+  Ensemble is a next generation service deployment and orchestration
+  framework.  It has been likened to APT for the cloud. With Ensemble,
+  different authors are able to create service formulas independently, and
+  make those services coordinate their communication through a simple
+  protocol.  Users can then take the product of different authors and very
+  comfortably deploy those services in an environment.  The result is
+  multiple machines and components transparently collaborating towards
+  providing the requested service.  Read more :doc:`about`
+
+When will it be ready for production?
+
+  As of Ubuntu Natty 11.04, Ensemble is a technology preview. It is not yet
+  ready to be used in production. However, adventurous users are encouraged to
+  evaluate it, study it, start writing formulas for it or start hacking on
+  Ensemble internals. The rough roadmap is to have Ensemble packaged for
+  Universe by 11.10 release and perhaps in main by 12.04
+
+What language is Ensemble developed in?
+
+  Ensemble itself is developed using python. However, writing formulas for
+  Ensemble can be done in any language. All Ensemble cares about is finding a
+  set of executable files, which it will trigger appropriately.
+
+How can I contact the Ensemble team?
+
+  User and formula author oriented resources
+   * Mailing list: https://lists.ubuntu.com/mailman/listinfo/Ubuntu-cloud
+   * IRC #ubuntu-cloud
+  Ensemble development
+   * Mailing list: https://lists.ubuntu.com/mailman/listinfo/Ensemble
+   * IRC #ubuntu-ensemble
+
+Where can I find out more about Ensemble?
+
+  * Project Site: https://launchpad.net/ensemble
+  * Ensemble Documentation: http://j.mp/ensemble-docs
+  * Work Items: http://people.canonical.com/~niemeyer/budapest.html
 
=== modified file 'docs/source/index.rst'
--- docs/source/index.rst	2011-05-05 09:41:11 +0000
+++ docs/source/index.rst	2011-05-05 13:46:31 +0000
@@ -10,7 +10,9 @@
    :maxdepth: 2
 
    about
+   faq
    getting-started
+   user-tutorial
    formula
    hook-debugging
    upgrades
 
=== added file 'docs/source/user-tutorial.rst'
--- docs/source/user-tutorial.rst	1970-01-01 00:00:00 +0000
+++ docs/source/user-tutorial.rst	2011-05-05 13:46:31 +0000
@@ -0,0 +1,324 @@
+.. _user-tutorial:
+
+User tutorial
+=============
+
+Introduction
+------------
+
+This tutorial demonstrates basic features of Ensemble from a user perspective.
+An Ensemble user would typically be a devops or a sys-admin who is interested in
+automated deployment and management of servers and services.
+
+Bootstrapping
+-------------
+
+The first step for deploying an Ensemble system is to perform bootstrapping.
+Bootstrapping launches a utility instance that is used in all subsequent
+operations to launch and orchestrate other instances::
+
+  $ bin/ensemble bootstrap
+
+Note that while the command should display a message indicating it has finished
+successfully, that does not mean the bootstrapping instance is immediately
+ready for usage. Bootstrapping an instance can require a couple of minutes. To
+check on the status of the Ensemble deployment, we can use the status command::
+
+  $ bin/ensemble status
+
+If the bootstrapping node has not yet completed bootstrapping, the status
+command may either mention the environment is not yet ready, or may display a
+connection timeout such as::
+
+  INFO Connecting to environment.
+  ERROR Connection refused
+  ProviderError: Interaction with machine provider failed:
+  ConnectionTimeoutException('could not connect before timeout after 2
+  retries',)
+  ERROR ProviderError: Interaction with machine
+  provider failed: ConnectionTimeoutException('could not connect before timeout
+  after 2 retries',)
+
+This is simply an indication the environment needs more time to complete
+initialization. It is recommended you retry every minute. Once the environment
+has properly initialized, the status command should display::
+
+  machines:
+    0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
+    services: {}
+
+Note the following, machine "0" has been started. This is the bootstrapping
+node and the first node to be started. The dns-name for the node is printed.
+Also the EC2 instance-id is printed. Since no services are yet deployed to the
+Ensemble system yet, the list of deployed services is empty
+
+Starting debug-log
+------------------
+
+While not a requirement, it is beneficial for the understanding of Ensemble to
+start a debug-log session. Ensemble's debug-log provides great insight into the
+execution of various hooks as they are triggered by various events. It is
+important to understand that debug-log shows events from a distributed
+environment (multiple-instances). This means that log lines will alternate
+between output from different instances. To start a debug-log session, from a
+secondary terminal issue::
+
+  $ bin/ensemble debug-log
+  INFO Connecting to environment.
+  INFO Enabling distributed debug log.
+  INFO Tailing logs - Ctrl-C to stop.
+
+This will connect to the environment, and start tailing logs.
+
+Deploying service units
+-----------------------
+
+Now that we have bootstrapped the Ensemble environment, and started the
+debug-log viewer, let's proceed by deploying a mysql service::
+
+  $ bin/ensemble deploy --repository=examples mysql
+  INFO Connecting to environment.
+  INFO Formula deployed as service: 'mysql'
+  INFO 'deploy' command finished successfully
+
+Checking the debug-log window, we can see the mysql service unit being
+downloaded and started::
+
+  Machine:1: ensemble.agents.machine DEBUG: Downloading formula
+  local:mysql-11...
+  Machine:1: ensemble.agents.machine INFO: Started service unit mysql/0
+
+It is important to note the different debug levels. DEBUG is used for very
+detailed logging messages, usually you should not care about reading such
+messages unless you are trying to debug (hence the name) a specific problem.
+INFO debugging level is used for slightly more important informational
+messages. In this case, these messages are generated as the mysql formula's
+hooks are being executed. Let's check the current status::
+
+  $ bin/ensemble status
+  machines:
+    0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
+    1: {dns-name: ec2-50-16-117-185.compute-1.amazonaws.com, instance-id: i-227e294d}
+  services:
+    mysql:
+      formula: local:mysql-11
+      relations: {}
+      units:
+        mysql/0:
+          machine: 1
+          relations: {}
+          state: null
+
+We can see a new EC2 instance has now been spun up for mysql. Information for
+this instance is displayed as machine number 1 and mysql is now listed under
+services. It is apparent the mysql service unit has no relations, since it has
+not been connected to wordpress yet. Since this is the first mysql service
+unit, it is being referred to as mysql/0, subsequent service units would be
+named mysql/1 and so on. 
+
+.. note::
+        An important distinction to make is the difference between a service
+        and a service unit. A service is a high level concept relating to an
+        end-user visible service such as mysql. The mysql service would be
+        composed of several mysql service units referred to as mysql/0, mysql/1
+        and so on. 
+
+The mysql service state is listed as null since it's not ready yet.
+Downloading, installing, configuring and starting mysql can take some time.
+However we don't have to wait for it to configure, let's proceed deploying
+wordpress::
+
+  $ bin/ensemble deploy --repository=examples wordpress
+
+Let's wait for a minute for all services to complete their configuration cycle and
+get properly started, then issue a status command::
+
+  $ bin/ensemble status
+  machines:
+    0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
+    1: {dns-name: ec2-50-16-117-185.compute-1.amazonaws.com, instance-id: i-227e294d}
+    2: {dns-name: ec2-184-72-156-54.compute-1.amazonaws.com, instance-id: i-9c7e29f3}
+  services:
+    mysql:
+      formula: local:mysql-11
+      relations: {}
+      units:
+        mysql/0:
+          machine: 1
+          relations: {}
+          state: started
+    wordpress:
+      formula: local:wordpress-29
+      relations: {}
+      units:
+        wordpress/0:
+          machine: 2
+          relations: {}
+          state: started
+
+mysql/0 as well as wordpress/0 are both now in the started state. Checking the
+debug-log would reveal wordpress has been started as well
+
+Adding a relation
+-----------------
+
+While mysql and wordpress service units have been started, they are still
+isolated from each other. An important concept for Ensemble is connecting
+various service units together to create a bigger ensemble! Adding a relation
+between service units causes hooks to trigger, in effect causing all service
+units to collaborate and work together to reach the desired end state. Adding a
+relation is extremely simple::
+
+  $ bin/ensemble add-relation wordpress mysql
+  INFO Connecting to environment.
+  INFO Added mysql relation to all service units.
+  INFO 'add_relation' command finished successfully
+
+Checking the Ensemble status we see that the db relation now exists with state
+up::
+
+  $ bin/ensemble status
+  machines:
+    0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
+    1: {dns-name: ec2-50-16-117-185.compute-1.amazonaws.com, instance-id: i-227e294d}
+    2: {dns-name: ec2-184-72-156-54.compute-1.amazonaws.com, instance-id: i-9c7e29f3}
+  services:
+    mysql:
+      formula: local:mysql-11
+      relations: {db: wordpress}
+      units:
+        mysql/0:
+          machine: 1
+          relations:
+            db: {state: up}
+          state: started
+    wordpress:
+      formula: local:wordpress-29
+      relations: {db: mysql}
+      units:
+        wordpress/0:
+          machine: 2
+          relations:
+            db: {state: up}
+          state: started
+
+You can now point your browser at the public dns-name for instance 2 (running
+wordpress) to view the wordpress blog
+
+Tracing hook execution
+----------------------
+
+An Ensemble user should never have to trace the execution order of hooks,
+however if you are the kind of person who enjoys looking under the hood, this
+section is for you. Understanding hook order execution, the parallel nature of
+hook execution across instances, and how relation-set in a hook can trigger the
+execution of another hook is quite interesting and provides insight into
+Ensemble internals
+
+Here are a few important messages from the debug-log of this Ensemble run.  The
+date field has been deliberately left in this log, in order to understand the
+parallel nature of hook execution.
+
+Things to consider while reading the log include:
+ * The time the log message was generated
+ * Which service unit is causing the log message (for example mysql/0)
+ * The message logging level. In this run DEBUG messages are generated by the
+   Ensemble core engine, while WARNING messages are generated by calling
+   ensemble-log from inside formulas (which you can read in the examples
+   folder)
+
+Let's view select debug-log messages which can help understand the execution
+order::
+
+  14:29:43,625 unit:mysql/0: hook.scheduler DEBUG: executing hook for wordpress/0:joined
+  14:29:43,626 unit:mysql/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-joined
+  14:29:43,660 unit:wordpress/0: hook.scheduler DEBUG: executing hook for mysql/0:joined
+  14:29:43,660 unit:wordpress/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-joined
+  14:29:43,661 unit:wordpress/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
+  14:29:43,789 unit:mysql/0: unit.hook.api WARNING: Creating new database and corresponding security settings
+  14:29:43,813 unit:wordpress/0: unit.hook.api WARNING: Retrieved hostname: ec2-184-72-156-54.compute-1.amazonaws.com
+  14:29:43,976 unit:mysql/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
+  14:29:43,997 unit:wordpress/0: hook.scheduler DEBUG: executing hook for mysql/0:modified
+  14:29:43,997 unit:wordpress/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
+  14:29:44,143 unit:wordpress/0: unit.hook.api WARNING: Retrieved hostname: ec2-184-72-156-54.compute-1.amazonaws.com
+  14:29:44,849 unit:wordpress/0: unit.hook.api WARNING: Creating appropriate upload paths and directories
+  14:29:44,992 unit:wordpress/0: unit.hook.api WARNING: Writing wordpress config file /etc/wordpress/config-ec2-184-72-156-54.compute-1.amazonaws.com.php
+  14:29:45,130 unit:wordpress/0: unit.hook.api WARNING: Writing apache config file /etc/apache2/sites-available/ec2-184-72-156-54.compute-1.amazonaws.com
+  14:29:45,301 unit:wordpress/0: unit.hook.api WARNING: Enabling apache modules: rewrite, vhost_alias
+  14:29:45,512 unit:wordpress/0: unit.hook.api WARNING: Enabling apache site: ec2-184-72-156-54.compute-1.amazonaws.com
+  14:29:45,688 unit:wordpress/0: unit.hook.api WARNING: Restarting apache2 service
+
+
+Scaling the ensemble
+--------------------
+
+Assuming your blog got really popular, is having high load and you decided to
+scale it up (it's a cloud deployment after all). Ensemble makes this magically
+easy. All what is needed is::
+
+  $ bin/ensemble add-unit wordpress
+  INFO Connecting to environment.
+  INFO Unit 'wordpress/1' added to service 'wordpress'
+  INFO 'add_unit' command finished successfully
+  $ bin/ensemble status
+  machines:
+    0: {dns-name: ec2-50-16-61-111.compute-1.amazonaws.com, instance-id: i-2a702745}
+    1: {dns-name: ec2-50-16-117-185.compute-1.amazonaws.com, instance-id: i-227e294d}
+    2: {dns-name: ec2-184-72-156-54.compute-1.amazonaws.com, instance-id: i-9c7e29f3}
+    3: {dns-name: ec2-50-16-156-106.compute-1.amazonaws.com, instance-id: i-ba6532d5}
+  services:
+    mysql:
+      formula: local:mysql-11
+      relations: {db: wordpress}
+      units:
+        mysql/0:
+          machine: 1
+          relations:
+            db: {state: up}
+          state: started
+    wordpress:
+      formula: local:wordpress-29
+      relations: {db: mysql}
+      units:
+        wordpress/0:
+          machine: 2
+          relations:
+            db: {state: up}
+          state: started
+        wordpress/1:
+          machine: 3
+          relations:
+            db: {state: up}
+          state: started
+
+
+The add-unit command starts a new wordpress instance (wordpress/1), which then
+joins the relation with the already existing mysql/0 instance. mysql/0 notices
+the database required has already been created and thus decides all needed
+configuration has already been done. On the other hand wordpress/1 reads
+service settings from mysql/0 and starts configuring itself and joining the
+ensemble. Let's review a short version of debug-log for adding wordpress/1::
+
+  14:36:19,755 unit:mysql/0: hook.scheduler DEBUG: executing hook for wordpress/1:joined
+  14:36:19,755 unit:mysql/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-joined
+  14:36:19,810 unit:wordpress/1: hook.scheduler DEBUG: executing hook for mysql/0:joined
+  14:36:19,811 unit:wordpress/1: unit.relation.lifecycle DEBUG: Executing hook db-relation-joined
+  14:36:19,811 unit:wordpress/1: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
+  14:36:19,918 unit:mysql/0: unit.hook.api WARNING: Database already exists, exiting
+  14:36:19,938 unit:mysql/0: unit.relation.lifecycle DEBUG: Executing hook db-relation-changed
+  14:36:19,990 unit:wordpress/1: unit.hook.api WARNING: Retrieved hostname: ec2-50-16-156-106.compute-1.amazonaws.com
+  14:36:20,757 unit:wordpress/1: unit.hook.api WARNING: Creating appropriate upload paths and directories
+  14:36:20,916 unit:wordpress/1: unit.hook.api WARNING: Writing wordpress config file /etc/wordpress/config-ec2-50-16-156-106.compute-1.amazonaws.com.php
+  14:36:21,088 unit:wordpress/1: unit.hook.api WARNING: Writing apache config file /etc/apache2/sites-available/ec2-50-16-156-106.compute-1.amazonaws.com
+  14:36:21,236 unit:wordpress/1: unit.hook.api WARNING: Enabling apache modules: rewrite, vhost_alias
+  14:36:21,476 unit:wordpress/1: unit.hook.api WARNING: Enabling apache site: ec2-50-16-156-106.compute-1.amazonaws.com
+  14:36:21,682 unit:wordpress/1: unit.hook.api WARNING: Restarting apache2 service
+
+Destroying the environment
+--------------------------
+
+Once you are done with an Ensemble deployment, you need to terminate all
+running instances in order to stop paying for them. The shutdown command helps
+terminate all running instances::
+
+  $ bin/ensemble shutdown