Merge lp:~caio1982/capomastro/releasedocs into lp:capomastro

[Caio Begotti]

Document the release process of Capomastro, also add deployment instructions.

This is for story 1339 -- as Capomastro developer I want to create a Capomastro release process so have a list of steps to follow for a release instead of winging it every time.

I've published on https://chinstrap.canonical.com/~caio1982/capomastro/releasing.html and https://chinstrap.canonical.com/~caio1982/capomastro/deployment.html the new files, as well an updated version of setup.html with a few different bits.

Since I'm used to the charm, the packaging and spec runs I'm biased to test it but I gave it all a try and it seemed okay.

Also, I'm afraid it's all too verbose, so please let me know what can be simplified and needs fixing!

[Caio Begotti (caio1982)]

I think these changes covers https://docs.google.com/document/d/14elkWYOLCeKKHaFagr4jzKb9TZYWEEdNJG0oqf2auso in pretty much everything but assumes many defaults so let me know if any part is still not very clear and that differs between the doc and the commits :-)

[Daniel Manrique (roadmr)]

I like this. I made two comments below. Since this is documentation I think we should fix those but they're minor so it should be easy.

review: Needs Fixing

[lp:~caio1982/capomastro/releasedocs]

202. By Caio Begotti on 2015-06-02

remove whitespaces found in blank lines during review, oops

203. By Caio Begotti on 2015-06-02

remove alien word glued to another, which probably means there is a "them" missing somewhere else now? dunno how it got here anyway

[Caio Begotti (caio1982)]

Thanks, Daniel. Just fixed those.

[Daniel Manrique (roadmr)]

+1 then, thanks!

review: Approve

=== modified file 'docs/source/conf.py'
--- docs/source/conf.py	2015-03-17 15:26:28 +0000
+++ docs/source/conf.py	2015-06-02 20:49:01 +0000
@@ -49,16 +49,16 @@
 
 # General information about the project.
 project = u'Capomastro'
-copyright = u'2014, Canonical'
+copyright = u'2015, Canonical'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
 # The short X.Y version.
-version = '20141030'
+version = '20150601'
 # The full version, including alpha/beta/rc tags.
-release = '20141030'
+release = '20150601'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

=== added file 'docs/source/deployment.rst'
--- docs/source/deployment.rst	1970-01-01 00:00:00 +0000
+++ docs/source/deployment.rst	2015-06-02 20:49:01 +0000
@@ -0,0 +1,149 @@
+==========
+Deployment
+==========
+
+.. Most of these notes came from https://docs.google.com/document/d/14elkWYOLCeKKHaFagr4jzKb9TZYWEEdNJG0oqf2auso
+
+Please follow the instructions below on how to deploy Capomastro, choosing what suits best in your scenario.
+
+With packages
+-------------
+
+This is the basis for cloud deployments as the software is pre-packaged so it is much easier to install. However, ensure your system provides Django 1.6 as this is the version currently supported on Capomastro.
+
+Add `ppa:ce-infrastructure/capomastro <https://launchpad.net/~ce-infrastructure/+archive/ubuntu/capomastro>`_ to your system, but note that it's a private PPA so you may need to request access to it. Then run:
+
+::
+
+  sudo apt-get update
+  sudo apt-get install capomastro
+
+The package should install and start up RabbitMQ and Celery automatically for you.
+
+`Install and configure a Jenkins server as documented upstream <http://pkg.jenkins-ci.org/debian/>`_.
+
+`Install its Notifications plugin <https://wiki.jenkins-ci.org/display/JENKINS/Notification+Plugin>`_ too, as Capomastro needs this to receive build status back.
+
+On your local system, edit ``/etc/hosts`` and add an entry called ``capomastro`` pointing to the IP address of your Capomastro server. For instance, if you're doing this in the same system you're using the browser on, add:
+
+::
+
+  127.0.0.1     capomastro
+
+Access http://capomastro and you should see the login page. The default credentials are ``capomastro:capomastro``.
+
+All configurations files of Capomastro are stored via symlinks inside ``/etc/capomastro/`` and you may need to tweak them depending on your setup.
+
+With Juju
+---------
+
+For a proper production deployment of Capomastro, it's best to have the corresponding units deployed separately as this allows better resilience and scalability. Unfortunately, since Capomastro is indeed a complex stack, deploying all these services and then connecting them appropriately has become a somewhat elaborate process.
+
+The Capomastro Juju charm lives in `lp:~ce-infrastructure/capomastro/charm <https://launchpad.net/~ce-infrastructure/capomastro/charm>`_ and it contains a pretty good ``README`` file which details how to prepare it with additional services to deploy, and how to connect them.
+
+However, this entire process is already scripted and automated using Mojo, so while manual deployment with Juju is possible we encourage using Mojo instead.
+
+Setup
++++++
+
+Ensure your ``.juju/environments.yaml`` has a section like the below, replacing the values found in your ``novarc`` (be careful with indenting, use spaces and not tabs). Option value ``YOUR_SSH_PUBLIC_KEY`` should be a key you have in your Launchpad account.
+
+::
+
+   environments:
+        canonistack:
+            type: openstack
+            auth-url: https://keystone.canonistack.canonical.com:443/v2.0/
+            default-series: trusty
+            username: $OS_USERNAME
+            tenant-name: $OS_TENANT_NAME
+            password: $OS_PASSWORD
+            region: $OS_REGION_NAME
+            authorized-keys: $YOUR_SSH_PUBLIC_KEY
+
+
+Switch to the ``canonistack`` environment by running ``juju switch canonistack``. Currently, `Canonistack <https://wiki.canonical.com/InformationInfrastructure/IS/CanonicalOpenstack>`_ is the best place to test charms if not locally.
+
+Bootstrap the first Juju node with ``juju bootstrap -v`` and wait a few minutes. You'll get the address of your bootstrap node and some info will be printed by Juju.
+
+If you have trouble related to network access or if Juju times out connecting to the instance, `ensure your VPN configuration is correct <https://wiki.canonical.com/InformationInfrastructure/IS/HowTo/CompanyOpenVPN>`_ and try SSH directly to the instance. If it fails, maybe the VPN is not working correctly.
+
+Now you can do e.g. ``juju status`` and deploy charms normally.
+
+With Mojo
+---------
+
+Mojo automates the complex process outlined in the Juju charm’s ``README`` file. Things begin to get a bit complex to do by hand, but fortunately all the services comprising the Capomastro stack can be magically deployed with Mojo.
+
+By default it tries to deploy Capomastro on an OpenStack-compliant cloud, and it will create and automatically attach volumes to the units that require them (System Image Server, Jenkins and the DB unit).
+
+Setup
++++++
+
+Install Mojo from ``ppa:mojo-maintainers/ppa`` and also ``python-novaclient``.
+
+Fetch `the current spec code for Capomastro that is kept in a repository owned by IS <https://launchpad.net/~canonical-is/canonical-mojo-specs/mojo-pes-capomastro/>`_. Capomastro's spec code is inside the directory ``pes/mojo-pes-capomastro``.
+
+::
+
+  bzr branch lp:~canonical-is/canonical-mojo-specs/mojo-pes-capomastro local-canonical-mojo-specs
+
+Whether you deploy with Mojo on Canonistack, on Staging or Production, you'll need 3 Ceph volumes of at least 30GB each. One of them will contain the DB, the other will be used by System Image Server and the last one will be Jenkins’s storage. They need to be created in advance because we want them to be persistent, so note the volume IDs for all. On production these volumes are likely to be created by IS, but on Staging it's all on you.
+
+You'll also need to provide a set of required secrets files for Mojo before proceeding.
+
+Please `check the README of the spec <https://bazaar.launchpad.net/~canonical-is/canonical-mojo-specs/mojo-pes-capomastro/view/head%3A/pes/mojo-pes-capomastro/README>`_ as it should explain how to get all these configured properly. In any case, the spec itself will abort and tell you what's wrong if anything is missing, so that you can fix the files before trying again.
+
+Capomastro provides a Makefile to ease setting up the project and running the spec, so once you have collected the needed secrets just use the ``make`` and its targets.
+
+On Canonistack
+++++++++++++++
+
+Getting a proper Canonistack setup should be outside of this document's scope, so please refer to `the official instructions <https://wiki.canonical.com/InformationInfrastructure/IS/CanonicalOpenstack>`_ and come back here once it's up and running. Please also `set up a VPN <https://wiki.canonical.com/InformationInfrastructure/IS/HowTo/CompanyOpenVPN>`_ to make things easier for yourself.
+
+Once the secrets files are placed in ``/srv/mojo/LOCAL/capomastro/pes/mojo-pes-capomastro/canonistack/`` you're ready to deploy. Using the Makefile:
+
+::
+
+  MOJO_STAGE_NAME=canonistack make deploy
+
+Variable ``MOJO_STAGE_NAME`` tells the spec to run from the ``canonistack`` directory. It defaults to ``staging`` so that's why you need to specify it for deployments on Canonistack.
+
+On Canonistack the deploy command will take between 2 to 3 hours, while it takes only 30 minutes on a faster cloud provider. Be patient, usually if it looks stuck it's because it is still working. If something fails, it will exit with suitable error messages.
+
+Once everything has been deployed and verified you should then be able to log into ``apache2`` unit's IP address and see the Capomastro login screen.
+
+On Staging
+++++++++++
+
+As the user ``stg-pes-capomastro`` on ``wendigo.canonical.com`` (the Staging server) you'll be doing a fresh deployment on staging so you can test new releases, running the full spec is recommended. Simply run the following and the spec Makefile will do the rest:
+
+::
+
+  make deploy
+
+
+On Production
++++++++++++++
+
+Production deployments are controlled by IS and we can't do them ourselves. However, we must guide them during upgrades so they run the Mojo bits we need for each different upgrade.
+
+Usually on production a Mojo upgrade is what you need, as you don't want to re-deploy the whole spec all the time. In this case, telling IS to run the  file ``manifest-upgrade`` is enough, but you can call it with our Makefile as such:
+
+::
+
+  make upgrade
+
+Clean up
+--------
+
+You'll probably want to destroy any previous deployment first before running the spec again or deploying existing charms manually. From the Capomastro directory inside the spec, run these to clean it all up:
+
+::
+
+  make destroy
+  juju destroy-environment --yes <juju environment name>
+  juju bootstrap
+
+
+Before redeploying again, depending on what you did previously, you may want to erase the Ceph volumes and create new ones. If that's the case see the current volumes available with ``nova volumes-list`` and delete each one of them with ``nova volume-delete <ID>``. Notice that in this case you might need to update the new IDs in the configuration files.

=== modified file 'docs/source/index.rst'
--- docs/source/index.rst	2015-03-17 15:05:05 +0000
+++ docs/source/index.rst	2015-06-02 20:49:01 +0000
@@ -38,9 +38,9 @@
    credentials
    stats
    development
-
-
- 
+   deployment
+   releasing
+
 
 Indices and tables
 ==================

=== added file 'docs/source/releasing.rst'
--- docs/source/releasing.rst	1970-01-01 00:00:00 +0000
+++ docs/source/releasing.rst	2015-06-02 20:49:01 +0000
@@ -0,0 +1,110 @@
+.. _deployment: deployment.html
+
+=========
+Releasing
+=========
+
+This document explains how Capomastro's release process works.
+
+Capomastro has no specific RC or QA process integrated with new releases. Instead, all new versions are supposed to be tested before they are made public through Juju and Mojo deployments in a self-contained environment, using a separate PPA containing the new release package. Capomastro doesn't have such broad audience yet that justify RC packages, though we can change this in the future.
+
+To make things short, essentially the release process goes like this:
+
+  #. `Merging of all pending branches`_
+  #. `Update of the package changelog`_
+  #. `Package release to a testing PPA`_
+  #. `Merge service charm and spec changes`_
+  #. `Staging deployment plus testing`_
+  #. `Production upgrade`_
+  #. `Release notifications`_
+
+Merging of all pending branches
+-------------------------------
+
+Make sure all relevant pending branches are merged automatically by Tarmac after being reviewed and approved. Tarmac will run the test suite itself so you don't need to worry about it (most of the time).
+
+
+Update of the package changelog
+-------------------------------
+
+The package changelog is also used as the source changelog for now, so use ``dch -v <version> -D <supported distro name> -u high`` to append a new changelog entry to ``debian/changelog`` directly.
+
+Currently the version string is the date of the package release, but we could and should use semver strings in the near future. E.g.:
+
+::
+
+  dch -v 20150601 -D trusty -u high
+
+Will render...
+
+::
+
+  capomastro (20150601) trusty; urgency=high
+ 
+    * Fix #123456 (short summary of the bug)
+ 
+   -- Caio Begotti <[e-mail]>  Mon, 01 Jun 2015 14:56:05 -0300
+
+Make sure you include the number and a short summary of all the bugs closed between the last and this new release. Hotfixes without bug reports must be included too, as well other changes needed for the service charm or spec.
+    
+.. Changelogs are manually crafted this way due to many auto merges we have, we have tried to use the Bazaar plugin Checkbox has to produce changelogs but it'd need too many changes to start off
+
+Changes to ``debian/changelog`` for a release should be minimal and their MRs can be self-approved on Launchpad as long as they only contain changes to this one file.
+
+Package release to a testing PPA
+--------------------------------
+
+Build the source package of Capomastro with ``debuild -S``, which will ask you for your GPG data to sign the new files. The resulting file ``../capomastro_<version>_source.changes`` will be used to upload this release to a PPA.
+
+You can now upload the source package to a testing PPA with ``dput ppa:<testing PPA name> ../capomastro_<version>_source.changes``. It may take Launchpad several minutes before the binary package is available for installation and testing from the PPA.
+
+Merge service charm and spec changes
+------------------------------------
+
+Some changes to Capomastro's source might require an update of the service charm, like when some default settings from ``/etc/capomastro`` needs an update as they're templated in the charm code too. Make sure the charm is always in sync with source changes before a release, otherwise we might break Juju deployments!
+
+If the charm gets a critical fix, bump its revno in the spec code inside ``common/collect`` so the next deployments will use that revno as base and people won't deploy broken code.
+
+Similarly, keep the Mojo spec up to date with any way the source expects to be deployed. If some infrastructure bit of Capomastro needs a change due to a new feature just released, make sure the spec supports it.
+
+Unfortunately, these steps need to be done manually as it's expected the people involved will know all these parts of the service.
+
+Staging deployment plus testing
+-------------------------------
+
+Do a full and clean Mojo run of the Capomastro spec on Wendigo, the staging environment, before releasing a new package. Ideally an upgrade of the spec using the last package released to this newest one is desired, but it's not always feasible.
+
+Please follow the deployment_ document to see how to do spec runs.
+
+After it's finished test some dependencies and projects builds. You should also test the archival of some artifacts to ensure nothing broke miserably in this release. The whole testing might take a while...
+
+Capomastro lacks a testing plan and some of the tests can't be fully automated we're afraid, so you'll need to use your best judgment here deciding what to test and what to skip during a release.
+
+Production upgrade
+------------------
+
+Once you know the new release works you can put it in the project's PPA with ``dput ppa:ce-infrastructure/capomastro  ../capomastro_<version>_source.changes``. This PPA is also used by the `CI server <https://ci.admin.canonical.com/>`_, so keep this up to date with new releases.
+
+Sometimes you may want to upgrade production to use the new release. Until Capomastro has a safe way to lock users out and pause its builds before an upgrade, you'll need to disable all Django accounts manually on its admin UI and wait until all builds are done before proceeding with a production upgrade.
+
+If it's the case during a particular new release, create a RT for IS to action the upgrade by sending an e-mail to [e-mail] with a meaningful subject line. Make sure you ask them:
+
+ #. To sync our PPA to the IS one, ``ppa:~canonical-losas/capomastro``
+ #. To run the latest spec revno on CI so the new release can be tested once more using a pristine environment controlled by IS.
+ #. To run the relevant Mojo manifest you need, usually ``manifest-upgrade``.
+
+Finally you can then raise the priority of the RT just created, ping the vanguard at Canonical's #webops IRC channel and follow up the upgrade. Ideally service upgrades should happen after 6PM UTC as it's a period of low service usage.
+
+Release notifications
+---------------------
+
+Every time there's a production change to Capomastro please send out notifications to its stakeholders and immediate users so they're aware of it.
+
+Try to keep the communication short:
+
+  * List critical issues that had been solved with the release
+  * List major features changes only
+ 
+If the release changes are not any of these then consider it to be a maintenance release and state it in the communication to the Phablet and PES mailing lists.
+    
+.. It would be nice to have e-mail templates somewhere for all sort of communication we have, like Hexr seems to have
\ No newline at end of file

=== modified file 'docs/source/setup.rst'
--- docs/source/setup.rst	2015-05-07 17:19:25 +0000
+++ docs/source/setup.rst	2015-06-02 20:49:01 +0000
@@ -2,14 +2,9 @@
 Setup
 =====
 
-This file was created based on the old historic README file from Capomastro, if
-anything seems to be missing here please refer to the whole documentation files
-in the ``docs/`` directory.
+This file was created based on the old historic README file from Capomastro, if anything seems to be missing here please refer to the whole documentation files in the ``docs/`` directory.
 
-If you're an old school Django guy, you can get started with local testing and
-development by following the next sections. You'll need the following services
-in order to bring the application up anyway, no matter if using the packaged
-code or not:
+If you're an old school Django guy, you can get started with local testing and development by following the next sections. You'll need the following services in order to bring the application up anyway, no matter if using the packaged code or not:
 
 * PostgreSQL, default database
 * Apache with mod WSGI, to expose the UI
@@ -19,18 +14,11 @@
 Environment
 -----------
 
-Create your environment either with ``mkvirtualenv <name>``, if you have
-virtualenvwrapper installed, or simply ``tox -e devenv`` since it does some
-extra steps for you. It's recommended to install ``tox`` and go with the latter
-option.
-
-If you decide for using ``mkvirtualenv`` you'll also need to manually install
-the Python dependencies with ``pip install -r requirements.txt`` or even ``pip
-install -r dev-requirements.txt`` if you're going to run the code tests.
-
-Then either activate the virtual environment manually with ``source
-devenv/bin/activate`` or explicitly call Python from the virtual environment
-with ``devenv/bin/python``.
+Create your environment either with ``mkvirtualenv <name>``, if you have virtualenvwrapper installed, or simply ``tox -e devenv`` since it does some extra steps for you. It's recommended to install ``tox`` and go with the latter option.
+
+If you decide for using ``mkvirtualenv`` you'll also need to manually install the Python dependencies with ``pip install -r requirements.txt`` or even ``pip install -r dev-requirements.txt`` if you're going to run the code tests.
+
+Then either activate the virtual environment manually with ``source devenv/bin/activate`` or explicitly call Python from the virtual environment with ``devenv/bin/python``.
 
 Database
 --------
@@ -42,44 +30,30 @@
 .. include:: ../../testing/requirements/capomastro.sql
    :literal:
 
-If needed, fixup the database entry in ``capomastro/db_settings.py``, but most
-likely the default values will work okay unless you're testing database
-changes.
+If needed, fixup the database entry in ``capomastro/db_settings.py``, but most likely the default values will work okay unless you're testing database changes.
 
-You'll need to prepare the database with ``./manage.py syncdb`` (and you'll be
-prompted to create a superuser at this time).
+You'll need to prepare the database with ``./manage.py syncdb`` (and you'll be prompted to create a superuser at this time).
 
 Web server
 ----------
+
+Start the Celery worker that will talk to the Rabbit queues with ``celery -A capomastro worker -l debug``. Capomastro will need it to be running first.
  
-You can manually start Capomastro from its source directory by executing
-``./manage.py runserver`` otherwise. However, if you want to set it up with
-WSGI, take a look at the file ``wsgi.conf``.
-
-All the remaining setup steps are through the ``/admin/`` interface of
-Capomastro, so go to http://localhost:8000/admin/ then login with the username
-and password created before and continue.
-
-If you prefer to test things with Gunicorn you'll need a Celery worker running
-already, as well as RabbitMQ.
-
-Just run ``gunicorn -b 0.0.0.0:8000 capomastro.wsgi:application`` and start the
-Celery worker that will talk to the Rabbit queue with ``celery -A capomastro
-worker -l debug``.
+You can manually start Capomastro from its source directory by executing ``./manage.py runserver``. However, if you want to set it up with WSGI, take a look at the file ``wsgi.conf``.
+
+All the remaining setup steps are through the ``/admin/`` interface of Capomastro, so go to http://localhost:8000/admin/ then login with the username and password created before and continue.
+
+If you prefer to test things with Gunicorn you'll need a Celery worker running already, as well as RabbitMQ. Just run ``gunicorn -b 0.0.0.0:8000 capomastro.wsgi:application``.
 
 Django Admin Setup
 ------------------
 
-All that remains now is the first the dependencies setup, then the projects
-setup, in that particular order. Creating of projects before its dependencies
-are set is not currently supported.
+All that remains now is the first the dependencies setup, then the projects setup, in that particular order. Creating projects before its dependencies is supported but has little purpose besides testing.
 
-In the ``/admin/`` interface of Capomastro you'll need to create a
-``jenkins.JenkinsServer`` object, with the correct credentials and
+In the ``/admin/`` interface of Capomastro you'll need to create a ``jenkins.JenkinsServer`` object, with the correct credentials and
 ``REMOTE_ADDR`` set up correctly, so that it can receive callbacks.
 
-Create a ``JobType`` and a ``Job`` for that ``JobType``. Then
-create a ``Dependency`` for that ``Job`` model.
+Create a ``JobType`` and a ``Job`` for that ``JobType``. Then create a ``Dependency`` for that ``Job`` model.
 
 You should have something like this:
 
@@ -91,33 +65,10 @@
      JenkinsServer -> Job;
    }
 
-You can create several dependencies, with different jobs. Your Jenkins server
-requires the ``notifications`` plugin installed, so make sure it's
-available otherwise you won't get status back from the builds.
-
-The jobs that are created must have a parameter ``BUILD_ID`` and they must have
-a notification setup, type ``http/json`` and with a callback address of
-``http://hostname/jenkins/notifications/``.
-
-Now you can create a ``Project``, associated with your dependencies, at
-``http://localhost:8000/projects/create/``. The option "auto track" means that
-the project will use the latest version of any dependencies automatically.
-
-Now you can build your project. This will create a project build, and trigger
-the tasks to build your project on the Jenkins server.
-   
-Packaged setup
---------------
-
-All the configurations of Capomastro (messaging, DB, the service etc) are
-stored via symlinks inside of ``/etc/capomastro/`` if you're using the Debian
-package of the application. This directory also contains the file ``wsgi.conf``
-which is symlinked to the Apache configuration so the site is enabled by
-default. The default Django superuser can be used to access the service for the
-first time, the username and password should be both ``capomastro`` and it's
-created by the package automatically.
-
-All code is actually validated through the application package, so although you
-can test and develop things really locally, only code from packages is
-considered deployable.
-
+You can create several dependencies, with different jobs. Your Jenkins server requires the ``Notifications`` plugin installed, so make sure it's available otherwise you won't get status back from the builds.
+
+The jobs that are created must have a parameter ``BUILD_ID`` and they must have a notification setup, type ``http/json`` and with a callback address of ``http://hostname/jenkins/notifications/``.
+
+Now you can create a ``Project``, associated with your dependencies, at ``http://localhost:8000/projects/create/``. The option "auto track" means that the project will use the latest version of any dependencies automatically.
+
+Now you can build your project. This will create a project build, and trigger the tasks to build your project on the Jenkins server.

=== modified file 'tox.ini'
--- tox.ini	2015-04-15 18:19:38 +0000
+++ tox.ini	2015-06-02 20:49:01 +0000
@@ -17,4 +17,4 @@
 [testenv:devenv]
 envdir = devenv
 basepython = python2.7
-commands =
+deps = -r{toxinidir}/dev-requirements.txt