Merge lp:~mthaddon/mojo/mojo-how-to-autodeploy into lp:mojo/mojo-specs

[Tom Haddon]

Add autodeployment mechanisms for mojo-how-to. This will only address updating built content, not charm upgrades, which will need further work.

[lp:~mthaddon/mojo/mojo-how-to-autodeploy]

12. By Tom Haddon on 2015-10-08

Pull in the desired and production build revnos from common script

13. By Tom Haddon on 2015-10-08

We need to actually build the env with current production revno on CI first as part of test-autodeploy before updating it

14. By Tom Haddon on 2015-10-08

Use the correct option to get stdout for wget

15. By Tom Haddon on 2015-10-09

We need bzr as a builddep too

16. By Tom Haddon on 2015-10-09

Arithmetic expression for comparing revnos

17. By Tom Haddon on 2015-10-09

We need to collect before running store_desired_revno on CI because that tells us what revno to start with

18. By Tom Haddon on 2015-10-09

Create dummy secrets file as part of a step that will get run for auto-deploy

19. By Tom Haddon on 2015-10-09

Print what revnos we have

20. By Tom Haddon on 2015-10-09

We need to pass the BUILD_LABEL to our upload and update script in autodeploy otherwise it'll just overwrite mojo.tar

21. By Tom Haddon on 2015-10-09

Print what revno we are publishing

22. By Tom Haddon on 2015-10-09

Also print what we're publishing in production

[Tom Haddon (mthaddon)]

Ok, this has been tested on https://ci.admin.canonical.com/job/autodeploy-is-mojo-dot-canonical-dot-com/ - not yet tested the actually rollout portions on production though

[lp:~mthaddon/mojo/mojo-how-to-autodeploy]

23. By Tom Haddon on 2015-10-12

Missing config from check-autodeploy

24. By Tom Haddon on 2015-10-12

Make sure we have the correct builddeps for performing the autodeploy

25. By Tom Haddon on 2015-10-12

We need to pass the packages variable, not config to builddeps

[Tom Haddon (mthaddon)]

And this has now been tested on production as well. Successfully performed the rollout, and then exited with "Production is on 206 and the desired revno is 206. Nothing to do, exiting..." when rerun.

[lp:~mthaddon/mojo/mojo-how-to-autodeploy]

26. By Tom Haddon on 2015-10-12

Add a comment when we've autodeployed so we can more easily look for this and report on it

[Paul Collins (pjdc)]

review: Approve

=== modified file 'mojo-how-to/README'
--- mojo-how-to/README	2015-05-07 14:11:05 +0000
+++ mojo-how-to/README	2015-10-12 10:20:25 +0000
@@ -20,3 +20,10 @@
         ksplice:
             options:
                 access-key: $YOUR_KSPLICE_ACCESS_KEY
+
+
+If you are using the autodeploy manifests, you will also need the following in
+a secrets file called "autodeploy-secrets":
+
+CI_SWIFT_CONTAINER=URL_OF_SWIFT_CONTAINER
+PROD_SWIFT_CONTAINER=URL_OF_SWIFT_CONTAINER

=== added file 'mojo-how-to/devel/build-desired-revno'
--- mojo-how-to/devel/build-desired-revno	1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/build-desired-revno	2015-10-12 10:20:25 +0000
@@ -0,0 +1,15 @@
+#!/bin/bash
+
+# Script to generate docs from Mojo source tree
+
+set -e
+set -u
+
+source ${MOJO_SPEC_DIR}/${MOJO_STAGE}/get_common_variables
+
+cd ${MOJO_BUILD_DIR}/mojo
+
+bzr pull -r $DESIRED_REVNO --overwrite .
+
+make generate-docs
+tar cvpf ${MOJO_LOCAL_DIR}/mojo.tar --directory=docs/www .

=== added file 'mojo-how-to/devel/build-production-revno'
--- mojo-how-to/devel/build-production-revno	1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/build-production-revno	2015-10-12 10:20:25 +0000
@@ -0,0 +1,25 @@
+#!/bin/bash
+
+# Script to generate docs from Mojo source tree
+
+set -e
+set -u
+
+source ${MOJO_SPEC_DIR}/${MOJO_STAGE}/get_common_variables
+
+cd ${MOJO_BUILD_DIR}/mojo
+
+bzr pull -r $PRODUCTION_REVNO --overwrite .
+
+make generate-docs
+tar cvpf ${MOJO_LOCAL_DIR}/mojo.tar --directory=docs/www .
+
+if [ ${MOJO_STAGE##*/} != "production" ]; then
+    # We don't deploy landscape in non-production environments, but we need a
+    # dummy secrets file
+    echo "Creating dummy services-secret file in ${MOJO_LOCAL_DIR}"
+    echo "mojo-how-to:
+    services:
+        nrpe:
+            charm: nrpe-external-master" > ${MOJO_LOCAL_DIR}/services-secret
+fi

=== added file 'mojo-how-to/devel/check_for_diff_production_desired_revno'
--- mojo-how-to/devel/check_for_diff_production_desired_revno	1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/check_for_diff_production_desired_revno	2015-10-12 10:20:25 +0000
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+source ${MOJO_SPEC_DIR}/${MOJO_STAGE}/get_common_variables
+
+if [[ $PRODUCTION_REVNO -ge $DESIRED_REVNO ]]; then
+    echo "Production is on ${PRODUCTION_REVNO} and the desired revno is ${DESIRED_REVNO}. Nothing to do, exiting..."
+    exit 1
+fi
+
+echo "Production is on ${PRODUCTION_REVNO} and the desired revno is ${DESIRED_REVNO}"

=== added file 'mojo-how-to/devel/get_common_variables'
--- mojo-how-to/devel/get_common_variables	1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/get_common_variables	2015-10-12 10:20:25 +0000
@@ -0,0 +1,5 @@
+#!/bin/bash
+
+# Source variables used by other scripts
+PRODUCTION_REVNO=$(cat ${MOJO_LOCAL_DIR}/production_revno)
+DESIRED_REVNO=$(cat ${MOJO_LOCAL_DIR}/desired_revno)

=== added file 'mojo-how-to/devel/publish_revno'
--- mojo-how-to/devel/publish_revno	1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/publish_revno	2015-10-12 10:20:25 +0000
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+source ${MOJO_SPEC_DIR}/${MOJO_STAGE}/get_common_variables
+
+echo $DESIRED_REVNO > ${MOJO_LOCAL_DIR}/tested_revno
+
+echo "Publishing $DESIRED_REVNO as tested_revno"
+
+cd ${MOJO_LOCAL_DIR}
+swift upload mojo-how-to-autodeploy tested_revno

=== added file 'mojo-how-to/devel/store_desired_revno'
--- mojo-how-to/devel/store_desired_revno	1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/store_desired_revno	2015-10-12 10:20:25 +0000
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# Store the desired revno we want to upgrade to. In our case this is going to
+# be the current tip of trunk whenever we run this script, which will have been
+# collected already, so we can just query the revno.
+
+DESIRED_REVNO=$(bzr revno ${MOJO_BUILD_DIR}/mojo)
+
+echo $DESIRED_REVNO > ${MOJO_LOCAL_DIR}/desired_revno

=== added file 'mojo-how-to/devel/store_production_revno'
--- mojo-how-to/devel/store_production_revno	1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/store_production_revno	2015-10-12 10:20:25 +0000
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# Store the current revno used on production (grab from production swift
+# container)
+
+. ${MOJO_LOCAL_DIR}/autodeploy-secrets
+PRODUCTION_REVNO=$(wget -q -O- ${PROD_SWIFT_CONTAINER}/production_revno)
+
+echo $PRODUCTION_REVNO > ${MOJO_LOCAL_DIR}/production_revno

=== added directory 'mojo-how-to/manifests'
=== added file 'mojo-how-to/manifests/check-autodeploy'
--- mojo-how-to/manifests/check-autodeploy	1970-01-01 00:00:00 +0000
+++ mojo-how-to/manifests/check-autodeploy	2015-10-12 10:20:25 +0000
@@ -0,0 +1,11 @@
+## This manifest should invoke prepare-autodeploy and then verify that the
+## proposed deployment can be done. This is usually done by invoking
+## utils/confirm-autodeployable, which will check that the current spec revno has
+## successfully been used in ci.admin and that the requested build label has been
+## tested. This manifest may also perhaps run a verify phase to ensure the
+## environment is healthy. This manifest is usually invoked by
+## deploy-from-blessed-branch.
+
+include config=manifests/prepare-autodeploy
+
+include config=manifest-verify

=== added file 'mojo-how-to/manifests/perform-autodeploy'
--- mojo-how-to/manifests/perform-autodeploy	1970-01-01 00:00:00 +0000
+++ mojo-how-to/manifests/perform-autodeploy	2015-10-12 10:20:25 +0000
@@ -0,0 +1,15 @@
+## This manifest should invoke check-autodeploy to ensure the deployment can be
+## done, and then perform the deployment, most preferably by invoking
+## perform-autodeploy-unchecked. The manifest may invoke database
+## migrations, if necessary and should also verify that the deployment was
+## successful. This manifest is usually invoked by deploy-from-blessed-branch.
+
+include config=manifests/check-autodeploy
+include config=manifests/perform-autodeploy-unchecked
+include config=manifest-verify
+
+script config=publish_revno
+
+## And now just print a comment that we've successfully auto-deployed so it's
+## easier to report on that
+# Successfully auto-deployed new code for mojo-how-to

=== added file 'mojo-how-to/manifests/perform-autodeploy-unchecked'
--- mojo-how-to/manifests/perform-autodeploy-unchecked	1970-01-01 00:00:00 +0000
+++ mojo-how-to/manifests/perform-autodeploy-unchecked	2015-10-12 10:20:25 +0000
@@ -0,0 +1,16 @@
+## This manifest should perform the "meat" of the autodeploy, *without*
+## checking whether or not do so. In general this manifest is invoked by
+## perform-autodeploy (below) and by test-autodeploy (further below). 
+
+# Pull down the latest content
+collect
+# Set builddeps
+builddeps packages=make,markdown,bzr
+# Run the build step
+build config=build-desired-revno
+# Copy our built resources to the instances
+script config=upload-built-content BUILD_LABEL=revno
+# Update the config variable so our content is updated
+script config=update-content BUILD_LABEL=revno
+# Sleep to let the update be applied
+sleep config=60

=== added file 'mojo-how-to/manifests/prepare-autodeploy'
--- mojo-how-to/manifests/prepare-autodeploy	1970-01-01 00:00:00 +0000
+++ mojo-how-to/manifests/prepare-autodeploy	2015-10-12 10:20:25 +0000
@@ -0,0 +1,13 @@
+## This manifest should prepare the workspace for a deployment test or
+## autodeploy. Tasks commonly performed here are storing the build label to be
+## deployed and fetching and storing the currently-live build label from the
+## production environment or elsewhere (but ideally the former). This manifest
+## exists in particular so that the build label to be deployed is specified in one
+## and only one location. 
+
+script config=store_production_revno
+collect
+script config=store_desired_revno
+
+# Now check there's actually a rollout that we want to attempt
+script config=check_for_diff_production_desired_revno

=== added file 'mojo-how-to/manifests/test-autodeploy'
--- mojo-how-to/manifests/test-autodeploy	1970-01-01 00:00:00 +0000
+++ mojo-how-to/manifests/test-autodeploy	2015-10-12 10:20:25 +0000
@@ -0,0 +1,36 @@
+## After invoking check-autodeploy-testable, this manifest should deploy an
+## environment to match the existing production environment and then upgrade to
+## the DESIRED BUILD LABEL, most preferably by invoking
+## perform-autodeploy-unchecked. Following the upgrade this manifest
+## should perform any verification needed, and then publish suitable objects to
+## the swift container recording the result.
+
+# Get what revnos we're interested in for this deployment
+include config=manifests/prepare-autodeploy
+
+# Having confirmed there's a deployment to test, we first need to bring the
+# environment up in it's current state on production
+
+# We need the markdown package to be able to generate the docs for Mojo
+builddeps packages=make,markdown,bzr
+# Run the collect step
+collect
+# Run the build step
+build config=build-production-revno
+# Deploy services only
+deploy config=services local=services-secret delay=0
+# Copy our built resources to the instances
+script config=upload-built-content
+# And now deploy relations as well
+deploy config=relations
+# Run verify steps
+include config=manifest-verify
+
+# Now perform the autodeploy
+include config=manifests/perform-autodeploy-unchecked
+
+# And verify our environment again
+include config=manifest-verify
+
+# Now publish the revno we've successfully rolled out
+script config=publish_revno

=== added symlink 'mojo-how-to/production/build-desired-revno'
=== target is u'../devel/build-desired-revno'
=== added symlink 'mojo-how-to/production/build-production-revno'
=== target is u'../devel/build-production-revno'
=== added symlink 'mojo-how-to/production/check_for_diff_production_desired_revno'
=== target is u'../devel/check_for_diff_production_desired_revno'
=== added symlink 'mojo-how-to/production/get_common_variables'
=== target is u'../devel/get_common_variables'
=== added file 'mojo-how-to/production/publish_revno'
--- mojo-how-to/production/publish_revno	1970-01-01 00:00:00 +0000
+++ mojo-how-to/production/publish_revno	2015-10-12 10:20:25 +0000
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+source ${MOJO_SPEC_DIR}/${MOJO_STAGE}/get_common_variables
+
+echo $DESIRED_REVNO > ${MOJO_LOCAL_DIR}/production_revno
+
+echo "Publishing $DESIRED_REVNO as production_revno"
+
+cd ${MOJO_LOCAL_DIR}
+swift upload mojo-how-to-autodeploy production_revno

=== added file 'mojo-how-to/production/store_desired_revno'
--- mojo-how-to/production/store_desired_revno	1970-01-01 00:00:00 +0000
+++ mojo-how-to/production/store_desired_revno	2015-10-12 10:20:25 +0000
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+# Store the desired revno we want to upgrade to. In our case this is going to
+# be the revno that's been tested on CI.
+
+. ${MOJO_LOCAL_DIR}/autodeploy-secrets
+DESIRED_REVNO=$(wget -q -O- ${CI_SWIFT_CONTAINER}/tested_revno)
+
+echo $DESIRED_REVNO > ${MOJO_LOCAL_DIR}/desired_revno

=== added file 'mojo-how-to/production/store_production_revno'
--- mojo-how-to/production/store_production_revno	1970-01-01 00:00:00 +0000
+++ mojo-how-to/production/store_production_revno	2015-10-12 10:20:25 +0000
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+# Store the current revno used on production (grab from production swift
+# container). Even though this is being run in the production stage, this is
+# still the most reliable way of getting this.
+
+. ${MOJO_LOCAL_DIR}/autodeploy-secrets
+PRODUCTION_REVNO=$(wget -q -O- ${PROD_SWIFT_CONTAINER}/production_revno)
+
+echo $PRODUCTION_REVNO > ${MOJO_LOCAL_DIR}/production_revno