Mojo: Continuous Delivery for Juju

Merge lp:~mthaddon/mojo/mojo-how-to-autodeploy into lp:mojo/mojo-specs

  1. mojo-how-to-autodeploy
  2. Merge into mojo-specs
Proposed by Tom Haddon
Status: Merged
Merged at revision: 11
Proposed branch: lp:~mthaddon/mojo/mojo-how-to-autodeploy
Merge into: lp:mojo/mojo-specs
Diff against target: 301 lines (+210/-0)
16 files modified
mojo-how-to/README (+7/-0)
mojo-how-to/devel/build-desired-revno (+15/-0)
mojo-how-to/devel/build-production-revno (+25/-0)
mojo-how-to/devel/check_for_diff_production_desired_revno (+10/-0)
mojo-how-to/devel/get_common_variables (+5/-0)
mojo-how-to/devel/publish_revno (+10/-0)
mojo-how-to/devel/store_desired_revno (+9/-0)
mojo-how-to/devel/store_production_revno (+9/-0)
mojo-how-to/manifests/check-autodeploy (+11/-0)
mojo-how-to/manifests/perform-autodeploy (+15/-0)
mojo-how-to/manifests/perform-autodeploy-unchecked (+16/-0)
mojo-how-to/manifests/prepare-autodeploy (+13/-0)
mojo-how-to/manifests/test-autodeploy (+36/-0)
mojo-how-to/production/publish_revno (+10/-0)
mojo-how-to/production/store_desired_revno (+9/-0)
mojo-how-to/production/store_production_revno (+10/-0)
To merge this branch: bzr merge lp:~mthaddon/mojo/mojo-how-to-autodeploy
Reviewer Review Type Date Requested Status
Paul Collins Approve
Review via email: mp+273817@code.launchpad.net

Description of the change

Add autodeployment mechanisms for mojo-how-to. This will only address updating built content, not charm upgrades, which will need further work.

To post a comment you must log in.
lp:~mthaddon/mojo/mojo-how-to-autodeploy updated
12. By Tom Haddon

Pull in the desired and production build revnos from common script

13. By Tom Haddon

We need to actually build the env with current production revno on CI first as part of test-autodeploy before updating it

14. By Tom Haddon

Use the correct option to get stdout for wget

15. By Tom Haddon

We need bzr as a builddep too

16. By Tom Haddon

Arithmetic expression for comparing revnos

17. By Tom Haddon

We need to collect before running store_desired_revno on CI because that tells us what revno to start with

18. By Tom Haddon

Create dummy secrets file as part of a step that will get run for auto-deploy

19. By Tom Haddon

Print what revnos we have

20. By Tom Haddon

We need to pass the BUILD_LABEL to our upload and update script in autodeploy otherwise it'll just overwrite mojo.tar

21. By Tom Haddon

Print what revno we are publishing

22. By Tom Haddon

Also print what we're publishing in production

Revision history for this message
Tom Haddon (mthaddon) wrote :

Ok, this has been tested on https://ci.admin.canonical.com/job/autodeploy-is-mojo-dot-canonical-dot-com/ - not yet tested the actually rollout portions on production though

lp:~mthaddon/mojo/mojo-how-to-autodeploy updated
23. By Tom Haddon

Missing config from check-autodeploy

24. By Tom Haddon

Make sure we have the correct builddeps for performing the autodeploy

25. By Tom Haddon

We need to pass the packages variable, not config to builddeps

Revision history for this message
Tom Haddon (mthaddon) wrote :

And this has now been tested on production as well. Successfully performed the rollout, and then exited with "Production is on 206 and the desired revno is 206. Nothing to do, exiting..." when rerun.

lp:~mthaddon/mojo/mojo-how-to-autodeploy updated
26. By Tom Haddon

Add a comment when we've autodeployed so we can more easily look for this and report on it

Revision history for this message
Paul Collins (pjdc) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'mojo-how-to/README'
--- mojo-how-to/README 2015-05-07 14:11:05 +0000
+++ mojo-how-to/README 2015-10-12 10:20:25 +0000
@@ -20,3 +20,10 @@
20 ksplice:20 ksplice:
21 options:21 options:
22 access-key: $YOUR_KSPLICE_ACCESS_KEY22 access-key: $YOUR_KSPLICE_ACCESS_KEY
23
24
25If you are using the autodeploy manifests, you will also need the following in
26a secrets file called "autodeploy-secrets":
27
28CI_SWIFT_CONTAINER=URL_OF_SWIFT_CONTAINER
29PROD_SWIFT_CONTAINER=URL_OF_SWIFT_CONTAINER
2330
=== added file 'mojo-how-to/devel/build-desired-revno'
--- mojo-how-to/devel/build-desired-revno 1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/build-desired-revno 2015-10-12 10:20:25 +0000
@@ -0,0 +1,15 @@
1#!/bin/bash
2
3# Script to generate docs from Mojo source tree
4
5set -e
6set -u
7
8source ${MOJO_SPEC_DIR}/${MOJO_STAGE}/get_common_variables
9
10cd ${MOJO_BUILD_DIR}/mojo
11
12bzr pull -r $DESIRED_REVNO --overwrite .
13
14make generate-docs
15tar cvpf ${MOJO_LOCAL_DIR}/mojo.tar --directory=docs/www .
016
=== added file 'mojo-how-to/devel/build-production-revno'
--- mojo-how-to/devel/build-production-revno 1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/build-production-revno 2015-10-12 10:20:25 +0000
@@ -0,0 +1,25 @@
1#!/bin/bash
2
3# Script to generate docs from Mojo source tree
4
5set -e
6set -u
7
8source ${MOJO_SPEC_DIR}/${MOJO_STAGE}/get_common_variables
9
10cd ${MOJO_BUILD_DIR}/mojo
11
12bzr pull -r $PRODUCTION_REVNO --overwrite .
13
14make generate-docs
15tar cvpf ${MOJO_LOCAL_DIR}/mojo.tar --directory=docs/www .
16
17if [ ${MOJO_STAGE##*/} != "production" ]; then
18 # We don't deploy landscape in non-production environments, but we need a
19 # dummy secrets file
20 echo "Creating dummy services-secret file in ${MOJO_LOCAL_DIR}"
21 echo "mojo-how-to:
22 services:
23 nrpe:
24 charm: nrpe-external-master" > ${MOJO_LOCAL_DIR}/services-secret
25fi
026
=== added file 'mojo-how-to/devel/check_for_diff_production_desired_revno'
--- mojo-how-to/devel/check_for_diff_production_desired_revno 1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/check_for_diff_production_desired_revno 2015-10-12 10:20:25 +0000
@@ -0,0 +1,10 @@
1#!/bin/bash
2
3source ${MOJO_SPEC_DIR}/${MOJO_STAGE}/get_common_variables
4
5if [[ $PRODUCTION_REVNO -ge $DESIRED_REVNO ]]; then
6 echo "Production is on ${PRODUCTION_REVNO} and the desired revno is ${DESIRED_REVNO}. Nothing to do, exiting..."
7 exit 1
8fi
9
10echo "Production is on ${PRODUCTION_REVNO} and the desired revno is ${DESIRED_REVNO}"
011
=== added file 'mojo-how-to/devel/get_common_variables'
--- mojo-how-to/devel/get_common_variables 1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/get_common_variables 2015-10-12 10:20:25 +0000
@@ -0,0 +1,5 @@
1#!/bin/bash
2
3# Source variables used by other scripts
4PRODUCTION_REVNO=$(cat ${MOJO_LOCAL_DIR}/production_revno)
5DESIRED_REVNO=$(cat ${MOJO_LOCAL_DIR}/desired_revno)
06
=== added file 'mojo-how-to/devel/publish_revno'
--- mojo-how-to/devel/publish_revno 1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/publish_revno 2015-10-12 10:20:25 +0000
@@ -0,0 +1,10 @@
1#!/bin/bash
2
3source ${MOJO_SPEC_DIR}/${MOJO_STAGE}/get_common_variables
4
5echo $DESIRED_REVNO > ${MOJO_LOCAL_DIR}/tested_revno
6
7echo "Publishing $DESIRED_REVNO as tested_revno"
8
9cd ${MOJO_LOCAL_DIR}
10swift upload mojo-how-to-autodeploy tested_revno
011
=== added file 'mojo-how-to/devel/store_desired_revno'
--- mojo-how-to/devel/store_desired_revno 1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/store_desired_revno 2015-10-12 10:20:25 +0000
@@ -0,0 +1,9 @@
1#!/bin/bash
2
3# Store the desired revno we want to upgrade to. In our case this is going to
4# be the current tip of trunk whenever we run this script, which will have been
5# collected already, so we can just query the revno.
6
7DESIRED_REVNO=$(bzr revno ${MOJO_BUILD_DIR}/mojo)
8
9echo $DESIRED_REVNO > ${MOJO_LOCAL_DIR}/desired_revno
010
=== added file 'mojo-how-to/devel/store_production_revno'
--- mojo-how-to/devel/store_production_revno 1970-01-01 00:00:00 +0000
+++ mojo-how-to/devel/store_production_revno 2015-10-12 10:20:25 +0000
@@ -0,0 +1,9 @@
1#!/bin/bash
2
3# Store the current revno used on production (grab from production swift
4# container)
5
6. ${MOJO_LOCAL_DIR}/autodeploy-secrets
7PRODUCTION_REVNO=$(wget -q -O- ${PROD_SWIFT_CONTAINER}/production_revno)
8
9echo $PRODUCTION_REVNO > ${MOJO_LOCAL_DIR}/production_revno
010
=== added directory 'mojo-how-to/manifests'
=== added file 'mojo-how-to/manifests/check-autodeploy'
--- mojo-how-to/manifests/check-autodeploy 1970-01-01 00:00:00 +0000
+++ mojo-how-to/manifests/check-autodeploy 2015-10-12 10:20:25 +0000
@@ -0,0 +1,11 @@
1## This manifest should invoke prepare-autodeploy and then verify that the
2## proposed deployment can be done. This is usually done by invoking
3## utils/confirm-autodeployable, which will check that the current spec revno has
4## successfully been used in ci.admin and that the requested build label has been
5## tested. This manifest may also perhaps run a verify phase to ensure the
6## environment is healthy. This manifest is usually invoked by
7## deploy-from-blessed-branch.
8
9include config=manifests/prepare-autodeploy
10
11include config=manifest-verify
012
=== added file 'mojo-how-to/manifests/perform-autodeploy'
--- mojo-how-to/manifests/perform-autodeploy 1970-01-01 00:00:00 +0000
+++ mojo-how-to/manifests/perform-autodeploy 2015-10-12 10:20:25 +0000
@@ -0,0 +1,15 @@
1## This manifest should invoke check-autodeploy to ensure the deployment can be
2## done, and then perform the deployment, most preferably by invoking
3## perform-autodeploy-unchecked. The manifest may invoke database
4## migrations, if necessary and should also verify that the deployment was
5## successful. This manifest is usually invoked by deploy-from-blessed-branch.
6
7include config=manifests/check-autodeploy
8include config=manifests/perform-autodeploy-unchecked
9include config=manifest-verify
10
11script config=publish_revno
12
13## And now just print a comment that we've successfully auto-deployed so it's
14## easier to report on that
15# Successfully auto-deployed new code for mojo-how-to
016
=== added file 'mojo-how-to/manifests/perform-autodeploy-unchecked'
--- mojo-how-to/manifests/perform-autodeploy-unchecked 1970-01-01 00:00:00 +0000
+++ mojo-how-to/manifests/perform-autodeploy-unchecked 2015-10-12 10:20:25 +0000
@@ -0,0 +1,16 @@
1## This manifest should perform the "meat" of the autodeploy, *without*
2## checking whether or not do so. In general this manifest is invoked by
3## perform-autodeploy (below) and by test-autodeploy (further below).
4
5# Pull down the latest content
6collect
7# Set builddeps
8builddeps packages=make,markdown,bzr
9# Run the build step
10build config=build-desired-revno
11# Copy our built resources to the instances
12script config=upload-built-content BUILD_LABEL=revno
13# Update the config variable so our content is updated
14script config=update-content BUILD_LABEL=revno
15# Sleep to let the update be applied
16sleep config=60
017
=== added file 'mojo-how-to/manifests/prepare-autodeploy'
--- mojo-how-to/manifests/prepare-autodeploy 1970-01-01 00:00:00 +0000
+++ mojo-how-to/manifests/prepare-autodeploy 2015-10-12 10:20:25 +0000
@@ -0,0 +1,13 @@
1## This manifest should prepare the workspace for a deployment test or
2## autodeploy. Tasks commonly performed here are storing the build label to be
3## deployed and fetching and storing the currently-live build label from the
4## production environment or elsewhere (but ideally the former). This manifest
5## exists in particular so that the build label to be deployed is specified in one
6## and only one location.
7
8script config=store_production_revno
9collect
10script config=store_desired_revno
11
12# Now check there's actually a rollout that we want to attempt
13script config=check_for_diff_production_desired_revno
014
=== added file 'mojo-how-to/manifests/test-autodeploy'
--- mojo-how-to/manifests/test-autodeploy 1970-01-01 00:00:00 +0000
+++ mojo-how-to/manifests/test-autodeploy 2015-10-12 10:20:25 +0000
@@ -0,0 +1,36 @@
1## After invoking check-autodeploy-testable, this manifest should deploy an
2## environment to match the existing production environment and then upgrade to
3## the DESIRED BUILD LABEL, most preferably by invoking
4## perform-autodeploy-unchecked. Following the upgrade this manifest
5## should perform any verification needed, and then publish suitable objects to
6## the swift container recording the result.
7
8# Get what revnos we're interested in for this deployment
9include config=manifests/prepare-autodeploy
10
11# Having confirmed there's a deployment to test, we first need to bring the
12# environment up in it's current state on production
13
14# We need the markdown package to be able to generate the docs for Mojo
15builddeps packages=make,markdown,bzr
16# Run the collect step
17collect
18# Run the build step
19build config=build-production-revno
20# Deploy services only
21deploy config=services local=services-secret delay=0
22# Copy our built resources to the instances
23script config=upload-built-content
24# And now deploy relations as well
25deploy config=relations
26# Run verify steps
27include config=manifest-verify
28
29# Now perform the autodeploy
30include config=manifests/perform-autodeploy-unchecked
31
32# And verify our environment again
33include config=manifest-verify
34
35# Now publish the revno we've successfully rolled out
36script config=publish_revno
037
=== added symlink 'mojo-how-to/production/build-desired-revno'
=== target is u'../devel/build-desired-revno'
=== added symlink 'mojo-how-to/production/build-production-revno'
=== target is u'../devel/build-production-revno'
=== added symlink 'mojo-how-to/production/check_for_diff_production_desired_revno'
=== target is u'../devel/check_for_diff_production_desired_revno'
=== added symlink 'mojo-how-to/production/get_common_variables'
=== target is u'../devel/get_common_variables'
=== added file 'mojo-how-to/production/publish_revno'
--- mojo-how-to/production/publish_revno 1970-01-01 00:00:00 +0000
+++ mojo-how-to/production/publish_revno 2015-10-12 10:20:25 +0000
@@ -0,0 +1,10 @@
1#!/bin/bash
2
3source ${MOJO_SPEC_DIR}/${MOJO_STAGE}/get_common_variables
4
5echo $DESIRED_REVNO > ${MOJO_LOCAL_DIR}/production_revno
6
7echo "Publishing $DESIRED_REVNO as production_revno"
8
9cd ${MOJO_LOCAL_DIR}
10swift upload mojo-how-to-autodeploy production_revno
011
=== added file 'mojo-how-to/production/store_desired_revno'
--- mojo-how-to/production/store_desired_revno 1970-01-01 00:00:00 +0000
+++ mojo-how-to/production/store_desired_revno 2015-10-12 10:20:25 +0000
@@ -0,0 +1,9 @@
1#!/bin/bash
2
3# Store the desired revno we want to upgrade to. In our case this is going to
4# be the revno that's been tested on CI.
5
6. ${MOJO_LOCAL_DIR}/autodeploy-secrets
7DESIRED_REVNO=$(wget -q -O- ${CI_SWIFT_CONTAINER}/tested_revno)
8
9echo $DESIRED_REVNO > ${MOJO_LOCAL_DIR}/desired_revno
010
=== added file 'mojo-how-to/production/store_production_revno'
--- mojo-how-to/production/store_production_revno 1970-01-01 00:00:00 +0000
+++ mojo-how-to/production/store_production_revno 2015-10-12 10:20:25 +0000
@@ -0,0 +1,10 @@
1#!/bin/bash
2
3# Store the current revno used on production (grab from production swift
4# container). Even though this is being run in the production stage, this is
5# still the most reliable way of getting this.
6
7. ${MOJO_LOCAL_DIR}/autodeploy-secrets
8PRODUCTION_REVNO=$(wget -q -O- ${PROD_SWIFT_CONTAINER}/production_revno)
9
10echo $PRODUCTION_REVNO > ${MOJO_LOCAL_DIR}/production_revno

Subscribers

People subscribed via source and target branches

to all changes: