Merge lp:~ce-infrastructure/capomastro/deploy-cleanup-for-mojo into lp:~ubuntuone-hackers/capomastro/deploy

Proposed by Caio Begotti on 2015-01-12
Status: Merged
Approved by: Daniel Manrique on 2015-01-12
Approved revision: 48
Merged at revision: 42
Proposed branch: lp:~ce-infrastructure/capomastro/deploy-cleanup-for-mojo
Merge into: lp:~ubuntuone-hackers/capomastro/deploy
Diff against target: 441 lines (+109/-235)
13 files modified
README.md (+96/-72)
bootstrap.sh (+0/-11)
config/apache-openid.yaml (+0/-4)
config/apache.yaml (+0/-3)
config/capomastro.yaml (+0/-3)
config/jenkins.yaml (+0/-5)
config/jenkins/hooks/install.d/bygmester (+0/-20)
deploy.sh (+0/-46)
helpers/establish_tunnel.sh (+0/-9)
init.sh (+0/-30)
mojo.sh (+13/-12)
post-deploy-config.sh (+0/-12)
upgrade.sh (+0/-8)
To merge this branch: bzr merge lp:~ce-infrastructure/capomastro/deploy-cleanup-for-mojo
Reviewer Review Type Date Requested Status
Daniel Manrique 2015-01-12 Approve on 2015-01-12
Review via email: mp+246119@code.launchpad.net

Description of the change

These changes should reflect the current state of our work with Mojo. Nothing from the previous script was needed after all, and I hope the new README changes are easy to follow and understand!

So, the idea is that people would just get the branch and follow along if they want to use the deployment script, otherwise they probably know what they are doing and they are probably IS as well.

To post a comment you must log in.
48. By Caio Begotti on 2015-01-12

typo

Daniel Manrique (roadmr) wrote :

Looks good, simplification is nice, I'm not 100% happy with removing the .yaml files because they were quite informative, but I guess that information is now in the mojo spec, so this should be OK, plus this is clearly meant for internal use so it's safer.

The mojo.sh script. We said that we had to keep mojo wrappers simple (otherwise it defeats the point and creates an onion of wrap layers). This is simple and nice, and set -exv will guarantee that if any step fails, it will exit. I'm particularly thinking of a user forgetting to create the secrets and keys files in /tmp, but that's OK because before doing mojo run, if the copy fails, the script will exit.

However, this brings me to my only question. If this happens, and the user goes back to create the required files, then reruns the script, I think it will again fail when trying to create the mojo workspace or project. I'm not sure if this is something we want to handle. As you rightly said, people using this script are expected to have a certain level of skill, so the script doesn't need to do so much handholding, but still it's something to consider.

Let me know your thoughts on this. I'll set as "needs information" as it's mainly just curiosity on my part, I don't think any actual fixes are needed.

review: Needs Information
Caio Begotti (caio1982) wrote :

Good point! The spec already checks for the existence of the secrets and
keys, so it will fail gracefully yeah. If the user then fixes the secrets
and the keys and re-run it, because we are using timestamps to make runs
unique, it will be a totally new environment, so it should not fail at
all... hopefully :-)

On Mon, Jan 12, 2015 at 3:32 PM, Daniel Manrique <
<email address hidden>> wrote:

> Review: Needs Information
>
> Looks good, simplification is nice, I'm not 100% happy with removing the
> .yaml files because they were quite informative, but I guess that
> information is now in the mojo spec, so this should be OK, plus this is
> clearly meant for internal use so it's safer.
>
> The mojo.sh script. We said that we had to keep mojo wrappers simple
> (otherwise it defeats the point and creates an onion of wrap layers). This
> is simple and nice, and set -exv will guarantee that if any step fails, it
> will exit. I'm particularly thinking of a user forgetting to create the
> secrets and keys files in /tmp, but that's OK because before doing mojo
> run, if the copy fails, the script will exit.
>
> However, this brings me to my only question. If this happens, and the user
> goes back to create the required files, then reruns the script, I think it
> will again fail when trying to create the mojo workspace or project. I'm
> not sure if this is something we want to handle. As you rightly said,
> people using this script are expected to have a certain level of skill, so
> the script doesn't need to do so much handholding, but still it's something
> to consider.
>
> Let me know your thoughts on this. I'll set as "needs information" as it's
> mainly just curiosity on my part, I don't think any actual fixes are needed.
> --
>
> https://code.launchpad.net/~ce-infrastructure/capomastro/deploy-cleanup-for-mojo/+merge/246119
> Your team CE Infrastructure is subscribed to branch
> lp:~ce-infrastructure/capomastro/deploy.
>

Daniel Manrique (roadmr) wrote :

Doh, I missed the MOJO_TIMESTAMP definition. Nothing to see here... approving. Thanks!

review: Approve
Daniel Manrique (roadmr) wrote :

oh btw, the deploy branch of capomastro is not handled by tarmac, so we need to merge this manually :/

Caio Begotti (caio1982) wrote :

Oh, no worries! I'll do that then :-)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'README.md'
2--- README.md 2014-11-13 03:44:08 +0000
3+++ README.md 2015-01-12 13:29:02 +0000
4@@ -5,80 +5,104 @@
5 ------------
6
7 * Launchpad:
8- * Access to ~canonical-losa owned charms
9- * Access to lp:bygmester
10+ * Access to the spec code at lp:~canonical-is/canonical-mojo-specs/mojo-pes-capomastro/
11+ * Access to Jenkins bot keys at lp:~ce-infrastructure/bygmester/ce-jenkins-bot
12+ * Access to the PPA at lp:~ce-infrastructure/+archive/ubuntu/capomastro
13 * OS:
14 * Trusty (14.04)
15- * python-virtualenv
16- * juju-core>=1.17.4-0ubuntu1
17- * juju-local>=1.17.4-0ubuntu1
18- * lxc>=1.0.1-0ubuntu1
19- * python-novaclient>=1:2.17.0-0ubuntu1
20- * python-swiftclient>=1:2.0.3-0ubuntu1
21- * python-cheetah
22
23-Steps
24+Setup
25 -----
26
27-### Deploy Application
28-
29- 1. Initialize deployment space
30-
31- ./init.sh
32-
33- 2. Bootstrap juju environment (assumes Canonistack)
34-
35- # The "local" environment is default. To switch environments do:
36- # export JUJU_ENV="canonistack"
37- ./bootstrap.sh
38-
39- NOTE: IF you are connecting to a private cloud provider like Canonistack
40- you can use ./helpers/establish_tunnel.sh to create the tunnel for you.
41-
42- 3. Deploy application
43-
44- ./deploy.sh
45-
46- 4. Post-deploy configuration (execute once all services have started)
47-
48- ./post-deploy-config.sh
49-
50- NOTE: This will attempt to bootstrap with the ~ce-jenkins-bot user keys.
51- Type in a bad password to proceed, if asked for one. Log into the system and do the
52- following:
53-
54- * Change to the 'jenkins' user
55-
56- sudo su - jenkins
57-
58- * Add a public/private keypair associated with your Launchpad account to
59- ~/.ssh/
60-
61- * Tell bzr which Launchpad account to use
62-
63- bzr launchpad-login ~me
64-
65- 5. Upgrade application
66-
67- ./upgrade.sh
68-
69-
70-### Add floating IPs (optional)
71-
72-If using a non-local provider you will need to associate a floating IP with
73-the "apache2/0" service unit for web access. Optionally associate a floating
74-IP with the "jenkins/0" service unit if you require web access to it as well.
75-
76-NOTE: Eventually apache will sit in front of haproxy and this documentation
77-will be updated when that happens.
78-
79-TODO
80-----
81-
82- * Go through and address FIXME's / things that were papered over
83- * Make sure we can deploy something similar to what IS will deploy
84- * Get upgrades working the IS-way
85- * Figure out / fix the psql ssl issue when deploying locally
86- * Add support for jenkins-slaves
87- * Potentially move the jenkins slave bootstrapping tarball into
88- lp:bygmester
89+Install the Mojo package from its PPA (and juju-local if doing local test):
90+
91+ sudo apt-add-repository --yes ppa:mojo-maintainers/ppa && sudo apt-get update
92+ sudo apt-get --yes install mojo juju-local
93+
94+Initialize the local environment for Juju so you can deploy Capomastro:
95+
96+ juju init
97+ juju switch local
98+ juju bootstrap
99+
100+It is also highly recommended to use the company VPN to run any deployment, see
101+https://wiki.canonical.com/InformationInfrastructure/IS/HowTo/CompanyOpenVPN
102+for instructions on how to set it up. You may have problems accessing IS-owned
103+resources if you do not use the VPN when running the Mojo script.
104+
105+
106+Mojo configuration
107+------------------
108+
109+There are two important vars in the deployment script: MOJO_REPO and MOJO_STAGE.
110+
111+MOJO_REPO should have the Launchpad address of the repository containing the spec
112+of Capomastro. It can be a local repository, i.e. a local directory in the file
113+system, as long as all changes have been committed (not necessarily pushed though).
114+
115+MOJO_STAGE point to the working spec directory inside the Mojo repository. You
116+may need to tweak that if you are working on a new spec which does not follow the
117+IS organization, but most likely that never needs to be changed.
118+
119+Also, before running the deployment script, you must have both the secret of Capomastro's
120+PPA and the SSH key pair used by the Jenkins bot to fetch and build images.
121+
122+The script expects the Capomastro PPA to be configured through a charm option
123+and it is copied over the rest of the Mojo files automatically. Just put the
124+option formatted as YAML in /tmp/secrets, like this:
125+
126+capomastro:
127+ services:
128+ capomastro:
129+ options:
130+ repository: "<private PPA credentials here>"
131+
132+As for the Jenkins bot configuration, simply create a directory /tmp/keys
133+and add both SSH key and its .pub inside it. The script will use them for some
134+post desployment steps, otherwise Jenkins won't be able to fetch and build
135+images correctly right after the setup of the service.
136+
137+All this ensures no secrets or credentials are stored anywhere in the spec.
138+
139+Deploying
140+---------
141+
142+You may need to provide a password for sudo at the start of it:
143+
144+ ./mojo.sh
145+
146+Considerations
147+--------------
148+
149+1. you should avoid using this script for any serious production deployment. This
150+was meant for testing and development only, though it could be used just fine
151+on staging servers if needed. Local deployments with Juju and Mojo are supported
152+but you may encounter unknown problems, so beware. Ideally you should always
153+deploy to Canonistack, but that is quite slow, that's why this script exists.
154+
155+2. if using a non-local provider you will need to associate a floating IP with
156+the "apache2/0" service unit for web access. Optionally associate a floating
157+IP with the "jenkins/0" service unit if you require web access to it as well,
158+but keep in mind that would be insecure as all artifacts by default are visible
159+in Jenkins.
160+
161+3. when using this locally, you won't be able to actually build images with
162+Capomastro right away as the default policy for LXC units is not to allow
163+nesting. You'll need to stop the Jenkins unit, update its policy to allow that,
164+restart the unit and then try again. These are the config options you'll need:
165+
166+ lxc.aa_profile = lxc-container-default-with-nesting
167+ lxc.mount.auto = cgroup:mixed
168+
169+Old references
170+-------------
171+
172+Previously, in the shellscripts used to deploy the Capomastro charm manually,
173+these were the equivalent to the new Mojo spec commands we are using:
174+
175+ init.sh -> collect script
176+ deploy.sh -> the manifest file at the root of the spec
177+ bootstrap.sh -> manually done only when needed now
178+ post-deploy-config.sh -> also part of the manifest file
179+ upgrade-charm.sh -> collect-upgrade script
180+ config/ directory -> services file
181
182=== removed file 'bootstrap.sh'
183--- bootstrap.sh 2014-11-12 01:01:38 +0000
184+++ bootstrap.sh 1970-01-01 00:00:00 +0000
185@@ -1,11 +0,0 @@
186-#!/bin/bash -xev
187-
188-echo "Bootstrapping the Juju environment we will deploy into..."
189-
190-JUJU_ENV=${JUJU_ENV:-"local"}
191-
192-if [[ "${JUJU_ENV}" == "local" ]]; then
193- juju bootstrap --series="trusty" --upload-tools --constraints="mem=1G"
194-else
195- juju bootstrap --constraints="mem=1G"
196-fi
197
198=== removed directory 'config'
199=== removed file 'config/apache-openid.yaml'
200--- config/apache-openid.yaml 2014-11-19 11:20:30 +0000
201+++ config/apache-openid.yaml 1970-01-01 00:00:00 +0000
202@@ -1,4 +0,0 @@
203-apache-openid:
204- debug: True
205- allowed_providers: "Launchpad=login.launchpad.net"
206- authorized_teams: "ce-infrastructure capomastro-team"
207
208=== removed file 'config/apache.yaml'
209--- config/apache.yaml 2014-11-12 01:05:31 +0000
210+++ config/apache.yaml 1970-01-01 00:00:00 +0000
211@@ -1,3 +0,0 @@
212-apache2:
213- servername: "capomastro"
214- enable_modules: "proxy proxy_http proxy_connect rewrite headers ssl python"
215
216=== removed file 'config/capomastro.yaml'
217--- config/capomastro.yaml 2014-11-13 03:35:35 +0000
218+++ config/capomastro.yaml 1970-01-01 00:00:00 +0000
219@@ -1,3 +0,0 @@
220-capomastro:
221- debug: True
222- repository: "deb https://caio1982:T9dBp1n6wD7BLNs2qDmw@private-ppa.launchpad.net/ce-infrastructure/capomastro/ubuntu trusty main"
223
224=== removed directory 'config/jenkins'
225=== removed file 'config/jenkins.yaml'
226--- config/jenkins.yaml 2014-11-24 12:23:27 +0000
227+++ config/jenkins.yaml 1970-01-01 00:00:00 +0000
228@@ -1,5 +0,0 @@
229-jenkins:
230- username: "admin"
231- password: "admin"
232- plugins: "notification"
233- release: "lts"
234
235=== removed directory 'config/jenkins/hooks'
236=== removed directory 'config/jenkins/hooks/install.d'
237=== removed file 'config/jenkins/hooks/install.d/bygmester'
238--- config/jenkins/hooks/install.d/bygmester 2014-11-24 11:55:11 +0000
239+++ config/jenkins/hooks/install.d/bygmester 1970-01-01 00:00:00 +0000
240@@ -1,20 +0,0 @@
241-#!/bin/bash
242-
243-# FIXME: currently there's no way to pass extra config.yaml to the jenkins charm
244-# so we can't simply hack its config.yaml to pass the following sources.list string
245-PPA="deb https://caio1982:T9dBp1n6wD7BLNs2qDmw@private-ppa.launchpad.net/ce-infrastructure/capomastro/ubuntu trusty main"
246-
247-apt-add-repository --yes "${PPA}" && apt-get update && apt-get install --assume-yes --force-yes bygmester
248-
249-echo "Defaults:jenkins !requiretty" >> /etc/sudoers.d/bygmester
250-echo "Defaults:jenkins !authenticate" >> /etc/sudoers.d/bygmester
251-
252-# now changes are in effect
253-addgroup jenkins sudo
254-service sudo restart
255-
256-# just to work around the first bygmester checkout annoying confirmation
257-sudo -H -u jenkins sh -c 'mkdir -p ~/.ssh'
258-sudo -H -u jenkins sh -c 'echo "StrictHostKeyChecking no" >> ~/.ssh/config'
259-
260-exit 0
261
262=== removed file 'deploy.sh'
263--- deploy.sh 2014-11-25 12:57:56 +0000
264+++ deploy.sh 1970-01-01 00:00:00 +0000
265@@ -1,46 +0,0 @@
266-#!/bin/bash -xev
267-
268-echo "Deploying into Juju environment..."
269-export JUJU_REPOSITORY=${PWD}/charms
270-
271-# Deploy the charms from the store with our config files, if needed
272-juju deploy --config=config/apache.yaml cs:apache2
273-juju deploy cs:rabbitmq-server
274-juju deploy cs:postgresql
275-
276-# Deploy our own stuff
277-juju deploy --config=config/jenkins.yaml local:trusty/jenkins --constraints="mem=10G root-disk=100G cpu-cores=10"
278-juju deploy --config=config/apache-openid.yaml local:trusty/apache-openid
279-juju deploy --config=config/capomastro.yaml local:trusty/capomastro
280-
281-while juju status | grep pending 2>&1> /dev/null; do
282- echo "Waiting pending deployments to finish, sleeping..."
283- sleep 60
284-done
285-
286-# Mandatory relations between the service charms
287-
288-# FIXME: these sleep commands are possibly a bad idea but we will
289-# need to check this with IS first because we don't quite get how
290-# sequential interdependent relations can be set safely...
291-# e.g. capomastro needs pgsql and rabbit but each relation
292-# needs to restart the service after config-changed is called
293-# and if they both do that at the same time there is a race here
294-
295-juju add-relation postgresql:db capomastro && sleep 60
296-juju add-relation rabbitmq-server capomastro && sleep 60
297-juju add-relation jenkins capomastro && sleep 60
298-juju add-relation apache2 apache-openid && sleep 60
299-
300-# FIXME: not in the trusty series, really?
301-# juju deploy cs:precise/nrpe-external-master
302-
303-# FIXME: principal and subordinate services' series must match
304-# juju add-relation nrpe-external-master capomastro && sleep 60
305-
306-# FIXME: how could this be inside the config.yaml for Apache instead?
307-juju set apache2 vhost_http_template="$(base64 < ${JUJU_REPOSITORY}/trusty/capomastro/templates/http_virtualhost.tmpl)"
308-juju add-relation apache2:reverseproxy capomastro:website && sleep 60
309-
310-# Expose web services
311-juju expose apache2
312
313=== removed directory 'helpers'
314=== removed file 'helpers/establish_tunnel.sh'
315--- helpers/establish_tunnel.sh 2014-11-13 03:10:38 +0000
316+++ helpers/establish_tunnel.sh 1970-01-01 00:00:00 +0000
317@@ -1,9 +0,0 @@
318-#!/bin/bash
319-
320-# first you will need the package python-nova for this to run at all
321-# and second you need to had set up your juju environment with nova credentials
322-# set otherwise it will not list anything because it is missing your account info
323-export JUJU_ADMIN=$(nova list | grep machine-0 | awk '{print $12}' | cut -d= -f2)
324-
325-sudo pkill -f sshuttle
326-sshuttle -D -r ${JUJU_ADMIN} 10.55.0.0/16
327
328=== removed file 'init.sh'
329--- init.sh 2014-11-25 16:08:54 +0000
330+++ init.sh 1970-01-01 00:00:00 +0000
331@@ -1,30 +0,0 @@
332-#!/bin/bash -xev
333-
334-echo "Creating a local charms repository..."
335-
336-JUJU_REPOSITORY=${PWD}/charms
337-
338-rm -rf ${JUJU_REPOSITORY}
339-mkdir -p ${JUJU_REPOSITORY}/trusty
340-
341-echo "Bzr branching the charms code we will need..."
342-# Checks out custom charm (i.e. not in the store)
343-bzr branch lp:~ce-infrastructure/capomastro/charm ${JUJU_REPOSITORY}/trusty/capomastro
344-
345-# Right now we only overlay a hook over the Jenkins
346-# charm from the charms store now that extensions
347-# are supported in its official charm (also by IS)
348-bzr branch lp:~caio1982/charms/trusty/jenkins/trunk ${JUJU_REPOSITORY}/trusty/jenkins
349-cp -R config/jenkins/* ${JUJU_REPOSITORY}/trusty/jenkins/
350-
351-# FIXME: https://bugs.launchpad.net/capomastro/+bug/1394527
352-bzr branch lp:~caio1982/charms/trusty/apache-openid/trunk ${JUJU_REPOSITORY}/trusty/apache-openid
353-
354-echo "Adding a new SSH keypair as a deployable asset to the capomastro charm..."
355-mkdir -p ${JUJU_REPOSITORY}/trusty/capomastro/files/keys
356-ssh-keygen -f ${JUJU_REPOSITORY}/trusty/capomastro/files/keys/capomastro -N ''
357-
358-echo "Adding the latest Jenkins jobs as deployable assets to the capomastro charm..."
359-cd ${JUJU_REPOSITORY}/trusty/capomastro/files && ./download-supported-job-types && cd -
360-
361-echo "Initialization done, hopefully :-)"
362
363=== modified file 'mojo.sh'
364--- mojo.sh 2015-01-05 17:19:50 +0000
365+++ mojo.sh 2015-01-12 13:29:02 +0000
366@@ -1,15 +1,7 @@
367 #!/bin/bash
368 #
369 # caio.begotti@canonical.com
370-#
371-# in case you need to setup the host machine:
372-#
373-# sudo apt-add-repository --yes ppa:mojo-maintainers/ppa && sudo apt-get update
374-# sudo apt-get --yes install mojo juju-local
375-#
376-# juju init
377-# juju switch local
378-# juju bootstrap
379+# Mon 2015-01-05 15:32:20 -0200
380
381 echo "
382 WARNING: this script is meant for testing mojo specs only,
383@@ -30,7 +22,11 @@
384 export MOJO_WORKSPACE=capomastro-${MOJO_TIMESTAMP}
385 export MOJO_SERIES=trusty
386 export MOJO_STAGE=pes/mojo-pes-capomastro/devel
387-export MOJO_REPO=lp:~ce-infrastructure/canonical-mojo-specs/capomastro
388+
389+# this can be a local directory in the filesystem,
390+# as long it is a working bzr repository and that you
391+# have committed any changes (no need to push them though)
392+export MOJO_REPO=lp:~canonical-is/canonical-mojo-specs/mojo-pes-capomastro/
393
394 set -xev
395
396@@ -44,8 +40,13 @@
397 mojo workspace-new --series ${MOJO_SERIES} --project ${MOJO_PROJECT} --stage ${MOJO_STAGE} ${MOJO_REPO} ${MOJO_WORKSPACE}
398
399 # prepare the secrets for local deployment
400-sudo mkdir -p /srv/mojo/LOCAL/${MOJO_PROJECT}/pes/mojo-pes-capomastro/devel/
401-sudo cp secrets /srv/mojo/LOCAL/${MOJO_PROJECT}/pes/mojo-pes-capomastro/devel/
402+sudo mkdir -p /srv/mojo/LOCAL/${MOJO_PROJECT}/pes/mojo-pes-capomastro/devel
403+
404+# private ppa option for the capomastro charm
405+sudo cp -a /tmp/secrets /srv/mojo/LOCAL/${MOJO_PROJECT}/pes/mojo-pes-capomastro/devel/
406+
407+# directory with key pair for ce-jenkins-bot
408+sudo cp -a /tmp/keys /srv/mojo/LOCAL/${MOJO_PROJECT}/pes/mojo-pes-capomastro/devel/
409
410 # will execute all phases listed inside the manifest file
411 mojo run --series ${MOJO_SERIES} --project ${MOJO_PROJECT} --stage ${MOJO_STAGE} ${MOJO_REPO} ${MOJO_WORKSPACE}
412
413=== removed file 'post-deploy-config.sh'
414--- post-deploy-config.sh 2014-11-12 01:33:41 +0000
415+++ post-deploy-config.sh 1970-01-01 00:00:00 +0000
416@@ -1,12 +0,0 @@
417-#!/bin/bash -xev
418-
419-echo "Exporting the CE Jenkins bot setup files..."
420-bzr export ce-jenkins-bot-setup.tar.gz lp:~ce-infrastructure/bygmester/ce-jenkins-bot
421-
422-echo "Setting up the Jenkins bot key..."
423-juju scp ce-jenkins-bot-setup.tar.gz jenkins/0:.
424-juju ssh jenkins/0 "tar zxvf ce-jenkins-bot-setup.tar.gz && cd ce-jenkins-bot-setup && ./ce-jenkins-bot-setup.sh"
425-juju ssh jenkins/0 "sudo -u jenkins sh -c \"echo $(cat charms/trusty/capomastro/files/keys/capomastro.pub) > /var/lib/jenkins/.ssh/authorized_keys\""
426-
427-echo "Cleaning up..."
428-rm ce-jenkins-bot-setup.tar.gz
429
430=== removed file 'upgrade.sh'
431--- upgrade.sh 2014-11-13 03:18:03 +0000
432+++ upgrade.sh 1970-01-01 00:00:00 +0000
433@@ -1,8 +0,0 @@
434-#!/bin/bash -xev
435-#
436-# Please see https://bugs.launchpad.net/juju-core/+bug/1297559
437-
438-JUJU_REPOSITORY=${PWD}/charms
439-
440-echo "Upgrading the Capomastro service charm..."
441-juju upgrade-charm capomastro

Subscribers

People subscribed via source and target branches