Ubuntu CI Services

Merge lp:~ev/ubuntu-ci-services-itself/phase0-ppa into lp:ubuntu-ci-services-itself

  1. phase0-ppa
  2. Merge into trunk
Proposed by Evan
Status: Merged
Approved by: Chris Johnston
Approved revision: 277
Merged at revision: 290
Proposed branch: lp:~ev/ubuntu-ci-services-itself/phase0-ppa
Merge into: lp:ubuntu-ci-services-itself
Diff against target: 345 lines (+249/-7)
9 files modified
charms/precise/rabbitmq-worker/Makefile (+24/-0)
charms/precise/rabbitmq-worker/charm-helpers.yaml (+4/-0)
charms/precise/rabbitmq-worker/cm.py (+172/-0)
charms/precise/rabbitmq-worker/config-manager.txt (+5/-0)
charms/precise/rabbitmq-worker/config.yaml (+25/-0)
charms/precise/rabbitmq-worker/hooks/hooks.py (+4/-1)
juju-deployer/branch-source-builder.yaml.tmpl (+5/-2)
juju-deployer/image-builder.yaml.tmpl (+5/-2)
juju-deployer/test-runner.yaml.tmpl (+5/-2)
To merge this branch: bzr merge lp:~ev/ubuntu-ci-services-itself/phase0-ppa
Reviewer Review Type Date Requested Status
Andy Doan (community) Approve
PS Jenkins bot (community) continuous-integration Approve
Review via email: mp+208811@code.launchpad.net

Commit message

- Bring charmhelpers into the rabbitmq-worker charm.
- Use PPA versions of the dependencies for the bsb, image builder, and test runner components.

Description of the change

This kills the remaining installs of pip packages to the production deployment. This ensures that if pypi goes down or the egg has a flaky install routine, we're still able to get through a deployment.

By using a "release" PPA, we ensure a predictable and reproducible environment by sticking to the same dependency versions unless we explicitly upload a new one.

- Bring charmhelpers into the rabbitmq-worker charm.
- Use PPA versions of the dependencies for the bsb, image builder, and test runner components.

To post a comment you must log in.
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

FAILED: Continuous integration, rev:277
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~ev/ubuntu-ci-services-itself/phase0-ppa/+merge/208811/+edit-commit-message

http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/250/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/250/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

PASSED: Continuous integration, rev:277
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/251/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/251/rebuild

review: Approve (continuous-integration)
Revision history for this message
Andy Doan (doanac) wrote :

makes me so happy.

NOTE: we have one pip dependency still lurking here:

 http://bazaar.launchpad.net/~ev/ubuntu-ci-services-itself/phase0-ppa/view/head:/charms/precise/restish/hooks/hooks.py#L93

but we can probably treat that in a separate MP

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== added file 'charms/precise/rabbitmq-worker/Makefile'
--- charms/precise/rabbitmq-worker/Makefile 1970-01-01 00:00:00 +0000
+++ charms/precise/rabbitmq-worker/Makefile 2014-02-28 14:19:37 +0000
@@ -0,0 +1,24 @@
1PWD := $(shell pwd)
2SOURCEDEPS_DIR ?= $(shell dirname $(PWD))/.sourcecode
3PYTHON := /usr/bin/env python
4CONFIGMANAGER := $(PWD)/bin/cm.py
5
6
7sourcedeps: $(PWD)/config-manager.txt clean
8 @echo Updating source dependencies...
9 @$(PYTHON) cm.py -c $(PWD)/config-manager.txt \
10 -p $(SOURCEDEPS_DIR) \
11 -t $(PWD)
12 @$(PYTHON) build/charm-helpers/tools/charm_helpers_sync/charm_helpers_sync.py \
13 -c charm-helpers.yaml \
14 -b build/charm-helpers \
15 -d hooks/charmhelpers
16 @echo Do not forget to commit the updated files if any.
17 #@cd $(PWD)/build/charm-helpers; \
18 #@$(PYTHON) setup.py install --install-purelib=$(PWD)/lib \
19 #--install-scripts=$(PWD)/lib/bin
20
21clean:
22 rm -fr hooks/charmhelpers build/charm-helpers ../.sourcecode
23
24.PHONY: sourcedeps
025
=== added directory 'charms/precise/rabbitmq-worker/build'
=== added file 'charms/precise/rabbitmq-worker/charm-helpers.yaml'
--- charms/precise/rabbitmq-worker/charm-helpers.yaml 1970-01-01 00:00:00 +0000
+++ charms/precise/rabbitmq-worker/charm-helpers.yaml 2014-02-28 14:19:37 +0000
@@ -0,0 +1,4 @@
1include:
2 - core
3 - fetch
4 - contrib.charmsupport
05
=== added file 'charms/precise/rabbitmq-worker/cm.py'
--- charms/precise/rabbitmq-worker/cm.py 1970-01-01 00:00:00 +0000
+++ charms/precise/rabbitmq-worker/cm.py 2014-02-28 14:19:37 +0000
@@ -0,0 +1,172 @@
1# Copyright 2014 Canonical Ltd. All rights reserved.
2
3import os
4import re
5import sys
6import errno
7import hashlib
8import subprocess
9import optparse
10import urllib
11
12from os import curdir
13from bzrlib.branch import Branch
14from bzrlib.plugin import load_plugins
15load_plugins()
16from bzrlib.plugins.launchpad import account as lp_account
17
18if 'GlobalConfig' in dir(lp_account):
19 from bzrlib.config import LocationConfig as LocationConfiguration
20 _ = LocationConfiguration
21else:
22 from bzrlib.config import LocationStack as LocationConfiguration
23 _ = LocationConfiguration
24
25
26def get_ubunet_branch_config(config_file):
27 """
28 Retrieves the sourcedeps configuration for an ubunet source dir.
29 Returns a dict of (branch, revspec) tuples, keyed by branch name.
30 """
31 branches = {}
32 with open(config_file, 'r') as stream:
33 for line in stream:
34 line = line.split('#')[0].strip()
35 match = re.match(r'(\S+)\s+'
36 'bzr\+ssh://([^/]+)/([^;]+)'
37 '(?:;revno=(\d+))?', line)
38 if match:
39 name, host, branch, revno = match.group(1, 2, 3, 4)
40 if revno is None:
41 revspec = "-1"
42 else:
43 revspec = revno
44 branches[name] = (host, branch, revspec)
45 return branches
46
47
48def main(config_file, parent_dir, target_dir, verbose):
49 """Do the deed."""
50
51 try:
52 os.makedirs(parent_dir)
53 except OSError, e:
54 if e.errno != errno.EEXIST:
55 raise
56
57 bzr_config = LocationConfiguration(parent_dir)
58 get_lp_login = lp_account.get_lp_login
59 username = get_lp_login(bzr_config) or get_lp_login()
60 if username is None:
61 raise RuntimeError("Unable to determine launchpad login")
62 quoted_username = urllib.quote(username)
63
64 branches = sorted(get_ubunet_branch_config(config_file).iteritems())
65 for branch_name, (host, quoted_branch_spec, revspec) in branches:
66 revno = int(revspec)
67
68 # qualify mirror branch name with hash of remote repo path to deal
69 # with changes to the remote branch URL over time
70 branch_spec_digest = hashlib.sha1(quoted_branch_spec).hexdigest()
71 branch_directory = branch_spec_digest
72
73 source_path = os.path.join(parent_dir, branch_directory)
74 destination_path = os.path.join(target_dir, branch_name)
75
76 # Remove leftover symlinks/stray files.
77 try:
78 os.remove(destination_path)
79 except OSError, e:
80 if e.errno != errno.EISDIR and e.errno != errno.ENOENT:
81 raise
82
83 branch_url = ("bzr+ssh://%s@%s/%s" %
84 (quoted_username, host, quoted_branch_spec))
85 lp_url = "lp:" + quoted_branch_spec.replace("+branch/", "")
86
87 # Create the local mirror branch if it doesn't already exist
88 if verbose:
89 sys.stderr.write('%30s: ' % (branch_name,))
90 sys.stderr.flush()
91
92 fresh = False
93 if not os.path.exists(source_path):
94 subprocess.check_call(['bzr', 'branch', '-q',
95 '--', branch_url, source_path])
96 fresh = True
97
98 source_branch = Branch.open(source_path)
99
100 # Freshen the source branch if required (-1 means we want tip).
101 if not fresh and (revno == -1 or revno > source_branch.revno()):
102 subprocess.check_call(['bzr', 'pull', '-q', '--overwrite', '-r',
103 str(revno), '-d', source_path,
104 '--', branch_url])
105
106 if os.path.exists(destination_path):
107 # Overwrite the destination with the appropriate revision.
108 subprocess.check_call(['bzr', 'clean-tree', '--force', '-q',
109 '--ignored', '-d', destination_path])
110 subprocess.check_call(['bzr', 'pull', '-q', '--overwrite',
111 '-r', str(revno),
112 '-d', destination_path, '--', source_path])
113 else:
114 # Create a new branch.
115 subprocess.check_call(['bzr', 'branch', '-q', '--hardlink',
116 '-r', str(revno),
117 '--', source_path, destination_path])
118
119 # Check the state of the destination branch.
120 destination_branch = Branch.open(destination_path)
121 destination_revno = destination_branch.revno()
122
123 if verbose:
124 sys.stderr.write('checked out %4s of %s\n' %
125 ("r" + str(destination_revno), lp_url))
126 sys.stderr.flush()
127
128 if revno != -1 and destination_revno != revno:
129 raise RuntimeError("Expected revno %d but got revno %d" %
130 (revno, destination_revno))
131
132if __name__ == '__main__':
133 parser = optparse.OptionParser(
134 usage="%prog [options]",
135 description=(
136 "Add a lightweight checkout in <target> for each "
137 "corresponding file in <parent>."),
138 add_help_option=False)
139 parser.add_option(
140 '-p', '--parent', dest='parent', default=None,
141 help=("The directory of the parent tree."),
142 metavar="DIR")
143 parser.add_option(
144 '-t', '--target', dest='target', default=curdir,
145 help=("The directory of the target tree."),
146 metavar="DIR")
147 parser.add_option(
148 '-c', '--config', dest='config', default=None,
149 help=("The config file to be used for config-manager."),
150 metavar="DIR")
151 parser.add_option(
152 '-q', '--quiet', dest='verbose', action='store_false',
153 help="Be less verbose.")
154 parser.add_option(
155 '-h', '--help', action='help',
156 help="Show this help message and exit.")
157 parser.set_defaults(verbose=True)
158
159 options, args = parser.parse_args()
160
161 if options.parent is None:
162 parser.error(
163 "Parent directory not specified.")
164
165 if options.target is None:
166 parser.error(
167 "Target directory not specified.")
168
169 sys.exit(main(config_file=options.config,
170 parent_dir=options.parent,
171 target_dir=options.target,
172 verbose=options.verbose))
0173
=== added file 'charms/precise/rabbitmq-worker/config-manager.txt'
--- charms/precise/rabbitmq-worker/config-manager.txt 1970-01-01 00:00:00 +0000
+++ charms/precise/rabbitmq-worker/config-manager.txt 2014-02-28 14:19:37 +0000
@@ -0,0 +1,5 @@
1# After making changes to this file, to ensure that the charm payload
2# is up-to-date do:
3#
4# make
5build/charm-helpers bzr+ssh://bazaar.launchpad.net/~charm-helpers/charm-helpers/devel;revno=99
06
=== modified file 'charms/precise/rabbitmq-worker/config.yaml'
--- charms/precise/rabbitmq-worker/config.yaml 2014-02-17 10:13:29 +0000
+++ charms/precise/rabbitmq-worker/config.yaml 2014-02-28 14:19:37 +0000
@@ -51,3 +51,28 @@
51 type: string51 type: string
52 default: '/'52 default: '/'
53 description: The vhost in the rabbitMQ server.53 description: The vhost in the rabbitMQ server.
54
55 # Apt
56 install_sources:
57 type: string
58 default: ""
59 description: |
60 YAML list of additional installation sources, as a string. The number
61 of install_sources must match the number of install_keys. For
62 example:
63 install_sources: |
64 - ppa:project1/ppa
65 - ppa:project2/ppa
66 install_keys:
67 type: string
68 default: ""
69 description: |
70 YAML list of GPG keys for installation sources, as a string. For apt
71 repository URLs, use the public key ID used to verify package
72 signatures. For other sources such as PPA, use empty string. This
73 list must have the same number of elements as install_sources, even
74 if the key items are all empty string. An example to go with the
75 above for install_sources:
76 install_keys: |
77 - ""
78 - ""
5479
=== modified file 'charms/precise/rabbitmq-worker/hooks/hooks.py'
--- charms/precise/rabbitmq-worker/hooks/hooks.py 2014-02-17 10:13:29 +0000
+++ charms/precise/rabbitmq-worker/hooks/hooks.py 2014-02-28 14:19:37 +0000
@@ -7,6 +7,7 @@
7import subprocess7import subprocess
8import sys8import sys
9import textwrap9import textwrap
10import charmhelpers.fetch
1011
1112
12def juju_info(msg):13def juju_info(msg):
@@ -80,13 +81,15 @@
8081
8182
82def install(config):83def install(config):
84 charmhelpers.fetch.configure_sources(update=True)
85
83 pkgs = [x for x in config.get('packages', '').split(' ') if x]86 pkgs = [x for x in config.get('packages', '').split(' ') if x]
84 pkgs.append('python-amqplib')87 pkgs.append('python-amqplib')
85 pkgs.append('python-pip')88 pkgs.append('python-pip')
86 pkgs.append('bzr')89 pkgs.append('bzr')
8790
88 juju_info('installing apt packages...')91 juju_info('installing apt packages...')
89 subprocess.check_call(['apt-get', 'install', '-y', '-q'] + pkgs)92 charmhelpers.fetch.apt_install(pkgs)
9093
91 pip_pkgs = [x for x in config.get('pip-packages', '').split(' ') if x]94 pip_pkgs = [x for x in config.get('pip-packages', '').split(' ') if x]
92 if pip_pkgs:95 if pip_pkgs:
9396
=== modified file 'juju-deployer/branch-source-builder.yaml.tmpl'
--- juju-deployer/branch-source-builder.yaml.tmpl 2014-02-18 23:43:39 +0000
+++ juju-deployer/branch-source-builder.yaml.tmpl 2014-02-28 14:19:37 +0000
@@ -23,8 +23,11 @@
23 branch: ${CI_BRANCH}23 branch: ${CI_BRANCH}
24 tarball: ${CI_PAYLOAD_URL}24 tarball: ${CI_PAYLOAD_URL}
25 unit-config: include-base64://configs/unit_config.yaml25 unit-config: include-base64://configs/unit_config.yaml
26 packages: dput26 packages: "dput python-dput"
27 pip-packages: dput27 install_sources: |
28 - ${CI_PPA}
29 install_keys: |
30 - ""
28 rabbit:31 rabbit:
29 branch: lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/rabbitmq-server@4632 branch: lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/rabbitmq-server@46
30 charm: rabbitmq33 charm: rabbitmq
3134
=== modified file 'juju-deployer/image-builder.yaml.tmpl'
--- juju-deployer/image-builder.yaml.tmpl 2014-02-26 05:13:02 +0000
+++ juju-deployer/image-builder.yaml.tmpl 2014-02-28 14:19:37 +0000
@@ -23,10 +23,13 @@
23 vcs: ${CI_CODE_SOURCE}23 vcs: ${CI_CODE_SOURCE}
24 branch: ${CI_BRANCH}24 branch: ${CI_BRANCH}
25 tarball: ${CI_PAYLOAD_URL}25 tarball: ${CI_PAYLOAD_URL}
26 packages: "qemu-utils"26 packages: "qemu-utils python-glanceclient python-swiftclient"
27 pip-packages: "python-glanceclient python-swiftclient==1.8.0"
28 uid: root27 uid: root
29 unit-config: include-base64://configs/unit_config.yaml28 unit-config: include-base64://configs/unit_config.yaml
29 install_sources: |
30 - ${CI_PPA}
31 install_keys: |
32 - ""
30 rabbit:33 rabbit:
31 branch: lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/rabbitmq-server@4634 branch: lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/rabbitmq-server@46
32 charm: rabbitmq35 charm: rabbitmq
3336
=== modified file 'juju-deployer/test-runner.yaml.tmpl'
--- juju-deployer/test-runner.yaml.tmpl 2014-02-19 23:49:36 +0000
+++ juju-deployer/test-runner.yaml.tmpl 2014-02-28 14:19:37 +0000
@@ -30,10 +30,13 @@
30 vcs: ${CI_CODE_SOURCE}30 vcs: ${CI_CODE_SOURCE}
31 branch: ${CI_BRANCH}31 branch: ${CI_BRANCH}
32 tarball: ${CI_PAYLOAD_URL}32 tarball: ${CI_PAYLOAD_URL}
33 packages: "python-requests"33 packages: "python-requests python-novaclient python-swiftclient"
34 pip-packages: "python-novaclient python-swiftclient==1.8.0"
35 unit-config: include-base64://configs/unit_config.yaml34 unit-config: include-base64://configs/unit_config.yaml
36 uid: root35 uid: root
36 install_sources: |
37 - ${CI_PPA}
38 install_keys: |
39 - ""
37 rabbit:40 rabbit:
38 branch: lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/rabbitmq-server@4641 branch: lp:~canonical-ci-engineering/charms/precise/ubuntu-ci-services-itself/rabbitmq-server@46
39 charm: rabbitmq42 charm: rabbitmq

Subscribers

People subscribed via source and target branches

to all changes: