Merge lp:~psivaa/uci-engine/install-cu2d-from-tarball into lp:uci-engine
- install-cu2d-from-tarball
- Merge into trunk
Status: | Work in progress |
---|---|
Proposed branch: | lp:~psivaa/uci-engine/install-cu2d-from-tarball |
Merge into: | lp:uci-engine |
Diff against target: |
176 lines (+99/-13) 4 files modified
charms/precise/chroot-builder/config.yaml (+17/-0) charms/precise/chroot-builder/hooks/hooks.py (+73/-12) cupstream2distro/chroot-tools/.pbuilderrc (+0/-1) juju-deployer/branch-source-builder.yaml.tmpl (+9/-0) |
To merge this branch: | bzr merge lp:~psivaa/uci-engine/install-cu2d-from-tarball |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
PS Jenkins bot (community) | continuous-integration | Approve | |
Evan (community) | Approve | ||
Review via email: mp+224080@code.launchpad.net |
Commit message
Install cupstream2distro from the uci-engine tarball and remove the eatmydata LD_PRELOAD errors seen during deployment.
Description of the change
To install cupstream2distro from the uci-engine tarball rather than from lp:lp:cupstream2dist
PS Jenkins bot (ps-jenkins) wrote : | # |
- 600. By Celso Providelo
-
[r=Andy Doan, PS Jenkins bot] Series of assorted small fixes related to the task-force for staging deployment and acceptance tests. from Celso Providelo
Francis Ginther (fginther) wrote : | # |
There is possibly another way to solve this problem, although I haven't figured out yet if it will work. The /cupstream2distro path is already being installed under /srv/${
The solution proposed here actually works :-), and I think is the better solution for now. I have no further comments, so once ev's are addressed, we can merge this.
- 601. By Joe Talbott
-
[r=Andy Doan, PS Jenkins bot] ticket-system - Add openid and oauth2 support.
* This is essentially a NOP at this point and shouldn't affect deployments as the
API does not yet require authentication.
from Joe Talbott - 602. By Andy Doan
-
[r=Evan Dandrea, PS Jenkins bot] rabbit-worker charm: update logs directory
This makes the upstart logging location for our rabbit workers consistent
to what i/s requires. The log directories will become:/srv/ci-
airline- imagebuild- worker/ logs/upstart. log
/srv/ci-airline- bsb-worker/ logs/upstart. log etc.
This had a side-effect of changing the upstart service names, which
I fixed the integration test cases for (as well as added one for
bsbuilder since one didn't exist)
from Andy Doan - 603. By Vincent Ladeuil
-
[r=PS Jenkins bot, Evan Dandrea] Ensures setup.sh fails when any setup.py component fails. 1333590 from Vincent Ladeuil
- 604. By Vincent Ladeuil
-
[r=Francis Ginther, PS Jenkins bot] Fallback to current dir (/srv/<service>) if HOME is not set for the test runner worker. 1331989 from Vincent Ladeuil
- 605. By Vincent Ladeuil
-
[r=PS Jenkins bot, Evan Dandrea] Add pep8/flake tests for juju-deployer and tests test files and fix found issues. from Vincent Ladeuil
- 606. By Andy Doan
-
[r=PS Jenkins bot, Evan Dandrea] lander charm: fix logging
i/s confirmed logging for this componet should go to:
/srv/ci-
airline- lander/ logs This required a few changes:
1) change the "service_dir" to use dashes and not underscores
This made a new upstart service name. This broke that json_status cgi
script *again*. So I tried to change this in a way that it can be
updated in the future without breakage.2) update the "lander logs" location
We were logging results from processing tickets under "logs". I chaged
this to "lander-logs" so it won't collide with our upstart logs.3) make the charm / upstart script smarter
This gets us logging in the correct place, but it also lets the upstart
job ensure the proper directories exist before the job is launched. This
makes things safer in the event someone deletes the logs directory and
restarts the service
from Andy Doan - 607. By Ursula Junque
-
[r=PS Jenkins bot, Celso Providelo] Fixes glitches on publisher from Ursula Junque
- 608. By Celso Providelo
-
[r=Andy Doan, PS Jenkins bot] Trivial fixes for ci-airline staging. from Celso Providelo
- 609. By Celso Providelo
-
[r=PS Jenkins bot, Andy Doan] Extend the current Webui to link to ticket artifacts using tempurls. from Celso Providelo
- 610. By Francis Ginther
-
[r=Andy Doan, PS Jenkins bot] Bump version of the upstart charm in production-
only.yaml to account for the revision bump in the charm itself. from Francis Ginther - 611. By Vincent Ladeuil
-
[r=Francis Ginther, PS Jenkins bot] Upload the run_test.py script explicitly without relying on cloud-init. 1333323 from Vincent Ladeuil
- 612. By Andy Doan
-
[r=Parameswaran Sivatharman, PS Jenkins bot] restish charm: fix to work with i/s requirement
As discussed with i/s this will get us going with:
/srv/ci-
airline- gatekeeper- restish/ logs/access. log
/srv/ci-airline- gatekeeper- restish/ logs/gunicorn. log from Andy Doan - 613. By Vincent Ladeuil
-
[r=Ursula Junque, PS Jenkins bot] Don't attempt to create ~/.ssh if it already exists. 1334146 from Vincent Ladeuil
- 614. By Para Siva
-
[r=Celso Providelo, PS Jenkins bot] This removes the creation and usage of ~/.ubuntu-ci. Consequently ubuntu-
ci-autoconfig was removed. Requires -D and -S for development and staging deployments from Parameswaran Sivatharman - 615. By Vincent Ladeuil
-
[r=Evan Dandrea, PS Jenkins bot, Andy Doan] Touch a stamp file to ensure the dependency is updated. from Vincent Ladeuil
- 616. By Andy Doan
-
[r=Paul Larson, PS Jenkins bot] django charm: remove need for hard-coded python paths
This lets us specify relative paths, so that our deployer files don't
require updates when we change the charm or service name from Andy Doan - 617. By Andy Doan
-
[r=PS Jenkins bot, Francis Ginther, Evan Dandrea] lander: don't crash when no juju-relations exist
We are seeing something in production where the lander is starting
before the juju-relations directory exist. I'm actually surprised
this code ever worked well. This does a check so that we'll wait
properly.I've also added a small engine-health-check so the exact error will be
visible in our webui.
from Andy Doan - 618. By Celso Providelo
-
[r=PS Jenkins bot, Francis Ginther, Evan Dandrea] Fixing TempUrl generation in canonistack/
prodstack. from Celso Providelo
Para Siva (psivaa) wrote : | # |
Thanks for the comments. Addressed where possible. charmhelpers dont seem to be much of help here. Just replied inline. Could you pls double check
- 619. By Ursula Junque
-
[r=Francis Ginther, Celso Providelo] Disables chroot-builder install during deployments, will be switched back on when fixing bug 1312271 from Ursula Junque
PS Jenkins bot (ps-jenkins) wrote : | # |
PASSED: Continuous integration, rev:602
http://
Executed test runs:
Click here to trigger a rebuild:
http://
Some small comments inline, but this is basically there. Resolve these and consider it an implicit +1 from me.
PS Jenkins bot (ps-jenkins) wrote : | # |
FAILED: Continuous integration, rev:603
http://
Executed test runs:
Click here to trigger a rebuild:
http://
PS Jenkins bot (ps-jenkins) wrote : | # |
PASSED: Continuous integration, rev:621
http://
Executed test runs:
Click here to trigger a rebuild:
http://
Para Siva (psivaa) wrote : | # |
Thanks for the comments again. Fixed them too. Holding it for https:/
Preview Diff
1 | === modified file 'charms/precise/chroot-builder/config.yaml' |
2 | --- charms/precise/chroot-builder/config.yaml 2014-05-27 06:04:10 +0000 |
3 | +++ charms/precise/chroot-builder/config.yaml 2014-06-26 14:33:53 +0000 |
4 | @@ -20,3 +20,20 @@ |
5 | If specified, this will be used as the base url for downloading |
6 | pre-built chroots. The format for the individual files is: |
7 | {chroot_base_url}/{series}-{architecture}.tgz |
8 | + install_root: |
9 | + type: string |
10 | + description: "The root directory the service will be installed in" |
11 | + default: "/srv/" |
12 | + tarball: |
13 | + type: string |
14 | + description: A URL to a tarball of the code |
15 | + vcs: |
16 | + default: "branch" |
17 | + type: string |
18 | + description: "Grab code from a bzr 'branch' or a URL to a 'tarball'" |
19 | + branch: |
20 | + type: string |
21 | + description: "BZR branch the service lives in" |
22 | + revno: |
23 | + type: string |
24 | + description: "Revision or tag to branch from" |
25 | |
26 | === modified file 'charms/precise/chroot-builder/hooks/hooks.py' |
27 | --- charms/precise/chroot-builder/hooks/hooks.py 2014-06-03 18:34:17 +0000 |
28 | +++ charms/precise/chroot-builder/hooks/hooks.py 2014-06-26 14:33:53 +0000 |
29 | @@ -29,6 +29,8 @@ |
30 | |
31 | CHROOT_USER_HOME = '/root' |
32 | CHROOT_ROOT_PATH = '/var/cache/pbuilder' |
33 | +EAT_MY_DATA_PRELOAD_LINE = "export LD_PRELOAD=\"${LD_PRELOAD:+$LD_PRELOAD:}\ |
34 | +/usr/lib/libeatmydata/libeatmydata.so\"\n" |
35 | |
36 | |
37 | def juju_info(msg, level='INFO'): |
38 | @@ -143,34 +145,93 @@ |
39 | return 0 |
40 | |
41 | |
42 | +def _service_name(config): |
43 | + '''Returns the service name in a form useful |
44 | + for using as a directory component, with |
45 | + characters that would need to be escaped replaced with _.''' |
46 | + unit = core.hookenv.service_name() |
47 | + for x in (':', '/', '"', "'"): |
48 | + unit = unit.replace(x, '_') |
49 | + return unit |
50 | + |
51 | + |
52 | +def _service_dir(config): |
53 | + return os.path.join(config['install_root'], _service_name(config)) |
54 | + |
55 | + |
56 | def setup_dependencies(config): |
57 | - required_packages = ['bzr', 'cowbuilder'] |
58 | + required_packages = ['bzr', 'cowbuilder', 'eatmydata'] |
59 | fetch.apt_update() |
60 | juju_info('installing apt packages...') |
61 | fetch.apt_install(required_packages, fatal=True) |
62 | |
63 | |
64 | def setup_pbuilderrc(config): |
65 | - cu2d_dir = '/srv/cu2d' |
66 | - if os.path.exists(cu2d_dir): |
67 | - subprocess.check_call(['bzr', 'update', cu2d_dir]) |
68 | + cu2d_dir = _service_dir(config) |
69 | + if config.get('vcs') == 'tarball': |
70 | + _install_from_tarball(config, os.path.exists(cu2d_dir)) |
71 | else: |
72 | - try: |
73 | - subprocess.check_call(['bzr', 'branch', |
74 | - 'lp:cupstream2distro/uci-airline', |
75 | - cu2d_dir]) |
76 | - except subprocess.CalledProcessError as e: |
77 | - raise EnvironmentError( |
78 | - 'Could not get cu2d (return: {})'.format(e.returncode)) |
79 | + _install_from_bzr(config, os.path.exists(cu2d_dir)) |
80 | |
81 | - shutil.copy(os.path.join(cu2d_dir, 'chroot-tools', '.pbuilderrc'), |
82 | + shutil.copy(os.path.join(cu2d_dir, 'cupstream2distro', 'chroot-tools', |
83 | + '.pbuilderrc'), |
84 | CHROOT_USER_HOME) |
85 | |
86 | |
87 | +def _install_from_tarball(config, upgrade): |
88 | + tarball_url = config['tarball'] |
89 | + juju_info('grabbing service from tarball {}...'.format(tarball_url)) |
90 | + cu2d_dir = _service_dir(config) |
91 | + if not upgrade: |
92 | + juju_info( |
93 | + 'Creating service directory: {}'.format(cu2d_dir)) |
94 | + os.mkdir(cu2d_dir) |
95 | + cmd = 'curl %s | tar -xzC %s' % (tarball_url, cu2d_dir) |
96 | + subprocess.check_call(cmd, shell=True) |
97 | + |
98 | + |
99 | +def _install_from_bzr(config, upgrade): |
100 | + juju_info('grabbing service from bzr...') |
101 | + |
102 | + cu2d_dir = _service_dir(config) |
103 | + rev = config.get('revno', '') |
104 | + |
105 | + if upgrade: |
106 | + args = ['bzr', 'pull'] |
107 | + if rev: |
108 | + args.extend(['-r', rev]) |
109 | + subprocess.check_call(args, cwd=cu2d_dir) |
110 | + return |
111 | + else: |
112 | + juju_info('deleting pre-existing service directory: %s' % cu2d_dir) |
113 | + shutil.rmtree(cu2d_dir) |
114 | + |
115 | + args = ['bzr', 'branch'] |
116 | + if rev: |
117 | + args.extend(['-r', rev]) |
118 | + args.append(config['branch']) |
119 | + args.append(cu2d_dir) |
120 | + subprocess.check_call(args) |
121 | + |
122 | + |
123 | +def update_pbuilderrc(config): |
124 | + pbuilderrc = os.path.join(CHROOT_USER_HOME, '.pbuilderrc') |
125 | + if EAT_MY_DATA_PRELOAD_LINE not in open(pbuilderrc, 'rb').readlines()[-1]: |
126 | + try: |
127 | + fh = open(pbuilderrc, 'a') |
128 | + fh.write(EAT_MY_DATA_PRELOAD_LINE) |
129 | + except IOError: |
130 | + print "Can't update pbuiderrc" |
131 | + else: |
132 | + print 'pbuilderrc updated for chroot' |
133 | + fh.close() |
134 | + |
135 | + |
136 | def config_changed(config): |
137 | setup_dependencies(config) |
138 | setup_pbuilderrc(config) |
139 | setup_chroots(config) |
140 | + update_pbuilderrc(config) |
141 | |
142 | |
143 | def main(): |
144 | |
145 | === modified file 'cupstream2distro/chroot-tools/.pbuilderrc' |
146 | --- cupstream2distro/chroot-tools/.pbuilderrc 2014-06-10 20:42:46 +0000 |
147 | +++ cupstream2distro/chroot-tools/.pbuilderrc 2014-06-26 14:33:53 +0000 |
148 | @@ -30,4 +30,3 @@ |
149 | # EXTRAPACKAGES is not obeyed by 'cowbuilder --create' but running |
150 | # 'cowbuilder--update' will happily fix that. |
151 | EXTRAPACKAGES="eatmydata pbuilder bzr bzr-builddeb" |
152 | -export LD_PRELOAD="${LD_PRELOAD:+$LD_PRELOAD:}/usr/lib/libeatmydata/libeatmydata.so" |
153 | |
154 | === modified file 'juju-deployer/branch-source-builder.yaml.tmpl' |
155 | --- juju-deployer/branch-source-builder.yaml.tmpl 2014-06-25 18:01:51 +0000 |
156 | +++ juju-deployer/branch-source-builder.yaml.tmpl 2014-06-26 14:33:53 +0000 |
157 | @@ -15,6 +15,14 @@ |
158 | - ${CI_PPA} |
159 | install_keys: | |
160 | - "" |
161 | + ci-airline-bsb-chroot-builder: |
162 | + charm: chroot-builder |
163 | + options: |
164 | + distros: trusty |
165 | + architectures: amd64 |
166 | + vcs: ${CI_CODE_SOURCE} |
167 | + branch: ${CI_BRANCH} |
168 | + tarball: ${CI_PAYLOAD_URL} |
169 | ci-airline-rabbit: |
170 | branch: lp:~canonical-ci-engineering/charms/precise/uci-engine/rabbitmq-server@47 |
171 | charm: rabbitmq |
172 | @@ -22,3 +30,4 @@ |
173 | management_plugin: true |
174 | relations: |
175 | - [ci-airline-bsb-worker, ci-airline-rabbit] |
176 | + - [ci-airline-bsb-worker, ci-airline-bsb-chroot-builder] |
PASSED: Continuous integration, rev:600 s-jenkins. ubuntu- ci:8080/ job/uci- engine- ci/910/
http://
Executed test runs:
Click here to trigger a rebuild: s-jenkins. ubuntu- ci:8080/ job/uci- engine- ci/910/ rebuild
http://