Merge lp:~chris.macnaughton/openstack-mojo-specs/upgrade-spec into lp:~ost-maintainers/openstack-mojo-specs/openstack-mojo-specs-1802

Proposed by Chris MacNaughton on 2018-03-14
Status: Needs review
Proposed branch: lp:~chris.macnaughton/openstack-mojo-specs/upgrade-spec
Merge into: lp:~ost-maintainers/openstack-mojo-specs/openstack-mojo-specs-1802
Diff against target: 357 lines (+297/-3)
6 files modified
helper/bundles/ubuntu-lite.yaml (+5/-0)
helper/collect/collect-ubuntu-lite (+1/-0)
helper/scripts/upgrade_all_units.py (+13/-0)
helper/utils/mojo_utils.py (+261/-3)
specs/dev/series_upgrade_ubuntu_lite/SPEC_INFO.txt (+1/-0)
specs/dev/series_upgrade_ubuntu_lite/manifest (+16/-0)
To merge this branch: bzr merge lp:~chris.macnaughton/openstack-mojo-specs/upgrade-spec
Reviewer Review Type Date Requested Status
David Ames 2018-03-14 Needs Fixing on 2018-03-19
Review via email: mp+341397@code.launchpad.net
To post a comment you must log in.
333. By Chris MacNaughton on 2018-03-14

rename upgrade script

David Ames (thedac) wrote :

I still need to do a full review but make lint fails with a number of lint problems.

review: Needs Fixing
David Ames (thedac) wrote :

Ok, so we have a lot of context in https://code.launchpad.net/~chris.macnaughton/openstack-mojo-specs/upgrade-spec/+merge/337646 which I would like to continue.

My recommendation is we merge the 18.02 branch without this change, abandon this MP and continue with the original MP which has several unaddressed comments.

334. By Chris MacNaughton on 2018-03-20

update to resolve lint issues and resolve discussions

- remove prints
- shift to use `juju run` where possible

Unmerged revisions

334. By Chris MacNaughton on 2018-03-20

update to resolve lint issues and resolve discussions

- remove prints
- shift to use `juju run` where possible

333. By Chris MacNaughton on 2018-03-14

rename upgrade script

332. By Chris MacNaughton on 2018-03-14

updating series upgrade spec

331. By Chris MacNaughton on 2018-03-14

update handlers to get units started correctly

330. By Chris MacNaughton on 2018-02-14

tidy a bit

329. By Chris MacNaughton on 2018-02-13

add helper to upgrade all units in a model

328. By Chris MacNaughton on 2018-02-06

starting framing upgrade spec

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'helper/bundles/ubuntu-lite.yaml'
2--- helper/bundles/ubuntu-lite.yaml 1970-01-01 00:00:00 +0000
3+++ helper/bundles/ubuntu-lite.yaml 2018-03-20 07:12:59 +0000
4@@ -0,0 +1,5 @@
5+series: "{{ series }}"
6+services:
7+ ubuntu-lite:
8+ charm: ubuntu-lite
9+ num_units: 3
10
11=== added file 'helper/collect/collect-ubuntu-lite'
12--- helper/collect/collect-ubuntu-lite 1970-01-01 00:00:00 +0000
13+++ helper/collect/collect-ubuntu-lite 2018-03-20 07:12:59 +0000
14@@ -0,0 +1,1 @@
15+ubuntu-lite cs:~jameinel/ubuntu-lite
16
17=== added file 'helper/scripts/upgrade_all_units.py'
18--- helper/scripts/upgrade_all_units.py 1970-01-01 00:00:00 +0000
19+++ helper/scripts/upgrade_all_units.py 2018-03-20 07:12:59 +0000
20@@ -0,0 +1,13 @@
21+#!/usr/bin/env python
22+import sys
23+import utils.mojo_utils as mojo_utils
24+import os
25+
26+
27+def main(argv):
28+ mojo_utils.setup_logging()
29+ mojo_utils.upgrade_all_units()
30+
31+
32+if __name__ == "__main__":
33+ sys.exit(main(sys.argv))
34
35=== modified file 'helper/utils/mojo_utils.py'
36--- helper/utils/mojo_utils.py 2018-02-28 10:53:42 +0000
37+++ helper/utils/mojo_utils.py 2018-03-20 07:12:59 +0000
38@@ -1,5 +1,5 @@
39 #!/usr/bin/env python
40-
41+import six
42 import logging
43 import os
44 import shutil
45@@ -465,8 +465,8 @@
46 files.append(os.path.join(os.path.dirname(__file__), filename))
47 # Up one directory from called file
48 files.append(os.path.join(
49- os.path.dirname(os.path.dirname(__file__)),
50- filename))
51+ os.path.dirname(os.path.dirname(__file__)),
52+ filename))
53
54 for file_path in files:
55 if os.path.isfile(file_path):
56@@ -563,6 +563,264 @@
57 time.sleep(30)
58
59
60+# Begin upgrade code
61+
62+def do_release_upgrade(unit):
63+ """Runs do-release-upgrade noninteractive"""
64+ logging.info('Upgrading ' + unit)
65+ subprocess.call([kiki.cmd(), 'run', '--unit', unit, 'status-set',
66+ 'maintenance', 'Doing release upgrade'])
67+ cmd = [kiki.cmd(), 'ssh', unit, 'sudo',
68+ 'do-release-upgrade', '-f', 'DistUpgradeViewNonInteractive']
69+ try:
70+ subprocess.check_call(cmd)
71+ except subprocess.CalledProcessError as e:
72+ logging.warn("Failed do-release-upgrade for {}".format(unit))
73+ logging.warn(e)
74+ return False
75+ finally:
76+ subprocess.call([kiki.cmd(), 'run', '--unit', unit, 'status-set',
77+ 'active'])
78+ return True
79+
80+
81+def reboot(unit):
82+ """Reboot machine"""
83+ cmd = [kiki.cmd(), 'run', '--unit', unit, 'sudo', 'reboot', '&&', 'exit']
84+ try:
85+ subprocess.check_call(cmd)
86+ except subprocess.CalledProcessError as e:
87+ logging.info(e)
88+ pass
89+
90+
91+def upgrade_machine(app_name, unit, machine, machine_num):
92+ """Run the upgrade process for a single machine"""
93+ cmd = [kiki.cmd(), 'run', '--unit', unit, 'status-set',
94+ 'maintenance', 'Upgrading series']
95+ subprocess.call(cmd)
96+ if not do_release_upgrade(unit):
97+ return False
98+ if machine["series"] == "trusty":
99+ upstart_to_systemd(machine_num)
100+ cmd = [kiki.cmd(), 'run', '--unit', unit, 'status-set',
101+ 'active']
102+ subprocess.call(cmd)
103+ logging.debug("Rebooting")
104+ reboot(unit)
105+ cmd = [kiki.cmd(), "ssh", unit, "exit"]
106+ while(True):
107+ try:
108+ subprocess.check_call(cmd)
109+ break
110+ except subprocess.CalledProcessError:
111+ logging.debug("Waiting 2 more seconds")
112+ time.sleep(2)
113+ update_machine_series(app_name, machine_num)
114+ return True
115+
116+
117+def update_machine_series(app_name, machine_num):
118+ cmd = [kiki.cmd(), 'run', '--machine',
119+ machine_num, 'lsb_release', '-c', '-s']
120+ codename = subprocess.check_output(cmd)
121+ if six.PY3:
122+ codename = codename.decode('utf-8')
123+ codename = codename.strip()
124+ logging.debug("Telling juju that {} series is {}".format(
125+ machine_num, codename))
126+ cmd = [kiki.cmd(), 'update-series', str(machine_num), codename]
127+ subprocess.call(cmd)
128+ cmd = [kiki.cmd(), 'update-series', app_name, codename]
129+ subprocess.call(cmd)
130+
131+
132+SYSTEMD_JUJU_MACHINE_AGENT_SCRIPT = """#!/usr/bin/env bash
133+
134+# Set up logging.
135+touch '/var/log/juju/machine-{machine_id}.log'
136+chown syslog:syslog '/var/log/juju/machine-{machine_id}.log'
137+chmod 0600 '/var/log/juju/machine-{machine_id}.log'
138+exec >> '/var/log/juju/machine-{machine_id}.log'
139+exec 2>&1
140+
141+# Run the script.
142+'/var/lib/juju/tools/machine-{machine_id}/jujud' machine --data-dir '/var/lib/juju' --machine-id {machine_id} --debug
143+""" # nopep8
144+
145+SYSTEMD_JUJU_UNIT_AGENT_SCRIPT = """#!/usr/bin/env bash
146+
147+# Set up logging.
148+touch '/var/log/juju/unit-{application_name}-{application_number}.log'
149+chown syslog:syslog '/var/log/juju/unit-{application_name}-{application_number}.log'
150+chmod 0600 '/var/log/juju/unit-{application_name}-{application_number}.log'
151+exec >> '/var/log/juju/unit-{application_name}-{application_number}.log'
152+exec 2>&1
153+
154+# Run the script.
155+'/var/lib/juju/tools/unit-{application_name}-{application_number}/jujud' unit --data-dir '/var/lib/juju' --unit-name {application} --debug
156+""" # nopep8
157+
158+SYSTEMD_JUJU_MACHINE_INIT_FILE = """[Unit]
159+Description=juju agent for machine-{name}
160+After=syslog.target
161+After=network.target
162+After=systemd-user-sessions.service
163+
164+[Service]
165+LimitNOFILE=20000
166+ExecStart=/var/lib/juju/init/jujud-machine-{name}/exec-start.sh
167+Restart=on-failure
168+TimeoutSec=300
169+
170+[Install]
171+WantedBy=multi-user.target
172+""" # nopep8
173+
174+
175+SYSTEMD_JUJU_UNIT_INIT_FILE = """[Unit]
176+Description=juju unit agent for unit-{application_name}-{application_number}
177+After=syslog.target
178+After=network.target
179+After=systemd-user-sessions.service
180+
181+[Service]
182+Environment="JUJU_CONTAINER_TYPE="
183+ExecStart=/var/lib/juju/init/jujud-unit-{application_name}-{application_number}/exec-start.sh
184+Restart=on-failure
185+TimeoutSec=300
186+
187+[Install]
188+WantedBy=multi-user.target
189+"""
190+
191+
192+def upstart_to_systemd(machine_number):
193+ """Upgrade upstart scripts to Systemd after upgrade from Trusty"""
194+ base_command = [kiki.cmd(), 'run', '--machine', str(machine_number), '--']
195+ commands = [
196+ base_command + [
197+ "sudo", "mkdir", "-p",
198+ "/var/lib/juju/init/jujud-machine-{}".format(machine_number)],
199+ base_command + [
200+ 'echo', SYSTEMD_JUJU_MACHINE_AGENT_SCRIPT.format(
201+ machine_id=machine_number),
202+ '|', 'sudo', 'tee',
203+ ('/var/lib/juju/init/jujud-machine-{machine_id}'
204+ '/exec-start.sh').format(
205+ machine_id=machine_number)],
206+ base_command + [
207+ 'echo', SYSTEMD_JUJU_MACHINE_INIT_FILE.format(
208+ name=machine_number), '|', 'sudo', 'tee',
209+ ('/var/lib/juju/init/jujud-machine-{name}'
210+ '/jujud-machine-{name}.service').format(
211+ name=machine_number)],
212+ base_command + [
213+ 'sudo', 'chmod', '755',
214+ ('/var/lib/juju/init/jujud-machine-{machine_id}/'
215+ 'exec-start.sh').format(machine_id=machine_number)],
216+ base_command + [
217+ 'sudo', 'ln', '-s',
218+ ('/var/lib/juju/init/jujud-machine-{machine_id}/'
219+ 'jujud-machine-{machine_id}.service').format(
220+ machine_id=machine_number
221+ ), '/etc/systemd/system/'],
222+ base_command + [
223+ 'sudo', 'ln', '-s',
224+ ('/var/lib/juju/init/jujud-machine-{machine_id}/'
225+ 'jujud-machine-{machine_id}.service').format(
226+ machine_id=machine_number),
227+ ('/etc/systemd/system/multi-user.target.wants/'
228+ 'jujud-machine-{machine_id}.service').format(
229+ machine_id=machine_number
230+ )
231+ ]
232+ ]
233+ commands += units_upstart_to_systemd_commands(machine_number)
234+ for cmd in commands:
235+ try:
236+ subprocess.check_call(cmd)
237+ except subprocess.CalledProcessError as e:
238+ logging.warn(e)
239+ return False
240+
241+
242+def units_upstart_to_systemd_commands(machine_number):
243+ """Upgrade a specific application unit from Upstart to Systemd"""
244+ units = get_juju_status(unit=str(machine_number))["applications"]
245+ base_command = [kiki.cmd(), 'run', '--machine', str(machine_number), '--']
246+ commands = []
247+ for (name, app_unit) in units.iteritems():
248+ for (unit_name, unit) in app_unit["units"].iteritems():
249+ logging.debug("Updating {} [{}]".format(name, unit_name))
250+ app_number = unit_name.split("/")[-1]
251+ systemd_file_name = ("jujud-unit-{app_name}"
252+ "-{app_number}.service").format(
253+ app_name=name, app_number=app_number)
254+ systemd_file_path = ('/var/lib/juju/init/jujud-unit-{app_name}'
255+ '-{app_number}/{file_name}').format(
256+ app_name=name,
257+ app_number=app_number,
258+ file_name=systemd_file_name)
259+ commands += [
260+ base_command + [
261+ "sudo", "mkdir", "-p",
262+ "/var/lib/juju/init/jujud-unit-{}-{}".format(
263+ name, app_number)],
264+ base_command + [
265+ 'echo', SYSTEMD_JUJU_UNIT_AGENT_SCRIPT.format(
266+ application=unit_name,
267+ application_name=name,
268+ application_number=app_number),
269+ '|', 'sudo', 'tee',
270+ ('/var/lib/juju/init/jujud-unit-{app_name}-'
271+ '{app_number}/exec-start.sh').format(
272+ app_name=name, app_number=app_number)],
273+ base_command + [
274+ 'echo', SYSTEMD_JUJU_UNIT_INIT_FILE.format(
275+ application_name=name, application_number=app_number),
276+ '|', 'sudo', 'tee', systemd_file_path],
277+ base_command + [
278+ 'sudo', 'chmod', '755',
279+ ('/var/lib/juju/init/jujud-unit-{app_name}-'
280+ '{app_number}/exec-start.sh').format(
281+ app_name=name, app_number=app_number)],
282+ base_command + [
283+ 'sudo', 'ln', '-s', systemd_file_path,
284+ '/etc/systemd/system/'],
285+ base_command + [
286+ 'sudo', 'ln', '-s', systemd_file_path,
287+ ('/etc/systemd/system/multi-user.target.wants/'
288+ '{file_name}').format(
289+ machine_id=machine_number,
290+ file_name=systemd_file_name)
291+ ]
292+ ]
293+ return commands
294+
295+
296+def upgrade_all_units(juju_status=None):
297+ if not juju_status:
298+ juju_status = get_juju_status()
299+ # Upgrade the rest
300+ upgraded_machines = []
301+ for (app_name, details) in juju_status['applications'].items():
302+ for name, unit_details in details['units'].items():
303+ logging.debug("Details for {}: {}".format(name, unit_details))
304+ machine_id = unit_details["machine"]
305+ if machine_id in upgraded_machines:
306+ continue
307+ if not upgrade_machine(
308+ app_name,
309+ name,
310+ juju_status["machines"][machine_id],
311+ machine_id
312+ ):
313+ logging.warn("No series upgrade found for {}".format(name))
314+ upgraded_machines.append(machine_id)
315+# End upgrade code
316+
317+
318 def parse_mojo_arg(options, mojoarg, multiargs=False):
319 if mojoarg.upper() in os.environ:
320 if multiargs:
321
322=== added directory 'specs/dev/series_upgrade_ubuntu_lite'
323=== added file 'specs/dev/series_upgrade_ubuntu_lite/SPEC_INFO.txt'
324--- specs/dev/series_upgrade_ubuntu_lite/SPEC_INFO.txt 1970-01-01 00:00:00 +0000
325+++ specs/dev/series_upgrade_ubuntu_lite/SPEC_INFO.txt 2018-03-20 07:12:59 +0000
326@@ -0,0 +1,1 @@
327+Deploys ubuntu-lite, and upgrades the charm between series.
328
329=== added symlink 'specs/dev/series_upgrade_ubuntu_lite/check_juju.py'
330=== target is u'../../../helper/tests/check_juju.py'
331=== added symlink 'specs/dev/series_upgrade_ubuntu_lite/collect-ubuntu-lite'
332=== target is u'../../../helper/collect/collect-ubuntu-lite'
333=== added file 'specs/dev/series_upgrade_ubuntu_lite/manifest'
334--- specs/dev/series_upgrade_ubuntu_lite/manifest 1970-01-01 00:00:00 +0000
335+++ specs/dev/series_upgrade_ubuntu_lite/manifest 2018-03-20 07:12:59 +0000
336@@ -0,0 +1,16 @@
337+# Collect the charm
338+collect config=collect-ubuntu-lite
339+
340+# Deploy ubuntu-lite bundle
341+deploy config=ubuntu-lite.yaml delay=0 wait=False
342+
343+# Wait for deployment to settle
344+verify config=check_juju.py
345+
346+# Upgrade units
347+script config=scripts/upgrade_all_units.py
348+
349+# Wait for deployment to settle
350+verify config=check_juju.py
351+
352+# Success
353
354=== added symlink 'specs/dev/series_upgrade_ubuntu_lite/scripts'
355=== target is u'../../../helper/scripts'
356=== added symlink 'specs/dev/series_upgrade_ubuntu_lite/ubuntu-lite.yaml'
357=== target is u'../../../helper/bundles/ubuntu-lite.yaml'

Subscribers

People subscribed via source and target branches