Merge lp:~chris.macnaughton/openstack-mojo-specs/upgrade-spec into lp:openstack-mojo-specs

[Chris MacNaughton]

[Ryan Beisner (1chb1n)]

Reference juju core bug: https://bugs.launchpad.net/juju/+bug/1749201

[David Ames (thedac)]

I understand this is a WIP. Just some notes for the next revision(s).

Most (all?) juju ssh $UNIT sudo $CMD should be juju run --unit $UNIT $CMD. You get root for free and various --format output options more convenient for python.

A number of WIP print statements that either need logging statements or removal before landing. Not also the use of logging.error or logging.warn where appropriate.

The juju series-update commands, when is it safe to set this for an application that may have multiple units?

Individual try/except blocks look good. But it would make sense to have a top level try/except that determines success or failure based on raised exceptions in nested tries.

Larger question. At what point does it make more sense to create a bash script and scp it to the unit to run. Rather, than piecemeal commands over ssh? That might be a cleaner approach and it would be re-usable outside mojo.

review: Needs Fixing

[Chris MacNaughton (chris.macnaughton)]

Thanks for the suggestions, in addition to the comments inline, changes are incoming!

Several comments inline; regarding making a bash script rather than this approach, at the moment it is fairly isolated, and runnable with the script in helper/scripts/upgrade_all_units.py. One of the goals of this bit of code is that we will be able to migrate it easily into zaza (or just something else) at some point in the future, as a generic set of helper utilities that are consumable outside of mojo.

[lp:~chris.macnaughton/openstack-mojo-specs/upgrade-spec]

330. By Chris MacNaughton on 2018-02-14

tidy a bit

331. By Chris MacNaughton on 2018-03-14

update handlers to get units started correctly

332. By Chris MacNaughton on 2018-03-14

updating series upgrade spec

333. By Chris MacNaughton on 2018-03-14

rename upgrade script

[David Ames (thedac)]

Need to fix linting errors. I expect this might be challenging as most of them are due to long lines in the bash snippets.

> Most (all?) juju ssh $UNIT sudo $CMD should be juju run --unit $UNIT $CMD. You get root for free and various --format output options more convenient for python.

The do-release-upgrade and the loop check after reboot makes sense to use SSH. The others, reboot and lsb_release check, make more sense as juju run.

Note: it turns out juju run --unit runs as root but juju run --machine runs as ubuntu user. Just for consistency.

> A number of WIP print statements that either need logging statements or removal before landing. Not also the use of logging.error or logging.warn where appropriate.

Still true. Some are commented but they should be removed or made logging statements. Use of logging.error and logging.warn suggestions still in effect.

review: Needs Fixing

[lp:~chris.macnaughton/openstack-mojo-specs/upgrade-spec]

334. By Chris MacNaughton on 2018-03-20

update to resolve lint issues and resolve discussions

- remove prints
- shift to use `juju run` where possible

[Chris MacNaughton (chris.macnaughton)]

A couple of things left to discuss:

> The juju series-update commands, when is it safe to set this for an application that may have multiple units?

At the moment, there is no possible config into the actual callable script, although it will only iterate through units, so, depending on the HA model of the hosted application, there shouldn't be service disruption as we will only be upgrading a single unit at a time for now.

> Individual try/except blocks look good. But it would make sense to have a top level try/except that determines success or failure based on raised exceptions in nested tries.

I'm open to working through doing something like this if we decide that it's necessary; however, I'm nervous to make this code perfect, given that the Juju core team has picked up a roadmap item that will reduce our "series_upgrade" function to something along the lines of `juju series-upgrade $machine_id`, at which point we will be removing almost all of the upgrade code. To make the cleanup of that easier, in the last commit I have comment fenced off the upgrade code, although I propose that it could be moved into a new helper alongside the mojo_utils.py for now to ensure isolation.

> Larger question. At what point does it make more sense to create a bash script and scp it to the unit to run. Rather, than piecemeal commands over ssh? That might be a cleaner approach and it would be re-usable outside mojo.

Rather than scp'ing a bash script, I am considering shifting the series of `juju ssh`'d commands into building a bash script that gets rendered (depends entirely on machine numbers, unit names & numbers, etc) and putting that into /tmp to execute this, as it would ease the run time a bit; however, the amount of time spent running these scripts pales in comparison to the runtime of the `do-release-upgrade`, so I'm not sure of the real value, except to improve readability (a good value, but questionable to me given that this code has been written to be deleted).

[David Ames (thedac)]

I think we are good at this point. The previous comment were topics I felt were answered previously. This looks good. I'll be merging. Several specs all back to trunk.

review: Approve

=== added file 'helper/bundles/ubuntu-lite.yaml'
--- helper/bundles/ubuntu-lite.yaml	1970-01-01 00:00:00 +0000
+++ helper/bundles/ubuntu-lite.yaml	2018-03-20 07:12:59 +0000
@@ -0,0 +1,5 @@
+series: "{{ series }}"
+services:
+  ubuntu-lite:
+    charm: ubuntu-lite
+    num_units: 3

=== added file 'helper/collect/collect-ubuntu-lite'
--- helper/collect/collect-ubuntu-lite	1970-01-01 00:00:00 +0000
+++ helper/collect/collect-ubuntu-lite	2018-03-20 07:12:59 +0000
@@ -0,0 +1,1 @@
+ubuntu-lite                 cs:~jameinel/ubuntu-lite

=== added file 'helper/scripts/upgrade_all_units.py'
--- helper/scripts/upgrade_all_units.py	1970-01-01 00:00:00 +0000
+++ helper/scripts/upgrade_all_units.py	2018-03-20 07:12:59 +0000
@@ -0,0 +1,13 @@
+#!/usr/bin/env python
+import sys
+import utils.mojo_utils as mojo_utils
+import os
+
+
+def main(argv):
+    mojo_utils.setup_logging()
+    mojo_utils.upgrade_all_units()
+
+
+if __name__ == "__main__":
+    sys.exit(main(sys.argv))

=== modified file 'helper/utils/mojo_utils.py'
--- helper/utils/mojo_utils.py	2018-01-19 12:15:27 +0000
+++ helper/utils/mojo_utils.py	2018-03-20 07:12:59 +0000
@@ -1,5 +1,5 @@
 #!/usr/bin/env python
-
+import six
 import logging
 import os
 import shutil
@@ -456,8 +456,8 @@
     files.append(os.path.join(os.path.dirname(__file__), filename))
     # Up one directory from called file
     files.append(os.path.join(
-                     os.path.dirname(os.path.dirname(__file__)),
-                     filename))
+        os.path.dirname(os.path.dirname(__file__)),
+        filename))
 
     for file_path in files:
         if os.path.isfile(file_path):
@@ -554,6 +554,264 @@
             time.sleep(30)
 
 
+# Begin upgrade code
+
+def do_release_upgrade(unit):
+    """Runs do-release-upgrade noninteractive"""
+    logging.info('Upgrading ' + unit)
+    subprocess.call([kiki.cmd(), 'run', '--unit', unit, 'status-set',
+                     'maintenance', 'Doing release upgrade'])
+    cmd = [kiki.cmd(), 'ssh', unit, 'sudo',
+           'do-release-upgrade', '-f', 'DistUpgradeViewNonInteractive']
+    try:
+        subprocess.check_call(cmd)
+    except subprocess.CalledProcessError as e:
+        logging.warn("Failed do-release-upgrade for {}".format(unit))
+        logging.warn(e)
+        return False
+    finally:
+        subprocess.call([kiki.cmd(), 'run', '--unit', unit, 'status-set',
+                         'active'])
+    return True
+
+
+def reboot(unit):
+    """Reboot machine"""
+    cmd = [kiki.cmd(), 'run', '--unit', unit, 'sudo', 'reboot', '&&', 'exit']
+    try:
+        subprocess.check_call(cmd)
+    except subprocess.CalledProcessError as e:
+        logging.info(e)
+        pass
+
+
+def upgrade_machine(app_name, unit, machine, machine_num):
+    """Run the upgrade process for a single machine"""
+    cmd = [kiki.cmd(), 'run', '--unit', unit, 'status-set',
+           'maintenance', 'Upgrading series']
+    subprocess.call(cmd)
+    if not do_release_upgrade(unit):
+        return False
+    if machine["series"] == "trusty":
+        upstart_to_systemd(machine_num)
+    cmd = [kiki.cmd(), 'run', '--unit', unit, 'status-set',
+           'active']
+    subprocess.call(cmd)
+    logging.debug("Rebooting")
+    reboot(unit)
+    cmd = [kiki.cmd(), "ssh", unit, "exit"]
+    while(True):
+        try:
+            subprocess.check_call(cmd)
+            break
+        except subprocess.CalledProcessError:
+            logging.debug("Waiting 2 more seconds")
+            time.sleep(2)
+    update_machine_series(app_name, machine_num)
+    return True
+
+
+def update_machine_series(app_name, machine_num):
+    cmd = [kiki.cmd(), 'run', '--machine',
+           machine_num, 'lsb_release', '-c', '-s']
+    codename = subprocess.check_output(cmd)
+    if six.PY3:
+        codename = codename.decode('utf-8')
+    codename = codename.strip()
+    logging.debug("Telling juju that {} series is {}".format(
+        machine_num, codename))
+    cmd = [kiki.cmd(), 'update-series', str(machine_num), codename]
+    subprocess.call(cmd)
+    cmd = [kiki.cmd(), 'update-series', app_name, codename]
+    subprocess.call(cmd)
+
+
+SYSTEMD_JUJU_MACHINE_AGENT_SCRIPT = """#!/usr/bin/env bash
+
+# Set up logging.
+touch '/var/log/juju/machine-{machine_id}.log'
+chown syslog:syslog '/var/log/juju/machine-{machine_id}.log'
+chmod 0600 '/var/log/juju/machine-{machine_id}.log'
+exec >> '/var/log/juju/machine-{machine_id}.log'
+exec 2>&1
+
+# Run the script.
+'/var/lib/juju/tools/machine-{machine_id}/jujud' machine --data-dir '/var/lib/juju' --machine-id {machine_id} --debug
+"""  # nopep8
+
+SYSTEMD_JUJU_UNIT_AGENT_SCRIPT = """#!/usr/bin/env bash
+
+# Set up logging.
+touch '/var/log/juju/unit-{application_name}-{application_number}.log'
+chown syslog:syslog '/var/log/juju/unit-{application_name}-{application_number}.log'
+chmod 0600 '/var/log/juju/unit-{application_name}-{application_number}.log'
+exec >> '/var/log/juju/unit-{application_name}-{application_number}.log'
+exec 2>&1
+
+# Run the script.
+'/var/lib/juju/tools/unit-{application_name}-{application_number}/jujud' unit --data-dir '/var/lib/juju' --unit-name {application} --debug
+"""  # nopep8
+
+SYSTEMD_JUJU_MACHINE_INIT_FILE = """[Unit]
+Description=juju agent for machine-{name}
+After=syslog.target
+After=network.target
+After=systemd-user-sessions.service
+
+[Service]
+LimitNOFILE=20000
+ExecStart=/var/lib/juju/init/jujud-machine-{name}/exec-start.sh
+Restart=on-failure
+TimeoutSec=300
+
+[Install]
+WantedBy=multi-user.target
+"""  # nopep8
+
+
+SYSTEMD_JUJU_UNIT_INIT_FILE = """[Unit]
+Description=juju unit agent for unit-{application_name}-{application_number}
+After=syslog.target
+After=network.target
+After=systemd-user-sessions.service
+
+[Service]
+Environment="JUJU_CONTAINER_TYPE="
+ExecStart=/var/lib/juju/init/jujud-unit-{application_name}-{application_number}/exec-start.sh
+Restart=on-failure
+TimeoutSec=300
+
+[Install]
+WantedBy=multi-user.target
+"""
+
+
+def upstart_to_systemd(machine_number):
+    """Upgrade upstart scripts to Systemd after upgrade from Trusty"""
+    base_command = [kiki.cmd(), 'run', '--machine', str(machine_number), '--']
+    commands = [
+        base_command + [
+            "sudo", "mkdir", "-p",
+            "/var/lib/juju/init/jujud-machine-{}".format(machine_number)],
+        base_command + [
+            'echo', SYSTEMD_JUJU_MACHINE_AGENT_SCRIPT.format(
+                machine_id=machine_number),
+            '|', 'sudo', 'tee',
+            ('/var/lib/juju/init/jujud-machine-{machine_id}'
+             '/exec-start.sh').format(
+                machine_id=machine_number)],
+        base_command + [
+            'echo', SYSTEMD_JUJU_MACHINE_INIT_FILE.format(
+                name=machine_number), '|', 'sudo', 'tee',
+            ('/var/lib/juju/init/jujud-machine-{name}'
+             '/jujud-machine-{name}.service').format(
+                name=machine_number)],
+        base_command + [
+            'sudo', 'chmod', '755',
+            ('/var/lib/juju/init/jujud-machine-{machine_id}/'
+             'exec-start.sh').format(machine_id=machine_number)],
+        base_command + [
+            'sudo', 'ln', '-s',
+            ('/var/lib/juju/init/jujud-machine-{machine_id}/'
+             'jujud-machine-{machine_id}.service').format(
+                 machine_id=machine_number
+              ), '/etc/systemd/system/'],
+        base_command + [
+            'sudo', 'ln', '-s',
+            ('/var/lib/juju/init/jujud-machine-{machine_id}/'
+             'jujud-machine-{machine_id}.service').format(
+                machine_id=machine_number),
+            ('/etc/systemd/system/multi-user.target.wants/'
+                'jujud-machine-{machine_id}.service').format(
+                    machine_id=machine_number
+                )
+        ]
+    ]
+    commands += units_upstart_to_systemd_commands(machine_number)
+    for cmd in commands:
+        try:
+            subprocess.check_call(cmd)
+        except subprocess.CalledProcessError as e:
+            logging.warn(e)
+            return False
+
+
+def units_upstart_to_systemd_commands(machine_number):
+    """Upgrade a specific application unit from Upstart to Systemd"""
+    units = get_juju_status(unit=str(machine_number))["applications"]
+    base_command = [kiki.cmd(), 'run', '--machine', str(machine_number), '--']
+    commands = []
+    for (name, app_unit) in units.iteritems():
+        for (unit_name, unit) in app_unit["units"].iteritems():
+            logging.debug("Updating {} [{}]".format(name, unit_name))
+            app_number = unit_name.split("/")[-1]
+            systemd_file_name = ("jujud-unit-{app_name}"
+                                 "-{app_number}.service").format(
+                                     app_name=name, app_number=app_number)
+            systemd_file_path = ('/var/lib/juju/init/jujud-unit-{app_name}'
+                                 '-{app_number}/{file_name}').format(
+                                     app_name=name,
+                                     app_number=app_number,
+                                     file_name=systemd_file_name)
+            commands += [
+                base_command + [
+                    "sudo", "mkdir", "-p",
+                    "/var/lib/juju/init/jujud-unit-{}-{}".format(
+                        name, app_number)],
+                base_command + [
+                    'echo', SYSTEMD_JUJU_UNIT_AGENT_SCRIPT.format(
+                        application=unit_name,
+                        application_name=name,
+                        application_number=app_number),
+                    '|', 'sudo', 'tee',
+                    ('/var/lib/juju/init/jujud-unit-{app_name}-'
+                     '{app_number}/exec-start.sh').format(
+                         app_name=name, app_number=app_number)],
+                base_command + [
+                    'echo', SYSTEMD_JUJU_UNIT_INIT_FILE.format(
+                        application_name=name, application_number=app_number),
+                    '|', 'sudo', 'tee', systemd_file_path],
+                base_command + [
+                    'sudo', 'chmod', '755',
+                    ('/var/lib/juju/init/jujud-unit-{app_name}-'
+                     '{app_number}/exec-start.sh').format(
+                         app_name=name, app_number=app_number)],
+                base_command + [
+                    'sudo', 'ln', '-s', systemd_file_path,
+                    '/etc/systemd/system/'],
+                base_command + [
+                    'sudo', 'ln', '-s', systemd_file_path,
+                    ('/etc/systemd/system/multi-user.target.wants/'
+                     '{file_name}').format(
+                         machine_id=machine_number,
+                         file_name=systemd_file_name)
+                ]
+            ]
+    return commands
+
+
+def upgrade_all_units(juju_status=None):
+    if not juju_status:
+        juju_status = get_juju_status()
+    # Upgrade the rest
+    upgraded_machines = []
+    for (app_name, details) in juju_status['applications'].items():
+        for name, unit_details in details['units'].items():
+            logging.debug("Details for {}: {}".format(name, unit_details))
+            machine_id = unit_details["machine"]
+            if machine_id in upgraded_machines:
+                continue
+            if not upgrade_machine(
+                app_name,
+                name,
+                juju_status["machines"][machine_id],
+                machine_id
+            ):
+                logging.warn("No series upgrade found for {}".format(name))
+            upgraded_machines.append(machine_id)
+# End upgrade code
+
+
 def parse_mojo_arg(options, mojoarg, multiargs=False):
     if mojoarg.upper() in os.environ:
         if multiargs:

=== added directory 'specs/dev/series_upgrade_ubuntu_lite'
=== added file 'specs/dev/series_upgrade_ubuntu_lite/SPEC_INFO.txt'
--- specs/dev/series_upgrade_ubuntu_lite/SPEC_INFO.txt	1970-01-01 00:00:00 +0000
+++ specs/dev/series_upgrade_ubuntu_lite/SPEC_INFO.txt	2018-03-20 07:12:59 +0000
@@ -0,0 +1,1 @@
+Deploys ubuntu-lite, and upgrades the charm between series.

=== added symlink 'specs/dev/series_upgrade_ubuntu_lite/check_juju.py'
=== target is u'../../../helper/tests/check_juju.py'
=== added symlink 'specs/dev/series_upgrade_ubuntu_lite/collect-ubuntu-lite'
=== target is u'../../../helper/collect/collect-ubuntu-lite'
=== added file 'specs/dev/series_upgrade_ubuntu_lite/manifest'
--- specs/dev/series_upgrade_ubuntu_lite/manifest	1970-01-01 00:00:00 +0000
+++ specs/dev/series_upgrade_ubuntu_lite/manifest	2018-03-20 07:12:59 +0000
@@ -0,0 +1,16 @@
+# Collect the charm
+collect config=collect-ubuntu-lite
+
+# Deploy ubuntu-lite bundle
+deploy config=ubuntu-lite.yaml delay=0 wait=False
+
+# Wait for deployment to settle
+verify config=check_juju.py
+
+# Upgrade units
+script config=scripts/upgrade_all_units.py
+
+# Wait for deployment to settle
+verify config=check_juju.py
+
+# Success

=== added symlink 'specs/dev/series_upgrade_ubuntu_lite/scripts'
=== target is u'../../../helper/scripts'
=== added symlink 'specs/dev/series_upgrade_ubuntu_lite/ubuntu-lite.yaml'
=== target is u'../../../helper/bundles/ubuntu-lite.yaml'