Merge lp:~doanac/lava-lab/lava-scripts into lp:lava-lab

[Andy Doan]

start work on some helper functions that should allow us to manage lava deployments using salt

[Antonio Terceiro (terceiro)]

This looks awesome!

One general comment: you don't need to run l-d-t as root. I did the following
on the vagrant-bootstrap script inside the l-d-t repo:

if ! test -d /srv/lava/instances/development; then
  cd $basedir
  su vagrant -c './lava-deployment-tool setup -nd'
  su vagrant -c './lava-deployment-tool install -nd development'
fi

so you could do something like that, and you won't need to chown/chmod away, or
even add that check on l-d-t to allow running as root.

--
Antonio Terceiro
Software Engineer - Linaro
http://www.linaro.org

[Andy Doan (doanac)]

On 02/05, Antonio Terceiro wrote:
> One general comment: you don't need to run l-d-t as root. I did the following
> on the vagrant-bootstrap script inside the l-d-t repo:
>
> if ! test -d /srv/lava/instances/development; then
> cd $basedir
> su vagrant -c './lava-deployment-tool setup -nd'
> su vagrant -c './lava-deployment-tool install -nd development'
> fi

You must be setting up your "vagrant" user to allow password-less sudo?
We don't do that for our instance-manager and it seems a little scare
(to me) to allow that.

> so you could do something like that, and you won't need to chown/chmod away, or
> even add that check on l-d-t to allow running as root.

[Antonio Terceiro (terceiro)]

On Tue, Feb 05, 2013 at 06:46:22PM -0000, Andy Doan wrote:
> On 02/05, Antonio Terceiro wrote:
> > One general comment: you don't need to run l-d-t as root. I did the following
> > on the vagrant-bootstrap script inside the l-d-t repo:
> >
> > if ! test -d /srv/lava/instances/development; then
> > cd $basedir
> > su vagrant -c './lava-deployment-tool setup -nd'
> > su vagrant -c './lava-deployment-tool install -nd development'
> > fi
>
> You must be setting up your "vagrant" user to allow password-less sudo?
> We don't do that for our instance-manager and it seems a little scare
> (to me) to allow that.

in that case we could setup password-less sudo for instance-manager
exclusively for running the lava-deployment-tool command.

--
Antonio Terceiro
Software Engineer - Linaro
http://www.linaro.org

[Michael Hudson-Doyle (mwhudson)]

Andy Doan <email address hidden> writes:

> Andy Doan has proposed merging lp:~doanac/lava-lab/lava-scripts into lp:lava-lab.
>
> Requested reviews:
> Linaro Validation Team (linaro-validation)
>
> For more details, see:
> https://code.launchpad.net/~doanac/lava-lab/lava-scripts/+merge/146547
>
> start work on some helper functions that should allow us to manage lava deployments using salt
> --
> https://code.launchpad.net/~doanac/lava-lab/lava-scripts/+merge/146547
> You are subscribed to branch lp:lava-lab.
> === modified file '_grains/lava.py'
> --- _grains/lava.py 2013-01-10 16:37:55 +0000
> +++ _grains/lava.py 2013-02-05 04:19:20 +0000
> @@ -9,4 +9,10 @@
> p = os.path.join(instdir, f)
> if os.path.isdir(p):
> insts.append(p)
> + # This seems to be a bug in salt. Even after you reload/resync grains
> + # the module still uses what it read at the time the minion started up.
> + # this makes this dynamic so we don't have to restart salt just to see
> + # a new instance
> + if 'lava_instances' in __grains__:
> + __grains__['lava_instances'] = insts

Thats because grains are supposed to be invariant. From
http://docs.saltstack.org/en/latest/topics/targeting/grains.html:

    It is important to remember that grains are bits of information
    loaded when the salt minion starts, so this information is
    static. This means that the information in grains is unchanging,
    therefore the nature of the data is static. So grains information
    are things like the running kernel, or the operating system.

I think grains-but-varying-things are "pillars", but not completely
sure. Might be worth asking #saltstack IRC...

[Andy Doan (doanac)]

On 02/05, Antonio Terceiro wrote:
> in that case we could setup password-less sudo for instance-manager
> exclusively for running the lava-deployment-tool command.

l-d-t runs too many sudo commands to make a sane sudoers entry. Its
really an all or nothing approach. So I'd say we just allow this hack in
l-d-t for now.

Unmerged revisions

69. By Andy Doan on 2013-02-05

minor bug from last commit

make sure we return the most interest part of this function

68. By Andy Doan on 2013-02-05

add support for installing a remote worker instance

this makes adding a new worker node as simple as possible

67. By Andy Doan on 2013-02-04

add some lava scripts to help manage lab

lava-status - shows the nodes contributing to a lava deployment, if
they are running, and which is the master instance

lava-{start/stop} - start/stop each lava-instance contributing to a lava deployment

lava-upgrade - performs an upgrade for a lava-deployment

=== modified file '_grains/lava.py'
--- _grains/lava.py	2013-01-10 16:37:55 +0000
+++ _grains/lava.py	2013-02-05 04:19:20 +0000
@@ -9,4 +9,10 @@
             p = os.path.join(instdir, f)
             if os.path.isdir(p):
                 insts.append(p)
+    # This seems to be a bug in salt. Even after you reload/resync grains
+    # the module still uses what it read at the time the minion started up.
+    # this makes this dynamic so we don't have to restart salt just to see
+    # a new instance
+    if 'lava_instances' in __grains__:
+        __grains__['lava_instances'] = insts
     return {'lava_instances': insts}

=== added directory 'scripts'
=== added file 'scripts/lava-add-worker'
--- scripts/lava-add-worker	1970-01-01 00:00:00 +0000
+++ scripts/lava-add-worker	2013-02-05 04:19:20 +0000
@@ -0,0 +1,22 @@
+#!/usr/bin/env python
+
+import argparse
+
+import lava_salt
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description=lava_salt.add_worker.__doc__)
+    parser.add_argument('minion', metavar='<minion>',
+                        help='The host to install the lava worker instance on.')
+    parser.add_argument('ip', metavar='<ip>',
+                        help='The public IP address for the minion.')
+    parser.add_argument('instance', metavar='<instance>',
+                        help='The instance name we are creating on the worker')
+    parser.add_argument('--dry-run', dest='dryrun', action='store_true',
+                        help='Just display what would be changed')
+    args = parser.parse_args()
+
+    client = lava_salt.salt_client()
+    ret = lava_salt.add_worker(client, args.minion, args.ip, args.instance, args.dryrun)
+    print ret[args.minion]

=== added file 'scripts/lava-start'
--- scripts/lava-start	1970-01-01 00:00:00 +0000
+++ scripts/lava-start	2013-02-05 04:19:20 +0000
@@ -0,0 +1,20 @@
+#!/usr/bin/env python
+
+import argparse
+
+import lava_salt
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description=lava_salt.start.__doc__)
+    parser.add_argument('instance', metavar='<instance>',
+                        help='The instance name to start')
+    args = parser.parse_args()
+
+    client = lava_salt.salt_client()
+    ret = lava_salt.start(client, args.instance)
+    if ret:
+        print 'salt started the following instances:'
+        print ret
+    else:
+        print 'Instance already running on all minions'

=== added file 'scripts/lava-status'
--- scripts/lava-status	1970-01-01 00:00:00 +0000
+++ scripts/lava-status	2013-02-05 04:19:20 +0000
@@ -0,0 +1,19 @@
+#!/usr/bin/env python
+
+import argparse
+
+import lava_salt
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description=lava_salt.info.__doc__)
+    parser.add_argument('instance', metavar='<instance>',
+                        help='The instance name to stop')
+    args = parser.parse_args()
+
+    client = lava_salt.salt_client()
+
+    for host, props in lava_salt.info(client, args.instance).iteritems():
+        running = lava_salt.STATE_STRING[props['running']]
+        master = props['master']
+        print '{0}: running({1}) master({2})'.format(host, running, master)

=== added file 'scripts/lava-stop'
--- scripts/lava-stop	1970-01-01 00:00:00 +0000
+++ scripts/lava-stop	2013-02-05 04:19:20 +0000
@@ -0,0 +1,22 @@
+#!/usr/bin/env python
+
+import argparse
+
+import lava_salt
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description=lava_salt.stop.__doc__)
+    parser.add_argument('--just-workers', dest='just_workers', action='store_true',
+                        help='Just stop worker instances, not the master')
+    parser.add_argument('instance', metavar='<instance>',
+                        help='The instance name to stop')
+    args = parser.parse_args()
+
+    client = lava_salt.salt_client()
+    ret = lava_salt.stop(client, args.instance, args.just_workers)
+    if ret:
+        print 'salt stopped the following instances:'
+        print ret
+    else:
+        print 'Instance not running on any minions'

=== added file 'scripts/lava-upgrade'
--- scripts/lava-upgrade	1970-01-01 00:00:00 +0000
+++ scripts/lava-upgrade	2013-02-05 04:19:20 +0000
@@ -0,0 +1,34 @@
+#!/usr/bin/env python
+
+import argparse
+
+import lava_salt
+
+
+def _indented(buff, indent_char):
+    indent_char = '\n' + indent_char
+    return '   ' + indent_char.join(buff.split('\n'))
+
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description=lava_salt.upgrade.__doc__)
+    parser.add_argument('instance', metavar='<instance>',
+                        help='The instance name to upgrade')
+    parser.add_argument('--dry-run', dest='dryrun', action='store_true',
+                        help='Just display what would be changed')
+    args = parser.parse_args()
+
+    client = lava_salt.salt_client()
+    m_ret, w_ret = lava_salt.upgrade(client, args.instance, args.dryrun)
+
+    print 'Master:'
+    for host, msg in m_ret.iteritems():
+        print ' {0}:'.format(host)
+        print '  upgrade:\n{0}'.format(_indented(msg, '   |'))
+
+    print '\nWorkers:'
+    for host, rets in w_ret.iteritems():
+        print ' {0}:'.format(host)
+        print '  stop:\n{0}'.format(_indented(rets['stop'], '   |'))
+        print '  upgrade:\n{0}'.format(_indented(rets['upgrade'], '   |'))
+        print '  start:\n{0}'.format(_indented(rets['start'], '   |'))

=== added file 'scripts/lava_salt.py'
--- scripts/lava_salt.py	1970-01-01 00:00:00 +0000
+++ scripts/lava_salt.py	2013-02-05 04:19:20 +0000
@@ -0,0 +1,218 @@
+import salt.client
+
+RUNNING = 0
+STOPPED = 1
+UNKNOWN = 3
+
+STATE_STRING = {
+    RUNNING: 'running',
+    STOPPED: 'stopped',
+    UNKNOWN: '???',
+}
+
+LDT = '/home/instance-manager/lava-deployment-tool/lava-deployment-tool'
+
+
+def salt_client():
+    return salt.client.LocalClient()
+
+
+def info(client, instance):
+    """
+    Shows whether an instance of LAVA is running or not on its configured hosts.
+    """
+    cmd = 'status lava-instance LAVA_INSTANCE={0}'.format(instance)
+    inst_path = '/srv/lava/instances/{0}'.format(instance)
+    worker_file = '{0}/sbin/mount-masterfs'.format(inst_path)
+
+    inf = {}
+
+    ret = client.cmd('*', 'grains.item', ['lava_instances'])
+    for k, v in ret.iteritems():
+        if inst_path in v:
+            ret = client.cmd(k, 'cmd.run', [cmd])
+            running = UNKNOWN
+            if ret[k] == 'status: Unknown instance: %s' % instance:
+                running = STOPPED
+            elif ret[k] == 'lava-instance (%s) start/running' % instance:
+                running = RUNNING
+
+            ret = client.cmd(k, 'file.file_exists', [worker_file])
+            master = not ret[k]
+
+            inf[k] = {'running': running, 'master': master}
+    return inf
+
+
+def stop(client, instance, just_workers=False):
+    """
+    Issues a command to stop a given instance name on all minions where the
+    LAVA instance appears to be running.
+    """
+    cmd = 'stop lava-instance LAVA_INSTANCE={0}'.format(instance)
+
+    hosts = []
+    for host, props in info(client, instance).iteritems():
+        if props['running'] != STOPPED:
+            if not just_workers or not props['master']:
+                hosts.append(host)
+
+    if len(hosts):
+        return client.cmd(hosts, 'cmd.run', [cmd], expr_form='list')
+
+
+def start(client, instance):
+    """
+    Issues a command to start a given instance name on all minions where the
+    LAVA instance appears to not be running.
+    """
+    cmd = 'start lava-instance LAVA_INSTANCE={0}'.format(instance)
+
+    hosts = []
+    for host, props in info(client, instance).iteritems():
+        if props['running'] != RUNNING:
+            hosts.append(host)
+
+    if len(hosts):
+        return client.cmd(hosts, 'cmd.run', [cmd], expr_form='list')
+
+
+def upgrade(client, instance, dry_run=True):
+    """
+    Runs lava-deployment-tool upgrade for a LAVA setup. It first shuts down
+    each worker node instance. Then it runs lava-deployment-tool upgrade on
+    the master. Lastly, it runs lava-deployment-tool upgradeworker on the
+    worker nodes.
+    """
+    timeout = 300  # 5 minutes
+    workers = []
+    master = None
+
+    for host, props in info(client, instance).iteritems():
+        if props['master']:
+            assert not master, 'Detected multiple master instances in LAVA deployment'
+            master = host
+        else:
+            workers.append(host)
+
+    assert master, 'No master instance found in LAVA deployment'
+
+    w_ret = {}
+    for h in workers:
+        w_ret[h] = {'stop': 'dry-run', 'upgrade': 'dry-run', 'start': 'dry-run'}
+
+    if dry_run:
+        m_ret = {master: 'dry-run: upgrade master'}
+        return m_ret, w_ret
+
+    # first stop workers. This prevents a DB error if the upgrade changes
+    # the schema.
+    ret = stop(client, instance, True)
+    if ret:
+        for host, msg in ret.iteritems():
+            w_ret[host]['stop'] = msg
+
+    # now upgrade the master node
+    cmd = 'SKIP_ROOT_CHECK=yes {0} upgrade {1}'.format(LDT, instance)
+    m_ret = client.cmd(master, 'cmd.run', [cmd], timeout=timeout)
+
+    # now upgrade the workers
+    cmd = 'SKIP_ROOT_CHECK=yes {0} upgradeworker {1}'.format(LDT, instance)
+    if len(workers):
+        ret = client.cmd(workers, 'cmd.run', [cmd], timeout=timeout, expr_form='list')
+        for host, msg in ret.iteritems():
+            w_ret[host]['upgrade'] = msg
+
+    ret = start(client, instance)
+    if ret:
+        for host, msg in ret.iteritems():
+            if host in w_ret:
+                w_ret[host]['start'] = msg
+
+    # last thing: l-d-t ran as root, lets chmod things
+    cmd = 'chown -R instance-manager:instance-manager /srv/lava/instances/{0}/code/*'.format(instance)
+    client.cmd(workers + [master], 'cmd.run', [cmd], expr_form='list')
+
+    return m_ret, w_ret
+
+
+def _update_props(inifile_content, props):
+    for line in inifile_content.split('\n'):
+        if not line.strip().startswith('#'):
+            key, val = line.split('=')
+            if key in props:
+                props[key] = val.replace("'", '')
+
+
+def add_worker(client, minion, minion_ip, instance, dry_run=True):
+    """
+    Creates a new lava workernode on a salt-minion.
+    """
+
+    args = {
+        'LAVA_SERVER_IP': None,
+        'LAVA_SYS_USER': None,
+        'LAVA_PROXY': None,
+        'masterdir': '/srv/lava/instances/{0}'.format(instance),
+        'workerip': minion_ip,
+        'ldt': LDT,
+        'instance': instance,
+        'dbuser': None,
+        'dbpass': None,
+        'dbname': None,
+        'dbserver': None,
+    }
+
+    # ensure the instance exists and isn't already installed on the minion
+    master = None
+    for host, props in info(client, instance).iteritems():
+        if props['master']:
+            assert not master, 'Detected multiple master instances in LAVA deployment'
+            master = host
+        assert minion != host, 'LAVA instance already deployed on minion'
+
+    assert master, 'No master instance found in LAVA deployment'
+
+    # determine settings needed by looking at master instance
+    cmd = 'cat {0}/instance.conf'.format(args['masterdir'])
+    ret = client.cmd(master, 'cmd.run', [cmd])
+    _update_props(ret[master], args)
+
+    # get the db information
+    cmd = 'cat {0}/etc/lava-server/default_database.conf'.format(args['masterdir'])
+    ret = client.cmd(master, 'cmd.run', [cmd])
+    _update_props(ret[master], args)
+    if not args['dbserver']:
+        args['dbserver'] = args['LAVA_SERVER_IP']
+
+    cmd = ('SKIP_ROOT_CHECK=yes '
+           'LAVA_DB_SERVER={dbserver} LAVA_DB_NAME={dbname} '
+           'LAVA_DB_USER={dbuser} LAVA_DB_PASSWORD={dbpass} '
+           'LAVA_REMOTE_FS_HOST={LAVA_SERVER_IP} '
+           'LAVA_REMOTE_FS_USER={LAVA_SYS_USER} LAVA_REMOTE_FS_DIR={masterdir} '
+           'LAVA_PROXY="{LAVA_PROXY}" LAVA_SERVER_IP={workerip} '
+           '{ldt} installworker -n {instance} 2>&1 | tee /tmp/ldt.log'.format(**args))
+
+    if dry_run:
+        return {minion: 'dry-run: {0}'.format(cmd)}
+
+    ret = client.cmd(minion, 'cmd.run', [cmd], timeout=600)
+
+    # l-d-t ran as root, lets chmod things
+    cmd = 'chown -R instance-manager:instance-manager /srv/lava/instances/{0}/code/*'.format(instance)
+    client.cmd(minion, 'cmd.run', [cmd])
+
+    # now add the pubkey of the minion to the master's list of authorized keys
+    cmd = 'cat /srv/lava/instances/{0}/home/.ssh/id_rsa.pub'.format(instance)
+    pubkey = client.cmd(minion, 'cmd.run', [cmd])
+    pubkey = pubkey[minion].replace('ssh key used by LAVA for sshfs', minion)
+    authorized_keys = '{0}/home/.ssh/authorized_keys'.format(args['masterdir'])
+    client.cmd(master, 'file.append', [authorized_keys, pubkey])
+
+    # salt doens't seem to properly deal with grains that are dynamic.
+    # and calling this only once doesn't get the grain updated
+    # see _grains/lava.py for further details
+    client.cmd(minion, 'saltutil.sync_grains', [])
+    client.cmd(minion, 'saltutil.sync_grains', [])
+
+    return ret