Merge lp:~doanac/lava-lab/lava-scripts into lp:lava-lab

Proposed by Andy Doan on 2013-02-05
Status: Needs review
Proposed branch: lp:~doanac/lava-lab/lava-scripts
Merge into: lp:lava-lab
Diff against target: 380 lines (+341/-0)
7 files modified
_grains/lava.py (+6/-0)
scripts/lava-add-worker (+22/-0)
scripts/lava-start (+20/-0)
scripts/lava-status (+19/-0)
scripts/lava-stop (+22/-0)
scripts/lava-upgrade (+34/-0)
scripts/lava_salt.py (+218/-0)
To merge this branch: bzr merge lp:~doanac/lava-lab/lava-scripts
Reviewer Review Type Date Requested Status
Linaro Validation Team 2013-02-05 Pending
Review via email: mp+146547@code.launchpad.net

Description of the change

start work on some helper functions that should allow us to manage lava deployments using salt

To post a comment you must log in.
Antonio Terceiro (terceiro) wrote :

This looks awesome!

One general comment: you don't need to run l-d-t as root. I did the following
on the vagrant-bootstrap script inside the l-d-t repo:

if ! test -d /srv/lava/instances/development; then
  cd $basedir
  su vagrant -c './lava-deployment-tool setup -nd'
  su vagrant -c './lava-deployment-tool install -nd development'
fi

so you could do something like that, and you won't need to chown/chmod away, or
even add that check on l-d-t to allow running as root.

--
Antonio Terceiro
Software Engineer - Linaro
http://www.linaro.org

Andy Doan (doanac) wrote :

On 02/05, Antonio Terceiro wrote:
> One general comment: you don't need to run l-d-t as root. I did the following
> on the vagrant-bootstrap script inside the l-d-t repo:
>
> if ! test -d /srv/lava/instances/development; then
> cd $basedir
> su vagrant -c './lava-deployment-tool setup -nd'
> su vagrant -c './lava-deployment-tool install -nd development'
> fi

You must be setting up your "vagrant" user to allow password-less sudo?
We don't do that for our instance-manager and it seems a little scare
(to me) to allow that.

> so you could do something like that, and you won't need to chown/chmod away, or
> even add that check on l-d-t to allow running as root.

Antonio Terceiro (terceiro) wrote :

On Tue, Feb 05, 2013 at 06:46:22PM -0000, Andy Doan wrote:
> On 02/05, Antonio Terceiro wrote:
> > One general comment: you don't need to run l-d-t as root. I did the following
> > on the vagrant-bootstrap script inside the l-d-t repo:
> >
> > if ! test -d /srv/lava/instances/development; then
> > cd $basedir
> > su vagrant -c './lava-deployment-tool setup -nd'
> > su vagrant -c './lava-deployment-tool install -nd development'
> > fi
>
> You must be setting up your "vagrant" user to allow password-less sudo?
> We don't do that for our instance-manager and it seems a little scare
> (to me) to allow that.

in that case we could setup password-less sudo for instance-manager
exclusively for running the lava-deployment-tool command.

--
Antonio Terceiro
Software Engineer - Linaro
http://www.linaro.org

Michael Hudson-Doyle (mwhudson) wrote :

Andy Doan <email address hidden> writes:

> Andy Doan has proposed merging lp:~doanac/lava-lab/lava-scripts into lp:lava-lab.
>
> Requested reviews:
> Linaro Validation Team (linaro-validation)
>
> For more details, see:
> https://code.launchpad.net/~doanac/lava-lab/lava-scripts/+merge/146547
>
> start work on some helper functions that should allow us to manage lava deployments using salt
> --
> https://code.launchpad.net/~doanac/lava-lab/lava-scripts/+merge/146547
> You are subscribed to branch lp:lava-lab.
> === modified file '_grains/lava.py'
> --- _grains/lava.py 2013-01-10 16:37:55 +0000
> +++ _grains/lava.py 2013-02-05 04:19:20 +0000
> @@ -9,4 +9,10 @@
> p = os.path.join(instdir, f)
> if os.path.isdir(p):
> insts.append(p)
> + # This seems to be a bug in salt. Even after you reload/resync grains
> + # the module still uses what it read at the time the minion started up.
> + # this makes this dynamic so we don't have to restart salt just to see
> + # a new instance
> + if 'lava_instances' in __grains__:
> + __grains__['lava_instances'] = insts

Thats because grains are supposed to be invariant. From
http://docs.saltstack.org/en/latest/topics/targeting/grains.html:

    It is important to remember that grains are bits of information
    loaded when the salt minion starts, so this information is
    static. This means that the information in grains is unchanging,
    therefore the nature of the data is static. So grains information
    are things like the running kernel, or the operating system.

I think grains-but-varying-things are "pillars", but not completely
sure. Might be worth asking #saltstack IRC...

Andy Doan (doanac) wrote :

On 02/05, Antonio Terceiro wrote:
> in that case we could setup password-less sudo for instance-manager
> exclusively for running the lava-deployment-tool command.

l-d-t runs too many sudo commands to make a sane sudoers entry. Its
really an all or nothing approach. So I'd say we just allow this hack in
l-d-t for now.

Unmerged revisions

69. By Andy Doan on 2013-02-05

minor bug from last commit

make sure we return the most interest part of this function

68. By Andy Doan on 2013-02-05

add support for installing a remote worker instance

this makes adding a new worker node as simple as possible

67. By Andy Doan on 2013-02-04

add some lava scripts to help manage lab

lava-status - shows the nodes contributing to a lava deployment, if
they are running, and which is the master instance

lava-{start/stop} - start/stop each lava-instance contributing to a lava deployment

lava-upgrade - performs an upgrade for a lava-deployment

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file '_grains/lava.py'
2--- _grains/lava.py 2013-01-10 16:37:55 +0000
3+++ _grains/lava.py 2013-02-05 04:19:20 +0000
4@@ -9,4 +9,10 @@
5 p = os.path.join(instdir, f)
6 if os.path.isdir(p):
7 insts.append(p)
8+ # This seems to be a bug in salt. Even after you reload/resync grains
9+ # the module still uses what it read at the time the minion started up.
10+ # this makes this dynamic so we don't have to restart salt just to see
11+ # a new instance
12+ if 'lava_instances' in __grains__:
13+ __grains__['lava_instances'] = insts
14 return {'lava_instances': insts}
15
16=== added directory 'scripts'
17=== added file 'scripts/lava-add-worker'
18--- scripts/lava-add-worker 1970-01-01 00:00:00 +0000
19+++ scripts/lava-add-worker 2013-02-05 04:19:20 +0000
20@@ -0,0 +1,22 @@
21+#!/usr/bin/env python
22+
23+import argparse
24+
25+import lava_salt
26+
27+
28+if __name__ == '__main__':
29+ parser = argparse.ArgumentParser(description=lava_salt.add_worker.__doc__)
30+ parser.add_argument('minion', metavar='<minion>',
31+ help='The host to install the lava worker instance on.')
32+ parser.add_argument('ip', metavar='<ip>',
33+ help='The public IP address for the minion.')
34+ parser.add_argument('instance', metavar='<instance>',
35+ help='The instance name we are creating on the worker')
36+ parser.add_argument('--dry-run', dest='dryrun', action='store_true',
37+ help='Just display what would be changed')
38+ args = parser.parse_args()
39+
40+ client = lava_salt.salt_client()
41+ ret = lava_salt.add_worker(client, args.minion, args.ip, args.instance, args.dryrun)
42+ print ret[args.minion]
43
44=== added file 'scripts/lava-start'
45--- scripts/lava-start 1970-01-01 00:00:00 +0000
46+++ scripts/lava-start 2013-02-05 04:19:20 +0000
47@@ -0,0 +1,20 @@
48+#!/usr/bin/env python
49+
50+import argparse
51+
52+import lava_salt
53+
54+
55+if __name__ == '__main__':
56+ parser = argparse.ArgumentParser(description=lava_salt.start.__doc__)
57+ parser.add_argument('instance', metavar='<instance>',
58+ help='The instance name to start')
59+ args = parser.parse_args()
60+
61+ client = lava_salt.salt_client()
62+ ret = lava_salt.start(client, args.instance)
63+ if ret:
64+ print 'salt started the following instances:'
65+ print ret
66+ else:
67+ print 'Instance already running on all minions'
68
69=== added file 'scripts/lava-status'
70--- scripts/lava-status 1970-01-01 00:00:00 +0000
71+++ scripts/lava-status 2013-02-05 04:19:20 +0000
72@@ -0,0 +1,19 @@
73+#!/usr/bin/env python
74+
75+import argparse
76+
77+import lava_salt
78+
79+
80+if __name__ == '__main__':
81+ parser = argparse.ArgumentParser(description=lava_salt.info.__doc__)
82+ parser.add_argument('instance', metavar='<instance>',
83+ help='The instance name to stop')
84+ args = parser.parse_args()
85+
86+ client = lava_salt.salt_client()
87+
88+ for host, props in lava_salt.info(client, args.instance).iteritems():
89+ running = lava_salt.STATE_STRING[props['running']]
90+ master = props['master']
91+ print '{0}: running({1}) master({2})'.format(host, running, master)
92
93=== added file 'scripts/lava-stop'
94--- scripts/lava-stop 1970-01-01 00:00:00 +0000
95+++ scripts/lava-stop 2013-02-05 04:19:20 +0000
96@@ -0,0 +1,22 @@
97+#!/usr/bin/env python
98+
99+import argparse
100+
101+import lava_salt
102+
103+
104+if __name__ == '__main__':
105+ parser = argparse.ArgumentParser(description=lava_salt.stop.__doc__)
106+ parser.add_argument('--just-workers', dest='just_workers', action='store_true',
107+ help='Just stop worker instances, not the master')
108+ parser.add_argument('instance', metavar='<instance>',
109+ help='The instance name to stop')
110+ args = parser.parse_args()
111+
112+ client = lava_salt.salt_client()
113+ ret = lava_salt.stop(client, args.instance, args.just_workers)
114+ if ret:
115+ print 'salt stopped the following instances:'
116+ print ret
117+ else:
118+ print 'Instance not running on any minions'
119
120=== added file 'scripts/lava-upgrade'
121--- scripts/lava-upgrade 1970-01-01 00:00:00 +0000
122+++ scripts/lava-upgrade 2013-02-05 04:19:20 +0000
123@@ -0,0 +1,34 @@
124+#!/usr/bin/env python
125+
126+import argparse
127+
128+import lava_salt
129+
130+
131+def _indented(buff, indent_char):
132+ indent_char = '\n' + indent_char
133+ return ' ' + indent_char.join(buff.split('\n'))
134+
135+
136+if __name__ == '__main__':
137+ parser = argparse.ArgumentParser(description=lava_salt.upgrade.__doc__)
138+ parser.add_argument('instance', metavar='<instance>',
139+ help='The instance name to upgrade')
140+ parser.add_argument('--dry-run', dest='dryrun', action='store_true',
141+ help='Just display what would be changed')
142+ args = parser.parse_args()
143+
144+ client = lava_salt.salt_client()
145+ m_ret, w_ret = lava_salt.upgrade(client, args.instance, args.dryrun)
146+
147+ print 'Master:'
148+ for host, msg in m_ret.iteritems():
149+ print ' {0}:'.format(host)
150+ print ' upgrade:\n{0}'.format(_indented(msg, ' |'))
151+
152+ print '\nWorkers:'
153+ for host, rets in w_ret.iteritems():
154+ print ' {0}:'.format(host)
155+ print ' stop:\n{0}'.format(_indented(rets['stop'], ' |'))
156+ print ' upgrade:\n{0}'.format(_indented(rets['upgrade'], ' |'))
157+ print ' start:\n{0}'.format(_indented(rets['start'], ' |'))
158
159=== added file 'scripts/lava_salt.py'
160--- scripts/lava_salt.py 1970-01-01 00:00:00 +0000
161+++ scripts/lava_salt.py 2013-02-05 04:19:20 +0000
162@@ -0,0 +1,218 @@
163+import salt.client
164+
165+RUNNING = 0
166+STOPPED = 1
167+UNKNOWN = 3
168+
169+STATE_STRING = {
170+ RUNNING: 'running',
171+ STOPPED: 'stopped',
172+ UNKNOWN: '???',
173+}
174+
175+LDT = '/home/instance-manager/lava-deployment-tool/lava-deployment-tool'
176+
177+
178+def salt_client():
179+ return salt.client.LocalClient()
180+
181+
182+def info(client, instance):
183+ """
184+ Shows whether an instance of LAVA is running or not on its configured hosts.
185+ """
186+ cmd = 'status lava-instance LAVA_INSTANCE={0}'.format(instance)
187+ inst_path = '/srv/lava/instances/{0}'.format(instance)
188+ worker_file = '{0}/sbin/mount-masterfs'.format(inst_path)
189+
190+ inf = {}
191+
192+ ret = client.cmd('*', 'grains.item', ['lava_instances'])
193+ for k, v in ret.iteritems():
194+ if inst_path in v:
195+ ret = client.cmd(k, 'cmd.run', [cmd])
196+ running = UNKNOWN
197+ if ret[k] == 'status: Unknown instance: %s' % instance:
198+ running = STOPPED
199+ elif ret[k] == 'lava-instance (%s) start/running' % instance:
200+ running = RUNNING
201+
202+ ret = client.cmd(k, 'file.file_exists', [worker_file])
203+ master = not ret[k]
204+
205+ inf[k] = {'running': running, 'master': master}
206+ return inf
207+
208+
209+def stop(client, instance, just_workers=False):
210+ """
211+ Issues a command to stop a given instance name on all minions where the
212+ LAVA instance appears to be running.
213+ """
214+ cmd = 'stop lava-instance LAVA_INSTANCE={0}'.format(instance)
215+
216+ hosts = []
217+ for host, props in info(client, instance).iteritems():
218+ if props['running'] != STOPPED:
219+ if not just_workers or not props['master']:
220+ hosts.append(host)
221+
222+ if len(hosts):
223+ return client.cmd(hosts, 'cmd.run', [cmd], expr_form='list')
224+
225+
226+def start(client, instance):
227+ """
228+ Issues a command to start a given instance name on all minions where the
229+ LAVA instance appears to not be running.
230+ """
231+ cmd = 'start lava-instance LAVA_INSTANCE={0}'.format(instance)
232+
233+ hosts = []
234+ for host, props in info(client, instance).iteritems():
235+ if props['running'] != RUNNING:
236+ hosts.append(host)
237+
238+ if len(hosts):
239+ return client.cmd(hosts, 'cmd.run', [cmd], expr_form='list')
240+
241+
242+def upgrade(client, instance, dry_run=True):
243+ """
244+ Runs lava-deployment-tool upgrade for a LAVA setup. It first shuts down
245+ each worker node instance. Then it runs lava-deployment-tool upgrade on
246+ the master. Lastly, it runs lava-deployment-tool upgradeworker on the
247+ worker nodes.
248+ """
249+ timeout = 300 # 5 minutes
250+ workers = []
251+ master = None
252+
253+ for host, props in info(client, instance).iteritems():
254+ if props['master']:
255+ assert not master, 'Detected multiple master instances in LAVA deployment'
256+ master = host
257+ else:
258+ workers.append(host)
259+
260+ assert master, 'No master instance found in LAVA deployment'
261+
262+ w_ret = {}
263+ for h in workers:
264+ w_ret[h] = {'stop': 'dry-run', 'upgrade': 'dry-run', 'start': 'dry-run'}
265+
266+ if dry_run:
267+ m_ret = {master: 'dry-run: upgrade master'}
268+ return m_ret, w_ret
269+
270+ # first stop workers. This prevents a DB error if the upgrade changes
271+ # the schema.
272+ ret = stop(client, instance, True)
273+ if ret:
274+ for host, msg in ret.iteritems():
275+ w_ret[host]['stop'] = msg
276+
277+ # now upgrade the master node
278+ cmd = 'SKIP_ROOT_CHECK=yes {0} upgrade {1}'.format(LDT, instance)
279+ m_ret = client.cmd(master, 'cmd.run', [cmd], timeout=timeout)
280+
281+ # now upgrade the workers
282+ cmd = 'SKIP_ROOT_CHECK=yes {0} upgradeworker {1}'.format(LDT, instance)
283+ if len(workers):
284+ ret = client.cmd(workers, 'cmd.run', [cmd], timeout=timeout, expr_form='list')
285+ for host, msg in ret.iteritems():
286+ w_ret[host]['upgrade'] = msg
287+
288+ ret = start(client, instance)
289+ if ret:
290+ for host, msg in ret.iteritems():
291+ if host in w_ret:
292+ w_ret[host]['start'] = msg
293+
294+ # last thing: l-d-t ran as root, lets chmod things
295+ cmd = 'chown -R instance-manager:instance-manager /srv/lava/instances/{0}/code/*'.format(instance)
296+ client.cmd(workers + [master], 'cmd.run', [cmd], expr_form='list')
297+
298+ return m_ret, w_ret
299+
300+
301+def _update_props(inifile_content, props):
302+ for line in inifile_content.split('\n'):
303+ if not line.strip().startswith('#'):
304+ key, val = line.split('=')
305+ if key in props:
306+ props[key] = val.replace("'", '')
307+
308+
309+def add_worker(client, minion, minion_ip, instance, dry_run=True):
310+ """
311+ Creates a new lava workernode on a salt-minion.
312+ """
313+
314+ args = {
315+ 'LAVA_SERVER_IP': None,
316+ 'LAVA_SYS_USER': None,
317+ 'LAVA_PROXY': None,
318+ 'masterdir': '/srv/lava/instances/{0}'.format(instance),
319+ 'workerip': minion_ip,
320+ 'ldt': LDT,
321+ 'instance': instance,
322+ 'dbuser': None,
323+ 'dbpass': None,
324+ 'dbname': None,
325+ 'dbserver': None,
326+ }
327+
328+ # ensure the instance exists and isn't already installed on the minion
329+ master = None
330+ for host, props in info(client, instance).iteritems():
331+ if props['master']:
332+ assert not master, 'Detected multiple master instances in LAVA deployment'
333+ master = host
334+ assert minion != host, 'LAVA instance already deployed on minion'
335+
336+ assert master, 'No master instance found in LAVA deployment'
337+
338+ # determine settings needed by looking at master instance
339+ cmd = 'cat {0}/instance.conf'.format(args['masterdir'])
340+ ret = client.cmd(master, 'cmd.run', [cmd])
341+ _update_props(ret[master], args)
342+
343+ # get the db information
344+ cmd = 'cat {0}/etc/lava-server/default_database.conf'.format(args['masterdir'])
345+ ret = client.cmd(master, 'cmd.run', [cmd])
346+ _update_props(ret[master], args)
347+ if not args['dbserver']:
348+ args['dbserver'] = args['LAVA_SERVER_IP']
349+
350+ cmd = ('SKIP_ROOT_CHECK=yes '
351+ 'LAVA_DB_SERVER={dbserver} LAVA_DB_NAME={dbname} '
352+ 'LAVA_DB_USER={dbuser} LAVA_DB_PASSWORD={dbpass} '
353+ 'LAVA_REMOTE_FS_HOST={LAVA_SERVER_IP} '
354+ 'LAVA_REMOTE_FS_USER={LAVA_SYS_USER} LAVA_REMOTE_FS_DIR={masterdir} '
355+ 'LAVA_PROXY="{LAVA_PROXY}" LAVA_SERVER_IP={workerip} '
356+ '{ldt} installworker -n {instance} 2>&1 | tee /tmp/ldt.log'.format(**args))
357+
358+ if dry_run:
359+ return {minion: 'dry-run: {0}'.format(cmd)}
360+
361+ ret = client.cmd(minion, 'cmd.run', [cmd], timeout=600)
362+
363+ # l-d-t ran as root, lets chmod things
364+ cmd = 'chown -R instance-manager:instance-manager /srv/lava/instances/{0}/code/*'.format(instance)
365+ client.cmd(minion, 'cmd.run', [cmd])
366+
367+ # now add the pubkey of the minion to the master's list of authorized keys
368+ cmd = 'cat /srv/lava/instances/{0}/home/.ssh/id_rsa.pub'.format(instance)
369+ pubkey = client.cmd(minion, 'cmd.run', [cmd])
370+ pubkey = pubkey[minion].replace('ssh key used by LAVA for sshfs', minion)
371+ authorized_keys = '{0}/home/.ssh/authorized_keys'.format(args['masterdir'])
372+ client.cmd(master, 'file.append', [authorized_keys, pubkey])
373+
374+ # salt doens't seem to properly deal with grains that are dynamic.
375+ # and calling this only once doesn't get the grain updated
376+ # see _grains/lava.py for further details
377+ client.cmd(minion, 'saltutil.sync_grains', [])
378+ client.cmd(minion, 'saltutil.sync_grains', [])
379+
380+ return ret

Subscribers

People subscribed via source and target branches