Merge lp:~abentley/lp-dev-utils/testr-remote into lp:lp-dev-utils

[Aaron Bentley]

Add testr-remote.

Add testr-remote to lp-dev-utils

[Benji York (benji)]

This branch looks good. Thanks for working on this. I have approved
the branch but I think you will want to address a couple of the things
below before landing it.

Do you really want the $() wrapping ./list-servers on line 62 of the
diff?

I do not understand the third and fourth caret characters in the regex
on line 291 of the diff). As far as I can tell they are unnecessary.

The commented-out code on line 334 and 343 of the diff should be
removed.

I doubt you intend to prescribe the location of the testr binary so
precisely on line 356 of the diff.

The exit(2) on line 347 of the diff has prompted me to climb the nearest
snow-covered peak and meditate for three days. I have yet to discover
the deep motivations that resulted in exit(2) instead of exit(1), but I
thank you for the opportunity to start on the path toward enlightenment.
Next I will try walking barefoot across a desert, that or you could add
a comment.

review: Approve (code)

Unmerged revisions

118. By Aaron Bentley on 2012-12-11

Doc tweaks.

117. By Aaron Bentley on 2012-12-11

Migrate ignore rules.

116. By Aaron Bentley on 2012-12-11

Added testr-remote.

115. By Aaron Bentley on 2012-11-28

Allow specifying the number of instances to run.

=== modified file '.bzrignore'
--- .bzrignore	2012-08-09 04:39:28 +0000
+++ .bzrignore	2012-12-11 16:13:28 +0000
@@ -8,3 +8,6 @@
 parts
 bin
 develop-eggs
+testr-remote/.testrepository
+testr-remote/servers
+testr-remote/server-locks/*

=== modified file 'cs-test'
--- cs-test	2012-11-09 16:37:50 +0000
+++ cs-test	2012-12-11 16:13:28 +0000
@@ -27,4 +27,9 @@
 su ubuntu -c 'make schema'
 EOT
 . $HOME/.canonistack/novarc
-euca-run-instances ami-00000168 -t m1.small -k ${OS_USERNAME}_$OS_REGION_NAME --user-data-file $cloud_init_script
+if [ -z "$1" ]; then
+  server_count=1;
+else
+  server_count=$1;
+fi
+euca-run-instances -n $server_count ami-00000168 -t m1.small -k ${OS_USERNAME}_$OS_REGION_NAME --user-data-file $cloud_init_script

=== added directory 'testr-remote'
=== added file 'testr-remote/.testr.conf'
--- testr-remote/.testr.conf	1970-01-01 00:00:00 +0000
+++ testr-remote/.testr.conf	2012-12-11 16:13:28 +0000
@@ -0,0 +1,4 @@
+[DEFAULT]
+test_command=./test-remote --subunit $IDOPTION $LISTOPT
+test_id_option=--load-list $IDFILE
+test_list_option=--list-tests

=== added file 'testr-remote/README'
--- testr-remote/README	1970-01-01 00:00:00 +0000
+++ testr-remote/README	2012-12-11 16:13:28 +0000
@@ -0,0 +1,53 @@
+How to use
+==========
+
+You'll need a version of test-repository that supports the --concurrency parameter.  Trunk will do.
+
+To use testr-remote, you first need to initialize the testr-remote directory as a test repository.
+
+To create your instances, use cs-test.
+
+Optionally, use instance-keys to add all the instances' keys to
+.ssh/known-hosts.  This is better than simply accepting the key proposed by
+ssh, because it verifies the key against the fingerprint from the console
+output.
+
+Wait for the instances to finish installing Launchpad.  This usually takes
+about half an hour, so it's good to do this before you need to run any tests.
+You can use a parallel ssh tool (e.g. cluster ssh) to tell you when they're
+done, because the load average will be near zero.
+
+Create a file called 'servers' in the tree root::
+
+  $(./list-servers) > servers
+
+This is just a list of the server names that can be used for ssh.  You can edit
+this list if you like.
+
+Run the tests using "testr-remote".  It's a wrapper around "testr-run" that
+provides the --parallel and --concurrency options, and takes the same
+parameters.  e.g. "testr-remote -- -t test_project" will run
+"bin/test -t test_project" remotely.  (Any parameters that will be passed on to
+bin/test need to be double-escaped.  This is a bug in testr that I have a local
+fix for.)
+
+When you commit new changes, you can use "update-all" to update all servers to
+the newer version.
+
+You can also use switch-all to switch all servers to a new branch.  (This does
+not re-run make).
+
+You can use "euca-terminate-instances $(./list-instance-ids)" to kill all the
+instances when you're done.
+
+How it works
+============
+testr requires a command that it can run to get subunit output, and it needs to
+run it multiple times simultaneously for concurrency.  test-remote is that
+command.  It provides subunit output by ssh-ing into the remote machine and
+running bin/test --subunit.  It uses the server-locks directory to acquire a
+free server.  If none are available, it blocks until one becomes available.
+
+When --parallel is specified, testr supplies a local file containing a list of
+test-ids to use.  test-remote intercepts that option, copies the file to the
+remote server, and supplies the remote path to bin/test.

=== added file 'testr-remote/instance-key'
--- testr-remote/instance-key	1970-01-01 00:00:00 +0000
+++ testr-remote/instance-key	2012-12-11 16:13:28 +0000
@@ -0,0 +1,142 @@
+#!/usr/bin/env python
+
+__metaclass__ = type
+
+import os.path
+from optparse import OptionParser
+import re
+from subprocess import Popen, PIPE
+import sys
+from tempfile import NamedTemporaryFile
+import time
+
+
+class Instance:
+
+    def __init__(self, instance_id, ip, servername):
+
+        self.instance_id = instance_id
+        self.ip = ip
+        self.servername = servername
+        self.instance_fingerprint = None
+        self.host_key = None
+
+    def find_fingerprint(self):
+        proc = Popen(['euca-get-console-output', self.instance_id],
+                     stdout=PIPE)
+        lines = iter(proc.stdout)
+        instance_key = None
+        for line in lines:
+            match = re.match('Generating public/private rsa key pair', line)
+            if match is not None:
+                [x for x in zip(lines, range(2))]
+                self.instance_fingerprint = lines.next()[:3*16-1]
+        proc.wait()
+
+
+def iter_instance():
+    proc = Popen(['euca-describe-instances'], stdout=PIPE)
+    try:
+        for num, line in enumerate(proc.stdout):
+            if not line.startswith('INSTANCE'):
+                continue
+            field = line.split('\t')
+            yield Instance(field[1], field[16], field[3])
+    finally:
+        proc.wait()
+
+
+def get_argmap(identifiers):
+    argmap = {}
+    for instance in iter_instance():
+        argmap[instance.instance_id] = instance
+        argmap[instance.ip] = instance
+        argmap[instance.servername] = instance
+    try:
+        return (argmap[identifier] for identifier in identifiers)
+    except KeyError as e:
+        sys.stderr.write('Cannot find %s\n' % e)
+        sys.exit(1)
+
+
+def find_host_keys(instances):
+    ips = [instance.ip for instance in instances]
+    proc = Popen(
+        ['ssh', 'chinstrap.canonical.com', 'ssh-keyscan'] + ips, stdout=PIPE,
+        stderr=PIPE)
+    keys = proc.communicate()[0]
+    key_map = {}
+    for key in keys.splitlines(True):
+        key_map[key.split(' ')[0]] = key
+    for instance in instances:
+        instance.host_key = key_map[instance.ip]
+
+
+def get_key_fingerprint(key):
+    with NamedTemporaryFile() as f:
+        f.write(key)
+        f.flush()
+        proc = Popen(['ssh-keygen', '-lf', f.name], stdout=PIPE)
+        host_fingerprint = proc.communicate()[0]
+        return host_fingerprint.split(' ')[1]
+
+
+def remove_key(known_hosts, hostname):
+    proc = Popen(['ssh-keygen', '-R', hostname, '-f', known_hosts],
+                 stderr=PIPE)
+    proc.communicate()
+
+
+def update_key(instance):
+    known_hosts = os.path.join(os.environ['HOME'], '.ssh/known_hosts')
+    hostnames = [instance.ip, instance.servername,
+                 instance.servername + '.canonistack']
+    for name in hostnames:
+        remove_key(known_hosts, name)
+    with file(known_hosts, 'ab') as f:
+        for name in hostnames:
+            f.write(instance.host_key.replace(instance.ip, name))
+
+def validate_instances(instances):
+    for instance in instances:
+        host_fingerprint = get_key_fingerprint(instance.host_key)
+        yield instance, bool(host_fingerprint == instance.instance_fingerprint)
+
+if __name__ == '__main__':
+    parser = OptionParser()
+    parser.add_option('--poll', action='store_true')
+    options, identifiers = parser.parse_args(sys.argv[1:])
+
+    if len(identifiers) > 0:
+        instances = list(get_argmap(identifiers))
+    else:
+        instances = list(iter_instance())
+
+    for instance in instances:
+        if options.poll:
+            attempt_count = 10
+        else:
+            attempt_count = 1
+        for attempt in range(attempt_count):
+            if attempt != 0:
+                sys.stdout.write('.')
+                sys.stdout.flush()
+                time.sleep(5)
+            instance.find_fingerprint()
+            if instance.instance_fingerprint is not None:
+                break
+        if attempt != 0:
+            sys.stdout.write('\n')
+
+        if instance.instance_fingerprint is None:
+            sys.stderr.write('Cannot find key in console output for %s\n' %
+                             instance.instance_id)
+            sys.exit(1)
+
+    find_host_keys(instances)
+    for instance, valid in validate_instances(instances):
+        if valid:
+            print 'Match.  Adding key for %s.' % instance.ip
+            update_key(instance)
+        else:
+            print "Mismatch.  Not adding key."

=== added file 'testr-remote/list-instance-ids'
--- testr-remote/list-instance-ids	1970-01-01 00:00:00 +0000
+++ testr-remote/list-instance-ids	2012-12-11 16:13:28 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+euca-describe-instances | grep '^INSTANCE' | cut -f2

=== added file 'testr-remote/list-servers'
--- testr-remote/list-servers	1970-01-01 00:00:00 +0000
+++ testr-remote/list-servers	2012-12-11 16:13:28 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+euca-describe-instances | grep '^INSTANCE' | cut -f4

=== added directory 'testr-remote/server-locks'
=== added file 'testr-remote/switch-all'
--- testr-remote/switch-all	1970-01-01 00:00:00 +0000
+++ testr-remote/switch-all	2012-12-11 16:13:28 +0000
@@ -0,0 +1,11 @@
+#!/bin/sh
+set -e
+push_branch=$(bzr config -d $1 push_location|sed 's/bzr+ssh:\/\/bazaar.launchpad.net\//lp:/')
+revision_id=$(bzr revision-info $1|cut -f2 -d' ')
+echo $push_branch
+echo $revision_id
+servers=$(cat servers)
+for server in $servers; do
+  echo Switching $server ...
+  ssh $server bzr switch -d /opt/launchpad/launchpad $push_branch -r revid:$revision_id
+done

=== added file 'testr-remote/test-remote'
--- testr-remote/test-remote	1970-01-01 00:00:00 +0000
+++ testr-remote/test-remote	2012-12-11 16:13:28 +0000
@@ -0,0 +1,71 @@
+#!/usr/bin/env python
+__metaclass__ = type
+
+import argparse
+import errno
+import fcntl
+import os.path
+from contextlib import contextmanager
+import re
+import subprocess
+import sys
+import time
+
+def escape(iput):
+    return re.sub('[^-^a-z^A-Z^_.]', lambda m: '\\'+m.group(0), iput)
+
+
+class ServerSet:
+
+    def __init__(self, servers):
+        self.servers = servers
+
+    @classmethod
+    def from_file(cls):
+        return cls(file('servers').read().splitlines())
+
+    @contextmanager
+    def selected_server(self):
+        while True:
+            for server in self.servers:
+                filename = os.path.join('server-locks', server)
+                with file(filename, 'wb') as lockfile:
+                    try:
+                        fcntl.lockf(lockfile, fcntl.LOCK_EX | fcntl.LOCK_NB)
+                    except IOError as e:
+                        if e.errno != errno.EWOULDBLOCK:
+                            raise
+                    else:
+                        try:
+                            yield server
+                        finally:
+                            fcntl.lockf(lockfile, fcntl.LOCK_UN)
+                            return
+            else:
+                time.sleep(1)
+
+
+parser = argparse.ArgumentParser(description='Run some tests remotely.',
+                                 add_help=False)
+parser.add_argument('--load-list', nargs=1, type=argparse.FileType())
+known, unknown = parser.parse_known_args()
+args = [escape(arg) for arg in unknown]
+try:
+    with ServerSet.from_file().selected_server() as server:
+        command = ['ssh', server]
+        if known.load_list is not None:
+            command.extend(['id_list=$(mktemp);',
+        #        'echo $id_list;',
+                'cat > $id_list;'])
+            args = ['--load-list', '$id_list'] + args
+            stdin = known.load_list[0]
+        else:
+            stdin = None
+        command.extend(['cd /opt/launchpad/launchpad;', 'xvfb-run',
+                        'bin/test'])
+        command.extend(args)
+        #print command
+        sys.exit(subprocess.call(command, stdin=stdin))
+except Exception as e:
+    sys.stderr.write(str(e) + '\n')
+    sys.exit(2)

=== added file 'testr-remote/testr-remote'
--- testr-remote/testr-remote	1970-01-01 00:00:00 +0000
+++ testr-remote/testr-remote	2012-12-11 16:13:28 +0000
@@ -0,0 +1,4 @@
+#!/bin/sh
+server_count=$(wc -l servers | cut -f 1 -d ' ')
+concurrency=$((server_count*2))
+~/hacking/testrepository/testr run --parallel --concurrency $concurrency "$@"

=== added file 'testr-remote/update-all'
--- testr-remote/update-all	1970-01-01 00:00:00 +0000
+++ testr-remote/update-all	2012-12-11 16:13:28 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+for server in $(cat servers); do ssh $server bzr update /opt/launchpad/launchpad; done