Merge lp:~christophe-lyon/cbuild/tools-fix-svn-name into lp:cbuild

[Christophe Lyon]

Attempt to avoid generation of ~/var/snapshots/svn* files and associated scheduler jobs.

[lp:~christophe-lyon/cbuild/tools-fix-svn-name]

108. By Christophe Lyon on 2014-07-01

Fix previous commit.

Unmerged revisions

108. By Christophe Lyon on 2014-07-01

Fix previous commit.

107. By Christophe Lyon on 2014-04-18

Do not generate tarball and delta if no name parameter is provided.

106. By Renato Golin on 2013-06-06

Adding Clang/Extra/RT to LLVM update

105. By Yvan Roux on 2013-06-07

Add gcc-linaro-4.8 branch in cron-often polling task.

104. By Linaro Toolchain Builder on 2013-05-29

Add support for toolchain64 host and GCC Linaro 4.8 branch

103. By Linaro Toolchain Builder on 2013-05-29

Update find to take gcc-4.8 branches into account.

102. By Christophe Lyon on 2013-04-11

Fix bug #1156536

101. By Michael Hope on 2013-01-28

Fix taker stuff

100. By Michael Hope on 2013-01-28

Make tabulate work on all GCC results.

99. By Michael Hope on 2013-01-28

Update to use gcc-google-4.7 and not gcc-google-4.6

=== added file 'MAINTAIN.txt'
--- MAINTAIN.txt	1970-01-01 00:00:00 +0000
+++ MAINTAIN.txt	2014-07-01 15:58:40 +0000
@@ -0,0 +1,92 @@
+cbuild maintenance tasks
+========================
+
+Users
+-----
+ * cbuild - pulls keys from http://launchpad.net/~cbuild
+
+Things to watch
+---------------
+
+Disks:
+
+ * / for free space
+ * /space for free space
+
+`~/var/snapshots/gcc-4.8?svn*`:
+
+ * Youngest is no more than 3 days old
+ * Check cbuild-tools/launcher.sh on fault
+
+`~/var/tcwg-web/*.pickle`:
+
+ * No more than four hours old
+ * Check lib/tcwg-web/update.sh on fault
+
+Common tasks
+------------
+
+Where have all my directories gone?
+
+ * ~/lib/cbuild
+ * ~/var/snapshots
+ * ~/public_html/build
+
+How do I spawn a release build via the shell?
+
+ * Use gcc-release-process.sh! else:
+ * scp gcc-linaro-4.7-2012.12.tar.bz2 [e-mail]:~/var/snapshots
+ * ssh [e-mail]
+ * ~/lib/cbuild-tools/spawn.sh ~/var/snapshots/gcc-linaro-4.7-2012.tar.bz2
+
+A fault with frequent tasks like merge reqeusts, tip builds or data
+being out of date?
+
+ * Check ~/lib/cbuild-tools/cron-often.sh
+
+A fault with daily tasks like upstream builds?
+
+ * Check ~/lib/cbuild-tools/launcher.sh
+
+A fault with one of the helpers?
+
+ * sudo service tcwg-web stop
+ * cd lib/tcwg-web
+ * Uncommet 'development = True' in twg-web.ini
+ * python index.py
+ * Run the request, see the backtrace
+
+Version out of date on one of the upstream builds?
+
+ * Edit launcher.sh
+
+Want to build/track a new upstream?
+
+ * git clone upstream-url  ~/repos/decent-name
+ * Add a new dow line to launcher.sh
+
+Want to track a new Linaro series?
+
+ * cd ~/repos/gcc-linaro
+ * bzr branch lp:gcc-linaro/4.8
+ * Add a new line to cron-often.sh
+ * Consider pushing a new gcc-linaro-4.8+bzr12345.{tar,tar.xz} to ~/var/snapshots/base
+
+Want to propogate a cbuild update to the slaves?
+
+ * cd ~/lib/cbuild
+ * bzr pull
+
+Want to add a new build queue?
+
+ * cd ~/var/scheduler/queue
+ * mkdir queue-name
+ * echo host1 host2 host3 > queue-name/hosts.txt
+ * echo config-fragments-if-any > queue-name/template.txt
+ * cd ../spawn/default
+ * ln -s ../../queue/queue-name
+
+Want to delete a job from the queue?
+
+ * cd ~/var/scheduler/queue
+ * `rm */job-name.job`
 
=== added file 'aarch64-merger.sh'
--- aarch64-merger.sh	1970-01-01 00:00:00 +0000
+++ aarch64-merger.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,172 @@
+#!/bin/bash
+#
+# Automatically merge and create a merge request for any new aarch64
+# commits.
+#
+# 
+
+set -e
+
+# Path to Linaro GCC 4.7
+tip=4.7
+# Path to the ARM bzr branch
+from=arm-aarch64-4.7
+# Fill this in with the last merged bzr revno
+# Example: bzr log 4.7 shows r193293..r193328.  bzr log arm-* --show-ids shows that
+# r193328 is bzr116407.
+current=116423
+
+get_revision() {
+    echo $(bzr log -c $1 --show-ids --short $2 | awk -F: '/revision-id:/ {print $NF}')
+}
+
+is_fsf_merge() {
+    (bzr log -c $1 -p $2 | grep -qF "+++ gcc/DATESTAMP") && echo y
+}
+
+all_revnos() {
+    bzr log -r $stop.. --log-format=line $1 | awk -F: '{print $1;}'
+}
+
+extract_changelog() {
+    # Pull the ChangeLog out
+    inside=
+
+    while read i; do
+        if [[ "$i" =~ ^=== ]]; then inside=; fi
+
+        if [ -n "$inside" ] && [[ "$i" =~ ^\+(.+) ]]; then
+            echo "${BASH_REMATCH[1]}"
+        fi
+
+        if [[ "$i" =~ ^\+\+\+\ .+/ChangeLog\.aarch64 ]]; then inside=y; fi
+
+    done < /tmp/bzr$revno.diff > /tmp/bzr$revno.changes
+}
+
+search_back() {
+    # Search back to the first non-merge
+    for i in $all; do
+        if [ -z "$(is_fsf_merge $i $from)" ]; then
+            last=$i
+            break
+        fi
+    done
+}
+
+search_forward() {
+    # Search forward to the first non-merge
+    for i in $(seq $((current+1)) 1 $last); do
+        if [ -z "$(is_fsf_merge $i $from)" ]; then
+            first=$i
+            break
+        fi
+    done
+}
+
+dir=`mktemp -td merger.XXXXXXXXXX`
+trap "rm -rf $dir" EXIT
+
+# End point for any scans
+stop=116380
+
+all=$(all_revnos $from)
+
+search_back
+search_forward
+
+sfirst=$(get_revision $first $from)
+slast=$(get_revision $last $from)
+
+name="merge-from-aarch64-4.7-r$sfirst-r$slast"
+
+# See if the merge request already exists
+if false && wget -q -O- https://code.launchpad.net/gcc-linaro | grep -q -F $name; then
+    echo "Branch $name already exists, nothing to do."
+    exit 0
+fi
+
+rm -rf $name
+bzr branch -q --hardlink $tip $name
+
+realname="Michael Hope"
+email="[e-mail]"
+login=$(bzr lp-login)
+
+short="Merge from FSF arm/aarch64-4.7-branch r$sfirst..r$slast."
+
+echo -e "$(date +%Y-%m-%d)  $realname  <$email>\n" > $dir/log
+echo -e "\t$short\n" >> $dir/log
+
+# Go through and merge each revision
+for revno in $(seq $first 1 $last); do
+    if [ -n "$(is_fsf_merge $revno $from)" ]; then
+        echo "Skipping the FSF merge at $revno"
+        continue
+    fi
+
+    svn=$(get_revision $revno $from)
+    stem=$dir/bzr$revno
+
+    echo "Merging $svn (bzr$revno)"
+    # bzr diff returns 1 on changed...
+    (cd $from && bzr diff -c $revno || true) > $stem.diff
+
+    if ! (cd $name && bzr patch $stem.diff); then
+        read -p "Conflicts found.  Fix in $name then press enter > "
+    fi
+
+    # Nuke any patch reminants
+    find $name -name *.orig -exec rm {} \;
+    find $name -name *.rej -exec rm {} \;
+
+    # Add any missing files.  bzr patch should have done this already
+    # but doesn't on conflict.
+    (cd $name && bzr add)
+
+    # Pull out the commit message
+    (cd $from && bzr log -c $revno) > $stem.log
+
+    inside=
+    while read i; do
+        if [ -n "$inside" ]; then
+            trimmed=$(echo "$i")
+            echo "$trimmed"
+        fi
+
+        if [[ "$i" =~ ^message: ]]; then inside=y; fi
+    done < $stem.log > $stem.message
+
+    # Commit each as a revision to make future tracking easier.  The
+    # merge will roll this up into one.
+    echo -e "Backport $from r$svn.\n" > $stem.commit
+    cat $stem.message >> $stem.commit
+
+    (cd $name && bzr commit -F $stem.commit)
+
+    echo -e "\tBackport $from r$svn:" >> $dir/log
+    while read i; do
+        echo -e "\t$i"
+    done < $stem.message >> $dir/log
+    
+    echo >> $dir/log
+done
+
+# Upate the ChangeLog and commit
+cat $dir/log $name/ChangeLog.linaro > $dir/new
+mv $dir/new $name/ChangeLog.linaro
+
+cd $name
+bzr commit -m"$short"
+bzr push "lp:~$login/gcc-linaro/$name"
+
+# propose-merge always runs an editor for the message.  Make a fake
+# editor that sets the text.
+cat > $dir/editor <<EOF
+#!/bin/bash
+cp $dir/log \$1
+EOF
+
+chmod +x $dir/editor
+
+VISUAL=$dir/editor bzr lp-propose-merge -m"$short"
 
=== added file 'addtestdiffs.py'
--- addtestdiffs.py	1970-01-01 00:00:00 +0000
+++ addtestdiffs.py	2014-07-01 15:58:40 +0000
@@ -0,0 +1,133 @@
+#!/usr/bin/python
+"""Record the difference between the test results of this build and
+it's ancestor or predecessor.
+"""
+
+import sys
+import re
+import os
+import pprint
+import subprocess
+import logging
+import time
+
+import findancestor
+import findlogs
+
+TEMPLATE = """Difference in testsuite results between:
+ %(current)s build %(buildid)s
+and the one before it:
+ %(other)s build %(otherbuildid)s
+
+------
+%(diff)s"""
+
+def make_testsuite_diff(left, right):
+    # PENDING: This is bad
+    return subprocess.check_output(['./difftests.sh', left, right])
+
+def make_benchmarks_diff(left, right, suffix):
+    return subprocess.check_output(['../linaro-toolchain-benchmarks/scripts/diffbench.py',
+                                    '%s/benchmarks-%s.txt' % (left, suffix),
+                                    '%s/benchmarks-%s.txt' % (right, suffix)
+                                    ])
+
+def run(allfiles, snapshots, builds, check, final, make_diff):
+    now = time.time()
+
+    # Make the search a bit easier by only keeping the .txt files
+    filtered = [x for x in allfiles if '.txt' in x[-1] or check in x[-1]]
+
+    # And cache by directory
+    directories = {}
+
+    for f in filtered:
+        directory = '/'.join(f[:-1])
+        directories.setdefault(directory, []).append(f)
+
+    prefiltered = findlogs.prefilter(allfiles, check)
+
+    # Find all builds that have finished
+    for build in [x for x in filtered if x[-1] == 'finished.txt']:
+        match = '/'.join(build[:-1])
+
+        # Skip if the build is too old.  Either we have a result or
+        # it's never coming in
+        try:
+            age = (now - os.path.getmtime('%s/%s/finished.txt' % (builds, match))) / (60*60*24)
+
+            if age >= 30:
+                continue
+        except OSError:
+            continue
+
+        # Find all of the files in the same directory
+        siblings = directories[match]
+
+        all = ' '.join(x[-1] for x in siblings)
+
+        # Are there any .sum style results?
+        if check not in all:
+            continue
+
+        # See if the diff already exists
+        if ' %s' % final in all:
+            continue
+
+        # Find the ancestor or predecessor
+        current = build[1]
+        other = findancestor.find(allfiles, current, snapshots)
+
+        if not other:
+            logging.info('No known ancestor to %s' % current)
+            continue
+
+        # Find the corresponding build
+        logs = ['/'.join(x) for x in siblings if check in x[-1]]
+        assert logs
+        log = logs[-1]
+
+        try:
+            otherlogs = findlogs.find(prefiltered, log, other)
+        except Exception, ex:
+            logging.info('Error %s while finding logs on %s vs %s' % (ex, log, other))
+            otherlogs = None
+
+        if not otherlogs:
+            logging.debug("Can't find %s of %s" % (log, other))
+            continue
+
+        other, otherbuildid = otherlogs
+        buildid = build[-2]
+        diff = None
+
+        try:
+            logging.debug('Generating a diff between predecessor %s/%s and %s' % (other, otherbuildid, os.path.dirname(log)))
+
+            left = '%s/%s/logs/%s' % (builds, other, otherbuildid)
+            right = os.path.join(builds, os.path.dirname(log))
+            diff = make_diff(left, right)
+
+        except Exception, ex:
+            logging.error(ex)
+
+        if diff == None:
+            continue
+
+        with open('%s/%s/%s' % (builds, match, final), 'w') as f:
+            print >> f, TEMPLATE % locals()
+
+def main():
+#    logging.basicConfig(level=logging.DEBUG)
+    allfiles, snapshots, builds = sys.argv[1:]
+
+    with open(allfiles) as f:
+        allf = [x.strip().split(os.sep) for x in f.readlines()]
+
+    run(allf, snapshots, builds, '.sum', 'testsuite-diff.txt', make_testsuite_diff)
+
+    for suffix in 'spec2000 eembc eembc_office'.split():
+        run(allf, snapshots, builds, 'benchmarks-%s.txt' % suffix, 'benchmarks-%s-diff.txt' % suffix, lambda x, y: make_benchmarks_diff(x, y, suffix))
+
+if __name__ == '__main__':
+    main()
 
=== added file 'cache-builds.py'
--- cache-builds.py	1970-01-01 00:00:00 +0000
+++ cache-builds.py	2014-07-01 15:58:40 +0000
@@ -0,0 +1,131 @@
+#!/usr/bin/python
+
+"""Scan a directory tree and cache the results of each version and
+build.
+"""
+
+import sys
+import os
+import pprint
+import cPickle
+import json
+
+def list(path):
+    try:
+        return os.walk(path).next()
+    except StopIteration:
+        return None, [], []
+
+def get_date(root, name):
+    return os.path.getmtime('%s/%s' % (root, name))
+
+def intern_all(values):
+    return tuple(intern(x) for x in values)
+
+def make_path(top, path):
+    root = top['root']
+
+    assert path.startswith(root)
+    return path[len(root)+1:]
+
+def add_build(top, name, aroot):
+    parts = intern_all(name.split('-') + [''])
+    arch, distro, cbuild, host, config = parts[:5]
+
+    root, dirs, files = list(aroot)
+    files = intern_all(files)
+
+    top['hosts'][host] = arch
+    top['arches'].setdefault(arch, {})
+    top['configs'].setdefault(config, {})
+
+    build = {
+        'name': name,
+        'arch': arch,
+        'distro': distro,
+        'cbuild': cbuild,
+        'host': host,
+        'config': config,
+        'abspath': aroot,
+        'path': make_path(top, aroot),
+        'files': files,
+        'failed': [],
+        'subfailures': [],
+        'testlog': None,
+        'started': None,
+        'finished': None,
+        'languages': [],
+        'binary': None,
+        }
+
+    for name in files:
+        if 'failed' in name:
+            build['failed'].append(name)
+
+            if '-failed' in name:
+                build['subfailures'].append(name.replace('-failed', ' ').split()[0])
+
+        elif name == 'started.txt':
+            build['started'] = get_date(root, name)
+        elif name == 'finished.txt':
+            build['finished'] = get_date(root, name)
+        elif name == 'gcc-configure.txt':
+            with open('%s/%s' % (root, name)) as f:
+                for line in f:
+                    if line.startswith('The following'):
+                        end = line.split(': ')[-1]
+                        build['languages'] = intern_all(end.strip().split(','))
+                        break
+
+        if name.endswith('-testsuite.txt'):
+            build['testlog'] = make_path(top, '%s/%s' % (root, name))
+
+    return build
+
+def add_version(top, name, path):
+    version = {
+        'name': name,
+        'abspath': path,
+        'builds': {}
+        }
+
+    root, dirs, files = list('%s/logs' % path)
+
+    for dir in dirs:
+        version['builds'][dir] = add_build(top, dir, '%s/%s' % (root, dir))
+
+    return version
+
+def add_versions(top, aroot, adirs):
+    for dir in adirs:
+        path = '%s/%s' % (aroot, dir)
+
+        root, dirs, files = list(path)
+
+        if 'logs' in dirs:
+            version = add_version(top, dir, path)
+            top['versions'][dir] = version
+
+def main():
+    top = {
+        'root': None,
+        'all': [],
+        'versions': {},
+        'arches': {},
+        'configs': {},
+        'hosts': {},
+        }
+
+    for path in sys.argv[1:]:
+        root, dirs, files = os.walk(path).next()
+        top['root'] = root
+        add_versions(top, root, dirs)
+
+    with open('builds.pickle', 'w') as f:
+        cPickle.dump(top, f, cPickle.HIGHEST_PROTOCOL)
+
+    with open('builds.json', 'w') as f:
+        json.dump(top, f, indent=2)
+
+if __name__ == '__main__':
+    main()
 
=== added file 'chain.sh'
--- chain.sh	1970-01-01 00:00:00 +0000
+++ chain.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,75 @@
+#!/bin/bash
+#
+# Chain a job on following another.  Generally works on the latest.
+#
+
+set -e
+# If the pattern fails then propogate the error
+set -o pipefail
+
+. ~/.config/cbuild/toolsrc
+
+config=cortexa9hf
+
+type=$1
+shift
+pattern=$1
+shift
+
+# Set to run=echo for testing
+run=
+
+function get_latest {
+    case $type in
+	benchmarks | ubutest)
+	    # See what binaries we have locally that match
+	    version=$( ls -t $binaries/*${config}r?.tar* | head -n1 )
+	    version=$( basename $( dirname $version ))
+	    ;;
+	trunk | oecore)
+	    # Pick the latest snapshot
+	    version=$( ls -t $snapshots/$pattern | head -n1 )
+	    version=$( basename $version )
+	    version=$( echo $version | sed -r "s/(.+)\.tar.+/\1/" )
+	    ;;
+	*)
+	    exit -1
+	    ;;
+    esac
+
+    [ -n $version ]
+}
+
+function spawn {
+    target=$queue/$2/$1.job
+
+    if [ ! -f $target ]; then
+        PYTHONPATH=$lib/tcwg-web python -m schedulejob $2 $1
+        echo Spawned $1 into $2
+    fi
+}
+
+function chain {
+    # Now do something
+    case $type in
+	benchmarks)
+	    spawn benchmarks-$version a9hf-ref
+	    spawn benchmarks-spec2000-$version a9hf-ref
+	    ;;
+	ubutest)
+	    spawn ubutest-$version a9hf-ref
+	    ;;
+	trunk)
+	    spawn $version a9hf-builder
+	    ;;
+	oecore)
+	    spawn test-oecore-$version x86_64-heavy
+	    ;;
+	*)
+	    exit -1
+	    ;;
+    esac
+}
+
+get_latest
+chain
 
=== added file 'check-leaks.sh'
--- check-leaks.sh	1970-01-01 00:00:00 +0000
+++ check-leaks.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,14 @@
+#!/bin/sh
+#
+# Watch the build directory for any results that shouldn't be there
+#
+
+. ~/.config/cbuild/toolsrc
+
+benchmarks="coremark eembc denbench spec"
+
+check=$www/build/
+
+for i in $benchmarks; do
+    find $check -name "$i*run.txt" | sed "s#$check##" | head -n 10
+done
 
=== added file 'closest.py'
--- closest.py	1970-01-01 00:00:00 +0000
+++ closest.py	2014-07-01 15:58:40 +0000
@@ -0,0 +1,63 @@
+#!/usr/bin/env python
+# Find the closest base filename.  Finds the one with the longest
+# common prefix
+#
+
+import sys
+import os.path
+import re
+
+def match(find, path):
+    """Check a build against a queue, returning the common prefix if
+    the needle starts with the queue name.
+
+    The longest match is the closest.
+
+    >>> match('foo', 'baz')
+    >>> match('gcc-foo-bar', 'gcc')
+    'gcc'
+    >>> match('gcc-foo-bar', 'gcc-foo')
+    'gcc-foo'
+    >>> match('gcc-foo-bar', 'default')
+    ''
+    """
+    base = os.path.basename(path)
+
+    if find.startswith(base):
+        return base
+    elif base == 'default':
+        return ''
+    else:
+        return None
+
+def match2(find, path):
+    base = os.path.basename(path)
+    return os.path.commonprefix([find, base])
+
+def main():
+    against = os.path.basename(sys.argv[1])
+
+    prefixes = []
+
+    for full in sys.argv[2:]:
+        key = match(against, full)
+
+        if key != None:
+            prefixes.append((full, key))
+
+    if not prefixes:
+        for full in sys.argv[2:]:
+            key = match2(against, full)
+
+            if key != None:
+                prefixes.append((full, key))
+
+    prefixes.sort(key=lambda x: -len(x[-1]))
+    print prefixes[0][0]
+
+def test():
+    import doctest
+    doctest.testmod()
+
+if __name__ == '__main__':
+    main()
 
=== added file 'cron-often.sh'
--- cron-often.sh	1970-01-01 00:00:00 +0000
+++ cron-often.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,56 @@
+#!/bin/bash
+#
+# Job that runs often, polling builds, branches, and others.
+#
+
+# Root of the www directory
+. ~/.config/cbuild/toolsrc
+
+timeout="timeout 3600"
+#timeout=
+
+cd $ctools
+
+# Update the list of all files and other caches
+function allfiles {
+	pushd $1 > /dev/null
+	find . | sort > /tmp/all-files.txt
+	
+        # Only update if changed
+	! cmp -s all-files.txt /tmp/all-files.txt \
+	&& python ~/lib/cbuild-tools/cache-builds.py . \
+	&& mv -f /tmp/all-files.txt . \
+	&& gzip -9fc --rsyncable all-files.txt > all-files.txt.gz \
+	&& gzip -9fc --rsyncable builds.json > builds.json.gz
+	
+	popd > /dev/null
+}
+
+# Tabulate any new benchmarks
+./tabulate.sh
+
+# Update the file list cache
+allfiles $build
+allfiles $benchmarks
+
+# Pull any new merge requests
+timeout 14400 python taker.py -f prompt=false
+
+# Add the testsuite diffs to anything new
+python2.7 addtestdiffs.py $build/all-files.txt $snapshots $build
+python2.7 addtestdiffs.py $benchmarks/all-files.txt $snapshots $benchmarks
+
+# Rebuild all files after the diffs are added
+allfiles $build
+allfiles $benchmarks
+
+# Poll all interesting Linaro branches
+$timeout ./export-git.sh $repos/meta-linaro meta-linaro-0~
+$timeout ./pull_branch.sh $repos/gcc-linaro-4.8/4.8 gcc-linaro-4.8
+$timeout ./pull_branch.sh $repos/gcc-linaro/4.7 gcc-linaro-4.7
+$timeout ./pull_branch.sh $repos/gcc-linaro/4.6 gcc-linaro-4.6
+$timeout ./pull_branch.sh $repos/crosstool-ng/linaro crosstool-ng-linaro-1.13.1
+$timeout ./pull_branch.sh $repos/gdb-linaro/7.5 gdb-linaro-7.5
+$timeout ./pull_branch.sh $repos/cortex-strings cortex-strings-1.0
+$timeout ./export-git.sh $repos/qemu-linaro qemu-linaro-1.0~
+$timeout ./export-git.sh $repos/boot-wrapper boot-wrapper-0~
 
=== added file 'ctimes.py'
--- ctimes.py	1970-01-01 00:00:00 +0000
+++ ctimes.py	2014-07-01 15:58:40 +0000
@@ -0,0 +1,100 @@
+"""Parse the scheduler log and print the time spent transitioning from
+state to state for each host.
+"""
+import sys
+import fileinput
+import collections
+import pprint
+
+Edge = collections.namedtuple('Edge', 'a b took time')
+
+def parse():
+    hosts = {}
+    lasts = {}
+    first = None
+
+    for line in fileinput.input():
+        parts = line.split()
+        stamp, host, state, arg = (parts + ['']*4)[:4]
+        stamp = float(stamp)
+
+        if first == None:
+            first = stamp
+
+        if arg:
+            state = arg
+
+        last = lasts.get(host, None)
+        lasts[host] = (stamp, state)
+
+        if last:
+            elapsed = stamp - last[0]
+            hosts.setdefault(host, []).append(Edge(last[1], state, elapsed, stamp - first))
+
+    return hosts
+
+def cost():
+    hosts = parse()
+
+    print('\t'.join('day host job took'.split()))
+
+    for host in hosts:
+        doing = None
+        start = None
+
+        for edge in hosts[host]:
+            if edge.b == 'idle':
+                if doing and start:
+                    pretty = doing.replace('-extract-top', '')
+                    print('%.1f\t%s\t\t%s\t%.1f' % (edge.time/3600/24, host, pretty, (edge.time - start)/3600))
+                    doing = None
+
+            if edge.b in ['lurking', 'idle']:
+                start = edge.time
+                doing = None
+
+            if edge.b in ['starting', 'idle', 'updating', 'running', 'lurking']:
+                # Skip
+                pass
+            elif 'fetch-binary' in edge.b:
+                # Skip
+                pass
+            else:
+                if doing == None:
+                    doing = edge.b
+
+def main():
+    hosts = parse()
+
+    # Go through and bin them
+    bins = {}
+
+    for host, edges in hosts.items():
+        bins[host] = {}
+
+        for edge in edges:
+            key = '%s -> %s' % (edge.a, edge.b)
+
+            if edge.a in ['lurking', 'running', 'idle']:
+                continue
+            
+            if key in ['idle -> updating', 'running -> idle', 'updating -> running']:
+                continue
+
+            bins[host].setdefault(key, []).append(edge.took)
+
+    for host in sorted(bins):
+        keys = bins[host]
+        print('%s:' % host)
+
+        for key in sorted(keys):
+            values = keys[key]
+            print('  %s: %.1f' % (key, sum(values)/len(values)), end=' ')
+
+            for value in values:
+                print('%.0f' % value, end=' ')
+
+            print()
+
+if __name__ == '__main__':
+    cost()
 
=== added file 'difftests.sh'
--- difftests.sh	1970-01-01 00:00:00 +0000
+++ difftests.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,45 @@
+#!/bin/bash
+#
+# Take all of the .sum files from two directories and generate the differences
+#
+# Example:
+#  difftests.sh \ 
+#   build/gcc-4.6+svn173209/logs/armv7l-maverick-cbuild113-ursa3-cortexa9r1 \
+#   build/gcc-4.6+svn173722/logs/armv7l-maverick-cbuild114-ursa4-cortexa9r1
+#
+
+set -e
+
+base=$1
+next=$2
+
+dir=`mktemp -d difftests.XXXXXXXXXX`
+trap "rm -rf $dir" EXIT
+
+mkdir -p $dir/base
+mkdir -p $dir/next
+
+# Copy across all logs
+cp $base/*.sum* $dir/base
+cp $next/*.sum* $dir/next
+unxz -f $dir/base/*.xz $dir/next/*.xz
+
+# Pull out jus the PASS/FAIL/etc lines and sort by test name
+#  * Change absolute path names to .../
+#  * Drop all limits tests
+#
+for i in `find $dir -name "*.sum"`; do
+    grep -E '^[A-Z]+:' $i \
+        | grep -Ev limits- \
+        | grep -Ev /guality/ \
+        | sed -r 's#/scratch/\w+/\w+/\w+/\w+/[^/]+#...#g' \
+        | sed -r "s#UNSUPPORTED: .+/testsuite/#UNSUPPORTED: #" \
+        | sort -k 2 > $i.tmp
+    mv $i.tmp $i
+done
+
+# diff returns non-zero if there is a difference
+set +e
+(cd $dir && diff -U 0 -r base next) > $dir/diff.txt
+# Drop anything but changes in test lines
+grep -E '^[+-][A-Z]' $dir/diff.txt || true
 
=== added file 'ec2-pull.sh'
--- ec2-pull.sh	1970-01-01 00:00:00 +0000
+++ ec2-pull.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,28 @@
+#!/bin/bash
+#
+# Pull the results from the cloud to the build server
+#
+
+set -e
+
+. ~/.config/cbuild/toolsrc
+
+s3=linaro-toolchain-builds
+
+dir=`mktemp -d`
+trap "rm -rf $dir" EXIT
+
+mkdir -p $dir/build
+cd $dir
+
+# Check what's there and fetch, aborting if it takes too long
+candidates=$( timeout 600 s3cmd ls s3://$s3 | awk '{print $4;}' | grep 'logs.tar.xz$' )
+
+for i in $candidates; do
+    base=$( basename $i )
+    # Get, extract, push, and remove
+    timeout 600 s3cmd get --skip-existing --no-progress $i && \
+    tar xaf $base && \
+    rsync -rtC build $www && \
+    s3cmd del $i
+done
 
=== added file 'ec2-spawn.sh'
--- ec2-spawn.sh	1970-01-01 00:00:00 +0000
+++ ec2-spawn.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,36 @@
+#!/bin/bash
+
+function spawn {
+    host=$1
+    ami=$2
+    type=$3
+
+    dir=/tmp/ec2slave/$host
+    mkdir -p $dir
+    cd $dir
+
+    ls /var/run/screen/S-cbuild/*.$host > /dev/null 2>&1 || \
+	screen -S $host -L -d -m bash -c "cd $HOME/lib/cbuild-tools && exec ./ec2slave.sh $host $ami $type"
+}
+
+# us-east1 x86_64 EBS Natty
+#spawn oort1 ami-fd589594 c1.xlarge
+# Precise
+spawn oort1 ami-a29943cb c1.xlarge
+#spawn oort3 ami-fd589594 c1.xlarge
+# us-east1 i686 EBS Natty
+#spawn oort2 ami-81c31ae8 c1.medium
+# Precise
+spawn oort2 ami-ac9943c5 c1.medium
+#spawn oort4 ami-ac9943c5 c1.medium
+spawn oort6 ami-ac9943c5 c1.medium
+#spawn oort8 ami-ac9943c5 c1.medium
+#spawn oort4 ami-e358958a c1.medium
+#spawn oort6 ami-e358958a c1.medium
+#spawn oort8 ami-e358958a c1.medium
+#spawn oort6 ami-e358958a c1.medium
+#spawn oort8 ami-06ad526f c1.medium
+#spawn oort11 ami-d5e54dbc c1.xlarge
+spawn oort12 ami-dfe54db6 c1.medium
+#spawn oort14 ami-37af765e c1.medium
+#spawn oort14 ami-87dd03ee c1.medium
 
=== added file 'ec2new.sh'
--- ec2new.sh	1970-01-01 00:00:00 +0000
+++ ec2new.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,23 @@
+#!/bin/bash
+#
+# Runs a EC2 slave for local use
+#
+# Usage: ec2new.sh [hostname [ami [type]]]
+#
+# such as:
+#  ec2new.sh oort51 ami-1aad5273 c1.xlarge
+#  ec2new.sh oort52 ami-06ad526f c1.medium
+#
+
+set -e
+
+# Don't kill on exit...
+kill=false
+. `dirname $0`/libec2.sh
+
+echo Starting an instance
+start_instance
+init_instance
+
+echo instance: $instance
+echo ip: $ip
 
=== added file 'ec2slave-init-cbuild.sh'
--- ec2slave-init-cbuild.sh	1970-01-01 00:00:00 +0000
+++ ec2slave-init-cbuild.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,36 @@
+#!/bin/sh
+#
+# Second stage where the cbuild user sets up
+#
+# Most configuration is in files shipped with this script
+
+touch ~/.in-cloud
+
+# Setup the SSH config
+cat > ~/.ssh/config <<EOF
+# Easier to set the port here then everywhere
+Host cbuild-master
+    ProxyCommand ssh cbuild.validation.linaro.org nc -q0 toolchain64 %p
+EOF
+
+# Set up the cbuild global config
+cat > ~/.config/cbuild/cbuildrc <<EOF
+# Nothing
+EOF
+
+# Seed the known hosts with keys
+for i in linaro-gateway bazaar.launchpad.net validation.linaro.org cbuild.validation.linaro.org $(hostname) localhost cbuild-master; do
+    ssh -o StrictHostKeyChecking=no -o BatchMode=yes $i true || true
+done
+
+# Checkout the cbuild scripts
+if [ ! -d /cbuild/.bzr ]; then
+    bzr lp-login cbuild
+    bzr branch --use-existing-dir lp:cbuild /cbuild
+fi
+
+cd /cbuild
+bzr pull
+
+# Seed files from the nice-and-close S3
+s3cmd sync s3://linaro-toolchain-files files
 
=== added file 'ec2slave-init.sh'
--- ec2slave-init.sh	1970-01-01 00:00:00 +0000
+++ ec2slave-init.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,105 @@
+#!/bin/sh
+
+# Fail on any error
+set -e
+
+# Automatically kill the slave if it runs for too long
+echo /sbin/poweroff | at now + 72 hours
+
+# Set the hostname
+hostname=${1:-oort99}
+
+echo $hostname > /etc/hostname
+hostname $hostname
+echo 127.0.0.1 $hostname proxy >> /etc/hosts
+
+# Bind mount /cbuild
+mkdir -p /cbuild /mnt/cbuild
+mount -o bind /mnt/cbuild /cbuild
+chmod a+w /cbuild
+
+# Use the S3 backed archive
+sed -i.dist 's,archive.ubuntu.com,archive.ubuntu.com.s3.amazonaws.com,g' /etc/apt/sources.list
+# without pipelining to make it more reliable
+echo "Acquire::http::Pipeline-Depth 0;" > /etc/apt/apt.conf.d/99no-pipelining
+
+flags="-yq --force-yes"
+
+# Install the basic packages
+apt-get update || true
+apt-get install $flags puppet jed python-software-properties
+
+# Add in the PPA
+#add-apt-repository ppa:linaro-toolchain-dev/build-deps
+#apt-get update || true
+
+# Basics
+apt-get build-dep $flags gcc binutils gdb
+apt-get build-dep $flags gcc gdb eglibc ffmpeg llvm python libvorbis
+apt-get build-dep $flags gcc-4.6 || \
+    apt-get build-dep $flags gcc-4.5 || \
+    apt-get build-dep $flags gcc-snapshot \
+
+# For llvm
+apt-get install $flags python-sphinx || true
+
+# For cross test.  Missing on lucid
+apt-get install $flags qemu-kvm qemu-user qemu-system || true
+
+# Needed for baremetal and graphite
+apt-get install $flags binutils-arm-none-eabi libcloog-ppl-dev || true
+
+# Needed for cbuild
+apt-get install $flags time python-irclib wget gnupg ccrypt rsync coreutils \
+    sendemail bzr xdelta3 xz-utils unzip openssh-client gdb libio-socket-ssl-perl libnet-ssleay-perl
+
+# Needed for ubutest
+apt-get build-dep $flags eglibc base-files base-passwd bash coreutils dash \
+    debianutils diffutils dpkg e2fsprogs findutils grep gzip hostname ncurses \
+    perl python-defaults sed shadow tar util-linux
+
+# Needed for qemu
+apt-get install $flags libpixman-1-dev
+
+apt-get build-dep $flags python2.7 || true
+# devscripts asks a question...
+echo 1 | apt-get install $flags devscripts || true
+
+# Needed for crosstool-NG LSB based builds
+apt-get install $flags lsb lsb-build-cc3 lsb-appchk3 ccache gcc-mingw32 flip tk tofrodos
+apt-get install $flags gcc-4.1 g++-4.1 || true
+apt-get install $flags gcc-4.1-multilib g++-4.1-multilib || true
+
+# Needed for cloud builds
+apt-get install $flags polipo s3cmd
+
+# Use Bash as /bin/sh.  Michael's fault...
+echo n | dpkg-reconfigure -f teletype dash
+
+# Set up the cbuild user
+cb=/home/cbuild
+
+adduser --disabled-password --gecos cbuild cbuild
+tar xaf *seed*.tar* -C /home/cbuild --strip-components=1
+
+chown -R cbuild.cbuild $cb
+chmod -R go-rwx $cb/.ssh
+
+# Finally put /tmp in RAM for faster builds
+mount -t tmpfs tmfs /tmp
+
+# Break multiarch
+multi=`dpkg-architecture -qDEB_BUILD_MULTIARCH || true`
+
+if [ ! -z "$multi" ]; then
+    pushd /usr/lib
+    cd /usr/lib && ls $multi/*crt*.o | xargs -L 1 ln -sf
+    cd /usr/include
+
+    for i in asm gnu bits sys; do
+	if [ -d $multi/$i ]; then
+	    rm -rf $i
+	    ln -sf $multi/$i
+	fi
+    done
+fi
 
=== added file 'ec2slave.sh'
--- ec2slave.sh	1970-01-01 00:00:00 +0000
+++ ec2slave.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,65 @@
+#!/bin/bash
+#
+# Runs a EC2 slave that builds cbuild jobs
+#
+# Usage: ec2slave.sh [hostname [ami [type]]]
+#
+# such as:
+#  ec2slave.sh oort2 ami-689c6701 c1.medium
+#  ec2slave.sh oort1 ami-40926929 c1.xlarge
+#
+
+set -e
+
+. `dirname $0`/libec2.sh
+
+function kill_old {
+    if [ "$instance" != "" ]; then
+        now=`date +%s`
+        elapsed=$(($now-$used))
+
+        if [ "$elapsed" -gt "600" ]; then
+            echo Killing $instance due to inactivity
+            $lock ec2kill $instance
+            instance=
+        fi
+    fi
+}
+
+function update_status {
+    if [ "$instance" == "" ]; then
+        $wget -O - $SCHEDULER_API/update/$hostname/lurking
+    else
+        $wget -O - $SCHEDULER_API/update/$hostname/idle
+    fi
+}
+
+DRIFT=$(($RANDOM % 30))
+
+while true; do
+    echo $hostname $instance `date`
+
+    sleep $((180 + $DRIFT))
+
+    kill_old
+    update_status
+
+    # See if there's a job waiting
+    next=`$wget -O - $SCHEDULER_API/peek/$hostname`
+
+    case "$next" in
+        False) continue ;;
+    esac
+
+    echo Running a job
+
+    if [ "$instance" == "" ]; then
+        echo Starting an instance
+        start_instance
+        init_instance
+        used=`date +%s`
+    fi
+
+    ssh -o StrictHostKeyChecking=no -o BatchMode=yes -t cbuild@$ip 'cd /cbuild && ./run.sh'
+    used=`date +%s`
+done
 
=== added file 'ec2watch.sh'
--- ec2watch.sh	1970-01-01 00:00:00 +0000
+++ ec2watch.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,18 @@
+#!/bin/bash
+#
+# Watch the EC2 instances so we can see where the money goes
+#
+# Terribly ugly hack.  Add this as a cronjob that runs once a half
+# hour.  Spits the results of ec2din into a timestamped file under
+# $HOME/var.
+#
+
+# Pull in the local configuration
+. ~/.private/ec2slave.rc
+
+dir=$HOME/var/log/ec2watch
+month=$(date +%Y-%m)
+now=$(date +%F-%H-%M)
+
+mkdir -p $dir/$month
+ec2din --show-empty-fields | gzip > $dir/$month/$now.txt.gz
 
=== added file 'enrich-gcc-linaro-repo'
--- enrich-gcc-linaro-repo	1970-01-01 00:00:00 +0000
+++ enrich-gcc-linaro-repo	2014-07-01 15:58:40 +0000
@@ -0,0 +1,45 @@
+#!/usr/bin/python
+# -*- coding: utf-8 -*-
+#
+# Copies revisions from branches into the dev focus branch as it's
+# also the stacked branch.  This speeds up branching when using a
+# shared repo.
+#
+# Runs as a cron job
+#
+# Author:  Loïc Minier <[e-mail]>
+#
+
+# took 6mn41s on the first run
+# 18s on the second run
+#
+
+import sys
+
+import bzrlib.repository
+import bzrlib.plugin
+
+#bzrlib.initialize()
+bzrlib.plugin.load_plugins()
+
+r_4_7 = bzrlib.repository.Repository.open('lp:gcc-linaro/4.7')
+r_4_6 = bzrlib.repository.Repository.open('lp:gcc-linaro/4.6')
+#r_4_5 = bzrlib.repository.Repository.open('lp:gcc-linaro/4.5')
+#r_4_4 = bzrlib.repository.Repository.open('lp:gcc-linaro/4.4')
+# copy revisions from 4.6 branches into 4.7 which is the devfocus
+# branch because it's used as a stacked branch
+# 4.4 and 4.5 aren't updated anymore
+r_4_7.fetch(r_4_6)
+#r_4_6.fetch(r_4_5)
+#r_4_6.fetch(r_4_4)
+# copy revisions from 4.6 into 4.4 and 4.5 once to workaround LP #388269; only
+# needed once ever
+# copying 4.5 into 4.4 took 52mn55s on the first run and 51s on second run
+# (with three up-to-date enriching branches enabled)
+# copying 4.5 into 4.6 took 1h45mn13s on the first run and 48s on second run
+# (with three maybe not fully up-to-date enriching branches enabled)
+#r_4_4.fetch(r_4_6)
+#r_4_5.fetch(r_4_6)
+r_4_6.fetch(r_4_7)
+
+print sys.argv[0], 'done'
 \ No newline at end of file
 
=== added file 'export-bzr.sh'
--- export-bzr.sh	1970-01-01 00:00:00 +0000
+++ export-bzr.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,85 @@
+#!/bin/sh
+# Update a branch and snapshot the latest revision
+# Can also snapshot past revisions and add a per-user
+# suffix
+#
+# Example: export-bzr.sh ~/repos/gcc-linaro/4.6 gcc-linaro-4.6 [revno [suffix [basedir]]]
+#
+set -e
+
+. ~/.config/cbuild/toolsrc
+
+dir=`mktemp -d`
+trap "rm -rf $dir" EXIT
+
+# Directory this script runs from
+here=`dirname $0`
+
+# Suffix for manual pulls
+SUFFIX=${4:-}
+
+# Pull a branch and tarball it up
+branch=$1
+head=$2
+
+latest=`bzr revno $branch`
+revno=${3:-$latest}
+
+base=${5:-$branch}
+
+stem="$head+bzr$revno$SUFFIX"
+tar="/tmp/$stem.tar"
+meta="/tmp/$stem.txt"
+diff="/tmp/$stem.diff"
+xdelta="$tar.xdelta3.xz"
+
+if [ -f $snapshots/`basename $xdelta` ]; then
+    exit 0
+fi
+
+closest=`$here/closest.py $tar $snapshots/base/*.tar`
+
+mkdir -p $snapshots
+
+ancestor=`bzr revision-info -d $branch -r ancestor:$base`
+
+# Record some meta data about the branch
+echo \# Branch summary > $meta
+echo name: $stem >> $meta
+echo snapshot-date: `date --rfc-3339=seconds` >> $meta
+bzr log -l1 $branch | grep -E '^(committer|timestamp):' >> $meta
+echo directory: $branch >> $meta
+echo branch: `bzr info $branch | grep parent | awk '{print $3;}'` >> $meta
+echo mainline-branch: $base >> $meta
+echo revno: $revno >> $meta
+echo revision: `bzr revision-info -d $branch -r $revno` >> $meta
+echo tip: `bzr revision-info -d $branch` >> $meta
+echo ancestor: $ancestor >> $meta
+echo closest: `basename $closest` >> $meta
+echo xdelta3: `basename $xdelta` >> $meta
+
+id=`echo $ancestor | awk '{ print $2; }'`
+
+# Create the diff
+if [ "$id" != "$revno" ]; then
+    bzr diff -r $id $branch > $diff || true
+fi
+
+echo Exporting and stamping $revno
+bzr export -r $revno $dir/$stem $branch
+REVNO=$revno $here/stamp_branch.sh $branch $dir/$stem
+
+# Create the tarball
+echo Creating $tar
+(cd $dir && find $stem -print0 | sort -z) | tar caf $tar -C $dir --no-recursion --null -T -
+
+# Create the delta
+echo Making the delta relative to $closest
+xdelta3 -fe -s $closest $tar $tar.xdelta3
+rm -f $tar
+xz -f $tar.xdelta3
+
+mv -f $tar*.xz $diff $meta $snapshots
+
+echo New snapshot `basename $tar` created
+echo stem: $stem
 
=== added file 'export-git.sh'
--- export-git.sh	1970-01-01 00:00:00 +0000
+++ export-git.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,75 @@
+#!/bin/sh
+#
+# Update a git branch and snapshot the latest revision
+#
+
+set -e
+
+. ~/.config/cbuild/toolsrc
+
+dir=`mktemp -d`
+trap "rm -rf $dir" EXIT
+
+# Directory this script runs from
+here=`dirname $0`
+
+# Suffix for manual pulls
+SUFFIX=${4:-}
+
+# Subdirectory to archive.  Needed for Android GCC
+#SUBDIR=
+
+# Pull a branch and tarball it up
+branch=$1
+head=${2:-`basename $branch`}
+id=${3:-HEAD}
+
+export GIT_DIR=$branch/.git
+export GIT_WORK_TREE=$branch
+
+(cd $branch && git reset --hard -q && git pull -q)
+latest=`git rev-parse $id`
+short=`echo $latest | cut -c -7`
+now=`date +%Y%m%d`
+
+name=$head$now+git$short$SUFFIX
+tar=/tmp/$name.tar
+xdelta=$tar.xdelta3.xz
+
+if ls $snapshots/$head*git$short*xz > /dev/null 2>&1; then
+    exit 0
+fi
+
+mkdir -p $snapshots
+echo Exporting $short
+git archive --format=tar --prefix=$name/ $id $SUBDIR | tar xf - -C $dir
+
+if [ -n "$SUBDIR" ]; then
+    # We've ended up with $dir/$head/$SUBDIR.  Shuffle them about
+    mv $dir/$head/$SUBDIR $dir
+    rm -r $dir/$head
+    mv $dir/$SUBDIR $dir/$head
+fi
+
+# Create the tarball
+echo Creating $tar
+(cd $dir && find $name -print0 | sort -z) | tar caf $tar -C $dir --no-recursion --null -T -
+
+# Use gzip as an approximation to how well it will compress
+size=$( gzip < $tar | wc -c )
+
+if [ "$size" -lt "10000000000" ]; then
+    # Small enough to just tar
+    xz -9f $tar
+else
+    # Create the delta
+    closest=`$here/closest.py $tar $snapshots/base/*.tar`
+    echo Making the delta relative to $closest
+    xdelta3 -fe -s $closest $tar $tar.xdelta3
+    rm -f $tar
+    xz -9f $tar.xdelta3
+fi
+
+mv $tar*.xz $snapshots
+echo New snapshot `basename $tar` created
+$here/spawn.sh $tar
 
=== added file 'export-hg.sh'
--- export-hg.sh	1970-01-01 00:00:00 +0000
+++ export-hg.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,61 @@
+#!/bin/sh
+# Update a branch and snapshot the latest revision
+# Can also snapshot past revisions and add a per-user
+# suffix
+#
+# Example: export-bzr.sh ~/repos/gcc-linaro/4.6 gcc-linaro-4.6 [revno [suffix [basedir]]]
+#
+set -e
+
+dir=`mktemp -d`
+trap "rm -rf $dir" EXIT
+
+# Directory this script runs from
+here=`dirname $0`
+
+# Suffix for manual pulls
+SUFFIX=${4:-}
+
+# Pull a branch and tarball it up
+branch=$1
+head=$2
+hgflags="-R $branch -y -q"
+
+hg $hgflags pull
+latest=`hg $hgflags -q tip | awk -F: '{print $1;}'`
+
+revno=${3:-$latest}
+
+base=${5:-$branch}
+
+snapshots=$HOME/snapshots
+stem=$head+hg$revno$SUFFIX
+tar=/tmp/$stem.tar
+xdelta=$tar.xdelta3.xz
+
+if [ -f $snapshots/`basename $xdelta` ]; then
+    exit 0
+fi
+
+closest=`$here/closest.py $tar $snapshots/base/*.tar`
+
+mkdir -p $snapshots
+
+echo Exporting and stamping $revno
+hg $hgflags archive -r $revno $dir/$head
+REVNO=$revno $here/stamp_branch.sh $branch $dir/$head
+
+# Create the tarball
+echo Creating $tar
+(cd $dir && find $head -print0 | sort -z) | tar caf $tar -C $dir --no-recursion --null -T -
+
+# Create the delta
+echo Making the delta relative to $closest
+xdelta3 -fe -s $closest $tar $tar.xdelta3
+rm -f $tar
+xz -f $tar.xdelta3
+
+mv -f $tar.xdelta3.xz $snapshots
+
+echo New snapshot `basename $tar` created
+echo stem: $stem
 
=== added file 'findancestor.py'
--- findancestor.py	1970-01-01 00:00:00 +0000
+++ findancestor.py	2014-07-01 15:58:40 +0000
@@ -0,0 +1,204 @@
+#!/usr/bin/python
+"""Given a build name, find the ancestor or the last working build of
+that series.
+
+Examples:
+ findancestor gcc-linaro-4.5+bzr99502~ams~merge-20110413 -> the ancestor
+ findancestor gcc-linaro-4.5+bzr99510 -> probably gcc-linaro-4.5+bzr99509
+ findancestor gcc-4.6+svn174684 -> the build from last week
+"""
+
+import sys
+import os.path
+import pprint
+import re
+import pdb
+import collections
+
+def parse(name):
+    """Parse the meta data file which is a series of name: value pairs"""
+    values = {}
+
+    with open(name) as f:
+        lines = [x.strip() for x in f.readlines()]
+
+        for line in lines:
+            if not line:
+                continue
+
+            if line.startswith('#'):
+                continue
+
+            name, value = [x.strip() for x in line.split(':', 1)]
+            values[name] = value
+
+    return values
+
+def revkey(record):
+    try:
+        if record.date:
+            return int(record.date.replace('~', ''))
+        else:
+            return int(record.revision)
+    except ValueError:
+        return record.revision
+
+def make_splitter():
+    """Splits a build name like gcc-4.5+svn1234,
+    gcc-linaro-4.6+bzr1234, and gdb-7.3~20110606+gitabcdef into the
+    product, series, optional date, version control system, and
+    revision.
+
+    >>> make_splitter().match('gcc-4.5~svn12345').groups()
+    ('gcc-', '4.5', None, '~svn', '12345')
+    >>> make_splitter().match('gcc-linaro-4.6+bzr4567').groups()
+    ('gcc-linaro-', '4.6', None, '+bzr', '4567')
+    >>> make_splitter().match('gdb-7.3~20110606+git057a947').groups()
+    ('gdb-', '7.3', '~20110606', '+git', '057a947')
+    """
+    return re.compile('([\D\-]+)([\d\.]+)([+~]\d+)?([+~][a-z]{3})(\w+)$')
+
+def make_release_splitter():
+    """Splits a release name like gcc-linaro-4.5-2011.03-0 into
+    product, series, and release.
+
+    >>> make_release_splitter().match('gcc-linaro-4.5-2011.03-0').groups()
+    ('gcc-linaro-', '4.5', '-2011.03-0')
+    >>> make_release_splitter().match('gcc-linaro-4.6-2012.03').groups()
+    ('gcc-linaro-', '4.6', '-2012.03')
+    """
+    return re.compile('([\D\-]+)([\d\.]+)(-[\d.\-]+)$')
+
+def find_via_meta(name, snapshots):
+    # See if we can find the meta data and extract the ancestor
+    metaname = os.path.join(snapshots, '%s.txt' % name)
+
+    try:
+        meta = parse(metaname)
+        revision, prev, mainline = meta['revision'], meta['ancestor'], meta['mainline-branch']
+
+        if revision != prev:
+            return '%s+bzr%s' % (mainline, prev.split()[0])
+    except Exception, ex:
+        pass
+
+    return ''
+
+def find_via_build(allfiles, name):
+    # Don't know the ancestor.  Find the previous build
+    Record = collections.namedtuple('Record', 'product series date vcs revision')
+
+    splitter = make_splitter()
+    builds = set([x[1] for x in allfiles if x[-1] == 'finished.txt'])
+    builds = [splitter.match(x) for x in sorted(builds)]
+    builds = [Record(*x.groups()) for x in builds if x]
+
+    match = splitter.match(name)
+
+    if match:
+        this = Record(*match.groups())
+
+        # Grab the builds that match
+        matches = [x for x in builds
+                   if x.product == this.product
+                   and x.series == this.series
+                   and x.vcs == this.vcs]
+
+        # Sort them by revno
+        matches.sort(key=revkey)
+
+        # Find this build and the one before
+        revnos = [x.revision for x in matches]
+
+        if this.revision in revnos:
+            idx = revnos.index(this.revision)
+
+            if idx > 0:
+                return ''.join(x for x in matches[idx-1] if x)
+
+    return ''
+
+def find_via_variant(allfiles, name):
+    # Don't know the ancestor.  Find the same build without the variant.
+    if '^' not in name:
+        return None
+
+    root, variant = name.split('^')
+
+    builds = [x for x in allfiles if x[-1] == 'finished.txt' and x[-4] == root]
+
+    if builds:
+        return root
+    
+    return ''
+
+def find_via_release(allfiles, name):
+    Record = collections.namedtuple('Record', 'product series release')
+
+    # Don't know the ancestor.  Find the previous release
+    splitter = make_release_splitter()
+    builds = set([x[1] for x in allfiles if x[-1] == 'finished.txt'])
+    builds = [splitter.match(x) for x in sorted(builds)]
+    builds = [Record(*x.groups()) for x in builds if x]
+
+    match = splitter.match(name)
+
+    if match:
+        this = Record(*match.groups())
+
+        # Grab the builds that match
+        matches = [x for x in builds
+                   if x.product == this.product
+                   and x.series == this.series]
+
+        # Sort them by release
+        matches.sort(key=lambda x: x.release)
+
+        # Find this build and the one before
+        releases = [x.release for x in matches]
+
+        if this.release in releases:
+            idx = releases.index(this.release)
+
+            if idx > 0:
+                return ''.join(x for x in matches[idx-1] if x)
+
+    return ''
+
+def find(allfiles, name, snapshots):
+    ancestor = find_via_meta(name, snapshots)
+
+    if ancestor:
+        return ancestor
+
+    ancestor = find_via_variant(allfiles, name)
+
+    if ancestor:
+        return ancestor
+
+    ancestor = find_via_build(allfiles, name)
+
+    if ancestor:
+        return ancestor
+
+    ancestor = find_via_release(allfiles, name)
+
+    if ancestor:
+        return ancestor
+
+    return ''
+
+def main():
+    allfiles, name = sys.argv[1:]
+
+    with open(allfiles) as f:
+        allf = [x.strip().split(os.sep) for x in f.readlines()]
+
+    print find(allf, name, '')
+
+def test():
+    import doctest
+    doctest.testmod()
+
+if __name__ == '__main__':
+    main()
 
=== added file 'findlogs.py'
--- findlogs.py	1970-01-01 00:00:00 +0000
+++ findlogs.py	2014-07-01 15:58:40 +0000
@@ -0,0 +1,113 @@
+#!/usr/bin/python
+"""Given a log file from a build and the build's ancestor, find the
+corresponding log file for the ancestor.
+
+Used to find the GCC testsuite results from the ancestor so that they
+can be compared.
+
+Example:
+ findlogs.py build/all-files.txt build/gcc-4.6+svn173722/logs/armv7l-maverick-cbuild114-ursa4-cortexa9r1/gcc.sum.xz gcc-4.6+svn173209
+ "gcc-4.6+svn173209 armv7l-maverick-cbuild113-ursa3-cortexa9r1"
+
+"""
+
+import sys
+import os.path
+import pprint
+import re
+import pdb
+import logging
+
+def prefilter(files, contains):
+    # Select those that could be a log
+    files = [x[-4:] for x in files if len(x) >= 4]
+    # ...and those that match the build log
+    files = [x for x in files if contains in x[-1]]
+
+    return files
+        
+def find(allfiles, log, ancestor):
+    logging.debug("FIND: input log %s" % (log))
+    logging.debug("FIND: input ancestor %s" % (ancestor))
+    log = log.split('/')
+    arch, osname, cbuild, host, config = log[-2].split('-')
+
+    # Tidy up the ancestor
+    match = re.match('lp\:([^/]+)(/.+)?([+~]\w+\d+)', ancestor)
+
+    if match:
+        branch, series, revision = match.groups()
+
+        if series:
+            series = series[1:]
+        elif branch == 'gcc-linaro':
+            # gcc-linaro has been different series at different
+            # times.  Patch.
+            match = re.match('\+bzr(\d+)', revision)
+
+            if match:
+                revno = int(match.group(1))
+                if revno < 100000:
+                    series = '4.5'
+                elif revno < 110000:
+                    series = '4.6'
+                elif revno < 121996:
+                    series = '4.7'
+                else:
+                    series = '4.8'
+
+            logging.debug("FIND: ancestor has branch %s series %s" % (branch, series))
+
+        elif branch == 'gcc':
+            # gcc is the same as gcc/4.8
+            match = re.match('\+bzr(\d+)', revision)
+
+            if match:
+                revno = int(match.group(1))
+                if revno < 120000:
+                    series = '4.7'
+                else:
+                  series = '4.8'
+        else:
+            assert False, 'Unrecognised development focus for %s' % branch
+    
+        ancestor = '%s-%s%s' % (branch, series, revision)
+        logging.debug("FIND: updated ancestor to %s" % (ancestor)) 
+
+    # Select those that could be a log
+    files = [x[-4:] for x in allfiles if len(x) >= 4]
+
+    # Select results from the ancestor
+    files = [x for x in files if x[-4] == ancestor]
+    # ...and those that match the build log
+    files = [x for x in files if x[-1] == log[-1]]
+
+    # Split out the build ID
+    for f in files:
+        f[-2] = (f[-2].split('-') + ['']*5)[:5]
+
+    # ...and that have the same architecture, config, and os
+    files = [x for x in files if x[-2][0] == arch and x[-2][-1] == config and x[-2][1] == osname]
+
+    # Sort by cbuild number
+    files.sort(key=lambda x: int(x[-2][2].replace('cbuild', '')))
+
+    if files:
+        best = files[-1]
+        return (best[0], '-'.join(best[2]))
+    else:
+        return None
+
+def main():
+    allfiles, log, ancestor = sys.argv[1:]
+
+    with open(allfiles) as f:
+        allf = [x.strip().split(os.sep) for x in f.readlines()]
+
+    best = find(allf, log, ancestor)
+
+    if best:
+        print ' '.join(best)
+
+if __name__ == '__main__':
+    main()
 
=== added file 'gcc-release.sh'
--- gcc-release.sh	1970-01-01 00:00:00 +0000
+++ gcc-release.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,168 @@
+#!/bin/bash
+#
+# Run the steps in the GCC release process
+#
+# gcc-release.sh lp:gcc-linaro/4.7 [respin]
+#
+
+set -e
+
+branch=$1
+[ -z "$branch" ] && echo 'Missing branch name' && exit -1
+
+if [ -n "$2" ]; then
+    respin=-$2
+    nextspin=$(($2 + 1))
+else
+    respin=
+    nextspin=1
+fi
+
+# Pull the author and email from bzr whoami
+me=$( bzr whoami )
+fullname=$( echo "$me" | awk -F\< '{print $1;}' | sed 's# *$##' )
+email=$( echo "$me" | awk -F\< '{print $2;}' | tr -d \> )
+
+year=$( date +%Y )
+month=$( date +%m )
+day=$( date +%d )
+
+series=$( echo $branch | sed -r "s#.+/(.+)#\1#" )
+# Die if the series can't be found in the branch name
+[ -n "$series" ]
+
+# For testing:
+# Stack on top of a local branch to cut the branch time
+#stacked=--stacked
+# Override the month
+#month=07
+
+# Use ~/.ssh/config to set up the validation lab bounce host
+host=[e-mail]
+snapshots="~/var/snapshots"
+
+release=$year.$month$respin
+
+mkdir -p release/$series-$release
+cd release/$series-$release
+ln -sf ~/linaro/gcc/.bzr
+
+# Create a new branch if needed
+if [ ! -d tree/.bzr ]; then
+    echo "Branching $branch"
+    bzr branch $stacked $branch tree
+fi
+
+# Add the release ChangeLog entry and bump the version
+if ! head -n 50 tree/ChangeLog.linaro | grep -q "GCC Linaro $series-$release released"; then
+    echo "Adding release entry to the ChangeLog"
+    cat > message <<EOF
+$year-$month-$day  $fullname  <$email>
+
+	GCC Linaro $series-$release released.
+
+	gcc/
+	* LINARO-VERSION: Update.
+
+EOF
+    cat message tree/ChangeLog.linaro > tmp
+    mv tmp tree/ChangeLog.linaro
+
+    echo $series-$release > tree/gcc/LINARO-VERSION
+
+    echo "Tagging and committing"
+    bzr commit -m "Make $series-$release release." tree
+    bzr tag -d tree gcc-linaro-$series-$release
+fi
+
+work=gcc-linaro-$series-$release
+
+if [ ! -d $work ]; then
+    echo "Exporting"
+    bzr export $work.tmp tree
+    mv $work.tmp $work
+fi
+
+cd $work
+
+# Update the manuals
+env SOURCEDIR=`pwd`/gcc/doc DESTDIR=`pwd`/INSTALL ./gcc/doc/install.texi2html
+
+# Build the tree and translations
+if [ ! -d ../objdir/gcc/po ]; then
+    echo "Building"
+    ./contrib/gcc_build -d `pwd` -o ../objdir -c "--enable-generated-files-in-srcdir --disable-multilib" -m \-sj4 build > ../build.txt 2>&1
+fi
+
+# Copy the translations across
+cp -uv ../objdir/gcc/po/*.gmo gcc/po/
+
+# Make the md5sums
+if [ ! -f MD5SUMS ]; then
+    echo "Making MD5SUMS"
+    cat > MD5SUMS <<EOF
+# This file contains the MD5 checksums of the files in the 
+# $work.tar.bz2 tarball.
+#
+# Besides verifying that all files in the tarball were correctly expanded,
+# it also can be used to determine if any files have changed since the
+# tarball was expanded or to verify that a patchfile was correctly applied.
+#
+# Suggested usage:
+# md5sum -c MD5SUMS | grep -v "OK\$"
+EOF
+    find . -type f | sed -e 's:^\./::' -e '/MD5SUMS/d' | sort | xargs md5sum >> MD5SUMS
+fi
+
+cd ..
+
+next=$series-$release-$nextspin~dev
+
+# Bump the version
+if ! grep -q $next tree/gcc/LINARO-VERSION; then
+    cat > message <<EOF
+$year-$month-$day  $fullname  <$email>
+
+	gcc/
+	* LINARO-VERSION: Bump version.
+
+EOF
+    cat message tree/ChangeLog.linaro > tmp
+    mv tmp tree/ChangeLog.linaro
+
+    echo $next > tree/gcc/LINARO-VERSION
+    bzr commit -m "Bump version number, post release." tree
+fi
+
+# Tar it up
+if [ ! -f $work.tar.bz2 ]; then
+    # Check for bad files
+    for ext in .orig .rej "~" .svn .bzr; do
+        if [ -n "$(find $work -name *$ext)" ]; then
+            echo "Found $ext files in the tarball"
+            exit -2
+        fi
+    done
+
+    echo "Making tarball"
+    tar cf $work.tar $work
+    bzip2 -9 $work.tar
+fi
+
+# Only sign on the build infrastructure
+if [ ! -f $work.tar.bz2.asc ]; then
+    read -p "Upload the tarball to snapshots and sign (y/n)? "
+
+    if [ "$REPLY" = "y" ]; then
+        echo "Pushing to snapshots"
+        rsync --progress -a $work.tar.bz2 "$host:$snapshots"
+        # Password is the EEMBC password on https://wiki.linaro.org/Internal/ToolChain
+        ssh -t $host gpg --no-use-agent -q --yes --passphrase-file /home/cbuild/.config/cbuild/password --armor --sign --detach-sig --default-key cbuild "$snapshots/$work.tar.bz2"
+        # Pull the signature back down and check
+        rsync -a "$host:$snapshots/$work*asc" .
+        gpg -q --verify $work.tar.bz2.asc
+    fi
+fi
+
+echo "Done.  Results are in $(pwd).  Spawn at http://ex.seabright.co.nz/helpers/scheduler/spawn."
+echo "Don't forget to push!"
 
=== added file 'getbody.py'
--- getbody.py	1970-01-01 00:00:00 +0000
+++ getbody.py	2014-07-01 15:58:40 +0000
@@ -0,0 +1,26 @@
+#!/usr/bin/python
+"""Extracts the body of a GCC contrib/test_summary email by extracting
+the lines between a cat <<'EOF' and the EOF.
+"""
+
+import fileinput
+import re
+
+def main():
+    lines = iter([x.rstrip() for x in fileinput.input()])
+
+    while True:
+        line = lines.next()
+        if re.match('cat\s*<<', line):
+            break
+
+    while True:
+        line = lines.next()
+
+        if re.match('EOF', line):
+            break
+        else:
+            print line
+
+if __name__ == '__main__':
+    main()
 
=== added file 'lab-pull.sh'
--- lab-pull.sh	1970-01-01 00:00:00 +0000
+++ lab-pull.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,22 @@
+#!/bin/bash
+#
+# Pull results from the lab into this server
+#
+
+dir=$HOME/incoming/validation
+www=/var/www/ex.seabright.co.nz
+
+mkdir -p $dir
+
+for j in build benchmarks; do
+    # Fetch everything
+    timeout 600 rsync -e "ssh -C" -itr "control.v:~/cbuild/$j/*/*-logs.*" $dir/$j
+
+    # Extract those that are new
+    cd $www
+
+    for i in $dir/$j/*-logs*.xz; do
+       if [ $i.done -nt $i ]; then continue; fi
+       tar xaf $i && touch $i.done
+    done
+done
 
=== added file 'latestami.sh'
--- latestami.sh	1970-01-01 00:00:00 +0000
+++ latestami.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,17 @@
+#!/bin/bash
+#
+# Get the AMI of the latest release of a series
+#
+
+distro=${1:-precise}
+arch=${2:-amd64}
+region=us-east-1
+storage=ebs
+
+wget -q -O - https://cloud-images.ubuntu.com/query/$distro/server/released.current.txt \
+    | grep $region \
+    | grep $arch \
+    | grep $storage \
+    | grep -v hvm \
+    | awk '{ print $8; }'
+
 
=== added file 'launcher.sh'
--- launcher.sh	1970-01-01 00:00:00 +0000
+++ launcher.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,146 @@
+#!/bin/bash
+#
+# Takes all incoming events and launches them.  Done as one large
+# script to keep everything together and remove contention and locking
+# issues.
+#
+
+. ~/.config/cbuild/toolsrc
+
+# Use run=echo to test
+run=
+
+#set -e
+
+# Pull in the local configuration
+hostrc=$( dirname $0 )/launcher-$( hostname ).rc
+[ -f $hostrc ] && . $hostrc
+
+function remove_old_jobs {
+    # Nuke any daily jobs early
+    $run find $scheduler -name "gcc-4.*svn*.job" -mtime +5 -exec rm -f {} \;
+    # Nuke any old jobs
+    $run find $scheduler -name "*.job" -mtime +14 -exec rm -f {} \;
+    # Nuke any old locks
+    $run find $scheduler -name "*.job.*" -mtime +20 -exec rm -f {} \;
+}
+
+function daily {
+    remove_old_jobs
+    # Pull the active trunk branches once a day
+    PRIORITY=daily $run $ctools/up_branch.sh $repos/gcc-trunk gcc-4.8~
+    PRIORITY=daily $run $ctools/up_branch.sh $repos/gcc-4.7 gcc-4.7+
+    # Update the md5sums on all snapshots
+    (cd $snapshots && $run rm -f md5sums && $run md5sum *z* > md5sums)
+    # Remove older files from S3
+    $run python $ctools/s3-diet.py linaro-toolchain-builds
+    # Check for leaking benchmark results
+    $run $ctools/check-leaks.sh
+    # Update the benchmarks graph
+    local pattern="logs/arm*precise*ursa1?-*a9hf*/benchmarks.txt"
+    local diffplot="python $lib/linaro-toolchain-benchmarks/scripts/diffplot.py"
+    (cd $benchmarks \
+	&& $run $diffplot gcc-4.8~svn.png gcc-4.7.0/$pattern gcc-4.8~svn??????/$pattern \
+	&& $run $diffplot gcc-linaro-4.7.png gcc-4.7.0/$pattern gcc-linaro-4.7-201?.??/$pattern \
+	&& $run $diffplot gcc-linaro-4.7+bzr.png gcc-linaro-4.7+bzr??????/$pattern \
+	)
+}
+
+function build_many {
+    # Build a range of branches
+    local dow=$1
+
+    case $dow in
+	1)
+	    # Benchmark the latest tip build
+	    $run $ctools/chain.sh benchmarks "gcc-linaro-4.7+bzr??????"
+	    $run $ctools/chain.sh oecore "gcc-linaro-4.7+bzr??????.*"
+	    ;;
+	2)
+	    $run $ctools/export-git.sh $repos/newlib newlib-1.21~
+	    $run $ctools/export-git.sh $repos/qemu-git qemu-1.4~
+	    $run $ctools/up_branch.sh $repos/gcc-arm-embedded-4.6 gcc-arm-embedded-4.6+
+	    $run $ctools/chain.sh ubutest "gcc-4.6+*"
+	    ;;
+	3)
+	    # Ensure we get at least one build of trunk a week
+	    $run $ctools/chain.sh trunk "gcc-4.8~*"
+	    $run $ctools/export-git.sh $repos/openembedded-core openembedded-core-1.4~
+	    $run $ctools/export-git.sh $repos/bitbake bitbake-1.15+
+	    ;;
+	4)
+	    $run $ctools/chain.sh trunk "gcc-4.7+*"
+	    $run $ctools/up_branch.sh $repos/gcc-arm-aarch64-4.7 gcc-arm-aaarch64-4.7+
+	    # Many updates for LLVM, only one tarball (no version number) at the end
+	    $run $ctools/up_branch.sh $repos/llvm/tools/clang
+	    $run $ctools/up_branch.sh $repos/llvm/tools/clang/tools/extra
+	    $run $ctools/up_branch.sh $repos/llvm/projects/compiler-rt
+	    $run $ctools/up_branch.sh $repos/llvm llvm-svn~
+	    ;;
+	5)
+	    # Benchmark the latest trunk build
+	    $run $ctools/chain.sh benchmarks "gcc-4.8~*"
+	    # Make sure 4.6 is built once a week
+	    $run $ctools/up_branch.sh $repos/gcc-4.6 gcc-4.6+
+	    $run $ctools/up_branch.sh $repos/eglibc/libc eglibc-2.18~
+	    $run $ctools/export-git.sh $repos/libffi libffi-3.1~
+	    ;;
+	6)
+	    $run $ctools/chain.sh ubutest "gcc-4.8~*"
+	    # Make sure 4.7 is built once a week
+	    $run $ctools/up_branch.sh $repos/gcc-4.7 gcc-4.7+
+	    $run $ctools/export-git.sh $repos/gdb gdb-7.6~
+	    $run $ctools/up_branch.sh $repos/gcc-google-4.7 gcc-google-4.7+
+	    ;;
+	7)
+	    $run $ctools/chain.sh ubutest "gcc-4.7+*"
+	    $run $ctools/export-git.sh $repos/binutils binutils-2.24~
+	    ;;
+    esac
+}
+
+function dow {
+    local dow=$1
+
+    build_many $dow
+
+    case $dow in
+	4)
+	    launch weekly
+	    ;;
+    esac
+}
+
+function weekly {
+    true
+}
+
+function often {
+    true
+}
+
+function launch {
+    local event=$1
+    local arg=$2
+
+    case $event in 
+	daily)
+	    launch dow $( date +%u )
+	    daily $arg
+	    ;;
+	dow)
+	    dow $arg
+	    ;;
+	weekly)
+	    weekly
+	    ;;
+	often)
+	    often
+	    ;;
+    esac
+}
+
+event=$1; shift
+
+echo launcher: running $event on $( date +%u )
+launch $event $1
 
=== added file 'libec2.sh'
--- libec2.sh	1970-01-01 00:00:00 +0000
+++ libec2.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,78 @@
+# Common functions for spawning and initialising an EC2 instace
+#
+
+. ~/.config/cbuild/toolsrc
+. $lib/cbuild/siterc
+
+wget="wget --no-proxy -q"
+this=ec2slave
+
+hostname=${1:-oort99}
+ami=${2:-missing-ami}
+# 64 bit high-CPU
+type=${3:-c1.xlarge}
+kill=${kill:-true}
+
+region=us-east-1
+
+lock="flock /tmp/heavy.lock"
+
+# Bring in any local configuration
+[ -f ~/.private/$this.rc ] && . ~/.private/$this.rc
+
+dir=/tmp/$this/$hostname
+
+function get_state {
+    $lock ec2din --show-empty-fields $instance > $dir/din
+    line=`grep ^INSTANCE $dir/din`
+    state=`echo $line | awk '{ print $6; }'`
+    if [ -f ~/.in-cloud ]; then
+	# Use the local IP
+	ip=`echo $line | awk '{ print $5; }'`
+    else
+	# Use the public IP
+	ip=`echo $line | awk '{ print $4; }'`
+    fi
+
+    echo get_state on $instance: $state $ip
+}
+
+function start_instance {
+    $wget -O - $SCHEDULER_API/update/$hostname/starting || true
+    mkdir -p $dir
+
+    $lock ec2run $ami -t $type --region $region -k $key \
+        --instance-initiated-shutdown-behavior terminate > $dir/run
+    instance=`grep ^INSTANCE $dir/run | awk '{ print $2; }'`
+
+    if [ "$kill" = "true" ]; then trap "$lock ec2kill $instance" EXIT; fi
+
+    get_state
+
+    while [ "$state" != "running" ]; do
+        sleep 5
+        get_state
+    done
+
+    # Wait for Ubuntu to boot
+    for i in {1..10}; do
+	if ssh -o StrictHostKeyChecking=no ubuntu@$ip true; then break; fi
+	sleep 10
+    done
+
+    # Test the connection
+    ssh ubuntu@$ip uname -a
+
+    echo Instance $instance started
+}
+
+function init_instance {
+    # Do the system wide setup
+    scp $this-init.sh ~/.private/$this-*seed* ubuntu@$ip:~
+    ssh ubuntu@$ip sudo bash $this-init.sh $hostname
+    # And the user specific
+    scp $this-init-cbuild.sh cbuild@$ip:~
+    ssh -t cbuild@$ip bash $this-init-cbuild.sh
+    # Copy the site configuration across
+    scp $lib/cbuild/siterc cbuild@$ip:/cbuild
+}
 
=== added file 'pull_branch.sh'
--- pull_branch.sh	1970-01-01 00:00:00 +0000
+++ pull_branch.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,21 @@
+#!/bin/sh
+# Update a branch, snapshot the latest revision, and spawn the jobs
+
+set -e
+
+# Directory this script runs from
+here=`dirname $0`
+
+# Pull a branch and tarball it up
+branch=$1
+head=$2
+
+# Silly bzr...
+rm -rf $branch/.bzr/checkout/limbo $branch/.bzr/checkout/pending-deletion
+bzr pull -q --overwrite -d $branch
+
+stem=`$here/export-bzr.sh $branch $head $3 $4 $5 $6 $7 | grep ^stem: | awk '{print $2;}'`
+
+if [ "$stem" != "" ]; then
+    $here/spawn.sh $stem
+fi
 
=== added file 's3-diet.py'
--- s3-diet.py	1970-01-01 00:00:00 +0000
+++ s3-diet.py	2014-07-01 15:58:40 +0000
@@ -0,0 +1,91 @@
+#!/usr/bin/env python
+
+"""Put a S3 bucket on a diet by removing old files.  Used to delete
+old builds from the cbuild archive.
+"""
+
+import collections
+import sys
+import datetime
+import re
+import os.path
+import ConfigParser
+import logging
+
+import simples3
+import hurry.filesize
+
+
+Entry = collections.namedtuple('Entry', 'key timestamp etag size basename')
+
+
+def main():
+    dry_run = False
+
+    logging.basicConfig(level=logging.INFO)
+
+    config = ConfigParser.SafeConfigParser()
+    config.read(os.path.expanduser('~/.s3cfg'))
+
+    bucket = simples3.S3Bucket(sys.argv[1],
+                               access_key=config.get('default', 'access_key'),
+                               secret_key=config.get('default', 'secret_key'))
+
+
+    by_key = {}
+    entries = []
+
+    for v in bucket.listdir():
+        entry = Entry(*(v + (os.path.basename(v[0]), )))
+        by_key[entry.key] = entry
+        entries.append(entry)
+
+    # Do some linting
+    to_delete = []
+
+    # Delete the signature if the original file is gone
+    for entry in entries:
+        if entry.key.endswith('.asc'):
+            original = entry.key.replace('.asc', '')
+
+            if original not in by_key:
+                logging.debug('Deleting orphan signature %s' % entry.basename)
+                to_delete.append(entry)
+
+    # Delete anything that's too old
+    now = datetime.datetime.utcnow()
+
+    for entry in entries:
+        age = now - entry.timestamp
+
+        is_merge = '+bzr' in entry.basename and len(entry.basename.split('~')) == 3
+
+        if re.search(r'-201.\.\d\d', entry.basename) and '+bzr' not in entry.basename:
+            logging.debug("Not deleting %s as it's special" % entry.basename)
+        elif is_merge and age.days >= 30:
+            logging.debug('Deleting %d day old merge %s' % (age.days, entry.basename))
+            to_delete.append(entry)
+        elif age.days >= 45:
+            logging.debug('Deleting %d day old %s' % (age.days, entry.basename))
+            to_delete.append(entry)
+        else:
+            pass
+
+    # Nuke any duplicates
+    to_delete = sorted(set(to_delete))
+
+    total = sum(x.size for x in entries)
+    deleting = sum(x.size for x in to_delete)
+
+    logging.info('Total stored: %s in %d entries' % (hurry.filesize.size(total), len(entries)))
+    logging.info('To delete: %s in %d entries' % (hurry.filesize.size(deleting), len(to_delete)))
+
+    for entry in sorted(to_delete, key=lambda x: -x.size):
+        logging.info('Deleting %s' % entry.basename)
+
+        if not dry_run:
+            del bucket[entry.key]
+
+if __name__ == '__main__':
+    main()
+
 
=== added file 'spawn.sh'
--- spawn.sh	1970-01-01 00:00:00 +0000
+++ spawn.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,36 @@
+#/bin/bash
+#
+# Spawn a job based on the queues
+#
+# Examples:
+#  spawn gcc-4.5+svn12345
+#  spawn.sh ~/path/to/thing-to-build.tar.xz
+#  spawn.sh ~/path/to/thing-to-build.tar.xz queue-name
+#
+
+set -e
+
+. ~/.config/cbuild/toolsrc
+
+# Directory this script runs from
+here=`dirname $0`
+
+job=$1
+
+priority=${PRIORITY:-.}
+
+# Tidy up job so that you can also supply a path
+job=`basename $job`
+job=`echo $job | sed -r 's/(.+)\.tar.*/\1/'`
+
+queue=${2:-$job}
+closest=$($here/closest.py $queue $scheduler/spawn/$priority/*)
+
+echo Spawning $job into the queue `basename $closest`
+
+# "Demux" via spawn queue into real job queues, use tcwg-web's schedulejob.py
+# script for scheduling jobs with LAVA support, etc.
+for i in $closest/*; do
+    PYTHONPATH=$lib/tcwg-web python -m schedulejob `basename $i` $job
+    echo Spawned into `basename $i`
+done
 
=== added file 'stamp_branch.sh'
--- stamp_branch.sh	1970-01-01 00:00:00 +0000
+++ stamp_branch.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,70 @@
+#!/bin/bash
+repo=$1
+into=$2
+
+# Check for known version control systems.
+if [ -d $repo/.bzr ]; then
+    GCC_GIT=${GCC_BZR-${BZR-bzr}}
+    vcs_type="bzr"
+elif [ -d $repo/.svn ]; then
+    GCC_SVN=${GCC_SVN-${SVN-svn}}
+    vcs_type="svn"
+else
+    echo "This does not seem to be a GCC GIT/Bzr tree!"
+    exit
+fi
+
+rm -f $into/LAST_UPDATED $into/gcc/REVISION
+
+case $vcs_type in
+    svn)
+	revision=${REVNO:-`svn info $1 | awk '/Revision:/ { print $2 }'`}
+	branch=`svn info $1 | sed -ne "/URL:/ {
+	    s,.*/trunk,trunk,
+	    s,.*/branches/,,
+	    s,.*/tags/,,
+	    p
+	}"`
+        ;;
+
+    bzr)
+        revision=${REVNO:-`bzr revno $repo`}
+	parent=`bzr info $repo | gawk '/parent branch/ { print $3; }'`
+
+	# Keep the branch name out of the version to prevent spurious
+	# test failures.
+	#
+	# ...net/%2Bbranch/gcc-linaro/4.7/ becomes gcc-linaro/4.7
+	# ...net/~michaelh1/gcc-linaro/better-longlong/ becomes gcc-linaro/~michaelh1
+	#
+	parts=( $(echo $parent | tr '/' '\n' | tac) )
+	echo ${parts[@]}
+	n3=${parts[0]}
+	n2=${parts[1]}
+	n1=${parts[2]}
+
+	case $n1,$n2,$n3 in
+	    *branch,*,*)  branch="$n2/$n3" ;;
+	    \~*,*,*)  branch="$n2/dev" ;;
+	    *)  branch="$n2" ;;
+	esac
+	;;
+esac
+
+{
+  date
+  echo "`TZ=UTC date` (revision $revision)"
+} > $into/LAST_UPDATED
+
+# And the product specific parts
+
+# GCC
+if [ -f $into/gcc/version.c ]; then
+    echo "[$branch revision $revision]" > $into/gcc/REVISION
+fi
+
+# crosstool-NG
+if [ -f $into/ct-ng.in ]; then
+    sed -i -r "s#bzr.+#bzr$revision#" $into/version.sh
+    sed -i -r "s#^REVISION\s*=.*#REVISION = +bzr$revision#" $into/contrib/linaro/build.mk
+fi
 
=== added file 'tabulate.sh'
--- tabulate.sh	1970-01-01 00:00:00 +0000
+++ tabulate.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,36 @@
+#!/bin/bash
+#
+# Tabulate any new benchmark results
+#
+
+. ~/.config/cbuild/toolsrc
+
+tmp=`mktemp -td tabulate.XXXXXXXXXX`
+trap "rm -rf $tmp" EXIT
+
+base=${1:-$benchmarks}
+scripts=${2:-$HOME/lib/linaro-toolchain-benchmarks/scripts}
+
+tabulate() {
+    filter=$1
+    suffix=$2
+
+    name=benchmarks$suffix.txt
+
+    for i in $base/gcc*/logs/*/$filter.txt; do
+        dir=$(dirname $i)
+    
+        # Skip if the tabulated version is newer
+        [ ! $i -nt $dir/$name ] && continue
+
+        echo $dir...
+        python $scripts/tabulate.py $dir/$filter.txt > $tmp/$name \
+	    && touch -r $i $tmp/$name \
+	    && mv $tmp/$name $dir
+    done
+}
+
+tabulate "*run" ""
+tabulate "spec2000-*run" "-spec2000"
+tabulate "eembc-*run" "-eembc"
+tabulate "eembc_office-*run" "-eembc_office"
 
=== added file 'taker.json'
--- taker.json	1970-01-01 00:00:00 +0000
+++ taker.json	2014-07-01 15:58:40 +0000
@@ -0,0 +1,82 @@
+{"build": "http://ex.seabright.co.nz/build/",
+ "instance": "production",
+ "projects": ["gcc-linaro", "gcc", "crosstool-ng", "gdb-linaro"],
+ "prompt": true,
+
+ "hosts": {
+        "crucis": {
+            "dry-run": true,
+            "repos": "/home/michaelh/linaro",
+            "all-files": "build/all-files.txt",
+            "tools": ".",
+            "build": "build"
+            },
+        "orion": {
+            "repos": "/home/cbuild/repos",
+            "all-files": "/var/www/ex.seabright.co.nz/build/all-files.txt",
+            "tools": "/home/cbuild/lib/cbuild-tools",
+            "build": "/var/www/ex.seabright.co.nz/build"
+            },
+        "cbuild-master": {
+            "repos": "/home/cbuild/var/repos",
+            "all-files": "/home/cbuild/public_html/build/all-files.txt",
+            "tools": "/home/cbuild/lib/cbuild-tools",
+            "build": "/home/cbuild/public_html/build"
+            },
+        "toolchain64": {
+            "repos": "/home/cbuild/var/repos",
+            "all-files": "/home/cbuild/public_html/build/all-files.txt",
+            "tools": "/home/cbuild/lib/cbuild-tools",
+            "build": "/home/cbuild/public_html/build"
+            }
+        },
+ "filters": {
+        "xinclude": ["lp663939"]
+        },
+ "branches": {
+        "lp:gcc-linaro": {
+            "prefix": "gcc-linaro-4.8",
+            "reporoot": "gcc-linaro-4.8"
+            },
+        "lp:gcc-linaro/4.5": {
+            "prefix": "gcc-linaro-4.5",
+            "reporoot": "gcc-linaro"
+            },
+        "lp:gcc-linaro/4.6": {
+            "prefix": "gcc-linaro-4.6",
+            "reporoot": "gcc-linaro"
+            },
+        "lp:gcc-linaro/4.7": {
+            "prefix": "gcc-linaro-4.7",
+            "reporoot": "gcc-linaro"
+            },
+        "lp:gcc-linaro/4.8": {
+            "prefix": "gcc-linaro-4.8",
+            "reporoot": "gcc-linaro-4.8"
+            },
+        "lp:gcc-linaro/4.6-2011.07-stable": {
+            "prefix": "gcc-linaro-4.6-2011.07-stable",
+            "reporoot": "gcc-linaro"
+            },
+        "lp:gcc-linaro/4.4": {
+            "prefix": "gcc-linaro-4.4",
+            "reporoot": "gcc-linaro"
+            },
+        "lp:gcc/4.6": {
+            "prefix": "gcc-4.6",
+            "reporoot": "gcc-linaro"
+            },
+        "lp:gcc": {
+            "prefix": "gcc-4.8",
+            "reporoot": "gcc-bzr"
+            },
+        "lp:gdb-linaro": {
+            "prefix": "gdb-linaro-7.5",
+            "reporoot": "gdb-linaro"
+            },
+        "lp:~linaro-toolchain-dev/crosstool-ng/linaro": {
+            "prefix": "crosstool-ng-linaro-1.13.1",
+            "reporoot": "crosstool-ng"
+            }
+        }
+}
 
=== added file 'taker.py'
--- taker.py	1970-01-01 00:00:00 +0000
+++ taker.py	2014-07-01 15:58:40 +0000
@@ -0,0 +1,396 @@
+"""Script that monitors merge requests, branches unseen ones, and sets
+up a build.
+"""
+
+import os
+import os.path
+import re
+import subprocess
+import json
+import socket
+import datetime
+import logging
+import getopt
+import sys
+import pprint
+import pdb
+import StringIO
+import findlogs
+
+import launchpadlib.launchpad
+
+# Message sent when a merge request is snapshotted and queued for build
+QUEUED = {
+    'subject': "[cbuild] Queued %(snapshot)s for build",
+    'body': """
+cbuild has taken a snapshot of this branch at r%(revno)s and queued it for build.
+
+The diff against the ancestor r%(ancestor)s is available at:
+ http://cbuild.validation.linaro.org/snapshots/%(snapshot)s.diff
+
+and will be built on the following builders:
+ %(builders)s
+
+You can track the build queue at:
+ http://cbuild.validation.linaro.org/helpers/scheduler
+
+cbuild-snapshot: %(snapshot)s
+cbuild-ancestor: %(target)s+bzr%(ancestor)s
+cbuild-state: check
+"""
+}
+
+# Message sent when a individual build completes
+CHECK_OK = {
+    'subject': "[cbuild] Build OK for %(snapshot)s on %(build)s",
+    'body': """
+cbuild successfully built this on %(build)s.
+
+The build results are available at:
+ http://cbuild.validation.linaro.org/build/%(snapshot)s/logs/%(build)s
+
+%(diff)s
+
+The full testsuite results are at:
+ http://cbuild.validation.linaro.org/build/%(snapshot)s/logs/%(build)s/gcc-testsuite.txt
+
+cbuild-checked: %(build)s
+"""
+}
+
+# Message sent when a individual build fails
+CHECK_FAILED = {
+    'subject': "[cbuild] Build failed for %(snapshot)s on %(build)s",
+    'body': """
+cbuild had trouble building this on %(build)s.
+See the following failure logs:
+ %(logs)s
+
+under the build results at:
+ http://cbuild.validation.linaro.org/build/%(snapshot)s/logs/%(build)s
+
+%(diff)s
+
+cbuild-checked: %(build)s
+""",
+    'vote': 'Needs Fixing',
+    'nice': False
+}
+
+CONFIG = json.load(open('taker.json'))
+
+def run(cmd, cwd=None, const=False):
+    logging.debug('Executing "%s" (cwd=%s)' % (' '.join(cmd), cwd))
+
+    if const or not get_config2(False, 'dry-run'):
+        child = subprocess.Popen(cmd, cwd=cwd, stdout=subprocess.PIPE)
+        stdout, stderr = child.communicate()
+        retcode = child.wait()
+
+        if retcode:
+            raise Exception('Child process returned %s' % retcode)
+
+        lines = [x.rstrip() for x in StringIO.StringIO(stdout).readlines()]
+        logging.debug('Stdout: %r' % lines[:20])
+
+        return lines
+    else:
+        return []
+
+def tidy_branch(v):
+    """Tidy up a branch name from one of the staging servers to the
+    top level server.
+    """
+    return v.replace('lp://qastaging/', 'lp:')
+
+def get_config_raw(v):
+    at = CONFIG
+
+    for name in v.split('!'):
+        at = at[name]
+
+    return at
+
+def get_config(*args):
+    v = '!'.join(args)
+
+    # See if the value exists in the hosts override first
+    try:
+        return get_config_raw('hosts!%s!%s' % (socket.gethostname(), v))
+    except KeyError:
+        return get_config_raw(v)
+
+def get_config2(default, *args):
+    """Get a configuration value, falling back to the default if not set."""
+    try:
+        return get_config(*args)
+    except KeyError:
+        return default
+
+class Proposal:
+    def __init__(self, proposal):
+        self.proposal = proposal
+        self.parse(proposal)
+
+    def parse(self, proposal):
+        """Parse the proposal by scanning all comments for
+        'cbuild-foo: bar' tokens.  Pull these into member variables.
+        """
+        self.state = None
+        self.ancestor = None
+        self.checked = []
+
+        for comment in proposal.all_comments:
+            lines = [x.strip() for x in comment.message_body.split('\n')]
+
+            for line in lines:
+                match = re.match('cbuild-(\w+):\s+(\S+)', line)
+
+                if match:
+                    name, value = match.groups()
+
+                    if name in ['checked']:
+                        getattr(self, name).append(value)
+                    else:
+                        setattr(self, name, value)
+
+def add_comment(proposal, template, **kwargs):
+    """Add a comment using the given template for the subject and
+    body text.
+    """
+    vote = template.get('vote', None)
+    # True if this is a 'happy' result.  Used in debugging to
+    # prevent unreversable results from going up to Launchpad.
+    nice = template.get('nice', True)
+
+    subject = (template['subject'] % kwargs).rstrip()
+    body = (template['body'] % kwargs).rstrip()
+
+    just_print = get_config2(False, 'dry-run')
+
+    if not nice and get_config2(True, 'cautious'):
+        # Running in a debug/test mode.  Don't push unreversable
+        # results.
+        just_print = True
+
+    if get_config2(False, 'prompt'):
+        logging.info("Want to add this comment: %s\n%s\n\nVote: %s\n" % (subject, body, vote))
+
+        if just_print:
+            got = raw_input('Add it anyway? > ')
+        else:
+            got = raw_input('OK to add? > ')
+
+        just_print = got.strip() not in ['Y', 'y']
+
+    if just_print:
+        logging.info("Would have added this comment: %s\n%s\n\nVote: %s\n" % (subject, body, vote))
+    else:
+        if vote:
+            proposal.proposal.createComment(subject=subject, content=body, vote=vote)
+        else:
+            proposal.proposal.createComment(subject=subject, content=body)
+
+def queue(proposal):
+    """Run the queue step by snapshoting and queuing the build."""
+    lp = proposal.proposal
+
+    source = lp.source_branch
+    owner = source.owner.name
+    branch = tidy_branch(source.bzr_identity)
+    revno = source.revision_count
+
+    # We might have hit this before Launchpad has scanned it
+    if not revno:
+        # Work around Launchpad being broken on 2011-09-07 by assuming
+        # the merge was against the latest revno
+        # Work around bzr revno bug with ghost ancestry
+        # (https://launchpad.net/bugs/1161018)
+        revno = run(['bzr', 'revno', 'nosmart+%s' % branch], const=True)
+        revno = int(revno[0])
+        assert revno
+
+    target = tidy_branch(lp.target_branch.bzr_identity)
+
+    # Find where to check it out
+    prefix = get_config('branches', target, 'prefix')
+    reporoot = get_config('branches', target, 'reporoot')
+
+    # Give it an archive name
+    suffix = '~%s~%s' % (owner, os.path.basename(branch))
+
+    snapshot = '%s+bzr%s%s' % (prefix, revno, suffix)
+    path = '%s/%s/%s' % (get_config('repos'), reporoot, snapshot)
+
+    # Remove the old directory
+    run(['rm', '-rf', path])
+    # Branch it
+    run(['bzr', 'branch', '--no-tree', branch, path], cwd=os.path.dirname(path))
+    # Find the common ancestor
+    lines = run(['bzr', 'revision-info', '-d', path, '-r', 'ancestor:%s' % target])
+
+    ancestor = lines[0].split()[0] if lines else 'unknown'
+
+    # Make the snapshot
+    lines = run(['%s/pull_branch.sh' % get_config('tools'), path, prefix, '%s' % revno, suffix, target])
+
+    # Scan the tool results to see which queues the build was
+    # pushed into
+    builders = []
+
+    for line in lines:
+        match = re.match('Spawned into (\S+)', line)
+        
+        if match:
+            builders.append(match.group(1))
+
+    add_comment(proposal, QUEUED, snapshot=snapshot, revno=revno, builders=' '.join(builders), ancestor=ancestor, target=target)
+
+def make_diff(proposal, all_files, check):
+    """Check if there are any DejaGNU summary files and diff them."""
+    sums = [x for x in check if '.sum' in x[-1]]
+
+    if not proposal.ancestor:
+        lines = ['The test suite was not checked as the branch point was not recorded.']
+    elif not sums:
+        lines = ['The test suite was not checked as this build has no .sum style test results']
+    else:
+        # Have some test results.  See if there's a corresponding
+        # version in the branch point build.
+        first = '/'.join(sums[0])
+        ref = findlogs.find(all_files, first, proposal.ancestor)
+
+        if not ref:
+            lines = ['The test suite was not checked as the branch point %s has nothing to compare against.' % proposal.ancestor]
+        else:
+            revision, build = ref
+
+            # Generate a unified diff between the two sets of results
+            diff = run(['%s/difftests.sh' % get_config('tools'),
+                        '%s/%s/logs/%s' % (get_config('build'), revision, build),
+                        '%s/%s' % (get_config('build'), os.path.dirname(first))],
+                       const=True)                           
+
+            if diff:
+                lines = ['The test suite results changed compared to the branch point %s:' % proposal.ancestor]
+                lines.extend([' %s' % x for x in diff])
+            else:
+                lines = ['The test suite results were unchanged compared to the branch point %s.' % proposal.ancestor]
+
+    if len(lines) > 40:
+        total = len(lines)
+        lines = lines[:40]
+        lines.append(' ...and %d more' % (total - len(lines)))
+            
+    return '\n'.join(lines)
+
+def check_build(proposal, all_files, build, matches):
+    """Check a single build to see how the results went."""
+
+    if build in proposal.checked:
+        logging.debug('Already posted a comment on %s' % build)
+    else:
+        check = [x for x in matches if len(x) >= 4 and x[2] == build]
+        failures = [x for x in check if 'failed' in x[-1]]
+        finished = 'finished.txt' in [x[-1] for x in check]
+
+        logging.info('Checking %s (finished=%s, failures=%r)' % (build, finished, failures))
+
+        if finished:
+            logs = ' '.join(x[-1] for x in failures)
+
+            diff = make_diff(proposal, all_files, check)
+
+            if failures:
+                add_comment(proposal, CHECK_FAILED, build=build, snapshot=proposal.snapshot, logs=logs, diff=diff)
+            else:
+                add_comment(proposal, CHECK_OK, build=build, snapshot=proposal.snapshot, logs=logs, diff=diff)
+
+def check(proposal, all_files):
+    """Check an already queued build for build results."""
+    # Find all log files for this snapshot
+    matches = [x for x in all_files if x[0] == proposal.snapshot]
+    matches = [x for x in matches if len(x) >= 3 and x[1] == 'logs']
+
+    if not matches:
+        logging.debug('No log files yet')
+        return
+
+    # Build a list of builds
+    builds = [x[2] for x in matches if len(x) == 3]
+
+    for build in builds:
+        check_build(proposal, all_files, build, matches)
+
+def run_proposal(proposal, all_files, allowed):
+    when = proposal.date_review_requested
+
+    # Only run recent proposals that have had review requested
+    if when:
+        now = datetime.datetime.now(when.tzinfo)
+        elapsed = (now - when).days
+
+        logging.info('Checking proposal %s which is %d days old' % (proposal, elapsed))
+
+        if elapsed <= get_config2(14, 'age-limit'):
+            p = Proposal(proposal)
+
+            state = p.state if p.state else 'queue'
+
+            if state in allowed:
+                # Could use a dodgy getattr()...
+                if state == 'queue':
+                    queue(p)
+                elif state == 'check':
+                    check(p, all_files)
+                else:
+                    assert False, 'Proposal %s is in the invalid state "%s"' % (proposal, p.state)
+            else:
+                logging.info("Skipping %s" % state)
+
+def main():
+    opts, args = getopt.getopt(sys.argv[1:], 'f:vs:')
+    verbose = int(get_config2(0, 'verbose'))
+    steps = get_config2('queue', 'steps').split(',')
+
+    for opt, arg in opts:
+        if opt == '-f':
+            name, value = arg.split('=')
+            option = json.loads('{ "%s": %s }' % (name, value))
+            CONFIG.update(option)
+        elif opt == '-v':
+            verbose += 1
+        elif opt == '-s':
+            steps = arg.split(',')
+
+    if verbose >= 2:
+        logging.basicConfig(level=logging.DEBUG)
+    elif verbose >= 1:
+        logging.basicConfig(level=logging.INFO)
+
+    # Parse all-files into a list of already split paths
+    with open(get_config('all-files')) as f:
+        all_files = f.readlines()
+
+    all_files = [x.rstrip().split('/') for x in all_files]
+    all_files = [x[1:] for x in all_files if len(x) >= 2]
+
+    # Login to Launchpad
+    launchpad = launchpadlib.launchpad.Launchpad.login_with('taker', get_config('instance'), credentials_file=os.path.expanduser('~/.launchpadlib/credentials'))
+
+    # Scan all projects...
+    for name in get_config('projects'):
+        logging.info('Checking project %s' % name)
+        project = launchpad.projects[name]
+         
+        # Scan all proposals in the project...
+        proposals = project.getMergeProposals()
+
+        for p in proposals:
+            try:
+                run_proposal(p, all_files, steps)
+            except Exception, ex:
+                logging.error('Error while processing %s: %s' % (p, ex))
+
+if __name__ == '__main__':
+    main()
 
=== added file 'track-tree.sh'
--- track-tree.sh	1970-01-01 00:00:00 +0000
+++ track-tree.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,16 @@
+#!/bin/sh
+
+set -e
+
+dir=`mktemp -d`
+trap "rm -rf $dir" EXIT
+
+find $@ -type f | sort > $dir/all-files.txt
+cp -a changes/all-files.txt $dir/previous.txt
+
+cmp -s $dir/previous.txt $dir/all-files.txt && exit
+
+stamp=`date +%Y-%m-%d-%H%M%S`
+
+(cd $dir && diff -U 0 previous.txt all-files.txt) | tee $dir/$stamp.diff
+mv $dir/$stamp.diff $dir/all-files.txt changes
 
=== added file 'up_branch.sh'
--- up_branch.sh	1970-01-01 00:00:00 +0000
+++ up_branch.sh	2014-07-01 15:58:40 +0000
@@ -0,0 +1,53 @@
+#!/bin/bash
+# Update a SVN branch and snapshot the latest revision
+# Can also snapshot past revisions and add a per-user
+# suffix
+
+set -e
+
+. ~/.config/cbuild/toolsrc
+
+# Directory this script runs from
+here=`dirname $0`
+
+# Suffix for manual pulls
+SUFFIX=${SUFFIX:-}
+
+# Pull a branch and tarball it up
+branch=$1
+head=$2
+
+svn up -q $branch
+# Recent SVN adds a last changed revision field.  Delete spaces to
+# strip the space from the revno and, coincidentally, make the field
+# easier to match.
+latest=$(svn info $branch | tr -d ' ' | awk -F: '/LastChangedRev/ { print $2 }')
+revno=${REVNO:-$latest}
+
+if [ "x${head}" != "x" ]; then
+    name=${head}svn$revno$SUFFIX
+    tar=/tmp/$name.tar
+    xdelta=$snapshots/$name.tar.xdelta3.xz
+
+    if [ ! -f $xdelta ]; then
+	echo Exporting and stamping $revno
+	tmp=`mktemp -d`
+	rm -rf $tmp/$name
+	svn export $branch $tmp/$name
+	REVNO=$revno $here/stamp_branch.sh $branch $tmp/$name
+
+        # Create the tarball
+	echo Creating $tar
+	(cd $tmp && find "$name" -print0 | sort -z) | tar caf $tar -C $tmp --no-recursion --null -T -
+	rm -rf $tmp
+        # Create the delta
+	closest=`$here/closest.py $tar $snapshots/base/*.tar`
+	echo Making the delta relative to $closest
+	xdelta3 -fe -s $closest $tar $tar.xdelta3
+	rm -f $tar
+	xz -f $tar.xdelta3
+	mv $tar.xdelta3.xz $snapshots
+	echo New snapshot $head $revno created
+	$here/spawn.sh $tar
+    fi
+fi
 
=== added file 'utilisation.py'
--- utilisation.py	1970-01-01 00:00:00 +0000
+++ utilisation.py	2014-07-01 15:58:40 +0000
@@ -0,0 +1,93 @@
+"""Plots the backlog and utilisation level of a class of build
+machines.
+"""
+
+import fileinput
+
+import matplotlib
+matplotlib.use('agg')
+
+from pylab import *
+
+def process(lines, name, matcher):
+    start = float(lines[0][0])
+    backlog = []
+    running = []
+    states = []
+    hosts = {}
+
+    for line in lines:
+        time, type, host, arg = line[:4]
+
+        if not matcher(host):
+            continue
+
+        offset = float(time) - start
+        days = offset / (60*60*24)
+
+        if type == 'backlog':
+            if hosts[host] == 'reserved':
+                # Reserved machines report as a zero backlog
+                pass
+            else:
+                backlog.append((days, int(arg)))
+        elif type == 'state':
+            if arg != 'updating':
+                states.append((host, arg, days))
+                hosts[host] = arg
+
+    # Scan through the states and remove the glitches
+    merged = []
+
+    for host in hosts:
+        just = [x for x in states if x[0] == host]
+        
+        for i in range(len(just) - 1):
+            this = just[i]
+            next = just[i+1]
+            elapsed = next[-1] - this[-1]
+
+            if this[1] == 'running' and next[1] == 'idle' and elapsed < 5.0/60/60/24:
+                # Drop this record
+                pass
+            else:
+                merged.append(this)
+
+    merged.sort(key=lambda x: x[-1])
+
+    # Scan through and figure out the number of hosts running at any one time
+    states = {}
+
+    for row in merged:
+        states[row[0]] = row[1]
+        
+        running.append((row[-1], sum(1 if x == 'running' else 0 for x in states.values())))
+
+    backlog = array(backlog)
+    running = array(running)
+
+    clf()
+    plot(backlog[:,0], backlog[:,1])
+    scatter(backlog[:,0], backlog[:,1], label='backlog')
+
+    plot(running[:,0], running[:,1], c='g', label='running')
+
+    ylim(0, ylim()[1]+1)
+    xlim(0, xlim()[1])
+    xlabel('Time (days)')
+    ylabel('Utilisation (#boards) & backlog (# jobs)')
+    title('%s utilisation' % name)
+    legend()
+    grid()
+    savefig('%s.png' % name)
+    show()
+
+def main():
+    lines = list(fileinput.input())
+    lines = [x.rstrip().split() for x in lines]
+    
+    process(lines, 'ursas', lambda x: x.startswith('ursa'))
+#    process(lines, 'cloud', lambda x: x.startswith('oort'))
+
+if __name__ == '__main__':
+    main()