Merge lp:~mwhudson/lava-deployment-tool/add-upgrade-staging-script into lp:~linaro-validation/lava-deployment-tool/trunk

[Michael Hudson-Doyle]

Hi,

This adds a script that I intend to use to automatically update staging on control.

It does two other things: stop l-d-t bundle from asking questions and improves the output of l-d-t preview.

If the code doesn't make sense, this box isn't the place to fix that problem :-)

Cheers,
mwh

[Michael Hudson-Doyle (mwhudson)]

Hm, it seems that using a bundle made with -e bzr:... lines does not interact entirely as expected with l-d-t upgrade. More investigation required.

[lp:~mwhudson/lava-deployment-tool/add-upgrade-staging-script]

176. By Michael Hudson-Doyle on 2012-05-02

a different approach, check out all the branches, build tarballs from them include those tarballs in the requirements file

177. By Michael Hudson-Doyle on 2012-05-02

actually install the bundle

178. By Michael Hudson-Doyle on 2012-05-02

have install_config_app start the instance

[Michael Hudson-Doyle (mwhudson)]

Ok, here is a new approach that seems more promising: don't let pip see anything to do with bzr, instead check out each bzr branch, run sdist and include the tarball in the requirements file that we feed to l-d-t bundle (this has the advantage that we don't need to include the hacked version of pip in the venv we build the bundle in).

[lp:~mwhudson/lava-deployment-tool/add-upgrade-staging-script]

179. By Michael Hudson-Doyle on 2012-05-03

remove code instead of just commenting it out

[Michael Hudson-Doyle (mwhudson)]

I should say that we probably should keep mirrors of the launchpad branches around to make this faster. But that can be a follow up improvement.

=== added file '_compare_manifests.py'
--- _compare_manifests.py	1970-01-01 00:00:00 +0000
+++ _compare_manifests.py	2012-05-03 00:26:18 +0000
@@ -0,0 +1,45 @@
+#!/usr/bin/python
+
+import sys
+
+def load(fname):
+    d = {}
+    for line in open(fname):
+        line = line.strip()
+        if line.startswith('-e'):
+            # This bit is a total hack.
+            _, stuff = line.split('#egg=')
+            package, version, _ = stuff.split('-', 2)
+            package = package.replace('_', '-')
+        else:
+            package, version = line.split('==')
+        d[package] = version
+    return d
+
+cur = load(sys.argv[1])
+new = load(sys.argv[2])
+
+not_present = set(cur) - set(new)
+new_packaged = set(new) - set(cur)
+changed = {}
+for package in set(cur) & set(new):
+    if cur[package] != new[package]:
+        changed[package] = (cur[package], new[package])
+
+if not_present:
+    print "Already installed, but not in bundle:"
+    w1 = max(len(p) for p in not_present)
+    for k in sorted(not_present):
+        print "    %-*s %s" % (w1, k, cur[k])
+if changed:
+    print "To be upgraded:"
+    w1 = max(len(p) for p in changed)
+    w2 = max(len(t[0]) for t in changed.itervalues())
+    w3 = max(len(t[1]) for t in changed.itervalues())
+    for k in sorted(changed):
+        print "    %-*s %-*s -> %-*s" % (w1, k, w2, changed[k][0], w3, changed[k][1])
+if new_packaged:
+    print "To be installed:"
+    w1 = max(len(p) for p in new_packaged)
+    for k in sorted(new_packaged):
+        print "    %-*s %s" % (w1, k, new[k])

=== modified file 'lava-deployment-tool'
--- lava-deployment-tool	2012-04-25 08:02:22 +0000
+++ lava-deployment-tool	2012-05-03 00:26:18 +0000
@@ -1020,8 +1020,10 @@
     # Get out of virtualenv
     deactivate
 
-    echo "Your instance is now ready, please start (or re-start) it with"
-    echo "sudo start lava-instance LAVA_INSTANCE=$LAVA_INSTANCE"
+    echo "Starting instance again..."
+    set -x
+    sudo start lava-instance LAVA_INSTANCE=$LAVA_INSTANCE
+    set +x
 }
 
 
@@ -1745,7 +1747,6 @@
 cmd_bundle() {
     LAVA_REQUIREMENT=${1:-requirements/requirements.txt}
     LAVA_BUNDLE=${2:-lava.pybundle}
-    unset PIP_DOWNLOAD_CACHE
     local step
     for step in $LAVA_INSTALL_STEPS; do
         defaults_$step
@@ -1754,13 +1755,6 @@
     for step in $LAVA_INSTALL_STEPS; do
         pkglist_$step >> "$LAVA_FULL_REQUIREMENT"
     done
-    echo "The resulting file can be copied for off-line installation"
-    echo "Do you wnt to create $LAVA_BUNDLE with the following packages:"
-    echo
-    cat "$LAVA_FULL_REQUIREMENT"
-    echo
-    read -p "Type BUNDLE to continue: " RESPONSE
-    test "$RESPONSE" = 'BUNDLE' || return
     set -x
     pip bundle "$LAVA_BUNDLE" $PIP_PROXY_OPTION --timeout=30 --use-mirrors --requirement="$LAVA_FULL_REQUIREMENT" || die "Failed to create bundle"
     set +x
@@ -1783,7 +1777,7 @@
     unzip -qc "${_bundle}" pip-manifest.txt | grep -v '^#' | tr '[A-Z]' '[a-z]' | sort > bundle-versions.txt
     $LAVA_PREFIX/$LAVA_INSTANCE/bin/pip freeze | tr '[A-Z]' '[a-z]' | sort > installed-versions.txt
 
-    diff -y installed-versions.txt bundle-versions.txt
+    $(dirname $(readlink -f "$0"))/_compare_manifests.py installed-versions.txt bundle-versions.txt
 }
 
 is_vhost(){

=== added file 'upgrade-staging'
--- upgrade-staging	1970-01-01 00:00:00 +0000
+++ upgrade-staging	2012-05-03 00:26:18 +0000
@@ -0,0 +1,61 @@
+#!/bin/bash
+
+# 1. Make virtualenv
+# 2. Patch pip in virtualenv
+# 3. Install bzrlib in virtualenv.
+# 4. Activate virtualenv
+# 5. Make bundle in virtualenv using requirements/requirements-trunk.txt
+# 6. Deactivate virtualenv
+# 7. Optionally run l-d-t preview and ask for confirmation
+# 8. Run l-d-t upgrade with bundle made in step 5.
+
+set -uxe
+
+allow-badness () {
+    set +uxe;
+    "$@";
+    set -uxe;
+}
+
+TARGET_INSTANCE="${1:-staging}"
+
+LAVA_ROOT=/srv/lava
+
+LDT_DIR="$(dirname $(readlink -f $0))"
+SOURCE_REQUIREMENTS="${LDT_DIR}/requirements/requirements-latest.txt"
+LDT="${LDT_DIR}/lava-deployment-tool"
+
+SCRATCH_DIR="$(mktemp -d)"
+
+trap "rm -rf \"${SCRATCH_DIR}\"" EXIT
+
+VENV="${SCRATCH_DIR}/env"
+BUNDLE="${SCRATCH_DIR}/lava.pybundle"
+REQS="${SCRATCH_DIR}/requirements.txt"
+
+virtualenv "${VENV}"
+
+allow-badness . "${VENV}/bin/activate"
+
+export PIP_DOWNLOAD_CACHE=$LAVA_ROOT/.downloads
+
+pip install -q bzr
+
+mkdir "${SCRATCH_DIR}/src"
+
+while read line
+do
+    bzr branch --stacked "lp:$line" "${SCRATCH_DIR}/src/$line"
+    pushd "${SCRATCH_DIR}/src/$line"
+    python setup.py sdist
+    echo $(readlink -f dist/*) >> "${REQS}"
+    popd
+done < "${SOURCE_REQUIREMENTS}"
+
+"${LDT}" bundle "${REQS}" "${BUNDLE}"
+
+allow-badness deactivate
+
+"${LDT}" preview "${TARGET_INSTANCE}" "${BUNDLE}"
+
+"${LDT}" upgrade "${TARGET_INSTANCE}" "${BUNDLE}"