Merge lp:~mwhudson/lava-deployment-tool/use-buildout into lp:~linaro-validation/lava-deployment-tool/trunk

[Michael Hudson-Doyle]

Hi,

This branch changes everything to use buildout for deployment rather than pip. It is intimately related to https://code.launchpad.net/~mwhudson/lava-server/use-buildout/+merge/109768. I'm sure the diff is going to be hard to read, so maybe just try it out? You'll need to run ./lava-deployment-tool setup again, but that shouldn't change anything that will break your old instances.

I've htmlified the new README and put it online at http://people.linaro.org/~mwh/ldt.html.

Cheers,
mwh

[Zygmunt Krynicki (zyga)]

There are some conflicts in the tree

766 +<<<<<<< TREE
767 if [ \! -e /etc/apache2/mods-available/uwsgi.load ]; then
768 logger "Building uWSGI apache module..."
769 # NOTE: we may not have this file cached as we used bundle to get it
770 @@ -691,6 +617,8 @@
771 INSTANCE_WSGI
772 fi
773
774 +=======
775 +>>>>>>> MERGE-SOURCE

[Michael Hudson-Doyle (mwhudson)]

Zygmunt Krynicki <email address hidden> writes:

> There are some conflicts in the tree

Thanks, fixed.

Cheers,
mwh

[lp:~mwhudson/lava-deployment-tool/use-buildout]

222. By Michael Hudson-Doyle on 2012-06-12

merge trunk

223. By Michael Hudson-Doyle on 2012-06-13

stoopid

224. By Michael Hudson-Doyle on 2012-06-13

run setup with -e ffs

225. By Michael Hudson-Doyle on 2012-06-13

chown -R

226. By Michael Hudson-Doyle on 2012-06-13

possible conversion code

227. By Michael Hudson-Doyle on 2012-06-13

more conversion

228. By Michael Hudson-Doyle on 2012-06-13

no plan survives contact etc

229. By Michael Hudson-Doyle on 2012-06-13

yet more details

230. By Michael Hudson-Doyle on 2012-06-13

ffs

231. By Michael Hudson-Doyle on 2012-06-13

mutilate the uwsgi.ini file too, and it works!

232. By Michael Hudson-Doyle on 2012-06-13

document upgrading from old world

233. By Michael Hudson-Doyle on 2012-06-13

start to use a pure manifest branch

234. By Michael Hudson-Doyle on 2012-06-14

point template_dirs and staticfiles_dirs at symlink to current lava-server code that the buildout now creates

235. By Michael Hudson-Doyle on 2012-06-14

rename variables to make sense now

236. By Michael Hudson-Doyle on 2012-06-14

update docs (so much simpler for the local development case!)

237. By Michael Hudson-Doyle on 2012-06-14

lets make everyone run setup again

238. By Michael Hudson-Doyle on 2012-06-14

do not download pre-seeded deps if they are already present!

239. By Michael Hudson-Doyle on 2012-06-15

do not depend on the version of distribute that virtualenv installs by default
also, check sha256s of downloaded files

240. By Michael Hudson-Doyle on 2012-06-15

no point having a PIP_PROXY_OPTION if we do not let pip touch the network

241. By Michael Hudson-Doyle on 2012-06-15

work with older virtualenvs

242. By Michael Hudson-Doyle on 2012-06-18

load configuration before determining $LAVA_REVNO (as that may depend on the branch the instance uses)

243. By Michael Hudson-Doyle on 2012-06-18

document lava-develop-branch/lava-undevelop-branch a bit

244. By Michael Hudson-Doyle on 2012-06-18

small tweak

245. By Michael Hudson-Doyle on 2012-06-18

i renamed the helper scripts

=== modified file 'README'
--- README	2012-03-23 01:30:07 +0000
+++ README	2012-06-18 06:02:17 +0000
@@ -5,42 +5,46 @@
 The tool is suitable for both personal and more "production" installations that
 are expected to perform adequately with more concurrent users. This tool is
 not, however, designed for multi-machine installation, including cloud
-deployment. For that it is likely we will turn to Juju
-(https://juju.ubuntu.com/)
+deployment. For that it is likely we will turn to Juju_.
+
+.. _Juju: https://juju.ubuntu.com/
+
+.. contents::
 
 Quickstart
 ^^^^^^^^^^
+
 For the impatient, or those just looking for a cheat sheet but know what they
-are doing otherwise, here are the basic set of commands to get an instance
-up and running based on the requirements-mini file provided as an example, 
-which only installs lava-server.
+are doing otherwise, here are the basic set of commands to get an instance::
 
  $ ./lava-deployment-tool setup
- $ ./lava-deployment-tool bundle requirements/requirements-mini.txt
- $ ./lava-deployment-tool install testinstance lava.pybundle
- $ sudo start lava-instance LAVA_INSTANCE=testinstance
+ $ ./lava-deployment-tool install testinstance
  $ ./lava-deployment-tool manage testinstance createsuperuser
-
-Assuming everything went well, you should be able to point a web browser
-at the system you did this, see the lava-server default page and sign-in.
-
+ $ ./lava-deployment-tool manage testinstance runserver
+
+Assuming everything went well, you should be able to point a web
+browser at http://127.0.0.1:8000/, see the lava-server default page
+and sign-in.
 
 Software Requirements
 ^^^^^^^^^^^^^^^^^^^^^
 
-This tool should work on Ubuntu versions starting with 10.10 release
-(Maverick). If you'd like to help us with other distributions feel free to
-contact us at validation (at) linaro (dot) org.
+We currently recommmend using Ubuntu 12.04.  This tool should work on
+Ubuntu versions 11.10 (Oneiric) and newer.
+
+If you'd like to help us with other distributions feel free to contact
+us at linaro-validation (at) lists (dot) linaro (dot) org.
 
 Hardware Requirements
 ^^^^^^^^^^^^^^^^^^^^^
 
-A small LAVA instance can be deployed on any modest hardware. We recommend at
-least one 1GB of RAM for runtime activity (this is shared, on a single host,
-among the database server, the application server and the web server). For
-storage please reserve about 20GB for application data, especially if you wish
-to mirror current public LAVA instance used by Linaro. LAVA uses append-only
-models so the storage requirements will grow at about several GB a year.
+A small LAVA instance can be deployed on any modest hardware. We
+recommend at least one 1GB of RAM for runtime activity (this is
+shared, on a single host, among the database server, the application
+server and the web server). For storage please reserve about 20GB for
+application data, especially if you wish to mirror current public LAVA
+instance used by Linaro.  LAVA uses append-only models so the storage
+requirements will grow at about several GB a year.
 
 Before installing
 ^^^^^^^^^^^^^^^^^
@@ -53,45 +57,62 @@
 development headers and a working compiler to build some of the native (C)
 extensions for python.
 
-This step is largely automated. To perform it run this command:
-
-$ ./lava-deployment-tool setup
-
-This step also prepares file-system places for LAVA. In particular it creates
-/srv/lava/ where all LAVA instances are later stored.
-
-Creating LAVA instance
-^^^^^^^^^^^^^^^^^^^^^^
-
-You can create multiple LAVA instances on a single machine. Being able to do
-so is very valuable for testing and developing LAVA itself. Before installing
-you must first create a bundle of sources you wish to install.  This simply 
-downloads all the things you intend to install into a single, local archive
-so that installation or updating is not impacted by any problems getting the
-sources later.  This also enables offline installs, or installs behind a
-restrictive firewall by allowing you to download the bundle in advance.
-To create the bundle, run:
-
-$ ./lava-deployment-tool bundle URL_to_requirements [bundle_file_name]
-If you do not specify the bundle_file_name, it will use the name 'lava.pybundle'
-
-To create a new instance run this command:
-
-$ ./lava-deployment-tool install NAME BUNDLE
-
-This will create a fresh instance called NAME, the instance will be composed of
-the software specified in the requirements file that was used to build the
-bundle and associated dependencies.
-
-To create an administrator user for this instance:
-
-$ ./lava-deployment-tool manage NAME createsuperuser
+This step is largely automated. To perform it run this command::
+
+ $ ./lava-deployment-tool setup
+
+This step also prepares file-system places for LAVA. In particular it
+creates ``/srv/lava/`` where all LAVA instances are later stored.
+
+Creating an instance of LAVA
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+An instance of LAVA is a self-contained installation of LAVA.  This
+means the code that makes up LAVA, configuration data, a Postgres
+database, and the log files and test result bundles produced by
+running tests.
+
+You can create multiple LAVA instances on a single machine, which is
+very valuable for testing and developing LAVA itself.
+
+The code part of an instance is described by a `buildout`_
+configuration file, which lives in a branch of the `lava-manifest`_
+project on Launchpad.  By default, lava-deployment-tool creates an
+instance that uses the buildout.cfg file from the most recent revision
+of the lp:lava-manifest branch.
+
+To create an instance, run::
+
+ $ ./lava-deployment-tool install NAME
+
+This will ask you a few questions and then create a fresh instance
+called NAME.
+
+You can override details about how to construct the instance:
+
+ 1. To override the branch to deploy from, set LAVA_MANIFEST_BRANCH in
+    the environment.
+
+ 2. To override the name of the buildout config file, set
+    LAVA_BUILDOUT_CFG.
+
+ 3. To deploy a specific revision of LAVA_MANIFEST_BRANCH, add the
+    revision number to the command line::
+
+     $ ./lava-deployment-tool install NAME <revno>
+
+See `different kinds of deployment`_ below for more about customizing
+instances.
+
+To create an administrator user for a newly-created instance::
+
+ $ ./lava-deployment-tool manage NAME createsuperuser
 
 This user can access the administration panel, edit permissions to other users
 and add bundle streams, devices, etc...
 
-The script produces verbose output, at the end it should say that everything
-went fine.
+.. _buildout: http://www.buildout.org/
+.. _lava-manifest: https://launchpad.net/lava-manifest
 
 Backing Up LAVA instance
 ^^^^^^^^^^^^^^^^^^^^^^^^
@@ -99,13 +120,14 @@
 LAVA instances store persistent data in two locations:
 
  * On the filesystem, in the directory
-   /srv/lava/$LAVA_INSTANCE/var/lib/lava-server/media
- * In a PostgreSQL database in the default cluster named $LAVA_INSTANCE
+   ``/srv/lava/$LAVA_INSTANCE/var/lib/lava-server/media``
+ * In a PostgreSQL database in the default cluster named
+   ``lava-$LAVA_INSTANCE``
 
 Backing up those two items is sufficient to preserve the entire system
-state.  You can do this by running:
+state.  You can do this by running::
 
-$ lava-deployment-tool backup $LAVA_INSTANCE
+ $ lava-deployment-tool backup $LAVA_INSTANCE
 
 which will create a backup with an ID based on the current date and
 time in a directory named
@@ -115,9 +137,9 @@
 
 Generally before backing up you should make sure that LAVA instance is
 turned off. This depends on how your instance is started. If you were
-using upstart the following shell command should turn LAVA off:
+using upstart the following shell command should turn LAVA off::
 
-$ sudo service lava stop
+ $ sudo stop lava
 
 If you take a backup while running, you will need to do some manual
 cleanup when you restore from it.
@@ -125,18 +147,18 @@
 Restoring from backup
 ^^^^^^^^^^^^^^^^^^^^^
 
-Running the command
+Running the command ::
 
-$ lava-deployment-tool restore $LAVA_INSTANCE $SNAPSHOT_ID
+ $ lava-deployment-tool restore $LAVA_INSTANCE $SNAPSHOT_ID
 
 will restore the given snapshot of the named instance.  This will
 first erase the database and media files of the named instance, so be
 careful what you type!
 
 You can restore an instance from a backup taken from a distinct
-instance with a command like:
+instance with a command like::
 
-$ lava-deployment-tool restore $TARGET_INSTANCE $SOURCE_INSTANCE/$SNAPSHOT_ID
+ $ lava-deployment-tool restore $TARGET_INSTANCE $SOURCE_INSTANCE/$SNAPSHOT_ID
 
 Currently to restore from a backup taken on a different machine, you
 have to put it under /srv/lava/backups, then run an appropriate
@@ -148,57 +170,77 @@
 Updating LAVA instance
 ^^^^^^^^^^^^^^^^^^^^^^
 
-LAVA is released periodically. Currently this is once a month but the release
-cycle becomes more and more fluid and eventually we'd like to release multiple
-tiny changes every day.
-
-Once you installed some LAVA components you can upgrade your installation to a
-more recent release using this deployment tool. There are some important
-upgrade considerations:
-
-1) Upgrades may alter the database or persistent media files. It is wise to
-perform a full system backup before each upgrade. While we don't anticipate
-catastrophic failures it's better to be safe than sorry. Refer to the previous
-chapter for details.
-
-2) Some database schema changes take a lot of time to finish. We try to
-minimize such changes but as you can install any third-party LAVA extensions we
-cannot predict the overall downtime in such case. For official Linaro releases
-please refer to our monthly release notes that are available at
-http://lava.readthedocs.org/
-
-3) Upgrades may introduce additional dependencies, which will be installed
-automatically. Periodically we make use of additional third party open source
-libraries. Those libraries will be installed for a single LAVA instance
-_only_. Your system libraries are not affected by this step. 
-
-4) Upgrades require network access. If you are behind a firewall or a
-corporate http proxy you may experience failures. Please note that the
-upgrade process does not install components without first downloading all of
-the required pieces so in case of a network failure your current installation
-should not be affected. While typically only HTTP and HTTPS protocols are
-being used at times you may see attempts to connect to native protocols used
-by git, bazaar or mercurial.
-
-5) Upgrading process rebuilds the collection of static assets served by
-Apache. During that moment you may encounter a very brief failure to resolve
-some of the static assets (typically images, cascading style sheets and
-javascript libraries)
-
-Prior to upgrading and instance, you will need to build a new bundle based
-on your new requirements file:
-
-$ ./lava-deployment-tool bundle URL-to-requirements [bundle_file_name]
-
-To upgrade an existing instance run the following command:
-
-$ ./lava-deployment-tool upgrade NAME BUNDLE
-
-Again the NAME and URL-to-requirements have the same meaning as in the
-install command mentioned in preceding chapter.
+In some sense, each revision of $LAVA_MANIFEST_BRANCH is a release and
+can be updated to (from time to time a revision of lp:lava-manifest will
+receive additional testing and be tagged as a release).  You can use
+lava-deployment-tool to update to a revision of the LAVA_MANIFEST_BRANCH
+that was used for that instance::
+
+ $ ./lava-deployment-tool upgrade <revno>
+
+There are some points to consider:
+
+1) Upgrades may alter the database or persistent media files. It is
+   wise to perform a full system backup before each upgrade. While we
+   don't anticipate catastrophic failures it's better to be safe than
+   sorry. Refer to the previous chapter for details.
+
+2) Upgrades may introduce additional dependencies, which will be
+   installed automatically. Periodically we make use of additional
+   third party open source libraries. Those libraries will be
+   installed for a single LAVA instance _only_. Your system libraries
+   are not affected by this step.
+
+3) Upgrades require network access. If you are behind a firewall or a
+   corporate http proxy you may experience failures. Please note that the
+   upgrade process does not install components without first downloading all of
+   the required pieces so in case of a network failure your current installation
+   should not be affected. While typically only HTTP and HTTPS protocols are
+   being used at times you may see attempts to connect to native protocols used
+   by git, bazaar or mercurial.
+
+4) The upgrade works by first installing the new code independently
+   from the currently running code, updates a symlink to make the new
+   code current, runs any database migrations that are needed and
+   restarts the services.  This means that many, but not all, kinds of
+   error during upgrade will not result in disruption to the running
+   service.
+
+5) Upgrading process rebuilds the collection of static assets served
+   by Apache.  During that moment you may encounter a very brief
+   failure to resolve some of the static assets (typically images,
+   cascading style sheets and javascript libraries)
+
+Upgrading from a pip-based instance to a buildout based instance
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In June 2012, we switched from a pip-based installation method to
+using buildout.  lava-deployment-tool can convert a pip-based
+installation to a buildout-based one, but without care this can result
+in changes to the set of packages/LAVA extensions installed.
+
+If you have no special requirements as to the extensions that should
+be installed you can just run "lava-deployment-tool upgrade
+$instance", and say y when prompted.
+
+If you have custom requirements, you should:
+
+ 1. create a test instance
+
+ 2. in this test instance, create custom buildout config file (see
+    `Limited Deployments`_ below) that installs the components you
+    want
+
+ 3. put this config file into the root of your existing instance as a
+    filed called, say, custom.cfg
+
+ 4. run LAVA_BUILDOUT_CFG=../../custom.cfg upgrade $instance
+
+Testing this process several times before running it on your
+production instance is advisable!
 
 Installing multiple LAVA instances on single IP machine
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 After installing and starting a LAVA instance using the instructions above,
 you can use "toggle" sub command to change how you access the instance.
@@ -210,21 +252,22 @@
 called "test1" and "test2", you can use the "location" toggle to set it
 so that you can access them on your local machine using:
 http://localhost/test1
-http://localhost/test2
+http://localhost/test2 ::
 
-$ lava-deployment-tool toggle $LAVA_INSTANCE location
+ $ lava-deployment-tool toggle $LAVA_INSTANCE location
 
 Then you can access the LAVA instance via:
     http://IP/$LAVA_INSTANCE
 
-You also can turn it back with the command below:
+You also can turn it back with the command below::
 
-$ lava-deployment-tool toggle $LAVA_INSTANCE vhost
+ $ lava-deployment-tool toggle $LAVA_INSTANCE vhost
 
 Then you can access the LAVA instance via:
     http://virtual-host/
 
-There is also a command to toggle all LAVA instances one time:
+There is also a command to toggle all LAVA instances one time::
+
     lava-deployment-tool toggle_all location
 
 Anatomy of a LAVA instance
@@ -232,11 +275,11 @@
 
 An instance is composed of several parts:
 
- - A new system user account called $LAVA_INSTANCE
+ - A new system user account called lava-$LAVA_INSTANCE
  - A directory tree similar to standard unix filesystem rooted
    in $LAVA_PREFIX/$LAVA_INSTANCE/
  - A postgres user and database in the default cluster, both named
-   $LAVA_INSTANCE
+   lava-$LAVA_INSTANCE
 
 A note on Postgres versions
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -247,6 +290,83 @@
 moving between versions is not technically difficult but not currently
 supported.
 
+Different kinds of deployment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Production-like
+---------------
+
+For our production deployment, we use the buildout-production.cfg
+buildout file which completely locks down the version of everything
+that's being deployed.  If you want to run the same kind of deployment
+as we do, set LAVA_BUILDOUT_CFG to ``buildout-production.cfg`` when
+creating the instance::
+
+ $ LAVA_BUILDOUT_CFG=buildout-production.cfg lava-deployment-tool install production
+
+To update the version of some component that is deployed we release
+that component, update the buildout-production.cfg file in
+lp:lava-manigest to refer to the new version and deploy the new
+revision.
+
+Local Development
+-----------------
+
+If you want an instance to use a custom branch of a component, you can
+drop the branch or a symlink to the branch in
+``/srv/lava/instances/$instances/code/current/local`` and re-run
+buildout.  For example::
+
+  $ bzr branch lp:lava-scheduler ~/src/my-scheduler-branch
+  $ cd /srv/lava/instances/$instance/code/current
+  $ ln -s ~/src/my-scheduler-branch local/ # The name of the symlink doesn't matter;
+                                           # buildout looks at the setup.py
+  $ ./bin/buildout
+
+Non-production instances contain some scripts that can help create and
+remove symlinks::
+
+  $ bzr branch lp:lava-scheduler ~/src/my-scheduler-branch
+  $ . /srv/lava/instances/testinstance/bin/activate
+  $ lava-develop-local ~/src/my-scheduler-branch
+  Determining egg name... lava-scheduler
+  + ln -sfT ~/src/my-scheduler-branch /srv/lava/instances/testinstance/code/current/local/lava-scheduler
+  ...
+  ... hack ...
+  $ lava-undevelop-local ~/src/my-scheduler-branch
+  removed /srv/lava/instances/testinstance/code/current/local/lava-scheduler
+
+Limited Deployments
+-------------------
+
+For a limited deployment, for example if you do not want to run the
+scheduler, you can set ``LAVA_BUILDOUT_CFG`` to point to a buildout
+config file you create.  Currently you can only do this after an
+instance is created (unfortunately).  So create an instance::
+
+ $ lava-deployment-tool install limited
+
+Create the custom buildout.cfg::
+
+ $ cat > /srv/lava/instances/limited/code/custom.cfg <<EOF
+ [buildout]
+ extends = buildout.cfg
+
+ [server]
+ eggs -= lava-scheduler
+ EOF
+
+And set it to be used in ``instance.conf``::
+
+ $ vim /srv/lava/instances/limited/code/custom.cfg
+ $ grep LAVA_BUILDOUT_CFG /srv/lava/instances/limited/code/custom.cfg
+ LAVA_BUILDOUT_CFG='../custom.cfg'
+
+Finally, 'upgrade' the instance to get it to use the new config file::
+
+ $ ./lava-deployment-tool upgrade limited
+
+
 Contact and bug reports
 ^^^^^^^^^^^^^^^^^^^^^^^
 

=== removed file '_compare_manifests.py'
--- _compare_manifests.py	2012-05-25 04:30:58 +0000
+++ _compare_manifests.py	1970-01-01 00:00:00 +0000
@@ -1,47 +0,0 @@
-#!/usr/bin/python
-
-import sys
-
-def load(fname):
-    d = {}
-    for line in open(fname):
-        line = line.strip()
-        if line.startswith('-e'):
-            # This bit is a total hack.
-            _, stuff = line.split('#egg=')
-            package, version, _ = stuff.split('-', 2)
-            package = package.replace('_', '-')
-        else:
-            package, version = line.split('==')
-        d[package] = version
-    return d
-
-cur = load(sys.argv[1])
-new = load(sys.argv[2])
-if 'pip' in new:
-    del new['pip']
-
-not_present = set(cur) - set(new)
-new_packaged = set(new) - set(cur)
-changed = {}
-for package in set(cur) & set(new):
-    if cur[package] != new[package]:
-        changed[package] = (cur[package], new[package])
-
-if not_present:
-    print "Already installed, but not in bundle:"
-    w1 = max(len(p) for p in not_present)
-    for k in sorted(not_present):
-        print "    %-*s %s" % (w1, k, cur[k])
-if changed:
-    print "To be upgraded:"
-    w1 = max(len(p) for p in changed)
-    w2 = max(len(t[0]) for t in changed.itervalues())
-    w3 = max(len(t[1]) for t in changed.itervalues())
-    for k in sorted(changed):
-        print "    %-*s %-*s -> %-*s" % (w1, k, w2, changed[k][0], w3, changed[k][1])
-if new_packaged:
-    print "To be installed:"
-    w1 = max(len(p) for p in new_packaged)
-    for k in sorted(new_packaged):
-        print "    %-*s %s" % (w1, k, new[k])

=== modified file 'lava-deployment-tool'
--- lava-deployment-tool	2012-06-11 10:02:34 +0000
+++ lava-deployment-tool	2012-06-18 06:02:17 +0000
@@ -1,5 +1,5 @@
 #!/bin/bash
-LAVA_DEPLOYMENT_TOOL_VERSION="0.4"
+LAVA_DEPLOYMENT_TOOL_VERSION="0.5"
 
 # Global Configuration
 
@@ -9,10 +9,14 @@
 # All LAVA instances are created relative to this path
 LAVA_PREFIX=$LAVA_ROOT/instances
 
-# Pip download cache 
-export PIP_DOWNLOAD_CACHE=$LAVA_ROOT/.downloads
+# Deploy revisions of this branch
+LAVA_MANIFEST_BRANCH="${LAVA_MANIFEST_BRANCH-lp:lava-manifest}"
+# Use this buildout config file from $LAVA_MANIFEST_BRANCH
+LAVA_BUILDOUT_CFG="${LAVA_BUILDOUT_CFG-buildout.cfg}"
 
-test -n "$http_proxy" && PIP_PROXY_OPTION="--proxy=$http_proxy"
+# Where to cache the eggs and downloads for buildbot
+LAVA_EGG_CACHE=$LAVA_ROOT/.cache/eggs
+LAVA_DOWNLOAD_CACHE=$LAVA_ROOT/.cache/download-cache
 
 # All LAVA uses this python version
 LAVA_PYTHON=python2.6
@@ -21,13 +25,13 @@
 LAVA_UWSGI=1.0.3
 
 # Current version of setup required by lava (global state)
-export LAVA_SETUP_REQUIRED_VERSION=20
+export LAVA_SETUP_REQUIRED_VERSION=22
 
 # Check if this installation is supported
 export LAVA_SUPPORTED=0
 
 # Installation and configuration steps (all the wizard_xxx install_xxx functions)
-LAVA_INSTALL_STEPS="user fs venv database broker web_hosting app config_app"
+LAVA_INSTALL_STEPS="user fs buildout database broker web_hosting app config_app"
 
 
 os_check() {
@@ -44,7 +48,7 @@
                 oneiric|precise)
                     export LAVA_PYTHON=python2.7
                     export LAVA_SUPPORTED=1
-                    LAVA_PKG_LIST="whiptail python-virtualenv python-pip git build-essential $LAVA_PYTHON-dev libxml2-dev apache2 apache2-dev postgresql rabbitmq-server mercurial conmux linaro-image-tools unzip"
+                    LAVA_PKG_LIST="whiptail python-virtualenv python-pip git build-essential $LAVA_PYTHON-dev libxml2-dev apache2 apache2-dev postgresql rabbitmq-server mercurial conmux linaro-image-tools unzip bzr"
                     ;;
             esac
             ;;
@@ -89,6 +93,9 @@
 LAVA_PREFIX='$LAVA_PREFIX'
 # Instance name
 LAVA_INSTANCE='$LAVA_INSTANCE'
+# The lava-server branch used by upgrade
+LAVA_MANIFEST_BRANCH='$LAVA_MANIFEST_BRANCH'
+LAVA_BUILDOUT_CFG='$LAVA_BUILDOUT_CFG'
 # System configuration (Unix-level)
 LAVA_SYS_USER='$LAVA_SYS_USER'
 # Apache configuration
@@ -112,31 +119,10 @@
 
 _save_config() {
     mkdir -p $LAVA_PREFIX/$LAVA_INSTANCE/
-    cat >$LAVA_PREFIX/$LAVA_INSTANCE/instance.conf <<INSTANCE_CONF
-# Installation prefix
-LAVA_PREFIX='$LAVA_PREFIX'
-# Instance name
-LAVA_INSTANCE='$LAVA_INSTANCE'
-# System configuration (Unix-level)
-LAVA_SYS_USER='$LAVA_SYS_USER'
-# Apache configuration
-LAVA_APACHE_VHOST='$LAVA_APACHE_VHOST'
-LAVA_DEV_MODE='$LAVA_DEV_MODE'
-# PostgreSQL configuration
-LAVA_DB_NAME='$LAVA_DB_NAME'
-LAVA_DB_USER='$LAVA_DB_USER'
-LAVA_DB_PASSWORD='$LAVA_DB_PASSWORD'
-# RabbitMQ configuration
-LAVA_RABBIT_VHOST='$LAVA_RABBIT_VHOST'
-LAVA_RABBIT_USER='$LAVA_RABBIT_USER'
-LAVA_RABBIT_PASSWORD='$LAVA_RABBIT_PASSWORD'
-# Scheduler configuration
-LAVA_SCHEDULER_ENABLED='$LAVA_SCHEDULER_ENABLED'
-INSTANCE_CONF
+    _show_config >$LAVA_PREFIX/$LAVA_INSTANCE/instance.conf
 }
 
 
-
 _configure() {
     set +x
     export LAVA_INSTANCE="$1"
@@ -193,7 +179,7 @@
 _install() {
     for install_step in $LAVA_INSTALL_STEPS; do 
         echo "Running installation step $install_step"
-        install_$install_step || die "Failed at $install_step"
+        install_$install_step
     done
     echo "All installation is done"
 }
@@ -204,11 +190,6 @@
 }
 
 
-pkglist_user() {
-    true
-}
-
-
 wizard_user() {
     export LAVA_SYS_USER_DESC="User for LAVA instance $LAVA_INSTANCE"
 
@@ -257,11 +238,6 @@
 }
 
 
-pkglist_fs() {
-    true
-}
-
-
 wizard_fs() {
     echo
     echo "Filesystem configuration"
@@ -316,11 +292,13 @@
     mkdir -p $LAVA_PREFIX/$LAVA_INSTANCE/var/log
     # Sockets and other runtime stuff
     mkdir -p $LAVA_PREFIX/$LAVA_INSTANCE/run
-    # Source code (used when tracking trunk)
-    mkdir -p $LAVA_PREFIX/$LAVA_INSTANCE/src
     # Temporary files
     mkdir -p $LAVA_PREFIX/$LAVA_INSTANCE/tmp
 
+    # For buildout
+    bzr init-repo $LAVA_PREFIX/$LAVA_INSTANCE/code
+    ln -s $LAVA_PREFIX/$LAVA_INSTANCE/code/current/bin $LAVA_PREFIX/$LAVA_INSTANCE
+
     # Allow apache (running as www-data) to read our public web files 
     sudo chgrp -R www-data $LAVA_PREFIX/$LAVA_INSTANCE/var/www/lava-server/
     sudo chmod -R g+rXs $LAVA_PREFIX/$LAVA_INSTANCE/var/www/lava-server/
@@ -350,58 +328,29 @@
 }
 
 
-defaults_venv() {
-    true
-}
-
-pkglist_venv() {
-    echo "https://github.com/zyga/pip/tarball/develop"
-}
-
-
-wizard_venv() {
-    echo
-    echo "Python virtual environment configuration"
-    echo "^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^"
-    echo
-    echo "We will use python virtualenv to isolate LAVA code from your system"
-    echo
-    echo "Python version: $LAVA_PYTHON"
-    echo
-    echo "You can interact with this source code (installation) as if it"
-    echo "was globally by invoking the following line:"
-    echo
-    echo "$ . $LAVA_PREFIX/$LAVA_INSTANCE/bin/activate"
-    echo
-    echo "(Note that the intent is to source (read) the file, not execute it)"
-    echo "When you are done just type:"
-    echo
-    echo "$ deactivate"
-    echo
-    echo "While inside your virtualenv you can invoke additional commands,"
-    echo "most notably, lava-server manage ..."
-    echo 
-    echo "(this is just a notification, it is not configurable)"
-    echo
-    echo "next    - continue"
-
-    read -p "Please decide what to do: " RESPONSE
-
-    case "$RESPONSE" in
-        next)
-            return 0  # loop complete
-    esac
-    return 1  # another loop please
-}
-
-
-install_venv() {
-    logger "Creating virtualenv using $LAVA_PYTHON for LAVA instance $LAVA_INSTANCE"
-
-    # Create virtualenv
-    virtualenv --no-site-packages --distribute $LAVA_PREFIX/$LAVA_INSTANCE -p $LAVA_PYTHON
-}
-
+defaults_buildout () {
+    true
+}
+
+wizard_buildout () {
+    true
+}
+
+install_buildout () {
+    set -xe
+    local venv="$LAVA_PREFIX/$LAVA_INSTANCE/var/lib/bootstrap-venv"
+    if [ $(virtualenv --version) '<' '1.7' ]; then
+        virtualenv --no-site-packages "${venv}"
+    else
+        virtualenv "${venv}"
+    fi
+    "${venv}/bin/pip" install \
+        $LAVA_DOWNLOAD_CACHE/dist/distribute-0.6.24.tar.gz \
+        $LAVA_DOWNLOAD_CACHE/dist/zc.buildout-1.5.2.tar.gz \
+        $LAVA_DOWNLOAD_CACHE/dist/uwsgi-$LAVA_UWSGI.tar.gz \
+        $LAVA_DOWNLOAD_CACHE/dist/bzr-2.5.1.tar.gz
+    set +xe
+}
 
 defaults_database() {
     export LAVA_DB_NAME="lava-$LAVA_INSTANCE"
@@ -410,11 +359,6 @@
 }
 
 
-pkglist_database() {
-    echo psycopg2 
-}
-
-
 wizard_database() {
     echo
     echo "PostgreSQL configuration"
@@ -491,11 +435,6 @@
 }
 
 
-pkglist_broker() {
-    true
-}
-
-
 wizard_broker() {
     echo
     echo "RabbitMQ configuration"
@@ -539,15 +478,6 @@
     export LAVA_DEV_MODE=no
 }
 
-
-pkglist_web_hosting() {
-    echo "http://projects.unbit.it/downloads/old/uwsgi-$LAVA_UWSGI.tar.gz"
-    echo "django-seatbelt"
-    echo "django-debian"
-    echo "django-celery"
-}
-
-
 wizard_web_hosting() {
     echo
     echo "Apache and uWSGI configuration"
@@ -615,82 +545,6 @@
 install_web_hosting() {
     logger "Installing uWSGI and other hosting parts for LAVA instance $LAVA_INSTANCE"
 
-    if [ \! -e /etc/apache2/mods-available/uwsgi.load ]; then
-        logger "Building uWSGI apache module..."
-        # NOTE: we may not have this file cached as we used bundle to get it
-        if [ ! -e $PIP_DOWNLOAD_CACHE/http%3A%2F%2Fprojects.unbit.it%2Fdownloads%2Fold%2Fuwsgi-$LAVA_UWSGI.tar.gz ]; then
-            pip install $PIP_PROXY_OPTION --no-install http://projects.unbit.it/downloads/old/uwsgi-$LAVA_UWSGI.tar.gz || die "Failed to download uWSGI"
-        fi
-        ( cd $LAVA_PREFIX/$LAVA_INSTANCE/tmp && tar zxf $PIP_DOWNLOAD_CACHE/http%3A%2F%2Fprojects.unbit.it%2Fdownloads%2Fold%2Fuwsgi-$LAVA_UWSGI.tar.gz )
-        ( cd $LAVA_PREFIX/$LAVA_INSTANCE/tmp/uwsgi-$LAVA_UWSGI/apache2 && sudo apxs2 -c -i -a mod_uwsgi.c )
-    fi
-
-    if [ "$LAVA_DEV_MODE" = "yes" ]; then
-        cat >$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/lava-server.wsgi <<INSTANCE_WSGI
-# This file was automatically generated by lava-deployment-tool.sh
-import os
-
-# Force django to use the specified settings module.
-os.environ['DJANGO_SETTINGS_MODULE'] = 'lava_server.settings.debian'
-# And force django-debian to look at our instance-specific configuration directory
-os.environ['DJANGO_DEBIAN_SETTINGS_TEMPLATE'] = '$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/{filename}.conf'
-
-# Setup django WSGI handler
-import django.core.handlers.wsgi
-
-application = django.core.handlers.wsgi.WSGIHandler()
-INSTANCE_WSGI
-    else
-        cat >$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/lava-server.wsgi <<INSTANCE_WSGI
-# This file was automatically generated by lava-deployment-tool.sh
-import os
-import sys
-from django_seatbelt import seatbelt
-
-# We need those to get stuff like threading that seems not to be supported when
-# running virtualenv. Despite virtualenv setting up symlinks for some of the
-# .so files this one does not seem to work. More insight on if this is really
-# the case welcome.
-core_python_paths = [
-    "/usr/lib/$LAVA_PYTHON",
-    "/usr/lib/$LAVA_PYTHON/lib-dynload"]
-
-# Construct new path starting with core python, followed by the rest
-sys.path = core_python_paths + [path for path in sys.path if path not in core_python_paths]
-
-# Filter path, allow only core python paths and prefix paths (no local/user/junk)
-def allow_core_python(path):
-    return path in core_python_paths
-
-# In virtualenv sys.prefix points to the root of the virtualenv, outside it
-# points to the prefix of the system python installation (typically /usr)
-def allow_sys_prefix(path):
-    return path.startswith(sys.prefix)
-
-# seatbelt.solder filters sys.path according to the callbacks specified below.
-seatbelt.solder(allow_callbacks=[allow_sys_prefix, allow_core_python])
-
-# Print summary (for debugging)
-# print "sys.prefix:", sys.prefix
-# print "sys.path:"
-# for path in sys.path:
-#    print " - %s" % path
-
-# Force django to use the specified settings module.
-os.environ['DJANGO_SETTINGS_MODULE'] = 'lava_server.settings.debian'
-# And force django-debian to look at our instance-specific configuration directory
-os.environ['DJANGO_DEBIAN_SETTINGS_TEMPLATE'] = '$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/{filename}.conf'
-
-# Setup django WSGI handler
-import django.core.handlers.wsgi
-
-# NOTE: Here one might also use applications = {'/': application} to define
-# namespace mapping. I'm not sure if this is used by uWSGI or by any handler
-# but I found that in the docs to uWSGI.
-application = django.core.handlers.wsgi.WSGIHandler()
-INSTANCE_WSGI
-    fi
-
     # Create apache2 site
     cat >$LAVA_PREFIX/$LAVA_INSTANCE/etc/apache2/sites-available/lava-server.conf <<INSTANCE_SITE
 <VirtualHost *:80>
@@ -742,10 +596,10 @@
     # Create uWSGI configuration file
     cat >$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/uwsgi.ini <<UWSGI_INI
 [uwsgi]
-home = $LAVA_PREFIX/$LAVA_INSTANCE
+home = $LAVA_PREFIX/$LAVA_INSTANCE/var/lib/bootstrap-venv
 socket = $LAVA_PREFIX/$LAVA_INSTANCE/run/uwsgi.sock
 chmod-socket = 660
-wsgi-file = $LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/lava-server.wsgi
+wsgi-file = $LAVA_PREFIX/$LAVA_INSTANCE/bin/lava-server.wsgi
 master = true
 workers = 8
 logto = $LAVA_PREFIX/$LAVA_INSTANCE/var/log/lava-uwsgi.log
@@ -767,11 +621,6 @@
 }
 
 
-pkglist_app() {
-    cat $LAVA_REQUIREMENT
-}
-
-
 wizard_app() {
     echo
     echo "LAVA application configuration"
@@ -782,86 +631,66 @@
 
 
 install_app() {
-    echo "Installing/upgrading application code from $LAVA_BUNDLE file"
-
-    # Clean all builds to allow package upgrading
-    rm -rf $LAVA_PREFIX/$LAVA_INSTANCE/build/*
-    . $LAVA_PREFIX/$LAVA_INSTANCE/bin/activate
-    set -x
-    pip install --upgrade $PIP_PROXY_OPTION "$LAVA_BUNDLE" || die "Failed to install application bundle"
-    set +x
-    deactivate
+    echo "Installing/upgrading application code from a revision of $LAVA_MANIFEST_BRANCH"
+
+    if [ ! -e $LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/settings.conf ]; then
+        echo "Generating $LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/settings.conf for PRODUCTION"
+        cat >$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/settings.conf <<SETTINGS_CONF
+{
+    "DEBUG": false,
+    "TEMPLATE_DIRS": [
+        "$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/templates",
+        "$LAVA_PREFIX/$LAVA_INSTANCE/code/current/server_code/templates/"
+    ],
+    "STATICFILES_DIRS": [
+        ["lava-server", "$LAVA_PREFIX/$LAVA_INSTANCE/code/current/server_code/htdocs/"]
+    ],
+    "MEDIA_ROOT": "$LAVA_PREFIX/$LAVA_INSTANCE/var/lib/lava-server/media",
+    "STATIC_ROOT": "$LAVA_PREFIX/$LAVA_INSTANCE/var/www/lava-server/static",
+    "MEDIA_URL": "/media/",
+    "STATIC_URL": "/static/",
+    "MOUNT_POINT": "/",
+    "LOGIN_URL": "/accounts/login/",
+    "LOGIN_REDIRECT_URL": "/",
+    "BROKER_USER": "$LAVA_RABBIT_USER",
+    "BROKER_VHOST": "$LAVA_RABBIT_VHOST",
+    "BROKER_PASSWORD": "$LAVA_RABBIT_PASSWORD",
+    "DATAREPORT_DIRS": [
+        "$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/reports"
+    ],
+    "DATAVIEW_DIRS": [
+        "$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/views"
+    ]
+}
+SETTINGS_CONF
+    fi
+
+    local venv="$LAVA_PREFIX/$LAVA_INSTANCE/var/lib/bootstrap-venv"
+    local code="$LAVA_PREFIX/$LAVA_INSTANCE/code/r$LAVA_REVNO"
+
+    set -ex
+    if [ ! -e ${code} ]; then
+        bzr branch -r $LAVA_REVNO $LAVA_MANIFEST_BRANCH $code
+        mkdir ${code}/bin
+        ln -s ${venv}/bin/{uwsgi,buildout} ${code}/bin/
+        ln -s $LAVA_DOWNLOAD_CACHE ${code}/download-cache
+        ln -s $LAVA_EGG_CACHE ${code}/eggs
+    fi
+    # If LAVA_BUILDOUT_CFG does not specify a file in the branch,
+    # symlink it in here before running buildout (as buildout works
+    # relative to the config file).
+    local cfg="$(basename "${LAVA_BUILDOUT_CFG}")"
+    if [ "${cfg}" != "${LAVA_BUILDOUT_CFG}" ]; then
+        ln -sfT "${LAVA_BUILDOUT_CFG}" "${cfg}"
+    fi
+    pushd $code
+    ${venv}/bin/buildout -c "${cfg}"
+    popd
+    set +ex
 
     gen_lava_dispatcher_config
 
-    if [ ! -e $LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/settings.conf ]; then
-        if [ -e $LAVA_PREFIX/$LAVA_INSTANCE/src/lava-server ]; then
-            # We're in editable server mode, let's use alternate paths for tempates and static files
-            echo "Generating $LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/settings.conf for DEVELOPMENT"
-            echo "This means that lava-server (which has to be special) is installed in editable mode"
-            echo "Some of the paths will refer to $LAVA_PREFIX/$LAVA_INSTANCE/src/lava-server."
-            echo "Even if you later on upgrade to non-editable lava-server those paths"
-            echo "will remain to be in use untill you wipe your old config and upgrade again"
-            cat >$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/settings.conf <<SETTINGS_CONF
-{
-    "DEBUG": false,
-    "TEMPLATE_DIRS": [
-        "$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/templates",
-        "$LAVA_PREFIX/$LAVA_INSTANCE/src/lava-server/lava_server/templates/"
-    ],
-    "STATICFILES_DIRS": [
-        ["lava-server", "$LAVA_PREFIX/$LAVA_INSTANCE/src/lava-server/lava_server/htdocs/"]
-    ],
-    "MEDIA_ROOT": "$LAVA_PREFIX/$LAVA_INSTANCE/var/lib/lava-server/media",
-    "STATIC_ROOT": "$LAVA_PREFIX/$LAVA_INSTANCE/var/www/lava-server/static",
-    "MEDIA_URL": "/media/",
-    "STATIC_URL": "/static/",
-    "MOUNT_POINT": "/",
-    "LOGIN_URL": "/accounts/login/",
-    "LOGIN_REDIRECT_URL": "/",
-    "BROKER_USER": "$LAVA_RABBIT_USER",
-    "BROKER_VHOST": "$LAVA_RABBIT_VHOST",
-    "BROKER_PASSWORD": "$LAVA_RABBIT_PASSWORD",
-    "DATAREPORT_DIRS": [
-        "$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/reports"
-    ],
-    "DATAVIEW_DIRS": [
-        "$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/views"
-    ]
-}
-SETTINGS_CONF
-        else
-            echo "Generating $LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/settings.conf for PRODUCTION"
-            cat >$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/settings.conf <<SETTINGS_CONF
-{
-    "DEBUG": false,
-    "TEMPLATE_DIRS": [
-        "$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/templates",
-        "$LAVA_PREFIX/$LAVA_INSTANCE/lib/$LAVA_PYTHON/site-packages/lava_server/templates/"
-    ],
-    "STATICFILES_DIRS": [
-        ["lava-server", "$LAVA_PREFIX/$LAVA_INSTANCE/lib/$LAVA_PYTHON/site-packages/lava_server/htdocs/"]
-    ],
-    "MEDIA_ROOT": "$LAVA_PREFIX/$LAVA_INSTANCE/var/lib/lava-server/media",
-    "STATIC_ROOT": "$LAVA_PREFIX/$LAVA_INSTANCE/var/www/lava-server/static",
-    "MEDIA_URL": "/media/",
-    "STATIC_URL": "/static/",
-    "MOUNT_POINT": "/",
-    "LOGIN_URL": "/accounts/login/",
-    "LOGIN_REDIRECT_URL": "/",
-    "BROKER_USER": "$LAVA_RABBIT_USER",
-    "BROKER_VHOST": "$LAVA_RABBIT_VHOST",
-    "BROKER_PASSWORD": "$LAVA_RABBIT_PASSWORD",
-    "DATAREPORT_DIRS": [
-        "$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/reports"
-    ],
-    "DATAVIEW_DIRS": [
-        "$LAVA_PREFIX/$LAVA_INSTANCE/etc/lava-server/views"
-    ]
-}
-SETTINGS_CONF
-    fi
-fi
+    ln -sfT $code $LAVA_PREFIX/$LAVA_INSTANCE/code/current
 }
 
 
@@ -870,22 +699,17 @@
 }
 
 
-pkglist_config_app() {
-    true
-}
-
-
 wizard_config_app() {
     local _question="Do you want to enable the scheduler in this instance?"
     whiptail --title "Enable the scheduler?" --yesno "$_question" 10 40 && LAVA_SCHEDULER_ENABLED=yes || LAVA_SCHEDULER_ENABLED=no
 
     local _question="Do you want to config the lava-dispatcher?"
     whiptail --title "Configurate the lava-dispatcher?" --yesno "$_question" 10 40 && LAVA_DISPATCHER_CONFIG=yes || LAVA_DISPATCHER_CONFIG=no
-    
+
     if [ $LAVA_DISPATCHER_CONFIG = yes ]; then
         config_lava_dispatcher
     fi
-           
+
     return 0
 }
 
@@ -998,8 +822,6 @@
     set -x
     rm -rf $LAVA_PREFIX/$LAVA_INSTANCE/var/www/lava-server/static
     lava-server manage \
-        --instance-template=$LAVA_ROOT/instances/{instance}/etc/lava-server/{{filename}}.conf \
-        --instance=$LAVA_INSTANCE \
         build_static --noinput --link || die "Failed to update the cache of static content"
     set +x
 
@@ -1011,16 +833,12 @@
     echo "Synchronizing database..."
     set -x
     lava-server manage \
-        --instance-template=$LAVA_ROOT/instances/{instance}/etc/lava-server/{{filename}}.conf \
-        --instance=$LAVA_INSTANCE \
         syncdb --noinput || die "Failed to synchronize database (run non-migration db updates)"
     set +x
 
     echo "Running migrations..."
     set -x
     lava-server manage \
-        --instance-template=$LAVA_ROOT/instances/{instance}/etc/lava-server/{{filename}}.conf \
-        --instance=$LAVA_INSTANCE \
         migrate --noinput || die "Failed to run database migrations"
     set +x
 
@@ -1045,6 +863,16 @@
     fi
 }
 
+ensure_downloaded_check_sha256 () {
+    local url="$1"
+    local sha256="$2"
+    local filename="$(basename "$url")"
+    if [ ! -f "$filename" ]; then
+        wget "$url"
+    fi
+    echo "$sha256  $filename" | sha256sum -c || die "checksum mismatch on $filename"
+}
+
 cmd_setup() {
     SETUP_VER=0
     if [ -e $LAVA_PREFIX/.setup ]; then
@@ -1060,15 +888,17 @@
         echo "System preparation steps:"
         echo " 1) Installing $LAVA_PKG_LIST"
         echo " 2) Setting up $LAVA_PREFIX owned by you"
-        echo " 3) Setting up $PIP_DOWNLOAD_CACHE for downloads"
+        echo " 3) Setting up the egg and download cache for buildbot"
         echo " 4) Setting up upstart jobs (incuding removal of stale jobs)"
         echo
         read -p "Type YES to continue: " RESPONSE
         test "$RESPONSE" = 'YES' || return
 
+        set -xe
+
         echo "Updating apt cache..."
         sudo apt-get update
-                                       
+
 	echo "Add Linaro tools ppa"
 	sudo apt-get install --yes python-software-properties
 	sudo add-apt-repository --yes ppa:linaro-maintainers/tools
@@ -1086,12 +916,29 @@
         echo "Creating LAVA filesystem in $LAVA_PREFIX"
         sudo mkdir -p $LAVA_PREFIX
         echo "Making $(whoami) the owner of that location"
-        sudo chown $(whoami):$(id -gn) $LAVA_PREFIX 
-
-        echo "Creating PIP download cache in $PIP_DOWNLOAD_CACHE and making it writable"
-        sudo mkdir -p $PIP_DOWNLOAD_CACHE
-        sudo chown $(whoami):$(id -gn) $PIP_DOWNLOAD_CACHE 
-
+        sudo chown $(whoami):$(id -gn) $LAVA_PREFIX
+
+        echo "Creating buildout caches"
+        sudo mkdir -p $LAVA_EGG_CACHE
+        sudo chown $(whoami):$(id -gn) $LAVA_EGG_CACHE
+        sudo mkdir -p $LAVA_DOWNLOAD_CACHE/dist
+        sudo chown -R $(whoami):$(id -gn) $LAVA_DOWNLOAD_CACHE
+        pushd $LAVA_DOWNLOAD_CACHE/dist
+        ensure_downloaded_check_sha256 \
+            http://pypi.python.org/packages/source/d/distribute/distribute-0.6.24.tar.gz \
+            c61fde9f388c9600eb8ee54bd7168039c5fb74fa334138bc49cdf6a6c1341627
+        ensure_downloaded_check_sha256 \
+            https://launchpad.net/bzr/2.5/2.5.1/+download/bzr-2.5.1.tar.gz \
+            967346fa6cc52971a8dfd5171ec22e1928cf455fb4317013b05b0915ff927982
+        ensure_downloaded_check_sha256 \
+            http://projects.unbit.it/downloads/old/uwsgi-$LAVA_UWSGI.tar.gz \
+            81dc137bc8b7b3962e527cafb1a719342c34eb07d568dc61c7766f9a9bcdee62
+        ensure_downloaded_check_sha256 \
+            http://pypi.python.org/packages/source/z/zc.buildout/zc.buildout-1.5.2.tar.gz \
+            0ac5a325d3ffbc5a988fb3ba87f4159d4769cc73e3331cb5234edc8839b6506b
+        popd
+
+        set +x
         echo "Creating upstart script for: lava"
         sudo sh -c "cat >/etc/init/lava.conf" <<LAVA_CONF
 author "Zygmunt Krynicki"
@@ -1434,6 +1281,21 @@
 end script
 LAVA_CONF
 
+        if [ \! -e /etc/apache2/mods-available/uwsgi.load ]; then
+            echo "Building uWSGI apache module..."
+            _tmpdir="$(mktemp -d)"
+
+            trap "rm -rf \"${_tmpdir}\"" EXIT
+            set -x
+            pushd $_tmpdir
+            tar -xzf $LAVA_DOWNLOAD_CACHE/dist/uwsgi-$LAVA_UWSGI.tar.gz --strip-components=1
+            cd apache2
+            apxs2 -c mod_uwsgi.c
+            sudo apxs2 -i -a mod_uwsgi.c
+            popd
+            set +x
+        fi
+
         # Store setup version
         echo $LAVA_SETUP_REQUIRED_VERSION > $LAVA_PREFIX/.setup
         echo "Setup complete, you can now install LAVA"
@@ -1459,7 +1321,7 @@
 
 cmd_install() {
     export LAVA_INSTANCE=$1
-    export LAVA_BUNDLE=$2
+    export LAVA_REVNO=$2
 
     check_min_version
     _banner
@@ -1469,9 +1331,10 @@
         return
     fi
 
-    if [ -z "$LAVA_BUNDLE" ]; then
-        echo "You need to pass the name of the bundle to use as second argument"
-        return
+    if [ -z "$LAVA_REVNO" ]; then
+        echo -n "Determining revno to deploy... "
+        LAVA_REVNO="$(bzr revno $LAVA_MANIFEST_BRANCH)"
+        echo $LAVA_REVNO
     fi
 
     # Sanity checking, ensure that instance does not exist yet
@@ -1489,7 +1352,7 @@
 
 cmd_upgrade() {
     export LAVA_INSTANCE=$1
-    export LAVA_BUNDLE=$2
+    export LAVA_REVNO=$2
 
     check_min_version
     _banner
@@ -1499,9 +1362,12 @@
         return
     fi
 
-    if [ -z "$LAVA_BUNDLE" ]; then
-        echo "You need to pass the name of the bundle to use as second argument"
-        return
+    _load_configuration "$LAVA_INSTANCE"
+
+    if [ -z "$LAVA_REVNO" ]; then
+        echo -n "Determining revno to deploy... "
+        LAVA_REVNO="$(bzr revno $LAVA_MANIFEST_BRANCH)"
+        echo $LAVA_REVNO
     fi
 
     # Sanity checking, ensure that instance exist
@@ -1510,9 +1376,58 @@
         return
     fi
 
-    _load_configuration "$LAVA_INSTANCE"
-    install_app || die "Failed to upgrade application code"
-    install_config_app || die "Failed to run post-upgrade configuration"
+    _maybe_pre_convert_to_buildout
+    install_app
+    _maybe_post_convert_to_buildout
+    install_config_app
+}
+
+
+_maybe_pre_convert_to_buildout () {
+    if [ -e $LAVA_PREFIX/$LAVA_INSTANCE/code ]; then
+        return
+    fi
+    if [ ! -e $LAVA_PREFIX/$LAVA_INSTANCE/lib ]; then
+        echo "Instance appears to be neither pip based or buildout based."
+        echo "Cowardly refusing to proceed."
+        exit 1;
+    fi
+
+    echo "$LAVA_INSTANCE appears to be a pip based instance"
+    echo "This script can attempt to convert it, but this may "
+    echo "lose any custom packages you have installed unless you "
+    echo "already have a plan to cope with that."
+    echo
+    read -p "Proceed? (y/n) " RESPONSE
+
+    if [ "$RESPONSE" != 'y' ]; then
+        exit 1;
+    fi
+
+    set -xe
+    _save_config
+    install_buildout
+    bzr init-repo $LAVA_PREFIX/$LAVA_INSTANCE/code
+    set +xe
+}
+
+
+_maybe_post_convert_to_buildout () {
+    local instpath=$LAVA_PREFIX/$LAVA_INSTANCE
+    set -xe
+    if [ ! -e $instpath/lib ]; then
+        return
+    fi
+    $instpath/bin/pip freeze > $instpath/last-pip-installed-versions.txt
+    rm -rf $instpath/{bin,lib,src,build,local}
+    rm $instpath/etc/lava-server/lava-server.wsgi
+    ln -s $instpath/code/current/bin $instpath
+    sed -i -e 's%lib/python2\../site-packages/lava_server%code/current/server_code%' \
+        $instpath/etc/lava-server/settings.conf
+    sed -i -e "s%home = $instpath%&/var/lib/bootstrap-venv%" \
+        -e "s%$instpath/etc/lava-server/lava-server\.wsgi%$instpath/bin/lava-server.wsgi%" \
+        $instpath/etc/lava-server/uwsgi.ini
+    set +xe
 }
 
 
@@ -1788,51 +1703,11 @@
 
     set -x
     lava-server manage \
-        --instance-template=$LAVA_ROOT/instances/{instance}/etc/lava-server/{{filename}}.conf \
-        --instance=$LAVA_INSTANCE \
         "$@"
     set +x
 }
 
 
-cmd_bundle() {
-    LAVA_REQUIREMENT=${1:-requirements/requirements.txt}
-    LAVA_BUNDLE=${2:-lava.pybundle}
-    local step
-    for step in $LAVA_INSTALL_STEPS; do
-        defaults_$step
-    done
-    LAVA_FULL_REQUIREMENT=$(mktemp)
-    for step in $LAVA_INSTALL_STEPS; do
-        pkglist_$step >> "$LAVA_FULL_REQUIREMENT"
-    done
-    set -x
-    pip bundle "$LAVA_BUNDLE" $PIP_PROXY_OPTION --timeout=30 --use-mirrors --requirement="$LAVA_FULL_REQUIREMENT" || die "Failed to create bundle"
-    set +x
-    rm -f "$LAVA_FULL_REQUIREMENT"
-}
-
-cmd_preview () {
-    export LAVA_INSTANCE=$1
-    export LAVA_BUNDLE=$2
-
-    check_min_version
-
-    local compare_manifests="$(dirname $(readlink -f "$0"))/_compare_manifests.py"
-
-    _bundle="$(readlink -f $LAVA_BUNDLE)"
-    _tmpdir="$(mktemp -d)"
-
-    trap "rm -rf \"${_tmpdir}\"" EXIT
-
-    cd "${_tmpdir}"
-
-    unzip -qc "${_bundle}" pip-manifest.txt | grep -v '^#' | tr '[A-Z]' '[a-z]' | sort > bundle-versions.txt
-    $LAVA_PREFIX/$LAVA_INSTANCE/bin/pip freeze | tr '[A-Z]' '[a-z]' | sort > installed-versions.txt
-
-    "${compare_manifests}" installed-versions.txt bundle-versions.txt
-}
-
 is_vhost(){
     export LAVA_INSTANCE=$1
     vhost_conf="/etc/apache2/sites-enabled/$LAVA_INSTANCE.conf"