Merge lp:~zyga/checkbox/selftest into lp:checkbox

[Zygmunt Krynicki]

This branch adds selftest, A standalone script that lets anyone use plainbox to test something.

We're going to use it to test checkbox.

There are some strings attached but I wanted to share it early
 - it doesn't yet handle package installation, it's easy to do, just not something I did yesterday
 - it doesn't build checkbox-gui

What does it do? It's easier to paste the tail of the output:
==================================[ Results ]===================================
 ☑ : Smoke test for selftest
 ☑ : Create a bare virtualenv
 ☑ : Populate virtualenv with additional environment variables
 ☑ : Clone the external-tarballs repository
 ☑ : Install all python dependencies
 ☑ : Meta-job for all bootstrap actions
 ☑ : Enumerate Python Projects
 ☑ : Enumerate Plainbox Providers
 ☑ : Develop Plainbox
 ☑ : Plainbox Integration Tests
 ☑ : Develop Checkbox Support
 ☑ : Develop Checkbox NG
 ☑ : Plainbox Unit Tests
 ☑ : Checkbox Support Unit Tests
 ☑ : Checkbox NG Unit Tests
 ☑ : Build Plainbox Internationalization Catalog
 ☑ : Build Checkbox Support Internationalization Catalog
 ☑ : Build Checkbox NG Internationalization Catalog
 ☑ : Build Plainbox HTML Documentation
 ☑ : Build Checkbox NG HTML Documentation
 ☑ : Build Plainbox Manual Pages
 ☑ : Develop Checkbox Provider for Plainbox
 ☑ : Develop Resource Provider for Plainbox
 ☑ : Develop Ubuntu Touch Provider for Plainbox
 ☑ : Develop Canonical Client Certification Provider for Plainbox
 ☑ : Develop Canonical Server Certification Provider for Plainbox
 ☑ : Validate Resource Provider for Plainbox
 ☑ : Validate Checkbox Provider for Plainbox
 ☑ : Validate Ubuntu Touch Provider for Plainbox
 ☑ : Validate Canonical Client Certification Provider for Plainbox
 ☑ : Validate Canonical Server Certification Provider for Plainbox
 ☑ : Build Checkbox Provider for Plainbox
 ☑ : Build Resource Provider for Plainbox
 ☑ : Build Ubuntu Touch Provider for Plainbox
 ☑ : Build Canonical Client Certification Provider for Plainbox
 ☑ : Build Canonical Server Certification Provider for Plainbox
 ☑ : Build Checkbox Provider for Plainbox Internationalization Catalog
 ☑ : Build Resource Provider for Plainbox Internationalization Catalog
 ☑ : Build Ubuntu Touch Provider for Plainbox Internationalization Catalog
 ☑ : Build Canonical Client Certification Provider for Plainbox Internationalization Catalog
 ☑ : Build Canonical Server Certification Provider for Plainbox Internationalization Catalog

09:54 <@zyga> this has a chance to replace 90% of support/
09:56 <@zyga> spineau: my goal is to get rid of all the testing scripts
09:57 <@zyga> spineau: and all the support/ directory
09:57 <@zyga> spineau: and all the cruft we have all over the tree
09:57 <@zyga> spineau: and put it all in the .selftest/ provider
09:57 <@zyga> spineau: and run it just like './selftest.sh' in any environment
09:57 <@zyga> spineau: native, in vagrant, in lxc, etc
09:57 <@zyga> spineau: and switch tarmac to that as well
09:58 <@spineau> zyga: we can relocate the release stuff if needed
09:58 <@zyga> spineau: no, I mean I don't mind keeping the support/ directory
09:58 <@zyga> spineau: just the explosion of scripts there that one has to understand
09:58 <@spineau> zyga: ok, I see
09:58 <@zyga> spineau: it also replaces mk-venv
09:59 <@zyga> spineau: though it's all incomplete, it does 90% of the work but not 100%
09:59 <@zyga> spineau: I wanted to get feedback from you guys
09:59 <@zyga> spineau: so that travis and tarmac and local testing is identical
09:59 <@zyga> spineau: and stuff like PROVIDERPATH not being set because it's hard to do so won't be a problem again
10:00 <@zyga> spineau: e.g. the new validation thing only fails because we have that issue
10:00 <@zyga> spineau: it's not broken, the test env is not setup correctly
10:00 <@zyga> spineau: I think to fully replace what we use today two more things need to happen
10:00 <@zyga> spineau: one is easy, add a few more jobs to install ppas, packages, and build checkbox-gui
10:00 <@zyga> spineau: that's really easy
10:01 <@zyga> spineau: the bigger thing is to improve plainbox so that plainbox run can be given a session directory and run (or re-run) tests in an existing session
10:01 <@zyga> spineau: the idea behind that is that now each ./selftest.sh run is standalone and cretes a new test session
10:01 <@zyga> spineau: but that's slow
10:01 <@zyga> spineau: for incremental tests we really want to reuse the session, reuse prior tess there that setup PLAINBOX_SESSION_STATE with data
10:02 <@zyga> spineau: e.g. tarballs, venvs, pip packages and more
10:02 <@zyga> spineau: so I'd like to bake it in trunk
10:02 <@zyga> spineau: and to the switch at around 0.22
10:02 <@zyga> spineau: and in the meantime test it in all the ways we can

[Daniel Manrique (roadmr)]

Cool! I tried this inside a container, just ran ./selftest.sh run, and it failed :( because I hadn't installed intltool, which goes to what you mentioned about package installation.

Another thing I noticed is that the exit code was 0 even though some of the tests failed, I wonder if we should have a proper non-success exit code for this.

I'm also thinking about whether it overlaps a bit with tools like tox, which we use in capomastro and hexr.

But in all, this is very nice and allows expressing the rather complicated setup for plainbox tests in a manner that's familiar to a developer who is already working with plainbox. Thanks!

[Zygmunt Krynicki (zyga)]

Yeah, it doesn't install any packages. I'll make it do that and
re-submit this branch.

The exit code is a plainbox deficiency. I will really start looking at
this so that 0.2{2,3,4} can improve.
As for tox, yes, but it does things that tox cannot do. Tox is like
pip, this is like dpkg. Self-test is a tox super-set.

Thanks for the feedback.
ZK

On Wed, Mar 25, 2015 at 7:14 PM, Daniel Manrique
<email address hidden> wrote:
> Cool! I tried this inside a container, just ran ./selftest.sh run, and it failed :( because I hadn't installed intltool, which goes to what you mentioned about package installation.
>
> Another thing I noticed is that the exit code was 0 even though some of the tests failed, I wonder if we should have a proper non-success exit code for this.
>
> I'm also thinking about whether it overlaps a bit with tools like tox, which we use in capomastro and hexr.
>
> But in all, this is very nice and allows expressing the rather complicated setup for plainbox tests in a manner that's familiar to a developer who is already working with plainbox. Thanks!
> --
> https://code.launchpad.net/~zyga/checkbox/selftest/+merge/253478
> You are the owner of lp:~zyga/checkbox/selftest.

[Maciej Kisielewski (kissiel)]

Could we run autopilot for CBT as well?

review: Needs Information

[Zygmunt Krynicki (zyga)]

Yes, sorry, this branch is not done by any means. I'll work on it
after the release. I want to land and replace all the magic with this.

Unmerged revisions

3623. By Zygmunt Krynicki on 2015-03-19

selftest: add a provider for selftest

Jobs in the selftest provider are dividied into three categories:
 - tests for a given component (e.g. plainbox or checkbox-gui)
 - configuration for a given component (e.g. installing required
   packages and other kinds of dependencies)
 - environment probing jobs that can be used by the earlier two,
   those can be used to restrict certain tests to a specific platform
   (e.g. don't build checkbox-gui on fedora, in travis CI or on precise)

Signed-off-by: Zygmunt Krynicki <email address hidden>

3622. By Zygmunt Krynicki on 2015-03-19

selftest: add the selftest tool

This patch adds a new tool, selftest.sh, coupld with any provider
(located in .selftest) for self-testing which replaces the ad-hoc
collection of scripts and tiny data files that we used to have for each
of the components. The provider can be invoked using the selftest.sh
shell script.

The script has the following modes of operation:

    selftest.sh run - run all tests
    (run is also implicit so just running the script is good enough)
    selftest.sh bootstrap - setup current environment for testing
    (bootstrap is also performed implicitly if needed)
    selftest.sh manage - run the management script
    selftest.sh shell - spawn a shell where plainbox sees the
                            local self-test provider
    selftest.sh validate - validate the selftest provider
    selftest.sh info - display information about the selftest provider

Signed-off-by: Zygmunt Krynicki <email address hidden>

=== added directory '.selftest'
=== added file '.selftest/manage.py'
--- .selftest/manage.py	1970-01-01 00:00:00 +0000
+++ .selftest/manage.py	2015-03-19 08:53:15 +0000
@@ -0,0 +1,20 @@
+#!/usr/bin/env python3
+from plainbox.provider_manager import setup
+from plainbox.provider_manager import N_
+
+# You can inject other stuff here but please don't go overboard.
+#
+# In particular, if you need comprehensive compilation support to get
+# your bin/ populated then please try to discuss that with us in the
+# upstream project IRC channel #checkbox on irc.freenode.net.
+
+# NOTE: one thing that you could do here, that makes a lot of sense,
+# is to compute version somehow. This may vary depending on the
+# context of your provider. Future version of PlainBox will offer git,
+# bzr and mercurial integration using the versiontools library
+# (optional)
+
+setup(
+    name='2015.com.canonical.checkbox.selftest:selftest', version="1.0",
+    description=N_("Self-testing provider for lp:checkbox"),
+)
 
=== added directory '.selftest/units'
=== added file '.selftest/units/coverage.pxu'
--- .selftest/units/coverage.pxu	1970-01-01 00:00:00 +0000
+++ .selftest/units/coverage.pxu	2015-03-19 08:53:15 +0000
@@ -0,0 +1,11 @@
+id: coverage/report
+_summary: Generate code coverage report
+_description:
+    This job creates a simple text coverage report for the entire project.
+depends: support/virtualenv
+plugin: shell
+estimated_duration: 1.0
+command:
+    . $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    coverage report
+flags: preserve-locale
 
=== added file '.selftest/units/enum.pxu'
--- .selftest/units/enum.pxu	1970-01-01 00:00:00 +0000
+++ .selftest/units/enum.pxu	2015-03-19 08:53:15 +0000
@@ -0,0 +1,66 @@
+id: enum/python-projects
+_summary: Enumerate Python Projects
+_description:
+ This job enumerates all of the python projects. It is a resource for a few
+ templates that simplify testing and maintenance of selftest.
+plugin: resource
+command:
+    echo "path: plainbox"
+    echo "name: plainbox"
+    echo "title: Plainbox"
+    echo "build_i18n: yes"
+    echo "build_sphinx_html: yes"
+    echo "build_sphinx_man: yes"
+    echo "depends:"
+    echo
+    echo "path: checkbox-support"
+    echo "name: checkbox-support"
+    echo "title: Checkbox Support"
+    echo "build_i18n: yes"
+    echo "build_sphinx_html: no"
+    echo "build_sphinx_man: no"
+    echo "depends:"
+    echo
+    echo "name: checkbox-ng"
+    echo "path: checkbox-ng"
+    echo "title: Checkbox NG"
+    echo "build_i18n: yes"
+    echo "build_sphinx_html: yes"
+    echo "build_sphinx_man: no"
+    echo "depends: test/plainbox/unit-tests, test/checkbox-support/unit-tests"
+estimated_duration: 1.0
+flags: preserve-locale
+
+id: enum/plainbox-providers
+_summary: Enumerate Plainbox Providers
+_description:
+ This job enumerates all of the plainbox providers. It is a resource for a few
+ templates that simplify testing and maintenance of selftest.
+plugin: resource
+command:
+    echo "path: providers/plainbox-provider-checkbox"
+    echo "name: plainbox-provider-checkbox"
+    echo "title: Checkbox Provider for Plainbox"
+    echo "depends: test/plainbox-provider-resource/validate"
+    echo
+    echo "path: providers/plainbox-provider-resource-generic"
+    echo "name: plainbox-provider-resource"
+    echo "title: Resource Provider for Plainbox"
+    echo "depends:"
+    echo
+    echo "path: providers/plainbox-provider-ubuntu-touch"
+    echo "name: plainbox-provider-ubuntu-touch"
+    echo "title: Ubuntu Touch Provider for Plainbox"
+    echo "depends:"
+    echo
+    echo "path: providers/plainbox-provider-certification-client"
+    echo "name: plainbox-provider-certification-client"
+    echo "title: Canonical Client Certification Provider for Plainbox"
+    echo "depends: test/plainbox-provider-checkbox/validate"
+    echo
+    echo "path: providers/plainbox-provider-certification-server"
+    echo "name: plainbox-provider-certification-server"
+    echo "title: Canonical Server Certification Provider for Plainbox"
+    echo "depends: test/plainbox-provider-checkbox/validate"
+estimated_duration: 1.0
+flags: preserve-locale
 
=== added file '.selftest/units/plans.pxu'
--- .selftest/units/plans.pxu	1970-01-01 00:00:00 +0000
+++ .selftest/units/plans.pxu	2015-03-19 08:53:15 +0000
@@ -0,0 +1,12 @@
+id: selftest
+unit: test plan
+_name: All Tests
+_description:
+ This test plan includes all of the self tests for lp:checkbox.
+ It is the most comprehensive and longest test plan to run.
+include:
+ selftest/smoke
+ meta/bootstrap
+ enum/.*
+ test/.*
+estimated_duration: 1.0
 
=== added file '.selftest/units/provider.pxu'
--- .selftest/units/provider.pxu	1970-01-01 00:00:00 +0000
+++ .selftest/units/provider.pxu	2015-03-19 08:53:15 +0000
@@ -0,0 +1,59 @@
+unit: template
+template-unit: job
+template-resource:enum/plainbox-providers
+id: test/{name}/develop
+_summary: Develop {title}
+_description:
+    This job develops the provider {name}.
+depends: meta/bootstrap, test/plainbox/unit-tests
+plugin: shell
+estimated_duration: 1.0
+command:
+    . $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    $SELFTEST_TOP/{path}/manage.py develop -d $PROVIDERPATH -f
+flags: preserve-locale
+
+unit: template
+template-unit: job
+template-resource:enum/plainbox-providers
+id: test/{name}/validate
+_summary: Validate {title}
+_description:
+    This job validates the provider {name}.
+depends: meta/bootstrap, test/{name}/develop, {depends}
+plugin: shell
+estimated_duration: 1.0
+command:
+    . $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    $SELFTEST_TOP/{path}/manage.py validate -N
+flags: preserve-locale
+
+unit: template
+template-unit: job
+template-resource:enum/plainbox-providers
+id: test/{name}/build
+_summary: Build {title}
+_description:
+    This job builds all buildable elements of {name}.
+depends: meta/bootstrap
+plugin: shell
+estimated_duration: 1.0
+command:
+    . $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    $SELFTEST_TOP/{path}/manage.py build
+flags: preserve-locale
+
+unit: template
+template-unit: job
+template-resource:enum/plainbox-providers
+id: test/{name}/build_i18n
+_summary: Build {title} Internationalization Catalog
+_description:
+    This job builds all of the localization catalogs for {name}.
+depends: meta/bootstrap
+plugin: shell
+estimated_duration: 1.0
+command:
+    . $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    $SELFTEST_TOP/{path}/manage.py i18n
+flags: preserve-locale
 
=== added file '.selftest/units/python.pxu'
--- .selftest/units/python.pxu	1970-01-01 00:00:00 +0000
+++ .selftest/units/python.pxu	2015-03-19 08:53:15 +0000
@@ -0,0 +1,114 @@
+unit: template
+template-unit: job
+template-resource:enum/python-projects
+id: test/{name}/develop
+_summary: Develop {title}
+_description:
+    This job runs setup.py develop for {name}.
+depends: meta/bootstrap
+plugin: shell
+estimated_duration: 1.0
+command:
+    . $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    # NOTE: Due to bug in how setup(scripts=[...]) works we need to be in the
+    # right directory as otherwise setuptools won't find one of the launcher
+    # scripts.
+    ( cd $SELFTEST_TOP/{path} && ./setup.py develop )
+flags: preserve-locale
+
+unit: template
+template-unit: job
+template-resource:enum/python-projects
+id: test/{name}/unit-tests
+_summary: {title} Unit Tests
+_description:
+    This job runs all of the unit tests for {name}. The tests are executed with
+    automatic coverage and branch analysis.
+depends: meta/bootstrap, test/{name}/develop, {depends}
+plugin: shell
+estimated_duration: 1
+command:
+    . $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    export LANGUAGE=en
+    export LANG=C.UTF-8
+    export COVERAGE_FILE=$PLAINBOX_SESSION_SHARE/coverage
+    $SELFTEST_TOP/{path}/setup.py test --quiet
+flags: preserve-locale
+
+unit: template
+template-unit: job
+template-resource:enum/python-projects
+template-filter: project.build_i18n == 'yes'
+template-imports: from 2015.com.canonical.checkbox.selftest import enum/python-projects as project
+id: test/{name}/build-i18n
+_summary: Build {title} Internationalization Catalog
+_description:
+    This job builds all of the localization catalogs for {name}.
+command:
+    . $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    $SELFTEST_TOP/{path}/setup.py build_i18n -m
+depends: meta/bootstrap
+plugin: shell
+estimated_duration: 1.0
+flags: preserve-locale
+
+unit: template
+template-unit: job
+template-resource:enum/python-projects
+template-filter: project.build_sphinx_html == 'yes'
+template-imports: from 2015.com.canonical.checkbox.selftest import enum/python-projects as project
+id: test/{name}/build-html-docs
+_summary: Build {title} HTML Documentation
+_description:
+    This job builds all of the HTML documentation for {name}.
+command:
+    . $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    # XXX: Due to a bug somewhere we cannot build manual pages from a temporary
+    # directory. I haven't figure out what is causing this yet.
+    cd $SELFTEST_TOP/{path}
+    $SELFTEST_TOP/{path}/setup.py build_sphinx -b html
+depends: meta/bootstrap
+plugin: shell
+estimated_duration: 1.0
+flags: preserve-locale
+
+unit: template
+template-unit: job
+template-resource:enum/python-projects
+template-filter: project.build_sphinx_man == 'yes'
+template-imports: from 2015.com.canonical.checkbox.selftest import enum/python-projects as project
+id: test/{name}/build-manpages
+_summary: Build {title} Manual Pages
+_description:
+    This job builds all of the manual pages for {name}.
+command:
+    . $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    . $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    # XXX: Due to a bug somewhere we cannot build manual pages from a temporary
+    # directory. I haven't figure out what is causing this yet.
+    cd $SELFTEST_TOP/{path}
+    $SELFTEST_TOP/{path}/setup.py build_sphinx -b man
+depends: meta/bootstrap
+plugin: shell
+estimated_duration: 1.0
+flags: preserve-locale
+
+id: test/plainbox/integration-tests
+_summary: Plainbox Integration Tests
+_description:
+    This job runs all of the "integration tests" for plainbox. Integration
+    tests are really mis-named and should eventually be removed and
+    transitioned into testing all of the providers that live in this project.
+    In practice integration tests run a set of automated tests on a small
+    subset of the "checkbox source" provider.
+command:
+    . $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    export LANGUAGE=en
+    export LANG=C.UTF-8
+    export COVERAGE_FILE=$PLAINBOX_SESSION_SHARE/coverage
+    coverage run --append --branch --module plainbox \
+        self-test --after-reexec --integration-tests --quiet
+depends: meta/bootstrap, test/plainbox/develop
+plugin: shell
+estimated_duration: 1.0
+flags: preserve-locale
 
=== added file '.selftest/units/smoke.pxu'
--- .selftest/units/smoke.pxu	1970-01-01 00:00:00 +0000
+++ .selftest/units/smoke.pxu	2015-03-19 08:53:15 +0000
@@ -0,0 +1,11 @@
+id: selftest/smoke
+unit: job
+_summary: Smoke test for selftest
+_description:
+ This smoke test can be used to check if selftest itself can run anything.
+ This tests simply prints a message and quits successfully.
+plugin: shell
+command:
+    echo "Smoke test successful"
+flags: preserve-locale
+estimated_duration: 1.0
 
=== added file '.selftest/units/support.pxu'
--- .selftest/units/support.pxu	1970-01-01 00:00:00 +0000
+++ .selftest/units/support.pxu	2015-03-19 08:53:15 +0000
@@ -0,0 +1,120 @@
+id: support/bare-virtualenv
+_summary: Create a bare virtualenv
+_description:
+    This job creates a new virtualenv that is required by other tests. This
+    virtualenv is created in the $PLAINBOX_SESSION_SHARE/virtualenv directory,
+    which is a temporary directory unique to each testing session.
+plugin: shell
+estimated_duration: 3
+command:
+    set -x
+    virtualenv \
+        --system-site-packages \
+        --python=/usr/bin/python3 \
+        $PLAINBOX_SESSION_SHARE/virtualenv
+flags: preserve-locale
+
+id: support/inject-extra-env
+_summary: Populate virtualenv with additional environment variables
+_description:
+    This job appends a number of checkbox-specific envirionment variables to the
+    activation script of the virtualenv. Since many other scripts source the same
+    file it is a simple and handy way to add some globals.
+    .
+    The variables are:
+     - PLAINBOX_LOCALE_DIR          - assits in plainbox i18n when uninstalled
+     - CHECKBOX_NG_LOCALE_DIR       - ditto
+     - CHECKBOX_PROVIDER_LOCALE_DIR - ditto
+     - PROVIDERPATH                 - tells plainbox where to look for providers
+    In addition, the directory specified by PROVIDERPATH is created.
+plugin: shell
+estimated_duration: 1
+depends: support/bare-virtualenv
+command:
+    set -e
+    set -x
+    echo "export PLAINBOX_LOCALE_DIR=$SELFTEST_TOP/plainbox/build/mo" >> $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    echo "export CHECKBOX_NG_LOCALE_DIR=$SELFTEST_TOP/checkbox-ng/build/mo" >> $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    echo "export CHECKBOX_PROVIDER_LOCALE_DIR=$SELFTEST_TOP/providers/plainbox-provider-checkbox/build/mo" >> $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    # Add PROVIDERPATH to the venv
+    echo "export PROVIDERPATH=$PLAINBOX_SESSION_SHARE/share/plainbox-providers-1" >> $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    mkdir -p "$PLAINBOX_SESSION_SHARE/share/plainbox-providers-1"
+flags: preserve-locale
+
+id: support/get-external-tarballs
+_summary: Clone the external-tarballs repository
+_description:
+    This job clones (or updates) the repository with external tarballs.  The
+    revision to switch to is stored in support/external-tarballs-revision.txt
+plugin: shell
+estimated_duration: 1
+command:
+    set -e
+    if [ ! -d "$PLAINBOX_SESSION_SHARE/external-tarballs" ]; then
+        echo "I: cloning external-tarballs repository..."
+        # Clone the external-tarballs repo
+        git clone git://github.com/checkbox/external-tarballs \
+            $PLAINBOX_SESSION_SHARE/external-tarballs
+    fi
+    cd $PLAINBOX_SESSION_SHARE/external-tarballs/
+    # Fetch so that we may have the needed revision
+    if ! git fetch origin --quiet; then
+        echo "W: cannot perform fetch, external-tarballs may be out of date"
+    fi
+    # Ensure that we are on the correct revision
+    if ! git checkout $(cat $SELFTEST_TOP/support/external-tarballs-revision.txt); then
+        echo "E: cannot switch to required commit!"
+        exit 1
+    fi
+flags: preserve-locale
+
+id: support/install-pip-dependencies
+_summary: Install all python dependencies
+_description:
+    This job installs all of the dependencis specified by various
+    requirement/pip-*.txt files into a python virtualenv.
+plugin: shell
+estimated_duration: 1
+depends: support/bare-virtualenv, support/get-external-tarballs
+command:
+    set -e
+    set -x
+    # Set a http_proxy to the local discard port to prevent setuptools from
+    # downloading anything.
+    export http_proxy=http://127.0.0.1:9
+    # Custom pip executable. Needed because the pip we want is called
+    # differently in different situations. Inside a virtualenv it is always
+    # 'pip' but outside it may be pip-3.2 or pip-3.3, depending on python
+    # version.
+    CHECKBOX_PIP=${CHECKBOX_PIP:-pip}
+    echo "I: using $CHECKBOX_PIP to install python packages"
+    # Construct a list of required python packages.
+    python_pkg_list="$(find "$SELFTEST_TOP" -path '*/requirements/pip-*.txt' -exec cat "{}" \; | grep -v '^#' | sort | uniq)"
+    # Install them
+    echo "I: installing following pip packages:" $python_pkg_list
+    $CHECKBOX_PIP install --no-index \
+        "--find-links=file://$PLAINBOX_SESSION_SHARE/external-tarballs/index.html" \
+        $python_pkg_list
+flags: preserve-locale
+
+id: meta/bootstrap
+_summary: Meta-job for all bootstrap actions
+_description:
+    This metajob depends on all of the bootstrapping jobs
+plugin: shell
+estimated_duration: 1
+depends: support/bare-virtualenv, support/inject-extra-env, support/install-pip-dependencies
+command: :
+flags: preserve-locale
+
+id: pip-freeze
+_summary: Run 'pip freeze' in the virtual environment
+_description:
+    This metajob depends on all of the bootstrapping jobs
+plugin: shell
+estimated_duration: 1
+depends: support/bare-virtualenv, support/install-pip-dependencies
+command:
+    . $PLAINBOX_SESSION_SHARE/virtualenv/bin/activate
+    pip freeze
+flags: preserve-locale
 
=== added file 'selftest.sh'
--- selftest.sh	1970-01-01 00:00:00 +0000
+++ selftest.sh	2015-03-19 08:53:15 +0000
@@ -0,0 +1,325 @@
+#!/bin/sh
+# This file is part of Checkbox.
+#
+# Copyright 2013-2015 Canonical Ltd.
+# Written by:
+#   Zygmunt Krynicki <[e-mail]>
+#
+# Checkbox is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3,
+# as published by the Free Software Foundation.
+#
+# Checkbox is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with selftest.  If not, see <http://www.gnu.org/licenses/>.
+
+# Feel free to customize those two lines for your project.
+SELFTEST_BOOTSTRAP_DIR=/tmp/selftest-bootstrap.venv
+SELFTEST_RUN_DEFAULT='-T 2015.com.canonical.checkbox.selftest::selftest'
+
+# The rest is generic, no need to change it.
+test -z "$SELFTEST_TOP" -a -d /vagrant && SELFTEST_TOP=/vagrant
+test -z "$SELFTEST_TOP" && SELFTEST_TOP="$(git rev-parse --show-toplevel 2>/dev/null)"
+test -z "$SELFTEST_TOP" && SELFTEST_TOP="$(bzr root)"
+export SELFTEST_TOP
+SELFTEST_PROVIDERPATH="$SELFTEST_BOOTSTRAP_DIR/share/plainbox-providers-1"
+SELFTEST_MANAGE="$SELFTEST_TOP/.selftest/manage.py"
+
+
+_selftest__detect_platform() {
+    echo "I: Detecting platform ..."
+    PLATFORM=$(
+        _selftest__load_os_release \
+            && echo "$ID-$VERSION_ID-$(uname -m | sed -e 's/x86_64/amd64/g')" \
+            || echo "error-error-error"
+    )
+    echo "N: Platform detected: $PLATFORM"
+}
+
+
+_selftest__load_os_release() {
+    # Try to guess where we are running
+    if [ -f /etc/os-release ]; then
+        # Load /etc/os-release if present
+        . /etc/os-release
+    elif [ "$(which lsb_release)" != "" ]; then
+        # Emulate /etc/os-release for well-known Ubuntu releases that lack it
+        # Ubuntu ships /etc/os-release starting with 12.04.2 or 12.10
+        # https://bugs.launchpad.net/ubuntu/+source/base-files/+bug/947236
+        case "$(lsb_release --short --description)" in
+            "Ubuntu 12.04 LTS")
+                NAME="Ubuntu"
+                VERSION="12.04 LTS, Precise Pangolin"
+                ID=ubuntu
+                ID_LIKE=debian
+                PRETTY_NAME="Ubuntu precise (12.04 LTS)"
+                VERSION_ID="12.04"
+                ;;
+            "Ubuntu 12.04.1 LTS")
+                NAME="Ubuntu"
+                VERSION="12.04.1 LTS, Precise Pangolin"
+                ID=ubuntu
+                ID_LIKE=debian
+                PRETTY_NAME="Ubuntu precise (12.04.1 LTS)"
+                VERSION_ID="12.04"
+                ;;
+            *)
+                NAME=Linux
+                ;;
+        esac
+    fi
+}
+
+
+
+_selftest__configure_platform() {
+    echo "I: Configuring platform ..."
+    case $PLATFORM in
+        ubuntu-*)
+            _selftest__configure_debian
+            ;;
+        elementary-*)
+            _selftest__configure_debian
+            ;;
+        debian-*)
+            _selftest__configure_debian
+            ;;
+        fedora-*)
+            _selftest__configure_fedora
+            ;;
+        *)
+            echo "E: Unable to configure platform $PLATFORM"
+            echo "E: Patch support/plainbox-selftest-provider/bootstrap"
+            echo "E: (look for _selftest__configure_platform)"
+            echo "E: If you think this should work then please report"
+            echo "E: a bug on https://bugs.launchpad.net/selftest/+filebug"
+            exit 1
+            ;;
+    esac
+}
+
+
+_selftest__configure_debian() {
+    export DEBIAN_FRONTEND=noninteractive
+    if ! sudo apt-get -qqy install python3 python-virtualenv; then
+        echo "E: Unable to instal core dependencies"
+        echo "E: Typically this happens when you are offline" 
+        exit 1
+    fi
+}
+
+
+_selftest__configure_fedora() {
+   if ! sudo yum -q install -y python3 python-virtualenv; then
+        echo "E: Unable to instal core dependencies"
+        echo "E: Typically this happens when you are offline" 
+        exit 1
+    fi
+}
+
+
+_selftest__install_stable_plainbox() {
+    echo "I: Creating python3 virtual environment"
+    if ! virtualenv --quiet -p python3 "$SELFTEST_BOOTSTRAP_DIR"; then
+        echo "E: Unable to create a virtualenv for python 3"
+        _selftest__bug
+    fi
+    _selftest__venv_enter
+    if ! pip freeze | grep plainbox >/dev/null; then
+        echo "I: Installing plainbox 0.20"
+        if ! pip install plainbox==0.20; then
+            echo "E: Unable to install plainbox 0.20"
+            echo "E: Typically this happens when you are offline"
+            exit 1
+        fi
+    fi
+    echo "I: Checking if plainbox is operational ..."
+    if [ "$(plainbox --version 2>&1 )" != 0.20 ]; then
+        echo "E: Installed plainbox doesn't work."
+        _selftest__bug
+    fi
+    _selftest__venv_exit
+}
+
+
+_selftest__develop_selftest_provider() {
+    echo "I: Developing the self-test provider ..."
+    _selftest__venv_enter
+    if ! "$SELFTEST_MANAGE" develop -f -d "$SELFTEST_PROVIDERPATH"; then
+        echo "E: Unable to develop the self-test provider"
+        _selftest__bug
+    fi
+    _selftest__venv_exit
+}
+
+_selftest__detect_broken_vagrant() {
+    if [ -d "/vagrant" ] ; then
+        # Vagrant sanity check, see if /vagrant really works.
+        #
+        # On various installations of Ubuntu this is not the case as the cloud images
+        # we depend on seem to have broken virtualbox tools installation inside.  This
+        # causes the filesystem mounted on /vagrant to crash horribly and kill all
+        # processes attempting to use it.
+        echo "I: Checking for broken /vagrant ..."
+        if ! find /vagrant -mindepth 1 -maxdepth 1 >/dev/null; then
+            cat <<__EOM__
+        E: It seems that /vagrant directory is broken
+        E: This is a known issue.
+        E: See this link for a workaround if you have recent-enough VirtualBox:
+        E: https://bugs.launchpad.net/ubuntu/+source/virtualbox/+bug/1252872
+        E: Alternatively you can use NFS on any version of VirtualBox:
+        E: http://docs.vagrantup.com/v2/synced-folders/nfs.html
+__EOM__
+            exit 1
+        else
+            echo "I: It seems that /vagrant is working okay"
+        fi
+    fi
+}
+
+
+_selftest__patch_broken_pkg_resources() {
+    if sudo patch -p0 /usr/lib/python3/dist-packages/pkg_resources.py << EOF
+--- pkg_resources.py.orig	2014-05-15 19:39:38.365807720 +0000
++++ pkg_resources.py	2014-05-15 19:44:50.449807720 +0000
+@@ -1749,11 +1749,16 @@
+                     for dist in find_distributions(os.path.join(path_item, entry)):
+                         yield dist
+                 elif not only and lower.endswith('.egg-link'):
+-                    for line in open(os.path.join(path_item, entry)):
+-                        if not line.strip(): continue
+-                        for item in find_distributions(os.path.join(path_item,line.rstrip())):
+-                            yield item
+-                        break
++                    try:
++                        entry_file = open(os.path.join(path_item, entry))
++                        for line in entry_file:
++                            if not line.strip(): continue
++                            for item in find_distributions(os.path.join(path_item,line.rstrip())):
++                                yield item
++                            break
++                    finally:
++                        entry_file.close()
++
+ register_finder(ImpWrapper,find_on_path)
+
+ _declare_state('dict', _namespace_handlers={})
+EOF
+    then
+        echo "I: Patching pkg_resources.py succeeded"
+    else
+        echo "E: Unable to patch pkg_resources.py :-("
+        _selftest__bug
+    fi
+}
+
+
+_selftest__bug() {
+    echo "E: Please report a bug on https://bugs.launchpad.net/checkbox/+filebug"
+    exit 1
+}
+
+
+_selftest__venv_enter() {
+    . "$SELFTEST_BOOTSTRAP_DIR/bin/activate"
+}
+
+
+_selftest__venv_exit() {
+    deactivate
+}
+
+
+_selftest__bootstrap() {
+    _selftest__detect_broken_vagrant
+    _selftest__detect_platform
+    _selftest__configure_platform
+    _selftest__install_stable_plainbox
+    _selftest__develop_selftest_provider
+}
+
+
+_selftest__maybe_bootstrap() {
+    if [ ! -e "$SELFTEST_BOOTSTRAP_DIR" ]; then
+        echo "N: Doing automatic bootstrap ..."
+        _selftest__bootstrap
+    fi
+}
+
+
+_selftest__run() {
+    _selftest__venv_enter
+    PROVIDERPATH="$SELFTEST_PROVIDERPATH" plainbox run ${@:-$SELFTEST_RUN_DEFAULT}
+    retval=$?
+    _selftest__venv_exit
+    return $retval
+}
+
+
+_selftest__manage() {
+    _selftest__venv_enter
+    "$SELFTEST_MANAGE" "$@"
+    _selftest__venv_exit
+}
+
+
+_selftest__shell() {
+    _selftest__venv_enter
+    echo "I: activating interactive selftest shell"
+    echo "I: type 'exit' to quit"
+    PROVIDERPATH="$SELFTEST_PROVIDERPATH" debian_chroot="selftest" bash "$@"
+    _selftest__venv_exit
+}
+
+
+_selftest__main() {
+    case "$1" in
+        '')
+            _selftest__maybe_bootstrap
+            _selftest__run "$@"
+            exit $?
+            ;;
+        run)
+            shift
+            _selftest__maybe_bootstrap
+            _selftest__run "$@"
+            exit $?
+            ;;
+        bootstrap)
+            shift
+            _selftest__bootstrap
+            exit $?
+            ;;
+        validate)
+            shift
+            _selftest__manage validate "$@"
+            exit $?
+            ;;
+        info)
+            shift
+            _selftest__manage info "$@"
+            exit $?
+            ;;
+        manage)
+            shift
+            _selftest__manage "$@"
+            exit $?
+            ;;
+        shell)
+            shift
+            _selftest__shell "$@"
+            exit $?
+            ;;
+        *)
+            echo "Usage: selftest {boostrap,run,validate,info,manage,shell}"
+            exit 1
+            ;;
+    esac
+}
+
+
+_selftest__main "$@"