Merge lp:~sergiusens/snapcraft/1500758 into lp:~dholbach/snapcraft/1498347

[Sergio Schvezov]

Adding a simple scons plugin

Unmerged revisions

216. By Sergio Schvezov on 2015-09-29

Adding tests for scons

215. By Sergio Schvezov on 2015-09-29

Simple scons plugin

214. By Sergio Schvezov on 2015-09-29

Detect tars in sources automatically

213. By Sergio Schvezov on 2015-09-29

Mask tar files so they are writable and set an extraction parameter to start with a clean build dir.

212. By Leo Arias on 2015-09-28

Fixed the heading on the build section of the docs. by elopio approved by dholbach

211. By Daniel Holbach on 2015-09-25

Reopen 0.3 for development (after 0.2.1 brown paperbag release). by dholbach approved by sergiusens

210. By Daniel Holbach on 2015-09-25

Brown paperbag release (install examples into the right directory). by dholbach approved by mvo

209. By Daniel Holbach on 2015-09-24

Open 0.3 for development. by dholbach approved by sergiusens

208. By Sergio Schvezov on 2015-09-24

Allow using local sources through an env var (mostly useful for launchpad building) by sergiusens approved by mvo,dholbach,cjwatson

207. By Sergio Schvezov on 2015-09-23

preparing a changelog by sergiusens approved by sergiusens,dholbach

=== modified file 'debian/changelog'
--- debian/changelog	2015-08-18 12:37:44 +0000
+++ debian/changelog	2015-09-29 10:44:16 +0000
@@ -1,8 +1,61 @@
-snapcraft (0.2) UNRELEASED; urgency=medium
+snapcraft (0.3) UNRELEASED; urgency=medium
 
   * 
 
- -- Daniel Holbach <[e-mail]>  Tue, 18 Aug 2015 14:23:24 +0200
+ -- Daniel Holbach <[e-mail]>  Fri, 25 Sep 2015 12:46:23 +0200
+
+snapcraft (0.2.1) wily; urgency=medium
+
+  * New upstream release (LP: #1499265)
+  * Brown-paper bag release:
+    - Fix install path of examples. Thanks Martin Pitt for noticing.
+    - Allow using local sources through an env var (mostly useful for
+      Launchpad building).
+
+ -- Daniel Holbach <[e-mail]>  Thu, 24 Sep 2015 11:37:36 +0200
+
+snapcraft (0.2) wily; urgency=medium
+
+  * Better setuptools support:
+    - Better support python-setuptools (LP: #1481864),
+    - Used builddir as a base for setup.py overrides (LP: #1498212),
+    - Fix build (LP: #1494825),
+  * Plugin fixes:
+    - Add demo.qml back to the snapcraft.yaml for the QML Demo (LP: #1491301),
+    - stage package libgudev-1.0-0, which makes the godd snap actually 
+      work (LP: #1498347),
+    - Wrap setup.py calls to configure the shebang writer (LP: #1486680),
+    - Update tomcat upstream URL, fixes example again (LP: #1491303),
+    - Add libgudev-1.0-dev as build-tools for godd example,
+      Ensure C library configuration tools don't use system 
+      paths (LP: #1496789),
+  * Snapcraft cli:
+    - On clean, check the contents of the parts dir only if it 
+      exists. (LP: #1497371)
+    - Load the config before trying to run (LP: #1498140),
+    - Notify user why the password is being requested (LP: #1481499),
+    - Fix numerous issues in snapcraft run (LP: #1486659),
+  * Snapcraft stage-packages:
+    - Enable ports for architectures that are not amd64 or 
+      i386 (LP: #1498157),
+    - Improving stage-package handling (LP: #1497453, LP: #1497582)
+    - Fetch all packages in a single download run with proper 
+      progress (LP: #1498333),
+  * Snapcraft wiki queries excessive (LP: #1496381),
+  * Snapcraft internals:
+    - Add 'simple' and 'has-leftovers' flags to all integration tests job
+      units. (LP: #1484596),
+    - Regex for binary and service names (LP: #1495662),
+    - Provide a nice error when tabs found in snapcraft.yaml (LP: #1477875),
+    - Use relative paths for image creation (LP: #1497108),
+    - Use the python logger (LP: #1476452),
+    - snapcraft now has less exit points (LP: #1477639),
+    - Depend and Build-Depend on python3-requests (LP: #1496363),
+    - Add autopkgtest, reshuffle build-deps accordingly (LP: #1496392),
+    - Fix "snapcraft run" (LP: #1484720),
+  * Examples packaged (LP: #1498189).
+
+ -- Sergio Schvezov <[e-mail]>  Wed, 24 Sep 2015 14:23:24 -0300
 
 snapcraft (0.1) wily; urgency=low
 

=== modified file 'debian/control'
--- debian/control	2015-09-16 14:26:10 +0000
+++ debian/control	2015-09-29 10:44:16 +0000
@@ -39,3 +39,14 @@
  a single toolset, but instead is a collection of tools that enable the
  natural workflow of an upstream to be extended with a simple release step
  into Snappy.
+
+Package: snapcraft-examples
+Architecture: all
+Depends: snapcraft (= ${binary:Version}), ${misc:Depends}, ${python3:Depends}
+Description: examples for snapcraft
+ Snapcraft aims to make upstream developers' lives easier and as such is not
+ a single toolset, but instead is a collection of tools that enable the
+ natural workflow of an upstream to be extended with a simple release step
+ into Snappy.
+ .
+ This package contains examples for snapcraft.

=== modified file 'debian/rules'
--- debian/rules	2015-09-16 14:26:10 +0000
+++ debian/rules	2015-09-29 10:44:16 +0000
@@ -3,3 +3,6 @@
 
 %:
 	dh $@ --with python3 --buildsystem=pybuild
+
+override_dh_compress:
+	dh_compress -Xusr/share/doc/snapcraft-examples/examples

=== added file 'debian/snapcraft-examples.examples'
--- debian/snapcraft-examples.examples	1970-01-01 00:00:00 +0000
+++ debian/snapcraft-examples.examples	2015-09-29 10:44:16 +0000
@@ -0,0 +1,1 @@
+examples/*

=== added file 'debian/snapcraft.install'
--- debian/snapcraft.install	1970-01-01 00:00:00 +0000
+++ debian/snapcraft.install	2015-09-29 10:44:16 +0000
@@ -0,0 +1,3 @@
+/usr/bin
+/usr/lib/python*
+/usr/share/snapcraft

=== modified file 'debian/source/options'
--- debian/source/options	2015-07-06 16:48:22 +0000
+++ debian/source/options	2015-09-29 10:44:16 +0000
@@ -1,7 +1,2 @@
 # Ignore standard files (like .bzr)
 tar-ignore
-
-# Ignore examples folder because it likely contains a bunch of stuff we don't
-# care about (like downloaded go1.4 binaries and such).  It's not important
-# for it to be distributed with the source.
-tar-ignore = "examples"

=== modified file 'debian/tests/control'
--- debian/tests/control	2015-09-17 14:56:20 +0000
+++ debian/tests/control	2015-09-29 10:44:16 +0000
@@ -1,5 +1,6 @@
-Test-Command: ./runtests.sh
-Restrictions: allow-stderr, isolation-container
+Tests: runtests
+       runexamples
+Restrictions: allow-stderr, isolation-container, rw-build-tree
 Depends: @,
          build-essential,
          pep8,

=== added file 'debian/tests/runexamples'
--- debian/tests/runexamples	1970-01-01 00:00:00 +0000
+++ debian/tests/runexamples	2015-09-29 10:44:16 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+./runtests.sh plainbox examples

=== added file 'debian/tests/runtests'
--- debian/tests/runtests	1970-01-01 00:00:00 +0000
+++ debian/tests/runtests	2015-09-29 10:44:16 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+./runtests.sh

=== modified file 'docs/intro.md'
--- docs/intro.md	2015-09-11 13:51:27 +0000
+++ docs/intro.md	2015-09-29 10:44:16 +0000
@@ -51,7 +51,7 @@
 sub-directories like `parts/part-name/src` for each part that contains
 the downloaded content.
 
-#### Build
+### Build
 
 The next step is that each part is built in its `parts/part-name/build`
 directory and installs itself into `parts/part-name/install`.

=== modified file 'docs/your-first-snap.md'
--- docs/your-first-snap.md	2015-09-14 20:53:28 +0000
+++ docs/your-first-snap.md	2015-09-29 10:44:16 +0000
@@ -268,8 +268,8 @@
     description: Exposes your webcam over a web UI
     icon: icon.png
     services:
-    - name: webcam-webui
-      start: bin/webcam-webui
+      webcam-webui:
+        start: bin/webcam-webui
 
     parts:
       golang-static-http:

=== modified file 'examples/downloader-with-wiki-parts/snapcraft.yaml'
--- examples/downloader-with-wiki-parts/snapcraft.yaml	2015-09-21 05:04:31 +0000
+++ examples/downloader-with-wiki-parts/snapcraft.yaml	2015-09-29 10:44:16 +0000
@@ -14,3 +14,4 @@
         source: .
         after:
             - curl
+build-packages: [libssl-dev]

=== added directory 'integration-tests/data/build-tools'
=== added file 'integration-tests/data/build-tools/Makefile'
--- integration-tests/data/build-tools/Makefile	1970-01-01 00:00:00 +0000
+++ integration-tests/data/build-tools/Makefile	2015-09-29 10:44:16 +0000
@@ -0,0 +1,6 @@
+
+all:
+	@echo "Makin' like a Makefile"
+
+install:
+	@echo "Installin' like a Makefile"

=== added file 'integration-tests/data/build-tools/icon.svg'
--- integration-tests/data/build-tools/icon.svg	1970-01-01 00:00:00 +0000
+++ integration-tests/data/build-tools/icon.svg	2015-09-29 10:44:16 +0000
@@ -0,0 +1,1 @@
+<svg/>

=== added file 'integration-tests/data/build-tools/snapcraft.yaml'
--- integration-tests/data/build-tools/snapcraft.yaml	1970-01-01 00:00:00 +0000
+++ integration-tests/data/build-tools/snapcraft.yaml	2015-09-29 10:44:16 +0000
@@ -0,0 +1,14 @@
+name: build-tools-test
+version: 0.1
+vendor: Napoleon Bonaparte <[e-mail]>
+summary: one line summary
+description: a longer description
+icon: icon.svg
+
+parts:
+  make-project:
+    source: .
+    stage-packages:
+      - libxml2-dev
+      - libxslt1-dev
+      - libglib2.0-dev

=== modified file 'integration-tests/data/pip-requirements/snapcraft.yaml'
--- integration-tests/data/pip-requirements/snapcraft.yaml	2015-09-09 20:19:12 +0000
+++ integration-tests/data/pip-requirements/snapcraft.yaml	2015-09-29 10:44:16 +0000
@@ -7,8 +7,8 @@
 
 parts:
   python2:
-    type: python2
+    type: python2-project
     requirements: requirements.txt
   python3:
-    type: python3
+    type: python3-project
     requirements: requirements.txt

=== added directory 'integration-tests/data/simple-scons'
=== added file 'integration-tests/data/simple-scons/SConstruct'
--- integration-tests/data/simple-scons/SConstruct	1970-01-01 00:00:00 +0000
+++ integration-tests/data/simple-scons/SConstruct	2015-09-29 10:44:16 +0000
@@ -0,0 +1,14 @@
+import os.path
+
+env = Environment()
+hello = env.Program('main.c')
+DESTDIR = os.environ.get('DESTDIR', '')
+install_dir = os.path.join('usr', 'bin')
+
+if DESTDIR:
+   install_dir = os.path.join(DESTDIR, install_dir)
+else:
+   install_dir = os.path.join(os.path.sep, install_dir)
+
+env.Install(install_dir,  hello)
+env.Alias('install', install_dir)

=== added file 'integration-tests/data/simple-scons/icon.png'
=== added file 'integration-tests/data/simple-scons/main.c'
--- integration-tests/data/simple-scons/main.c	1970-01-01 00:00:00 +0000
+++ integration-tests/data/simple-scons/main.c	2015-09-29 10:44:16 +0000
@@ -0,0 +1,5 @@
+#include <stdio.h>
+
+int main(int argc, char** argv) {
+	printf("Hello world\n");
+}

=== added file 'integration-tests/data/simple-scons/snapcraft.yaml'
--- integration-tests/data/simple-scons/snapcraft.yaml	1970-01-01 00:00:00 +0000
+++ integration-tests/data/simple-scons/snapcraft.yaml	2015-09-29 10:44:16 +0000
@@ -0,0 +1,11 @@
+name: test-package
+version: 0.1
+vendor: Sergio Schvezov <[e-mail]>
+summary: test a simple scons project
+description: a longer description
+icon: icon.png
+
+parts:
+  scons-project:
+    type: scons-project
+    source: .

=== added file 'integration-tests/data/simple-tar/readonly-files.tar.gz'
Binary files integration-tests/data/simple-tar/readonly-files.tar.gz	1970-01-01 00:00:00 +0000 and integration-tests/data/simple-tar/readonly-files.tar.gz	2015-09-29 10:44:16 +0000 differ
=== modified file 'integration-tests/data/simple-tar/snapcraft.yaml'
--- integration-tests/data/simple-tar/snapcraft.yaml	2015-09-08 11:48:30 +0000
+++ integration-tests/data/simple-tar/snapcraft.yaml	2015-09-29 10:44:16 +0000
@@ -27,6 +27,9 @@
   evil-slash:
     type: tar-content
     source: evil-slash.tar.gz
+  readonly-files:
+    type: tar-content
+    source: readonly-files.tar.gz
   project:
     type: make-project
     source: project.tar.xz

=== modified file 'integration-tests/runtests.sh'
--- integration-tests/runtests.sh	2015-08-04 23:46:54 +0000
+++ integration-tests/runtests.sh	2015-09-29 10:44:16 +0000
@@ -17,24 +17,43 @@
 
 set -e
 
-if [ -z "$SNAPCRAFT" ]; then
-    export SNAPCRAFT=snapcraft
-fi
+run_test_plan(){
+    TEST_PLAN=$1
+    
+    if [ -z "$SNAPCRAFT" ]; then
+        export SNAPCRAFT=snapcraft
+    fi
 
-# Create a temporary directory so that we can run 'manage.py develop' and
-# create the .provider file there
-temp_dir=$(mktemp -d)
-# Develop the provider, this will let us run tests on it
-./manage.py develop -d $temp_dir
-# Set PROVIDERPATH (see plainbox(1)) so that we can see the provider
-# without installing it.
-export PROVIDERPATH=$PROVIDERPATH:$temp_dir
-# Run the 'normal' test plan
-plainbox run \
-    -T 2015.com.canonical.snapcraft::normal \
-    -f json -o $temp_dir/result.json
-# Analyze the result and fail if there are any failures
-python3 - << __PYTHON__
+    # Create a temporary directory so that we can run 'manage.py develop' and
+    # create the .provider file there
+    temp_dir=$(mktemp -d)
+    # Develop the provider, this will let us run tests on it
+    ./manage.py develop -d $temp_dir
+    # Set PROVIDERPATH (see plainbox(1)) so that we can see the provider
+    # without installing it.
+    export PROVIDERPATH=$PROVIDERPATH:$temp_dir
+    # create symlink from the provider's data directory to examples for them to
+    # be available to the tests
+    if [ "$TEST_PLAN" = "examples" ]; then
+        BASEDIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )"/.. && pwd )
+        TARGET_PATH=${BASEDIR}/integration-tests/data/examples
+        if [ -z $ADT_ARTIFACTS ]; then
+            EXAMPLES_PATH=${BASEDIR}/examples
+        else
+            EXAMPLES_PATH=/usr/share/snapcraft/examples
+        fi
+        ln -fs ${EXAMPLES_PATH} ${TARGET_PATH}
+    fi
+    # Run the test plan
+    plainbox run \
+             -T 2015.com.canonical.snapcraft::"$TEST_PLAN" \
+             -f json -o $temp_dir/result.json
+    # remove the examples symlink
+    if [ "$TEST_PLAN" = "examples" ]; then
+        rm ${TARGET_PATH}
+    fi
+    # Analyze the result and fail if there are any failures
+    python3 - << __PYTHON__
 import json
 with open("$temp_dir/result.json", "rt", encoding="utf-8") as stream:
     results = json.load(stream)
@@ -46,3 +65,11 @@
 print("Overall: {0}".format("fail" if failed else "pass"))
 raise SystemExit(failed)
 __PYTHON__
+}
+
+if [ -z "$1" ]; then
+    echo "need test plan as first argument"
+    exit 1
+fi
+
+run_test_plan "$@"

=== added file 'integration-tests/units/examples.pxu'
--- integration-tests/units/examples.pxu	1970-01-01 00:00:00 +0000
+++ integration-tests/units/examples.pxu	2015-09-29 10:44:16 +0000
@@ -0,0 +1,10 @@
+id: snapcraft/examples/build
+plugin: shell
+command:
+    set -e
+    for example in ${PLAINBOX_PROVIDER_DATA}/examples/*
+    do
+      cp -rT ${example//:} ${PLAINBOX_SESSION_SHARE}
+      cd ${PLAINBOX_SESSION_SHARE} && ${SNAPCRAFT}
+      rm -rf ${PLAINBOX_SESSION_SHARE}/*
+    done

=== modified file 'integration-tests/units/jobs.pxu'
--- integration-tests/units/jobs.pxu	2015-09-21 19:20:43 +0000
+++ integration-tests/units/jobs.pxu	2015-09-29 10:44:16 +0000
@@ -6,6 +6,7 @@
     OUTPUT=$(${SNAPCRAFT} pull 2>&1)
     test $? = 1 || exit 1
     echo $OUTPUT | grep "Could not find snapcraft\.yaml\."
+flags: simple has-leftovers
 
 id: snapcraft/normal/no-yaml-run
 plugin: shell
@@ -26,6 +27,7 @@
     # care where (and you don't need snappy-metadata to find it)
     touch copy-meta # instructs Makefile to copy snappy/ itself
     ${SNAPCRAFT} assemble
+flags: simple has-leftovers
 
 id: snapcraft/normal/assemble-meta
 plugin: shell
@@ -48,6 +50,7 @@
     if [ -e ./snap/bin/not-wrapped.wrapper ]; then
         exit 1
     fi
+flags: simple has-leftovers
 
 id: snapcraft/normal/local-source
 plugin: shell
@@ -60,6 +63,7 @@
     test "$(readlink parts/make-project/build)" = "$(pwd)"
     ${SNAPCRAFT} stage
     test -e stamp-install
+flags: simple has-leftovers
 
 id: snapcraft/normal/local-plugin
 plugin: shell
@@ -69,6 +73,7 @@
     cp -rT $PLAINBOX_PROVIDER_DATA/local-plugin .
     ${SNAPCRAFT} stage
     test -e stage/build-stamp
+flags: simple has-leftovers
 
 id: snapcraft/normal/simple-tar
 plugin: shell
@@ -88,7 +93,11 @@
     test -e stage/notopdir
     test -e stage/parent
     test -e stage/slash
+    test -e stage/readonly_file
     test "$(./stage/bin/test)" = "tarproject"
+    # Regression test for https://bugs.launchpad.net/snapcraft/+bug/1500728
+    ${SNAPCRAFT} pull --force
+flags: simple has-leftovers
 
 id: snapcraft/normal/pypi-config
 plugin: shell
@@ -103,6 +112,7 @@
     test -f ./snap/usr/bin/python3
     test -f ./snap/meta/hooks/config
     grep -q 'exec "usr/bin/config.py" $*' ./snap/meta/hooks/config
+flags: simple has-leftovers
 
 id: snapcraft/normal/simple-make
 plugin: shell
@@ -112,6 +122,7 @@
     cp -rT $PLAINBOX_PROVIDER_DATA/simple-make .
     ${SNAPCRAFT} stage
     test "$(./stage/bin/test)" = "Hello world"
+flags: simple has-leftovers
 
 id: snapcraft/normal/simple-make-filesets
 plugin: shell
@@ -130,6 +141,7 @@
     grep -q 'file1' ./snap/share/file1
     grep -q 'file2' ./snap/share/file2
     grep -q 'file1' ./snap/new/dir2/file1
+flags: simple has-leftovers
 
 id: snapcraft/normal/simple-make-clean
 plugin: shell
@@ -149,6 +161,7 @@
     # Clean a second time doesn't fail.
     # Regression test for https://bugs.launchpad.net/snapcraft/+bug/1497371
     ${SNAPCRAFT} clean
+flags: simple has-leftovers
 
 id: snapcraft/normal/simple-cmake
 plugin: shell
@@ -158,6 +171,7 @@
     cp -rT $PLAINBOX_PROVIDER_DATA/simple-cmake .
     ${SNAPCRAFT} stage
     test "$(./stage/bin/simple-cmake)" = "It's a CMake world"
+flags: simple has-leftovers
 
 id: snapcraft/normal/simple-copy
 plugin: shell
@@ -167,6 +181,7 @@
     cp -rT $PLAINBOX_PROVIDER_DATA/simple-copy .
     ${SNAPCRAFT} stage
     test "$(cat ./stage/dst)" = "I got copied"
+flags: simple has-leftovers
 
 id: snapcraft/normal/conflicts
 plugin: shell
@@ -177,6 +192,7 @@
     OUTPUT=$(${SNAPCRAFT} stage 2>&1)
     test $? = 1 || exit 1
     echo $OUTPUT | grep "Error: parts p1 and p2 have the following file paths in common which have different contents: bin/test" # squished output by bash
+flags: simple has-leftovers
 
 id: snapcraft/normal/dependencies
 plugin: shell
@@ -189,6 +205,7 @@
     test "$(${SNAPCRAFT} shell p3)" = 'p1
     p1
     p2'
+flags: simple has-leftovers
 
 id: snapcraft/normal/dependencies-circular
 plugin: shell
@@ -200,6 +217,7 @@
     OUTPUT=$(${SNAPCRAFT} pull 2>&1)
     test $? = 1 || exit 1
     echo $OUTPUT | grep -i "circular dependency"
+flags: simple has-leftovers
 
 id: snapcraft/normal/dependencies-fail
 plugin: shell
@@ -210,6 +228,7 @@
     sed -i '/after/d' snapcraft.yaml
     ${SNAPCRAFT} build
     test $? = 1 || exit 1
+flags: simple has-leftovers
 
 id: snapcraft/normal/bzr-head
 plugin: shell
@@ -225,6 +244,7 @@
     test "$(bzr revno -r -1 parts/bzr/src)" = "2"
     ${SNAPCRAFT} pull # test pull doesn't fail
     test "$(bzr revno -r -1 parts/bzr/src)" = "2"
+flags: simple has-leftovers
 
 id: snapcraft/normal/bzr-tag
 plugin: shell
@@ -241,6 +261,7 @@
     test "$(bzr revno -r -1 parts/bzr/src)" = "1"
     ${SNAPCRAFT} pull # test pull doesn't fail
     test "$(bzr revno -r -1 parts/bzr/src)" = "1"
+flags: simple has-leftovers
 
 id: snapcraft/normal/git-head
 plugin: shell
@@ -257,6 +278,7 @@
     test "$(git -C parts/git/src log -1 --oneline | cut -d' ' -f2)" = "2"
     ${SNAPCRAFT} pull
     test "$(git -C parts/git/src log -1 --oneline | cut -d' ' -f2)" = "2"
+flags: simple has-leftovers
 
 id: snapcraft/normal/git-tag
 plugin: shell
@@ -274,6 +296,7 @@
     test "$(git -C parts/git/src log -1 --oneline | cut -d' ' -f2)" = "1"
     ${SNAPCRAFT} pull
     test "$(git -C parts/git/src log -1 --oneline | cut -d' ' -f2)" = "1"
+flags: simple has-leftovers
 
 id: snapcraft/normal/git-branch
 plugin: shell
@@ -294,6 +317,7 @@
     test "$(git -C parts/git/src log -2 --oneline | cut -d' ' -f2 | tr '\n' ' ')" = "3 1 "
     ${SNAPCRAFT} pull
     test "$(git -C parts/git/src log -2 --oneline | cut -d' ' -f2 | tr '\n' ' ')" = "3 1 "
+flags: simple has-leftovers
 
 id: snapcraft/normal/hg-head
 plugin: shell
@@ -310,6 +334,7 @@
     test "$(hg log --cwd parts/mercurial/src --template "{desc}" -r -1)" = "2"
     ${SNAPCRAFT} pull
     test "$(hg log --cwd parts/mercurial/src --template "{desc}" -r -1)" = "2"
+flags: simple has-leftovers
 
 id: snapcraft/normal/hg-tag
 plugin: shell
@@ -327,6 +352,7 @@
     test "$(ls -1 parts/mercurial/src/ | wc -l )" = "1"
     ${SNAPCRAFT} pull
     test "$(ls -1 parts/mercurial/src/ | wc -l )" = "1"
+flags: simple has-leftovers
 
 id: snapcraft/normal/hg-branch
 plugin: shell
@@ -345,6 +371,7 @@
     test -e parts/mercurial/src/second
     ${SNAPCRAFT} pull
     test -e parts/mercurial/src/second
+flags: simple has-leftovers
 
 id: snapcraft/normal/pip-requirements
 plugin: shell
@@ -355,3 +382,16 @@
     ${SNAPCRAFT} pull
     test -f parts/python2/install/usr/lib/python2.7/argparse.py
     test -f parts/python3/install/usr/lib/python3.4/argparse.py
+flags: simple has-leftovers
+
+id: snapcraft/normal/build-tool-paths
+plugin: shell
+estimated_duration: 30
+command:
+    set -ex
+    cp -rT $PLAINBOX_PROVIDER_DATA/build-tools .
+    ${SNAPCRAFT} stage
+    ${SNAPCRAFT} shell pkg-config --cflags glib-2.0 | grep -q `pwd`
+    ${SNAPCRAFT} shell xml2-config --cflags | grep -q `pwd`
+    ${SNAPCRAFT} shell xslt-config --cflags | grep -q `pwd`
+flags: simple has-leftovers

=== modified file 'integration-tests/units/testplans.pxu'
--- integration-tests/units/testplans.pxu	2015-07-02 18:29:47 +0000
+++ integration-tests/units/testplans.pxu	2015-09-29 10:44:16 +0000
@@ -1,4 +1,9 @@
 unit: test plan
 id: normal
-name: Tests for Snapcraft 
+name: Tests for Snapcraft
 include: snapcraft/normal/.*
+
+unit: test plan
+id: examples
+name: Test for building Snapcraft examples
+include: snapcraft/examples/.*

=== modified file 'plugins/python2-project.yaml'
--- plugins/python2-project.yaml	2015-09-22 03:08:07 +0000
+++ plugins/python2-project.yaml	2015-09-29 10:44:16 +0000
@@ -1,8 +1,8 @@
-requires:
-  - python2
 options:
+  requirements:
+    required: false
   source:
-    required: true
+    required: false
   source-type:
   source-tag:
   source-branch:

=== removed file 'plugins/python2.yaml'
--- plugins/python2.yaml	2015-09-01 14:10:18 +0000
+++ plugins/python2.yaml	1970-01-01 00:00:00 +0000
@@ -1,3 +0,0 @@
-options:
-  requirements:
-    required: false

=== modified file 'plugins/python3-project.yaml'
--- plugins/python3-project.yaml	2015-09-22 03:08:07 +0000
+++ plugins/python3-project.yaml	2015-09-29 10:44:16 +0000
@@ -1,8 +1,8 @@
-requires:
-  - python3
 options:
+  requirements:
+    required: false
   source:
-    required: true
+    required: false
   source-type:
   source-tag:
   source-branch:

=== removed file 'plugins/python3.yaml'
--- plugins/python3.yaml	2015-09-09 21:35:18 +0000
+++ plugins/python3.yaml	1970-01-01 00:00:00 +0000
@@ -1,3 +0,0 @@
-options:
-  requirements:
-    required: false

=== added file 'plugins/scons-project.yaml'
--- plugins/scons-project.yaml	1970-01-01 00:00:00 +0000
+++ plugins/scons-project.yaml	2015-09-29 10:44:16 +0000
@@ -0,0 +1,8 @@
+build-packages:
+  - scons
+options:
+  source:
+    required: true
+  source-type:
+  source-tag:
+  source-branch:

=== modified file 'runtests.sh'
--- runtests.sh	2015-09-17 07:31:24 +0000
+++ runtests.sh	2015-09-29 10:44:16 +0000
@@ -20,39 +20,62 @@
 export PATH=$(pwd)/bin:$PATH
 export PYTHONPATH=$(pwd):$PYTHONPATH
 
-SRC_PATHS="bin snapcraft snapcraft/tests"
-
-# These three checks could easily be done with flake8 in one shot if
-# we had python3-flake8 provide flake8
-# Ignore 501 (line-too-long)
-pep8 $SRC_PATHS --ignore=E501
-
-pyflakes3 $SRC_PATHS
-
-# mccabe in 'warning' mode as we have high complexity
-mccabe_list=
-for unit in $(find snapcraft -type f -name '*.py')
-do
-  output=$(python3 -m mccabe --min 10 "$unit")
-  [ -n "$output" ] && mccabe_list="- $unit:\n  $output\n$mccabe_list"
-done
-
-if [ -n "$mccabe_list" ]; then
-  echo -e "\e[1;31mThe project has gotten complex\e[0m."
-  echo "Here's the list of units exceeding 10:"
-  echo -e "$mccabe_list"
-fi
-
-if which python3-coverage >/dev/null 2>&1; then
-    python3-coverage erase
-    python3-coverage run --branch --source snapcraft -m unittest
-    mv .coverage .coverage.unit
-else
-    python3 -m unittest
-fi
-
-if [ -z "$SNAPCRAFT_TESTS_SKIP_PLAINBOX" ]; then
-(
+parseargs(){
+    if [[ "$#" -eq 0 ]] || [[ "$1" == "all" ]]; then
+        export RUN_UNIT="true"
+        export RUN_PLAINBOX="true"
+        export PLAINBOX_TEST_PLANS="normal"
+    else
+        if [ "$1" == "unit" ] ; then
+            export RUN_UNIT="true"
+        elif [ "$1" == "plainbox" ] ; then
+            export RUN_PLAINBOX="true"
+            if [ "$#" -gt 1 ]; then
+                export PLAINBOX_TEST_PLANS="$2"
+            else
+                export PLAINBOX_TEST_PLANS="normal"
+            fi
+        else
+            echo "Not recognized option, should be one of all, unit or plainbox"
+            exit 1
+        fi
+    fi
+}
+
+run_unit_tests(){
+    SRC_PATHS="bin snapcraft snapcraft/tests"
+
+    # These three checks could easily be done with flake8 in one shot if
+    # we had python3-flake8 provide flake8
+    # Ignore 501 (line-too-long)
+    pep8 $SRC_PATHS --ignore=E501
+
+    pyflakes3 $SRC_PATHS
+
+    # mccabe in 'warning' mode as we have high complexity
+    mccabe_list=
+    for unit in $(find snapcraft -type f -name '*.py')
+    do
+        output=$(python3 -m mccabe --min 10 "$unit")
+        [ -n "$output" ] && mccabe_list="- $unit:\n  $output\n$mccabe_list"
+    done
+
+    if [ -n "$mccabe_list" ]; then
+        echo -e "\e[1;31mThe project has gotten complex\e[0m."
+        echo "Here's the list of units exceeding 10:"
+        echo -e "$mccabe_list"
+    fi
+
+    if which python3-coverage >/dev/null 2>&1; then
+        python3-coverage erase
+        python3-coverage run --branch --source snapcraft -m unittest
+        mv .coverage .coverage.unit
+    else
+        python3 -m unittest
+    fi
+}
+
+run_plainbox(){
     # well, well, what can we do
     if ! which plainbox >/dev/null; then
         cat <<EOF
@@ -73,8 +96,17 @@
 
     # Go to the plainbox provider of snapcraft tests
     cd integration-tests
-    ./runtests.sh
-)
+    ./runtests.sh $PLAINBOX_TEST_PLANS
+}
+
+parseargs "$@"
+
+if [ ! -z "$RUN_UNIT" ]; then
+    run_unit_tests
+fi
+
+if [ -z "$SNAPCRAFT_TESTS_SKIP_PLAINBOX" ] && [ ! -z "$RUN_PLAINBOX" ] ; then
+    run_plainbox
 fi
 
 if which python3-coverage >/dev/null 2>&1; then

=== modified file 'snapcraft/__init__.py'
--- snapcraft/__init__.py	2015-09-21 14:16:55 +0000
+++ snapcraft/__init__.py	2015-09-29 10:44:16 +0000
@@ -16,6 +16,7 @@
 
 import logging
 import os
+import re
 
 import snapcraft.common
 import snapcraft.sources
@@ -69,8 +70,17 @@
             cwd = self.builddir
         if True:
             print(' '.join(cmd))
+        self.makedirs(cwd)
         return snapcraft.common.run(cmd, cwd=cwd, **kwargs)
 
+    def run_output(self, cmd, cwd=None, **kwargs):
+        if cwd is None:
+            cwd = self.builddir
+        if True:
+            print(' '.join(cmd))
+        self.makedirs(cwd)
+        return snapcraft.common.run_output(cmd, cwd=cwd, **kwargs)
+
     def isurl(self, url):
         return snapcraft.common.isurl(url)
 
@@ -108,6 +118,13 @@
             ubuntu = snapcraft.repo.Ubuntu(self.ubuntudir, sources=self.PLUGIN_STAGE_SOURCES)
             ubuntu.get(self.PLUGIN_STAGE_PACKAGES + part_stage_packages)
             ubuntu.unpack(self.installdir)
+            self._fixup(self.installdir)
+
+    def _fixup(self, root):
+        if os.path.isfile(os.path.join(root, 'usr', 'bin', 'xml2-config')):
+            self.run(['sed', '-i', '-e', 's|prefix=/usr|prefix={}/usr|'.format(root), os.path.join(root, 'usr', 'bin', 'xml2-config')])
+        if os.path.isfile(os.path.join(root, 'usr', 'bin', 'xslt-config')):
+            self.run(['sed', '-i', '-e', 's|prefix=/usr|prefix={}/usr|'.format(root), os.path.join(root, 'usr', 'bin', 'xslt-config')])
 
 
 def _get_source_handler(source_type, source):
@@ -134,6 +151,8 @@
         source_type = 'bzr'
     elif source.startswith("git:"):
         source_type = 'git'
+    elif re.compile(r'.*\.((tar\.(xz|gz|bz2))|tgz)$').match(source):
+        source_type = 'tar'
     elif snapcraft.common.isurl(source):
         raise ValueError()
 

=== modified file 'snapcraft/cmds.py'
--- snapcraft/cmds.py	2015-09-21 19:20:43 +0000
+++ snapcraft/cmds.py	2015-09-29 10:44:16 +0000
@@ -267,6 +267,8 @@
             for f in common:
                 this = os.path.join(part.installdir, f)
                 other = os.path.join(parts_files[other_part_name]['installdir'], f)
+                if os.path.islink(this) and os.path.islink(other):
+                    continue
                 if not filecmp.cmp(this, other, shallow=False):
                     conflict_files.append(f)
 

=== modified file 'snapcraft/common.py'
--- snapcraft/common.py	2015-09-17 18:02:05 +0000
+++ snapcraft/common.py	2015-09-29 10:44:16 +0000
@@ -49,6 +49,17 @@
         return subprocess.call(['/bin/sh', f.name] + cmd, **kwargs) == 0
 
 
+def run_output(cmd, **kwargs):
+    assert isinstance(cmd, list), "run command must be a list"
+    # FIXME: This is gross to keep writing this, even when env is the same
+    with tempfile.NamedTemporaryFile(mode='w+') as f:
+        f.write(assemble_env())
+        f.write('\n')
+        f.write('exec $*')
+        f.flush()
+        return subprocess.check_output(['/bin/sh', f.name] + cmd, **kwargs).decode('utf8').strip()
+
+
 def fatal():
     sys.exit(1)
 

=== modified file 'snapcraft/plugin.py'
--- snapcraft/plugin.py	2015-09-22 03:08:07 +0000
+++ snapcraft/plugin.py	2015-09-29 10:44:16 +0000
@@ -39,15 +39,8 @@
 }
 
 
-def is_local_plugin(name):
-    return name.startswith("x-")
-
-
-def plugindir(name):
-    if is_local_plugin(name):
-        return os.path.abspath(os.path.join('parts', 'plugins'))
-    else:
-        return common.get_plugindir()
+def _local_plugindir():
+    return os.path.abspath(os.path.join('parts', 'plugins'))
 
 
 class PluginError(Exception):
@@ -87,10 +80,12 @@
             return
 
     def _load_config(self, name):
-        configPath = os.path.join(plugindir(name), name + ".yaml")
-        if not os.path.exists(configPath):
+        config_path = os.path.join(common.get_plugindir(), name + ".yaml")
+        if not os.path.exists(config_path):
+            config_path = os.path.join(_local_plugindir(), name + ".yaml")
+        if not os.path.exists(config_path):
             raise PluginError('Unknown plugin: {}'.format(name))
-        with open(configPath, 'r') as fp:
+        with open(config_path, 'r') as fp:
             self.config = yaml.load(fp) or {}
 
     def _make_options(self, name, properties):
@@ -121,14 +116,15 @@
         options = self._make_options(name, properties)
         module_name = name.replace('-', '_')
 
-        # Load code from local plugin dir if it is there
-        if is_local_plugin(name):
-            sys.path = [plugindir(name)] + sys.path
-        else:
-            module_name = 'snapcraft.plugins.' + module_name
+        try:
+            module = importlib.import_module('snapcraft.plugins.' + module_name)
+        except ImportError:
+            module = None
 
-        module = importlib.import_module(module_name)
-        if is_local_plugin(name):
+        if not module:
+            logger.info('Searching for local plugin for %s', name)
+            sys.path = [_local_plugindir()] + sys.path
+            module = importlib.import_module(module_name)
             sys.path.pop(0)
 
         for prop_name in dir(module):

=== modified file 'snapcraft/plugins/go_project.py'
--- snapcraft/plugins/go_project.py	2015-07-31 15:16:47 +0000
+++ snapcraft/plugins/go_project.py	2015-09-29 10:44:16 +0000
@@ -27,7 +27,9 @@
             self.fullname = self.options.source.split("://")[1]
 
     def pull(self):
-        return self.run(['go', 'get', '-t', self.fullname])
+        # use -d to only download (build will happen later)
+        # use -t to also get the test-deps
+        return self.run(['go', 'get', '-t', '-d', self.fullname])
 
     def build(self):
         if not self.run(['go', 'build', self.fullname]):

=== removed file 'snapcraft/plugins/python2.py'
--- snapcraft/plugins/python2.py	2015-09-17 15:14:49 +0000
+++ snapcraft/plugins/python2.py	1970-01-01 00:00:00 +0000
@@ -1,59 +0,0 @@
-# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
-#
-# Copyright (C) 2015 Canonical Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 3 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import os
-import snapcraft
-
-
-class Python2Plugin(snapcraft.BasePlugin):
-
-    _PLUGIN_STAGE_PACKAGES = [
-        'python-dev',
-        'python-setuptools',
-    ]
-
-    def __init__(self, name, options):
-        if options.requirements:
-            self.requirements = options.requirements
-            self._PLUGIN_STAGE_PACKAGES.extend(['python-pkg-resources', 'python-setuptools'])
-        else:
-            self.requirements = None
-        super().__init__(name, options)
-
-    def snap_fileset(self):
-        return [
-            '-usr/share',
-        ]
-
-    # note that we don't need to set PYTHONHOME here,
-    # python discovers this automatically from it installed
-    # location.  And PATH is automatically set by snapcraft.
-
-    def env(self, root):
-        return ["PYTHONPATH=%s" % os.path.join(
-            root, 'usr', 'lib', 'python2.7', 'dist-packages')]
-
-    def pull(self):
-        # A nice idea here would be to be asking setup tools
-        # to use the deb layout, but that doesn't work with
-        # prefix sadly
-
-        if self.requirements and not (self.run(
-                ['ln', '-s', os.path.join(self.installdir, 'usr', 'lib', 'python2.7', 'dist-packages'), os.path.join(self.installdir, 'usr', 'lib', 'python2.7', 'site-packages')]) and self.run(
-                ['python2', os.path.join(self.installdir, 'usr', 'bin', 'easy_install'), '--prefix', os.path.join(self.installdir, 'usr'), 'pip']) and self.run(
-                ['python2', os.path.join(self.installdir, 'usr', 'bin', 'pip2'), 'install', '--target', os.path.join(self.installdir, 'usr', 'lib', 'python2.7', 'site-packages'), '--requirement', os.path.join(os.getcwd(), self.requirements)])):
-            return False
-        return True

=== modified file 'snapcraft/plugins/python2_project.py'
--- snapcraft/plugins/python2_project.py	2015-09-21 22:22:36 +0000
+++ snapcraft/plugins/python2_project.py	2015-09-29 10:44:16 +0000
@@ -22,20 +22,76 @@
 
 class Python2ProjectPlugin(snapcraft.BasePlugin):
 
-    # note that we don't need to setup env(), python figures it out
-    # see python2.py for more details
+    _PLUGIN_STAGE_PACKAGES = [
+        'python-dev',
+        'python-pkg-resources',
+        'python-setuptools',
+    ]
+
+    def __init__(self, name, options):
+        super().__init__(name, options)
+        self.requirements = options.requirements
+        self.source = options.source
+
+    def env(self, root):
+        return ["PYTHONPATH=%s" % os.path.join(
+            root, 'usr', 'lib', self.python_version, 'dist-packages')]
 
     def pull(self):
-        return self.handle_source_options()
+        if self.source and not self.handle_source_options():
+            return False
+
+        return self._pip()
+
+    def _pip(self):
+        setup = 'setup.py'
+        if os.listdir(self.sourcedir):
+            setup = os.path.join(self.sourcedir, 'setup.py')
+
+        if self.requirements:
+            requirements = os.path.join(os.getcwd(), self.requirements)
+
+        if not os.path.exists(setup) and not self.requirements:
+            return True
+
+        easy_install = os.path.join(
+            self.installdir, 'usr', 'bin', 'easy_install')
+        prefix = os.path.join(self.installdir, 'usr')
+        site_packages_dir = os.path.join(
+            prefix, 'lib', self.python_version, 'site-packages')
+
+        if not os.path.exists(site_packages_dir):
+            os.symlink(
+                os.path.join(prefix, 'lib', self.python_version, 'dist-packages'),
+                site_packages_dir)
+
+        if not self.run(['python2', easy_install, '--prefix', prefix, 'pip']):
+            return False
+
+        pip2 = os.path.join(self.installdir, 'usr', 'bin', 'pip2')
+        pip_install = ['python2', pip2, 'install', '--target',
+                       site_packages_dir]
+
+        if self.requirements and not self.run(
+                pip_install + ['--requirement', requirements]):
+            return False
+
+        if os.path.exists(setup) and not self.run(pip_install + ['.', ]):
+            return False
+
+        return True
 
     def build(self):
         # If setuptools is used, it tries to create files in the
         # dist-packages dir and import from there, so it needs to exist
         # and be in the PYTHONPATH. It's harmless if setuptools isn't
         # used.
+
+        if not os.path.exists(os.path.join(self.builddir, 'setup.py')):
+            return True
+
         os.makedirs(self.dist_packages_dir, exist_ok=True)
         setuptemp = self.copy_setup()
-
         return self.run(
             ['python2', setuptemp.name, 'install', '--install-layout=deb',
              '--prefix={}/usr'.format(self.installdir)], cwd=self.builddir)
@@ -43,7 +99,11 @@
     @property
     def dist_packages_dir(self):
         return os.path.join(
-            self.installdir, 'usr', 'lib', 'python2.7', 'dist-packages')
+            self.installdir, 'usr', 'lib', self.python_version, 'dist-packages')
+
+    @property
+    def python_version(self):
+        return self.run_output(['pyversions', '-i'])
 
     # Takes the setup.py file and puts a couple little gems on the
     # front to make things work better.

=== removed file 'snapcraft/plugins/python3.py'
--- snapcraft/plugins/python3.py	2015-09-17 15:14:49 +0000
+++ snapcraft/plugins/python3.py	1970-01-01 00:00:00 +0000
@@ -1,61 +0,0 @@
-# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
-#
-# Copyright (C) 2015 Canonical Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 3 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import os
-import snapcraft
-
-
-class Python3Plugin(snapcraft.BasePlugin):
-
-    _PLUGIN_STAGE_PACKAGES = [
-        'python3-dev',
-        'python3-setuptools',
-    ]
-
-    def __init__(self, name, options):
-        if options.requirements:
-            self.requirements = options.requirements
-            self._PLUGIN_STAGE_PACKAGES.extend(['python3-pkg-resources', 'python3-setuptools'])
-        else:
-            self.requirements = None
-        super().__init__(name, options)
-
-    def snap_fileset(self):
-        return [
-            '-usr/share',
-        ]
-
-    # note that we don't need to set PYTHONHOME here,
-    # python discovers this automatically from it installed
-    # location, see https://code.launchpad.net/~mvo/snapcraft/python3-project/+merge/264521/comments/664308
-    #
-    # PATH is automatically set by snapcraft
-
-    def env(self, root):
-        return ["PYTHONPATH=%s" % os.path.join(
-            root, 'usr', 'lib', 'python3', 'dist-packages')]
-
-    def pull(self):
-        # A nice idea here would be to be asking setup tools
-        # to use the deb layout, but that doesn't work with
-        # prefix sadly
-
-        if self.requirements and not (self.run(
-                ['ln', '-s', os.path.join(self.installdir, 'usr', 'lib', 'python3', 'dist-packages'), os.path.join(self.installdir, 'usr', 'lib', 'python3.4', 'site-packages')]) and self.run(
-                ['python3', os.path.join(self.installdir, 'usr', 'bin', 'easy_install3'), '--prefix', os.path.join(self.installdir, 'usr'), 'pip']) and self.run(
-                ['python3', os.path.join(self.installdir, 'usr', 'bin', 'pip3'), 'install', '--target', os.path.join(self.installdir, 'usr', 'lib', 'python3.4', 'site-packages'), '--requirement', os.path.join(os.getcwd(), self.requirements)])):
-            return False
-        return True

=== modified file 'snapcraft/plugins/python3_project.py'
--- snapcraft/plugins/python3_project.py	2015-09-21 22:22:36 +0000
+++ snapcraft/plugins/python3_project.py	2015-09-29 10:44:16 +0000
@@ -22,21 +22,74 @@
 
 class Python3ProjectPlugin(snapcraft.BasePlugin):
 
-    # note that we don't need to setup env(), python figures it out
-    # see python3.py for more details
+    _PLUGIN_STAGE_PACKAGES = [
+        'python3-dev',
+        'python3-pkg-resources',
+        'python3-setuptools',
+    ]
+
+    def __init__(self, name, options):
+        super().__init__(name, options)
+        self.requirements = options.requirements
+        self.source = options.source
 
     def env(self, root):
         return ["PYTHONPATH=%s" % os.path.join(
-            root, 'usr', 'lib', 'python3', 'dist-packages')]
+            root, 'usr', 'lib', self.python_version, 'dist-packages')]
 
     def pull(self):
-        return self.handle_source_options()
+        if self.source and not self.handle_source_options():
+            return False
+
+        return self._pip()
+
+    def _pip(self):
+        setup = 'setup.py'
+        if os.listdir(self.sourcedir):
+            setup = os.path.join(self.sourcedir, 'setup.py')
+
+        if self.requirements:
+            requirements = os.path.join(os.getcwd(), self.requirements)
+
+        if not os.path.exists(setup) and not self.requirements:
+            return True
+
+        easy_install = os.path.join(
+            self.installdir, 'usr', 'bin', 'easy_install3')
+        prefix = os.path.join(self.installdir, 'usr')
+        site_packages_dir = os.path.join(
+            prefix, 'lib', self.python_version, 'site-packages')
+
+        if not os.path.exists(site_packages_dir):
+            os.symlink(
+                os.path.join(prefix, 'lib', 'python3', 'dist-packages'),
+                site_packages_dir)
+
+        if not self.run(['python3', easy_install, '--prefix', prefix, 'pip']):
+            return False
+
+        pip3 = os.path.join(self.installdir, 'usr', 'bin', 'pip3')
+        pip_install = ['python3', pip3, 'install', '--target',
+                       site_packages_dir]
+
+        if self.requirements and not self.run(
+                pip_install + ['--requirement', requirements]):
+            return False
+
+        if os.path.exists(setup) and not self.run(pip_install + ['.', ]):
+            return False
+
+        return True
 
     def build(self):
         # If setuptools is used, it tries to create files in the
         # dist-packages dir and import from there, so it needs to exist
         # and be in the PYTHONPATH. It's harmless if setuptools isn't
         # used.
+
+        if not os.path.exists(os.path.join(self.builddir, 'setup.py')):
+            return True
+
         os.makedirs(self.dist_packages_dir, exist_ok=True)
         setuptemp = self.copy_setup()
         return self.run(
@@ -46,7 +99,11 @@
     @property
     def dist_packages_dir(self):
         return os.path.join(
-            self.installdir, 'usr', 'lib', 'python3', 'dist-packages')
+            self.installdir, 'usr', 'lib', self.python_version, 'dist-packages')
+
+    @property
+    def python_version(self):
+        return self.run_output(['py3versions', '-i'])
 
     # Takes the setup.py file and puts a couple little gems on the
     # front to make things work better.

=== added file 'snapcraft/plugins/scons_project.py'
--- snapcraft/plugins/scons_project.py	1970-01-01 00:00:00 +0000
+++ snapcraft/plugins/scons_project.py	2015-09-29 10:44:16 +0000
@@ -0,0 +1,31 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright (C) 2015 Canonical Ltd
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+import snapcraft
+
+
+class SconsPlugin(snapcraft.BasePlugin):
+
+    def pull(self):
+        return self.handle_source_options()
+
+    def build(self):
+        env = os.environ.copy()
+        env['DESTDIR'] = self.installdir
+        return self.run(['scons', 'prefix=/usr']) and \
+            self.run(['scons', 'install'], env=env)

=== modified file 'snapcraft/repo.py'
--- snapcraft/repo.py	2015-09-21 15:21:15 +0000
+++ snapcraft/repo.py	2015-09-29 10:44:16 +0000
@@ -14,6 +14,7 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+import apt
 import glob
 import itertools
 import os
@@ -22,18 +23,19 @@
 import urllib
 import urllib.request
 
-import apt
 from xml.etree import ElementTree
 
-_DEFAULT_SOURCES = '''deb http://${mirror}archive.ubuntu.com/ubuntu/ vivid main restricted
-deb http://${mirror}archive.ubuntu.com/ubuntu/ vivid-updates main restricted
-deb http://${mirror}archive.ubuntu.com/ubuntu/ vivid universe
-deb http://${mirror}archive.ubuntu.com/ubuntu/ vivid-updates universe
-deb http://${mirror}archive.ubuntu.com/ubuntu/ vivid multiverse
-deb http://${mirror}archive.ubuntu.com/ubuntu/ vivid-updates multiverse
-deb http://security.ubuntu.com/ubuntu vivid-security main restricted
-deb http://security.ubuntu.com/ubuntu vivid-security universe
-deb http://security.ubuntu.com/ubuntu vivid-security multiverse
+import snapcraft.common
+
+_DEFAULT_SOURCES = '''deb http://${prefix}.ubuntu.com/${suffix}/ ${release} main restricted
+deb http://${prefix}.ubuntu.com/${suffix}/ ${release}-updates main restricted
+deb http://${prefix}.ubuntu.com/${suffix}/ ${release} universe
+deb http://${prefix}.ubuntu.com/${suffix}/ ${release}-updates universe
+deb http://${prefix}.ubuntu.com/${suffix}/ ${release} multiverse
+deb http://${prefix}.ubuntu.com/${suffix}/ ${release}-updates multiverse
+deb http://${security}.ubuntu.com/${suffix} ${release}-security main restricted
+deb http://${security}.ubuntu.com/${suffix} ${release}-security universe
+deb http://${security}.ubuntu.com/${suffix} ${release}-security multiverse
 '''
 _GEOIP_SERVER = "http://geoip.ubuntu.com/lookup"
 
@@ -61,11 +63,16 @@
 class Ubuntu:
 
     def __init__(self, rootdir, recommends=False, sources=_DEFAULT_SOURCES):
-        sources = sources or _DEFAULT_SOURCES
         self.downloaddir = os.path.join(rootdir, 'download')
         self.rootdir = rootdir
-        self.apt_cache = _setup_apt_cache(rootdir, sources)
         self.recommends = recommends
+        sources = sources or _DEFAULT_SOURCES
+        local = False
+        if 'SNAPCRAFT_LOCAL_SOURCES' in os.environ:
+            print('using local sources')
+            sources = _get_local_sources_list()
+            local = True
+        self.apt_cache = _setup_apt_cache(rootdir, sources, local)
 
     def get(self, package_names):
         os.makedirs(self.downloaddir, exist_ok=True)
@@ -78,18 +85,35 @@
             except KeyError:
                 raise PackageNotFoundError(name)
 
+        skipped_essential = []
+        skipped_blacklisted = []
+
+        # unmark some base packages here
+        # note that this will break the consistency check inside apt_cache
+        # (self.apt_cache.broken_count will be > 0)
+        # but that is ok as it was consistent before we excluded
+        # these base package
         for pkg in self.apt_cache:
             # those should be already on each system, it also prevents
             # diving into downloading libc6
             if (pkg.candidate.priority in 'essential' and
                pkg.name not in package_names):
-                print('Skipping priority essential/imporant %s' % pkg.name)
+                skipped_essential.append(pkg.name)
+                pkg.mark_keep()
                 continue
             if (pkg.name in manifest_dep_names and pkg.name not in package_names):
-                print('Skipping blacklisted from manifest package %s' % pkg.name)
+                skipped_blacklisted.append(pkg.name)
+                pkg.mark_keep()
                 continue
-            if pkg.marked_install:
-                pkg.candidate.fetch_binary(destdir=self.downloaddir)
+
+        if skipped_essential:
+            print('Skipping priority essential packages:', skipped_essential)
+        if skipped_blacklisted:
+            print('Skipping blacklisted from manifest packages:', skipped_blacklisted)
+
+        # download the remaining ones with proper progress
+        apt.apt_pkg.config.set("Dir::Cache::Archives", self.downloaddir)
+        self.apt_cache.fetch_archives()
 
     def unpack(self, rootdir):
         pkgs_abs_path = glob.glob(os.path.join(self.downloaddir, '*.deb'))
@@ -114,7 +138,19 @@
         return manifest_dep_names
 
 
-def get_geoip_country_code_prefix():
+def _get_local_sources_list():
+    sources_list = glob.glob('/etc/apt/sources.list.d/*.list')
+    sources_list.append('/etc/apt/sources.list')
+
+    sources = ''
+    for source in sources_list:
+        with open(source) as f:
+            sources += f.read()
+
+    return sources
+
+
+def _get_geoip_country_code_prefix():
     try:
         with urllib.request.urlopen(_GEOIP_SERVER) as f:
             xml_data = f.read()
@@ -122,20 +158,41 @@
         cc = et.find("CountryCode")
         if cc is None:
             return ""
-        return cc.text.lower() + "."
+        return cc.text.lower()
     except (ElementTree.ParseError, urllib.error.URLError):
         pass
-    return ""
-
-
-def _setup_apt_cache(rootdir, sources):
+    return ''
+
+
+def _format_sources_list(sources, arch, release='vivid'):
+    if arch in ('amd64', 'i386'):
+        geoip_prefix = _get_geoip_country_code_prefix()
+        prefix = geoip_prefix + '.archive' if geoip_prefix else 'archive'
+        suffix = 'ubuntu'
+        security = 'security'
+    else:
+        prefix = 'ports'
+        suffix = 'ubuntu-ports'
+        security = 'ports'
+
+    return string.Template(sources).substitute({
+        'prefix': prefix,
+        'release': release,
+        'suffix': suffix,
+        'security': security,
+    })
+
+
+def _setup_apt_cache(rootdir, sources, local=False):
     os.makedirs(os.path.join(rootdir, 'etc', 'apt'), exist_ok=True)
     srcfile = os.path.join(rootdir, 'etc', 'apt', 'sources.list')
+
+    if not local:
+        sources = _format_sources_list(sources, snapcraft.common.get_arch())
+
     with open(srcfile, 'w') as f:
-        mirror_prefix = get_geoip_country_code_prefix()
-        sources_list = string.Template(sources).substitute(
-            {"mirror": mirror_prefix})
-        f.write(sources_list)
+        f.write(sources)
+
     progress = apt.progress.text.AcquireProgress()
     apt_cache = apt.Cache(rootdir=rootdir, memonly=True)
     apt_cache.update(fetch_progress=progress, sources_list=srcfile)

=== modified file 'snapcraft/sources.py'
--- snapcraft/sources.py	2015-09-15 05:17:14 +0000
+++ snapcraft/sources.py	2015-09-29 10:44:16 +0000
@@ -17,6 +17,7 @@
 import logging
 import os
 import os.path
+import shutil
 import tarfile
 import re
 
@@ -130,13 +131,17 @@
         else:
             return True
 
-    def provision(self, dst):
+    def provision(self, dst, clean_target=True):
         # TODO add unit tests.
         if snapcraft.common.isurl(self.source):
             tarball = os.path.join(self.source_dir, os.path.basename(self.source))
         else:
             tarball = os.path.abspath(self.source)
 
+        if clean_target:
+            shutil.rmtree(dst)
+            os.makedirs(dst)
+
         with tarfile.open(tarball) as tar:
             def filter_members(tar):
                 """Filters members and member names:
@@ -163,6 +168,7 @@
                         m.name = m.name[len(common + "/"):]
                     # strip leading "/", "./" or "../" as many times as needed
                     m.name = re.sub(r'^(\.{0,2}/)*', r'', m.name)
+                    m.mode = m.mode | 0o222
                     yield m
 
             tar.extractall(members=filter_members(tar), path=dst)

=== modified file 'snapcraft/tests/test_base_plugin.py'
--- snapcraft/tests/test_base_plugin.py	2015-08-28 20:26:41 +0000
+++ snapcraft/tests/test_base_plugin.py	2015-09-29 10:44:16 +0000
@@ -57,6 +57,19 @@
         plugin.makedirs(path)
         self.assertTrue(os.path.exists(path))
 
+    def test_get_tar_source_from_uri(self):
+        sources = [
+            'https://golang.tar.gz',
+            'https://golang.tar.xz',
+            'https://golang.tar.bz2',
+            'https://golang.tar.tgz',
+        ]
+
+        for source in sources:
+            with self.subTest(key=source):
+                self.assertEqual(
+                    snapcraft._get_source_type_from_uri(source), 'tar')
+
 
 class GetSourceWithBranches(tests.TestCase):
 

=== modified file 'snapcraft/tests/test_plugin.py'
--- snapcraft/tests/test_plugin.py	2015-09-22 03:08:07 +0000
+++ snapcraft/tests/test_plugin.py	2015-09-29 10:44:16 +0000
@@ -16,7 +16,6 @@
 
 import logging
 import os
-import sys
 import tempfile
 
 import fixtures
@@ -210,19 +209,6 @@
 
         self.assertEqual('test stage mock-part\n', fake_logger.output)
 
-    def test_local_plugins(self):
-        """Ensure local plugins are loaded from parts/plugins"""
-        def mock_import_modules(module_name):
-            # called with the name only and sys.path set
-            self.assertEqual(module_name, "x_mock")
-            self.assertTrue(sys.path[0].endswith("parts/plugins"))
-            return mock_plugin
-        with patch("importlib.import_module", side_effect=mock_import_modules):
-            plugin.PluginHandler(
-                "x-mock", "mock-part", {}, load_config=False, load_code=True)
-        # sys.path is cleaned afterwards
-        self.assertFalse(sys.path[0].endswith("parts/plugins"))
-
     def test_non_local_plugins(self):
         """Ensure regular plugins are loaded from snapcraft only"""
         def mock_import_modules(module_name):

=== modified file 'snapcraft/tests/test_repo.py'
--- snapcraft/tests/test_repo.py	2015-09-21 15:14:29 +0000
+++ snapcraft/tests/test_repo.py	2015-09-29 10:44:16 +0000
@@ -16,6 +16,7 @@
 
 import os
 import tempfile
+import unittest.mock
 
 from snapcraft import repo
 from snapcraft import tests
@@ -23,6 +24,43 @@
 
 class UbuntuTestCase(tests.TestCase):
 
+    @unittest.mock.patch('snapcraft.repo._get_geoip_country_code_prefix')
+    def test_sources_amd64_vivid(self, mock_cc):
+        mock_cc.return_value = 'ar'
+
+        sources_list = repo._format_sources_list(
+            repo._DEFAULT_SOURCES, 'amd64', 'vivid')
+
+        expected_sources_list = '''deb http://ar.archive.ubuntu.com/ubuntu/ vivid main restricted
+deb http://ar.archive.ubuntu.com/ubuntu/ vivid-updates main restricted
+deb http://ar.archive.ubuntu.com/ubuntu/ vivid universe
+deb http://ar.archive.ubuntu.com/ubuntu/ vivid-updates universe
+deb http://ar.archive.ubuntu.com/ubuntu/ vivid multiverse
+deb http://ar.archive.ubuntu.com/ubuntu/ vivid-updates multiverse
+deb http://security.ubuntu.com/ubuntu vivid-security main restricted
+deb http://security.ubuntu.com/ubuntu vivid-security universe
+deb http://security.ubuntu.com/ubuntu vivid-security multiverse
+'''
+        self.assertEqual(sources_list, expected_sources_list)
+
+    @unittest.mock.patch('snapcraft.repo._get_geoip_country_code_prefix')
+    def test_sources_armhf_trusty(self, mock_cc):
+        sources_list = repo._format_sources_list(
+            repo._DEFAULT_SOURCES, 'armhf', 'trusty')
+
+        expected_sources_list = '''deb http://ports.ubuntu.com/ubuntu-ports/ trusty main restricted
+deb http://ports.ubuntu.com/ubuntu-ports/ trusty-updates main restricted
+deb http://ports.ubuntu.com/ubuntu-ports/ trusty universe
+deb http://ports.ubuntu.com/ubuntu-ports/ trusty-updates universe
+deb http://ports.ubuntu.com/ubuntu-ports/ trusty multiverse
+deb http://ports.ubuntu.com/ubuntu-ports/ trusty-updates multiverse
+deb http://ports.ubuntu.com/ubuntu-ports trusty-security main restricted
+deb http://ports.ubuntu.com/ubuntu-ports trusty-security universe
+deb http://ports.ubuntu.com/ubuntu-ports trusty-security multiverse
+'''
+        self.assertEqual(sources_list, expected_sources_list)
+        self.assertFalse(mock_cc.called)
+
     def test_fix_symlinks(self):
         tempdirObj = tempfile.TemporaryDirectory()
         self.addCleanup(tempdirObj.cleanup)

=== modified file 'snapcraft/yaml.py'
--- snapcraft/yaml.py	2015-09-18 18:04:48 +0000
+++ snapcraft/yaml.py	2015-09-29 10:44:16 +0000
@@ -166,14 +166,46 @@
 
     def runtime_env(self, root):
         env = []
-        env.append('PATH="{0}/bin:{0}/usr/bin:$PATH"'.format(root))
-        env.append('LD_LIBRARY_PATH="{0}/lib:{0}/usr/lib:{0}/lib/{1}:{0}/usr/lib/{1}:$LD_LIBRARY_PATH"'.format(root, snapcraft.common.get_arch_triplet()))
+        env.append('PATH="' + ':'.join([
+            '{0}/bin',
+            '{0}/usr/bin',
+            '$PATH'
+        ]).format(root) + '"')
+        env.append('LD_LIBRARY_PATH="' + ':'.join([
+            '{0}/lib',
+            '{0}/usr/lib',
+            '{0}/lib/{1}',
+            '{0}/usr/lib/{1}',
+            '$LD_LIBRARY_PATH'
+        ]).format(root, snapcraft.common.get_arch_triplet()) + '"')
         return env
 
     def build_env(self, root):
         env = []
-        env.append('CFLAGS="-I{0}/include -I{0}/usr/include -I{0}/include/{1} -I{0}/usr/include/{1} $CFLAGS"'.format(root, snapcraft.common.get_arch_triplet()))
-        env.append('LDFLAGS="-L{0}/lib -L{0}/usr/lib -L{0}/lib/{1} -L{0}/usr/lib/{1} $LDFLAGS"'.format(root, snapcraft.common.get_arch_triplet()))
+        env.append('CFLAGS="' + ' '.join([
+            '-I{0}/include',
+            '-I{0}/usr/include',
+            '-I{0}/include/{1}',
+            '-I{0}/usr/include/{1}',
+            '$CFLAGS'
+        ]).format(root, snapcraft.common.get_arch_triplet()) + '"')
+        env.append('LDFLAGS="' + ' '.join([
+            '-L{0}/lib',
+            '-L{0}/usr/lib',
+            '-L{0}/lib/{1}',
+            '-L{0}/usr/lib/{1}',
+            '$LDFLAGS'
+        ]).format(root, snapcraft.common.get_arch_triplet()) + '"')
+        env.append('PKG_CONFIG_SYSROOT_DIR={0}'.format(root))
+        env.append('PKG_CONFIG_PATH=' + ':'.join([
+            '{0}/usr/lib/pkgconfig',
+            '{0}/usr/lib/{1}/pkgconfig',
+            '{0}/usr/share/pkgconfig',
+            '{0}/usr/local/lib/pkgconfig',
+            '{0}/usr/local/lib/{1}/pkgconfig',
+            '{0}/usr/local/share/pkgconfig',
+            '$PKG_CONFIG_PATH'
+        ]).format(root, snapcraft.common.get_arch_triplet()))
         return env
 
     def build_env_for_part(self, part):