Merge lp:~sergiusens/snapcraft/sources_for_sources into lp:~dholbach/snapcraft/1498347

[Sergio Schvezov]

Allow using local sources through an env var (mostly useful for launchpad building)

In launchpad, geoip does not work so our code gets stuck. For now we will use whatever the builder is using. We need a longer plan strategy for this one.

Unmerged revisions

209. By Sergio Schvezov on 2015-09-24

Be more resilient to geop failing

208. By Sergio Schvezov on 2015-09-23

Add flag to use local sources

207. By Sergio Schvezov on 2015-09-23

preparing a changelog by sergiusens approved by sergiusens,dholbach

206. By Michael Vogt on 2015-09-23

The go plugin should only download in the pull phase by mvo approved by sergiusens

205. By Ted Gould on 2015-09-23

Ensure C library configuration tools don't use system paths by ted approved by sergiusens

204. By Sergio Schvezov on 2015-09-23

Show skipped packages only once per filter by sergiusens approved by chipaca

203. By Sergio Schvezov on 2015-09-23

do NOT print in tests by sergiusens approved by fgimenez

202. By Sergio Schvezov on 2015-09-23

Make the py 2 and 3 projects smarter and have dependencies pulled during the pull phase by sergiusens approved by chipaca

201. By Michael Vogt on 2015-09-23

Add quotes around $TEST_PLAN and pass the $1 directly to run_test_plan instead of indirectly in runtests.sh by mvo approved by sergiusens,fgimenez,dholbach

200. By Zygmunt Krynicki on 2015-09-23

Add 'simple' and 'has-leftovers' flags to all integration tests job units. by zyga approved by dholbach,elopio

=== modified file 'debian/changelog'
--- debian/changelog	2015-08-18 12:37:44 +0000
+++ debian/changelog	2015-09-24 04:06:53 +0000
@@ -1,8 +1,45 @@
-snapcraft (0.2) UNRELEASED; urgency=medium
-
-  * 
-
- -- Daniel Holbach <[e-mail]>  Tue, 18 Aug 2015 14:23:24 +0200
+snapcraft (0.2) wily; urgency=medium
+
+  * Better setuptools support:
+    - Better support python-setuptools (LP: #1481864),
+    - Used builddir as a base for setup.py overrides (LP: #1498212),
+    - Fix build (LP: #1494825),
+  * Plugin fixes:
+    - Add demo.qml back to the snapcraft.yaml for the QML Demo (LP: #1491301),
+    - stage package libgudev-1.0-0, which makes the godd snap actually 
+      work (LP: #1498347),
+    - Wrap setup.py calls to configure the shebang writer (LP: #1486680),
+    - Update tomcat upstream URL, fixes example again (LP: #1491303),
+    - Add libgudev-1.0-dev as build-tools for godd example,
+      Ensure C library configuration tools don't use system 
+      paths (LP: #1496789),
+  * Snapcraft cli:
+    - On clean, check the contents of the parts dir only if it 
+      exists. (LP: #1497371)
+    - Load the config before trying to run (LP: #1498140),
+    - Notify user why the password is being requested (LP: #1481499),
+    - Fix numerous issues in snapcraft run (LP: #1486659),
+  * Snapcraft stage-packages:
+    - Enable ports for architectures that are not amd64 or 
+      i386 (LP: #1498157),
+    - Improving stage-package handling (LP: #1497453, LP: #1497582)
+    - Fetch all packages in a single download run with proper 
+      progress (LP: #1498333),
+  * Snapcraft wiki queries excessive (LP: #1496381),
+  * Snapcraft internals:
+    - Add 'simple' and 'has-leftovers' flags to all integration tests job
+      units. (LP: #1484596),
+    - Regex for binary and service names (LP: #1495662),
+    - Provide a nice error when tabs found in snapcraft.yaml (LP: #1477875),
+    - Use relative paths for image creation (LP: #1497108),
+    - Use the python logger (LP: #1476452),
+    - snapcraft now has less exit points (LP: #1477639),
+    - Depend and Build-Depend on python3-requests (LP: #1496363),
+    - Add autopkgtest, reshuffle build-deps accordingly (LP: #1496392),
+    - Fix "snapcraft run" (LP: #1484720),
+  * Examples packaged (LP: #1498189).
+
+ -- Sergio Schvezov <[e-mail]>  Wed, 24 Sep 2015 14:23:24 -0300
 
 snapcraft (0.1) wily; urgency=low
 

=== modified file 'debian/control'
--- debian/control	2015-09-16 14:26:10 +0000
+++ debian/control	2015-09-24 04:06:53 +0000
@@ -39,3 +39,14 @@
  a single toolset, but instead is a collection of tools that enable the
  natural workflow of an upstream to be extended with a simple release step
  into Snappy.
+
+Package: snapcraft-examples
+Architecture: all
+Depends: snapcraft (= ${binary:Version}), ${misc:Depends}, ${python3:Depends}
+Description: examples for snapcraft
+ Snapcraft aims to make upstream developers' lives easier and as such is not
+ a single toolset, but instead is a collection of tools that enable the
+ natural workflow of an upstream to be extended with a simple release step
+ into Snappy.
+ .
+ This package contains examples for snapcraft.

=== modified file 'debian/rules'
--- debian/rules	2015-09-16 14:26:10 +0000
+++ debian/rules	2015-09-24 04:06:53 +0000
@@ -3,3 +3,6 @@
 
 %:
 	dh $@ --with python3 --buildsystem=pybuild
+
+override_dh_compress:
+	dh_compress -X/usr/share/snapcraft/examples

=== added file 'debian/snapcraft-examples.install'
--- debian/snapcraft-examples.install	1970-01-01 00:00:00 +0000
+++ debian/snapcraft-examples.install	2015-09-24 04:06:53 +0000
@@ -0,0 +1,1 @@
+examples		/usr/share/snapcraft

=== added file 'debian/snapcraft.install'
--- debian/snapcraft.install	1970-01-01 00:00:00 +0000
+++ debian/snapcraft.install	2015-09-24 04:06:53 +0000
@@ -0,0 +1,3 @@
+/usr/bin
+/usr/lib/python*
+/usr/share/snapcraft

=== modified file 'debian/source/options'
--- debian/source/options	2015-07-06 16:48:22 +0000
+++ debian/source/options	2015-09-24 04:06:53 +0000
@@ -1,7 +1,2 @@
 # Ignore standard files (like .bzr)
 tar-ignore
-
-# Ignore examples folder because it likely contains a bunch of stuff we don't
-# care about (like downloaded go1.4 binaries and such).  It's not important
-# for it to be distributed with the source.
-tar-ignore = "examples"

=== modified file 'debian/tests/control'
--- debian/tests/control	2015-09-17 14:56:20 +0000
+++ debian/tests/control	2015-09-24 04:06:53 +0000
@@ -1,5 +1,6 @@
-Test-Command: ./runtests.sh
-Restrictions: allow-stderr, isolation-container
+Tests: runtests
+       runexamples
+Restrictions: allow-stderr, isolation-container, rw-build-tree
 Depends: @,
          build-essential,
          pep8,

=== added file 'debian/tests/runexamples'
--- debian/tests/runexamples	1970-01-01 00:00:00 +0000
+++ debian/tests/runexamples	2015-09-24 04:06:53 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+./runtests.sh plainbox examples

=== added file 'debian/tests/runtests'
--- debian/tests/runtests	1970-01-01 00:00:00 +0000
+++ debian/tests/runtests	2015-09-24 04:06:53 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+./runtests.sh

=== modified file 'docs/your-first-snap.md'
--- docs/your-first-snap.md	2015-09-14 20:53:28 +0000
+++ docs/your-first-snap.md	2015-09-24 04:06:53 +0000
@@ -268,8 +268,8 @@
     description: Exposes your webcam over a web UI
     icon: icon.png
     services:
-    - name: webcam-webui
-      start: bin/webcam-webui
+      webcam-webui:
+        start: bin/webcam-webui
 
     parts:
       golang-static-http:

=== modified file 'examples/downloader-with-wiki-parts/snapcraft.yaml'
--- examples/downloader-with-wiki-parts/snapcraft.yaml	2015-09-21 05:04:31 +0000
+++ examples/downloader-with-wiki-parts/snapcraft.yaml	2015-09-24 04:06:53 +0000
@@ -14,3 +14,4 @@
         source: .
         after:
             - curl
+build-packages: [libssl-dev]

=== added directory 'integration-tests/data/build-tools'
=== added file 'integration-tests/data/build-tools/Makefile'
--- integration-tests/data/build-tools/Makefile	1970-01-01 00:00:00 +0000
+++ integration-tests/data/build-tools/Makefile	2015-09-24 04:06:53 +0000
@@ -0,0 +1,6 @@
+
+all:
+	@echo "Makin' like a Makefile"
+
+install:
+	@echo "Installin' like a Makefile"

=== added file 'integration-tests/data/build-tools/icon.svg'
--- integration-tests/data/build-tools/icon.svg	1970-01-01 00:00:00 +0000
+++ integration-tests/data/build-tools/icon.svg	2015-09-24 04:06:53 +0000
@@ -0,0 +1,1 @@
+<svg/>

=== added file 'integration-tests/data/build-tools/snapcraft.yaml'
--- integration-tests/data/build-tools/snapcraft.yaml	1970-01-01 00:00:00 +0000
+++ integration-tests/data/build-tools/snapcraft.yaml	2015-09-24 04:06:53 +0000
@@ -0,0 +1,14 @@
+name: build-tools-test
+version: 0.1
+vendor: Napoleon Bonaparte <[e-mail]>
+summary: one line summary
+description: a longer description
+icon: icon.svg
+
+parts:
+  make-project:
+    source: .
+    stage-packages:
+      - libxml2-dev
+      - libxslt1-dev
+      - libglib2.0-dev

=== modified file 'integration-tests/data/pip-requirements/snapcraft.yaml'
--- integration-tests/data/pip-requirements/snapcraft.yaml	2015-09-09 20:19:12 +0000
+++ integration-tests/data/pip-requirements/snapcraft.yaml	2015-09-24 04:06:53 +0000
@@ -7,8 +7,8 @@
 
 parts:
   python2:
-    type: python2
+    type: python2-project
     requirements: requirements.txt
   python3:
-    type: python3
+    type: python3-project
     requirements: requirements.txt

=== modified file 'integration-tests/runtests.sh'
--- integration-tests/runtests.sh	2015-08-04 23:46:54 +0000
+++ integration-tests/runtests.sh	2015-09-24 04:06:53 +0000
@@ -17,24 +17,43 @@
 
 set -e
 
-if [ -z "$SNAPCRAFT" ]; then
-    export SNAPCRAFT=snapcraft
-fi
+run_test_plan(){
+    TEST_PLAN=$1
+    
+    if [ -z "$SNAPCRAFT" ]; then
+        export SNAPCRAFT=snapcraft
+    fi
 
-# Create a temporary directory so that we can run 'manage.py develop' and
-# create the .provider file there
-temp_dir=$(mktemp -d)
-# Develop the provider, this will let us run tests on it
-./manage.py develop -d $temp_dir
-# Set PROVIDERPATH (see plainbox(1)) so that we can see the provider
-# without installing it.
-export PROVIDERPATH=$PROVIDERPATH:$temp_dir
-# Run the 'normal' test plan
-plainbox run \
-    -T 2015.com.canonical.snapcraft::normal \
-    -f json -o $temp_dir/result.json
-# Analyze the result and fail if there are any failures
-python3 - << __PYTHON__
+    # Create a temporary directory so that we can run 'manage.py develop' and
+    # create the .provider file there
+    temp_dir=$(mktemp -d)
+    # Develop the provider, this will let us run tests on it
+    ./manage.py develop -d $temp_dir
+    # Set PROVIDERPATH (see plainbox(1)) so that we can see the provider
+    # without installing it.
+    export PROVIDERPATH=$PROVIDERPATH:$temp_dir
+    # create symlink from the provider's data directory to examples for them to
+    # be available to the tests
+    if [ "$TEST_PLAN" = "examples" ]; then
+        BASEDIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )"/.. && pwd )
+        TARGET_PATH=${BASEDIR}/integration-tests/data/examples
+        if [ -z $ADT_ARTIFACTS ]; then
+            EXAMPLES_PATH=${BASEDIR}/examples
+        else
+            EXAMPLES_PATH=/usr/share/snapcraft/examples
+        fi
+        ln -fs ${EXAMPLES_PATH} ${TARGET_PATH}
+    fi
+    # Run the test plan
+    plainbox run \
+             -T 2015.com.canonical.snapcraft::"$TEST_PLAN" \
+             -f json -o $temp_dir/result.json
+    # remove the examples symlink
+    if [ "$TEST_PLAN" = "examples" ]; then
+        rm ${TARGET_PATH}
+    fi
+    # Analyze the result and fail if there are any failures
+    python3 - << __PYTHON__
 import json
 with open("$temp_dir/result.json", "rt", encoding="utf-8") as stream:
     results = json.load(stream)
@@ -46,3 +65,11 @@
 print("Overall: {0}".format("fail" if failed else "pass"))
 raise SystemExit(failed)
 __PYTHON__
+}
+
+if [ -z "$1" ]; then
+    echo "need test plan as first argument"
+    exit 1
+fi
+
+run_test_plan "$@"

=== added file 'integration-tests/units/examples.pxu'
--- integration-tests/units/examples.pxu	1970-01-01 00:00:00 +0000
+++ integration-tests/units/examples.pxu	2015-09-24 04:06:53 +0000
@@ -0,0 +1,10 @@
+id: snapcraft/examples/build
+plugin: shell
+command:
+    set -e
+    for example in ${PLAINBOX_PROVIDER_DATA}/examples/*
+    do
+      cp -rT ${example//:} ${PLAINBOX_SESSION_SHARE}
+      cd ${PLAINBOX_SESSION_SHARE} && ${SNAPCRAFT}
+      rm -rf ${PLAINBOX_SESSION_SHARE}/*
+    done

=== modified file 'integration-tests/units/jobs.pxu'
--- integration-tests/units/jobs.pxu	2015-09-21 19:20:43 +0000
+++ integration-tests/units/jobs.pxu	2015-09-24 04:06:53 +0000
@@ -6,6 +6,7 @@
     OUTPUT=$(${SNAPCRAFT} pull 2>&1)
     test $? = 1 || exit 1
     echo $OUTPUT | grep "Could not find snapcraft\.yaml\."
+flags: simple has-leftovers
 
 id: snapcraft/normal/no-yaml-run
 plugin: shell
@@ -26,6 +27,7 @@
     # care where (and you don't need snappy-metadata to find it)
     touch copy-meta # instructs Makefile to copy snappy/ itself
     ${SNAPCRAFT} assemble
+flags: simple has-leftovers
 
 id: snapcraft/normal/assemble-meta
 plugin: shell
@@ -48,6 +50,7 @@
     if [ -e ./snap/bin/not-wrapped.wrapper ]; then
         exit 1
     fi
+flags: simple has-leftovers
 
 id: snapcraft/normal/local-source
 plugin: shell
@@ -60,6 +63,7 @@
     test "$(readlink parts/make-project/build)" = "$(pwd)"
     ${SNAPCRAFT} stage
     test -e stamp-install
+flags: simple has-leftovers
 
 id: snapcraft/normal/local-plugin
 plugin: shell
@@ -69,6 +73,7 @@
     cp -rT $PLAINBOX_PROVIDER_DATA/local-plugin .
     ${SNAPCRAFT} stage
     test -e stage/build-stamp
+flags: simple has-leftovers
 
 id: snapcraft/normal/simple-tar
 plugin: shell
@@ -89,6 +94,7 @@
     test -e stage/parent
     test -e stage/slash
     test "$(./stage/bin/test)" = "tarproject"
+flags: simple has-leftovers
 
 id: snapcraft/normal/pypi-config
 plugin: shell
@@ -103,6 +109,7 @@
     test -f ./snap/usr/bin/python3
     test -f ./snap/meta/hooks/config
     grep -q 'exec "usr/bin/config.py" $*' ./snap/meta/hooks/config
+flags: simple has-leftovers
 
 id: snapcraft/normal/simple-make
 plugin: shell
@@ -112,6 +119,7 @@
     cp -rT $PLAINBOX_PROVIDER_DATA/simple-make .
     ${SNAPCRAFT} stage
     test "$(./stage/bin/test)" = "Hello world"
+flags: simple has-leftovers
 
 id: snapcraft/normal/simple-make-filesets
 plugin: shell
@@ -130,6 +138,7 @@
     grep -q 'file1' ./snap/share/file1
     grep -q 'file2' ./snap/share/file2
     grep -q 'file1' ./snap/new/dir2/file1
+flags: simple has-leftovers
 
 id: snapcraft/normal/simple-make-clean
 plugin: shell
@@ -149,6 +158,7 @@
     # Clean a second time doesn't fail.
     # Regression test for https://bugs.launchpad.net/snapcraft/+bug/1497371
     ${SNAPCRAFT} clean
+flags: simple has-leftovers
 
 id: snapcraft/normal/simple-cmake
 plugin: shell
@@ -158,6 +168,7 @@
     cp -rT $PLAINBOX_PROVIDER_DATA/simple-cmake .
     ${SNAPCRAFT} stage
     test "$(./stage/bin/simple-cmake)" = "It's a CMake world"
+flags: simple has-leftovers
 
 id: snapcraft/normal/simple-copy
 plugin: shell
@@ -167,6 +178,7 @@
     cp -rT $PLAINBOX_PROVIDER_DATA/simple-copy .
     ${SNAPCRAFT} stage
     test "$(cat ./stage/dst)" = "I got copied"
+flags: simple has-leftovers
 
 id: snapcraft/normal/conflicts
 plugin: shell
@@ -177,6 +189,7 @@
     OUTPUT=$(${SNAPCRAFT} stage 2>&1)
     test $? = 1 || exit 1
     echo $OUTPUT | grep "Error: parts p1 and p2 have the following file paths in common which have different contents: bin/test" # squished output by bash
+flags: simple has-leftovers
 
 id: snapcraft/normal/dependencies
 plugin: shell
@@ -189,6 +202,7 @@
     test "$(${SNAPCRAFT} shell p3)" = 'p1
     p1
     p2'
+flags: simple has-leftovers
 
 id: snapcraft/normal/dependencies-circular
 plugin: shell
@@ -200,6 +214,7 @@
     OUTPUT=$(${SNAPCRAFT} pull 2>&1)
     test $? = 1 || exit 1
     echo $OUTPUT | grep -i "circular dependency"
+flags: simple has-leftovers
 
 id: snapcraft/normal/dependencies-fail
 plugin: shell
@@ -210,6 +225,7 @@
     sed -i '/after/d' snapcraft.yaml
     ${SNAPCRAFT} build
     test $? = 1 || exit 1
+flags: simple has-leftovers
 
 id: snapcraft/normal/bzr-head
 plugin: shell
@@ -225,6 +241,7 @@
     test "$(bzr revno -r -1 parts/bzr/src)" = "2"
     ${SNAPCRAFT} pull # test pull doesn't fail
     test "$(bzr revno -r -1 parts/bzr/src)" = "2"
+flags: simple has-leftovers
 
 id: snapcraft/normal/bzr-tag
 plugin: shell
@@ -241,6 +258,7 @@
     test "$(bzr revno -r -1 parts/bzr/src)" = "1"
     ${SNAPCRAFT} pull # test pull doesn't fail
     test "$(bzr revno -r -1 parts/bzr/src)" = "1"
+flags: simple has-leftovers
 
 id: snapcraft/normal/git-head
 plugin: shell
@@ -257,6 +275,7 @@
     test "$(git -C parts/git/src log -1 --oneline | cut -d' ' -f2)" = "2"
     ${SNAPCRAFT} pull
     test "$(git -C parts/git/src log -1 --oneline | cut -d' ' -f2)" = "2"
+flags: simple has-leftovers
 
 id: snapcraft/normal/git-tag
 plugin: shell
@@ -274,6 +293,7 @@
     test "$(git -C parts/git/src log -1 --oneline | cut -d' ' -f2)" = "1"
     ${SNAPCRAFT} pull
     test "$(git -C parts/git/src log -1 --oneline | cut -d' ' -f2)" = "1"
+flags: simple has-leftovers
 
 id: snapcraft/normal/git-branch
 plugin: shell
@@ -294,6 +314,7 @@
     test "$(git -C parts/git/src log -2 --oneline | cut -d' ' -f2 | tr '\n' ' ')" = "3 1 "
     ${SNAPCRAFT} pull
     test "$(git -C parts/git/src log -2 --oneline | cut -d' ' -f2 | tr '\n' ' ')" = "3 1 "
+flags: simple has-leftovers
 
 id: snapcraft/normal/hg-head
 plugin: shell
@@ -310,6 +331,7 @@
     test "$(hg log --cwd parts/mercurial/src --template "{desc}" -r -1)" = "2"
     ${SNAPCRAFT} pull
     test "$(hg log --cwd parts/mercurial/src --template "{desc}" -r -1)" = "2"
+flags: simple has-leftovers
 
 id: snapcraft/normal/hg-tag
 plugin: shell
@@ -327,6 +349,7 @@
     test "$(ls -1 parts/mercurial/src/ | wc -l )" = "1"
     ${SNAPCRAFT} pull
     test "$(ls -1 parts/mercurial/src/ | wc -l )" = "1"
+flags: simple has-leftovers
 
 id: snapcraft/normal/hg-branch
 plugin: shell
@@ -345,6 +368,7 @@
     test -e parts/mercurial/src/second
     ${SNAPCRAFT} pull
     test -e parts/mercurial/src/second
+flags: simple has-leftovers
 
 id: snapcraft/normal/pip-requirements
 plugin: shell
@@ -355,3 +379,16 @@
     ${SNAPCRAFT} pull
     test -f parts/python2/install/usr/lib/python2.7/argparse.py
     test -f parts/python3/install/usr/lib/python3.4/argparse.py
+flags: simple has-leftovers
+
+id: snapcraft/normal/build-tool-paths
+plugin: shell
+estimated_duration: 30
+command:
+    set -ex
+    cp -rT $PLAINBOX_PROVIDER_DATA/build-tools .
+    ${SNAPCRAFT} stage
+    ${SNAPCRAFT} shell pkg-config --cflags glib-2.0 | grep -q `pwd`
+    ${SNAPCRAFT} shell xml2-config --cflags | grep -q `pwd`
+    ${SNAPCRAFT} shell xslt-config --cflags | grep -q `pwd`
+flags: simple has-leftovers

=== modified file 'integration-tests/units/testplans.pxu'
--- integration-tests/units/testplans.pxu	2015-07-02 18:29:47 +0000
+++ integration-tests/units/testplans.pxu	2015-09-24 04:06:53 +0000
@@ -1,4 +1,9 @@
 unit: test plan
 id: normal
-name: Tests for Snapcraft 
+name: Tests for Snapcraft
 include: snapcraft/normal/.*
+
+unit: test plan
+id: examples
+name: Test for building Snapcraft examples
+include: snapcraft/examples/.*

=== modified file 'plugins/python2-project.yaml'
--- plugins/python2-project.yaml	2015-09-22 03:08:07 +0000
+++ plugins/python2-project.yaml	2015-09-24 04:06:53 +0000
@@ -1,8 +1,8 @@
-requires:
-  - python2
 options:
+  requirements:
+    required: false
   source:
-    required: true
+    required: false
   source-type:
   source-tag:
   source-branch:

=== removed file 'plugins/python2.yaml'
--- plugins/python2.yaml	2015-09-01 14:10:18 +0000
+++ plugins/python2.yaml	1970-01-01 00:00:00 +0000
@@ -1,3 +0,0 @@
-options:
-  requirements:
-    required: false

=== modified file 'plugins/python3-project.yaml'
--- plugins/python3-project.yaml	2015-09-22 03:08:07 +0000
+++ plugins/python3-project.yaml	2015-09-24 04:06:53 +0000
@@ -1,8 +1,8 @@
-requires:
-  - python3
 options:
+  requirements:
+    required: false
   source:
-    required: true
+    required: false
   source-type:
   source-tag:
   source-branch:

=== removed file 'plugins/python3.yaml'
--- plugins/python3.yaml	2015-09-09 21:35:18 +0000
+++ plugins/python3.yaml	1970-01-01 00:00:00 +0000
@@ -1,3 +0,0 @@
-options:
-  requirements:
-    required: false

=== modified file 'runtests.sh'
--- runtests.sh	2015-09-17 07:31:24 +0000
+++ runtests.sh	2015-09-24 04:06:53 +0000
@@ -20,39 +20,62 @@
 export PATH=$(pwd)/bin:$PATH
 export PYTHONPATH=$(pwd):$PYTHONPATH
 
-SRC_PATHS="bin snapcraft snapcraft/tests"
-
-# These three checks could easily be done with flake8 in one shot if
-# we had python3-flake8 provide flake8
-# Ignore 501 (line-too-long)
-pep8 $SRC_PATHS --ignore=E501
-
-pyflakes3 $SRC_PATHS
-
-# mccabe in 'warning' mode as we have high complexity
-mccabe_list=
-for unit in $(find snapcraft -type f -name '*.py')
-do
-  output=$(python3 -m mccabe --min 10 "$unit")
-  [ -n "$output" ] && mccabe_list="- $unit:\n  $output\n$mccabe_list"
-done
-
-if [ -n "$mccabe_list" ]; then
-  echo -e "\e[1;31mThe project has gotten complex\e[0m."
-  echo "Here's the list of units exceeding 10:"
-  echo -e "$mccabe_list"
-fi
-
-if which python3-coverage >/dev/null 2>&1; then
-    python3-coverage erase
-    python3-coverage run --branch --source snapcraft -m unittest
-    mv .coverage .coverage.unit
-else
-    python3 -m unittest
-fi
-
-if [ -z "$SNAPCRAFT_TESTS_SKIP_PLAINBOX" ]; then
-(
+parseargs(){
+    if [[ "$#" -eq 0 ]] || [[ "$1" == "all" ]]; then
+        export RUN_UNIT="true"
+        export RUN_PLAINBOX="true"
+        export PLAINBOX_TEST_PLANS="normal"
+    else
+        if [ "$1" == "unit" ] ; then
+            export RUN_UNIT="true"
+        elif [ "$1" == "plainbox" ] ; then
+            export RUN_PLAINBOX="true"
+            if [ "$#" -gt 1 ]; then
+                export PLAINBOX_TEST_PLANS="$2"
+            else
+                export PLAINBOX_TEST_PLANS="normal"
+            fi
+        else
+            echo "Not recognized option, should be one of all, unit or plainbox"
+            exit 1
+        fi
+    fi
+}
+
+run_unit_tests(){
+    SRC_PATHS="bin snapcraft snapcraft/tests"
+
+    # These three checks could easily be done with flake8 in one shot if
+    # we had python3-flake8 provide flake8
+    # Ignore 501 (line-too-long)
+    pep8 $SRC_PATHS --ignore=E501
+
+    pyflakes3 $SRC_PATHS
+
+    # mccabe in 'warning' mode as we have high complexity
+    mccabe_list=
+    for unit in $(find snapcraft -type f -name '*.py')
+    do
+        output=$(python3 -m mccabe --min 10 "$unit")
+        [ -n "$output" ] && mccabe_list="- $unit:\n  $output\n$mccabe_list"
+    done
+
+    if [ -n "$mccabe_list" ]; then
+        echo -e "\e[1;31mThe project has gotten complex\e[0m."
+        echo "Here's the list of units exceeding 10:"
+        echo -e "$mccabe_list"
+    fi
+
+    if which python3-coverage >/dev/null 2>&1; then
+        python3-coverage erase
+        python3-coverage run --branch --source snapcraft -m unittest
+        mv .coverage .coverage.unit
+    else
+        python3 -m unittest
+    fi
+}
+
+run_plainbox(){
     # well, well, what can we do
     if ! which plainbox >/dev/null; then
         cat <<EOF
@@ -73,8 +96,17 @@
 
     # Go to the plainbox provider of snapcraft tests
     cd integration-tests
-    ./runtests.sh
-)
+    ./runtests.sh $PLAINBOX_TEST_PLANS
+}
+
+parseargs "$@"
+
+if [ ! -z "$RUN_UNIT" ]; then
+    run_unit_tests
+fi
+
+if [ -z "$SNAPCRAFT_TESTS_SKIP_PLAINBOX" ] && [ ! -z "$RUN_PLAINBOX" ] ; then
+    run_plainbox
 fi
 
 if which python3-coverage >/dev/null 2>&1; then

=== modified file 'snapcraft/__init__.py'
--- snapcraft/__init__.py	2015-09-21 14:16:55 +0000
+++ snapcraft/__init__.py	2015-09-24 04:06:53 +0000
@@ -69,8 +69,17 @@
             cwd = self.builddir
         if True:
             print(' '.join(cmd))
+        self.makedirs(cwd)
         return snapcraft.common.run(cmd, cwd=cwd, **kwargs)
 
+    def run_output(self, cmd, cwd=None, **kwargs):
+        if cwd is None:
+            cwd = self.builddir
+        if True:
+            print(' '.join(cmd))
+        self.makedirs(cwd)
+        return snapcraft.common.run_output(cmd, cwd=cwd, **kwargs)
+
     def isurl(self, url):
         return snapcraft.common.isurl(url)
 
@@ -108,6 +117,13 @@
             ubuntu = snapcraft.repo.Ubuntu(self.ubuntudir, sources=self.PLUGIN_STAGE_SOURCES)
             ubuntu.get(self.PLUGIN_STAGE_PACKAGES + part_stage_packages)
             ubuntu.unpack(self.installdir)
+            self._fixup(self.installdir)
+
+    def _fixup(self, root):
+        if os.path.isfile(os.path.join(root, 'usr', 'bin', 'xml2-config')):
+            self.run(['sed', '-i', '-e', 's|prefix=/usr|prefix={}/usr|'.format(root), os.path.join(root, 'usr', 'bin', 'xml2-config')])
+        if os.path.isfile(os.path.join(root, 'usr', 'bin', 'xslt-config')):
+            self.run(['sed', '-i', '-e', 's|prefix=/usr|prefix={}/usr|'.format(root), os.path.join(root, 'usr', 'bin', 'xslt-config')])
 
 
 def _get_source_handler(source_type, source):

=== modified file 'snapcraft/cmds.py'
--- snapcraft/cmds.py	2015-09-21 19:20:43 +0000
+++ snapcraft/cmds.py	2015-09-24 04:06:53 +0000
@@ -267,6 +267,8 @@
             for f in common:
                 this = os.path.join(part.installdir, f)
                 other = os.path.join(parts_files[other_part_name]['installdir'], f)
+                if os.path.islink(this) and os.path.islink(other):
+                    continue
                 if not filecmp.cmp(this, other, shallow=False):
                     conflict_files.append(f)
 

=== modified file 'snapcraft/common.py'
--- snapcraft/common.py	2015-09-17 18:02:05 +0000
+++ snapcraft/common.py	2015-09-24 04:06:53 +0000
@@ -49,6 +49,17 @@
         return subprocess.call(['/bin/sh', f.name] + cmd, **kwargs) == 0
 
 
+def run_output(cmd, **kwargs):
+    assert isinstance(cmd, list), "run command must be a list"
+    # FIXME: This is gross to keep writing this, even when env is the same
+    with tempfile.NamedTemporaryFile(mode='w+') as f:
+        f.write(assemble_env())
+        f.write('\n')
+        f.write('exec $*')
+        f.flush()
+        return subprocess.check_output(['/bin/sh', f.name] + cmd, **kwargs).decode('utf8').strip()
+
+
 def fatal():
     sys.exit(1)
 

=== modified file 'snapcraft/plugin.py'
--- snapcraft/plugin.py	2015-09-22 03:08:07 +0000
+++ snapcraft/plugin.py	2015-09-24 04:06:53 +0000
@@ -39,15 +39,8 @@
 }
 
 
-def is_local_plugin(name):
-    return name.startswith("x-")
-
-
-def plugindir(name):
-    if is_local_plugin(name):
-        return os.path.abspath(os.path.join('parts', 'plugins'))
-    else:
-        return common.get_plugindir()
+def _local_plugindir():
+    return os.path.abspath(os.path.join('parts', 'plugins'))
 
 
 class PluginError(Exception):
@@ -87,10 +80,12 @@
             return
 
     def _load_config(self, name):
-        configPath = os.path.join(plugindir(name), name + ".yaml")
-        if not os.path.exists(configPath):
+        config_path = os.path.join(common.get_plugindir(), name + ".yaml")
+        if not os.path.exists(config_path):
+            config_path = os.path.join(_local_plugindir(), name + ".yaml")
+        if not os.path.exists(config_path):
             raise PluginError('Unknown plugin: {}'.format(name))
-        with open(configPath, 'r') as fp:
+        with open(config_path, 'r') as fp:
             self.config = yaml.load(fp) or {}
 
     def _make_options(self, name, properties):
@@ -121,14 +116,15 @@
         options = self._make_options(name, properties)
         module_name = name.replace('-', '_')
 
-        # Load code from local plugin dir if it is there
-        if is_local_plugin(name):
-            sys.path = [plugindir(name)] + sys.path
-        else:
-            module_name = 'snapcraft.plugins.' + module_name
+        try:
+            module = importlib.import_module('snapcraft.plugins.' + module_name)
+        except ImportError:
+            module = None
 
-        module = importlib.import_module(module_name)
-        if is_local_plugin(name):
+        if not module:
+            logger.info('Searching for local plugin for %s', name)
+            sys.path = [_local_plugindir()] + sys.path
+            module = importlib.import_module(module_name)
             sys.path.pop(0)
 
         for prop_name in dir(module):

=== modified file 'snapcraft/plugins/go_project.py'
--- snapcraft/plugins/go_project.py	2015-07-31 15:16:47 +0000
+++ snapcraft/plugins/go_project.py	2015-09-24 04:06:53 +0000
@@ -27,7 +27,9 @@
             self.fullname = self.options.source.split("://")[1]
 
     def pull(self):
-        return self.run(['go', 'get', '-t', self.fullname])
+        # use -d to only download (build will happen later)
+        # use -t to also get the test-deps
+        return self.run(['go', 'get', '-t', '-d', self.fullname])
 
     def build(self):
         if not self.run(['go', 'build', self.fullname]):

=== removed file 'snapcraft/plugins/python2.py'
--- snapcraft/plugins/python2.py	2015-09-17 15:14:49 +0000
+++ snapcraft/plugins/python2.py	1970-01-01 00:00:00 +0000
@@ -1,59 +0,0 @@
-# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
-#
-# Copyright (C) 2015 Canonical Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 3 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import os
-import snapcraft
-
-
-class Python2Plugin(snapcraft.BasePlugin):
-
-    _PLUGIN_STAGE_PACKAGES = [
-        'python-dev',
-        'python-setuptools',
-    ]
-
-    def __init__(self, name, options):
-        if options.requirements:
-            self.requirements = options.requirements
-            self._PLUGIN_STAGE_PACKAGES.extend(['python-pkg-resources', 'python-setuptools'])
-        else:
-            self.requirements = None
-        super().__init__(name, options)
-
-    def snap_fileset(self):
-        return [
-            '-usr/share',
-        ]
-
-    # note that we don't need to set PYTHONHOME here,
-    # python discovers this automatically from it installed
-    # location.  And PATH is automatically set by snapcraft.
-
-    def env(self, root):
-        return ["PYTHONPATH=%s" % os.path.join(
-            root, 'usr', 'lib', 'python2.7', 'dist-packages')]
-
-    def pull(self):
-        # A nice idea here would be to be asking setup tools
-        # to use the deb layout, but that doesn't work with
-        # prefix sadly
-
-        if self.requirements and not (self.run(
-                ['ln', '-s', os.path.join(self.installdir, 'usr', 'lib', 'python2.7', 'dist-packages'), os.path.join(self.installdir, 'usr', 'lib', 'python2.7', 'site-packages')]) and self.run(
-                ['python2', os.path.join(self.installdir, 'usr', 'bin', 'easy_install'), '--prefix', os.path.join(self.installdir, 'usr'), 'pip']) and self.run(
-                ['python2', os.path.join(self.installdir, 'usr', 'bin', 'pip2'), 'install', '--target', os.path.join(self.installdir, 'usr', 'lib', 'python2.7', 'site-packages'), '--requirement', os.path.join(os.getcwd(), self.requirements)])):
-            return False
-        return True

=== modified file 'snapcraft/plugins/python2_project.py'
--- snapcraft/plugins/python2_project.py	2015-09-21 22:22:36 +0000
+++ snapcraft/plugins/python2_project.py	2015-09-24 04:06:53 +0000
@@ -22,20 +22,76 @@
 
 class Python2ProjectPlugin(snapcraft.BasePlugin):
 
-    # note that we don't need to setup env(), python figures it out
-    # see python2.py for more details
+    _PLUGIN_STAGE_PACKAGES = [
+        'python-dev',
+        'python-pkg-resources',
+        'python-setuptools',
+    ]
+
+    def __init__(self, name, options):
+        super().__init__(name, options)
+        self.requirements = options.requirements
+        self.source = options.source
+
+    def env(self, root):
+        return ["PYTHONPATH=%s" % os.path.join(
+            root, 'usr', 'lib', self.python_version, 'dist-packages')]
 
     def pull(self):
-        return self.handle_source_options()
+        if self.source and not self.handle_source_options():
+            return False
+
+        return self._pip()
+
+    def _pip(self):
+        setup = 'setup.py'
+        if os.listdir(self.sourcedir):
+            setup = os.path.join(self.sourcedir, 'setup.py')
+
+        if self.requirements:
+            requirements = os.path.join(os.getcwd(), self.requirements)
+
+        if not os.path.exists(setup) and not self.requirements:
+            return True
+
+        easy_install = os.path.join(
+            self.installdir, 'usr', 'bin', 'easy_install')
+        prefix = os.path.join(self.installdir, 'usr')
+        site_packages_dir = os.path.join(
+            prefix, 'lib', self.python_version, 'site-packages')
+
+        if not os.path.exists(site_packages_dir):
+            os.symlink(
+                os.path.join(prefix, 'lib', self.python_version, 'dist-packages'),
+                site_packages_dir)
+
+        if not self.run(['python2', easy_install, '--prefix', prefix, 'pip']):
+            return False
+
+        pip2 = os.path.join(self.installdir, 'usr', 'bin', 'pip2')
+        pip_install = ['python2', pip2, 'install', '--target',
+                       site_packages_dir]
+
+        if self.requirements and not self.run(
+                pip_install + ['--requirement', requirements]):
+            return False
+
+        if os.path.exists(setup) and not self.run(pip_install + ['.', ]):
+            return False
+
+        return True
 
     def build(self):
         # If setuptools is used, it tries to create files in the
         # dist-packages dir and import from there, so it needs to exist
         # and be in the PYTHONPATH. It's harmless if setuptools isn't
         # used.
+
+        if not os.path.exists(os.path.join(self.builddir, 'setup.py')):
+            return True
+
         os.makedirs(self.dist_packages_dir, exist_ok=True)
         setuptemp = self.copy_setup()
-
         return self.run(
             ['python2', setuptemp.name, 'install', '--install-layout=deb',
              '--prefix={}/usr'.format(self.installdir)], cwd=self.builddir)
@@ -43,7 +99,11 @@
     @property
     def dist_packages_dir(self):
         return os.path.join(
-            self.installdir, 'usr', 'lib', 'python2.7', 'dist-packages')
+            self.installdir, 'usr', 'lib', self.python_version, 'dist-packages')
+
+    @property
+    def python_version(self):
+        return self.run_output(['pyversions', '-i'])
 
     # Takes the setup.py file and puts a couple little gems on the
     # front to make things work better.

=== removed file 'snapcraft/plugins/python3.py'
--- snapcraft/plugins/python3.py	2015-09-17 15:14:49 +0000
+++ snapcraft/plugins/python3.py	1970-01-01 00:00:00 +0000
@@ -1,61 +0,0 @@
-# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
-#
-# Copyright (C) 2015 Canonical Ltd
-#
-# This program is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License version 3 as
-# published by the Free Software Foundation.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-import os
-import snapcraft
-
-
-class Python3Plugin(snapcraft.BasePlugin):
-
-    _PLUGIN_STAGE_PACKAGES = [
-        'python3-dev',
-        'python3-setuptools',
-    ]
-
-    def __init__(self, name, options):
-        if options.requirements:
-            self.requirements = options.requirements
-            self._PLUGIN_STAGE_PACKAGES.extend(['python3-pkg-resources', 'python3-setuptools'])
-        else:
-            self.requirements = None
-        super().__init__(name, options)
-
-    def snap_fileset(self):
-        return [
-            '-usr/share',
-        ]
-
-    # note that we don't need to set PYTHONHOME here,
-    # python discovers this automatically from it installed
-    # location, see https://code.launchpad.net/~mvo/snapcraft/python3-project/+merge/264521/comments/664308
-    #
-    # PATH is automatically set by snapcraft
-
-    def env(self, root):
-        return ["PYTHONPATH=%s" % os.path.join(
-            root, 'usr', 'lib', 'python3', 'dist-packages')]
-
-    def pull(self):
-        # A nice idea here would be to be asking setup tools
-        # to use the deb layout, but that doesn't work with
-        # prefix sadly
-
-        if self.requirements and not (self.run(
-                ['ln', '-s', os.path.join(self.installdir, 'usr', 'lib', 'python3', 'dist-packages'), os.path.join(self.installdir, 'usr', 'lib', 'python3.4', 'site-packages')]) and self.run(
-                ['python3', os.path.join(self.installdir, 'usr', 'bin', 'easy_install3'), '--prefix', os.path.join(self.installdir, 'usr'), 'pip']) and self.run(
-                ['python3', os.path.join(self.installdir, 'usr', 'bin', 'pip3'), 'install', '--target', os.path.join(self.installdir, 'usr', 'lib', 'python3.4', 'site-packages'), '--requirement', os.path.join(os.getcwd(), self.requirements)])):
-            return False
-        return True

=== modified file 'snapcraft/plugins/python3_project.py'
--- snapcraft/plugins/python3_project.py	2015-09-21 22:22:36 +0000
+++ snapcraft/plugins/python3_project.py	2015-09-24 04:06:53 +0000
@@ -22,21 +22,74 @@
 
 class Python3ProjectPlugin(snapcraft.BasePlugin):
 
-    # note that we don't need to setup env(), python figures it out
-    # see python3.py for more details
+    _PLUGIN_STAGE_PACKAGES = [
+        'python3-dev',
+        'python3-pkg-resources',
+        'python3-setuptools',
+    ]
+
+    def __init__(self, name, options):
+        super().__init__(name, options)
+        self.requirements = options.requirements
+        self.source = options.source
 
     def env(self, root):
         return ["PYTHONPATH=%s" % os.path.join(
-            root, 'usr', 'lib', 'python3', 'dist-packages')]
+            root, 'usr', 'lib', self.python_version, 'dist-packages')]
 
     def pull(self):
-        return self.handle_source_options()
+        if self.source and not self.handle_source_options():
+            return False
+
+        return self._pip()
+
+    def _pip(self):
+        setup = 'setup.py'
+        if os.listdir(self.sourcedir):
+            setup = os.path.join(self.sourcedir, 'setup.py')
+
+        if self.requirements:
+            requirements = os.path.join(os.getcwd(), self.requirements)
+
+        if not os.path.exists(setup) and not self.requirements:
+            return True
+
+        easy_install = os.path.join(
+            self.installdir, 'usr', 'bin', 'easy_install3')
+        prefix = os.path.join(self.installdir, 'usr')
+        site_packages_dir = os.path.join(
+            prefix, 'lib', self.python_version, 'site-packages')
+
+        if not os.path.exists(site_packages_dir):
+            os.symlink(
+                os.path.join(prefix, 'lib', 'python3', 'dist-packages'),
+                site_packages_dir)
+
+        if not self.run(['python3', easy_install, '--prefix', prefix, 'pip']):
+            return False
+
+        pip3 = os.path.join(self.installdir, 'usr', 'bin', 'pip3')
+        pip_install = ['python3', pip3, 'install', '--target',
+                       site_packages_dir]
+
+        if self.requirements and not self.run(
+                pip_install + ['--requirement', requirements]):
+            return False
+
+        if os.path.exists(setup) and not self.run(pip_install + ['.', ]):
+            return False
+
+        return True
 
     def build(self):
         # If setuptools is used, it tries to create files in the
         # dist-packages dir and import from there, so it needs to exist
         # and be in the PYTHONPATH. It's harmless if setuptools isn't
         # used.
+
+        if not os.path.exists(os.path.join(self.builddir, 'setup.py')):
+            return True
+
         os.makedirs(self.dist_packages_dir, exist_ok=True)
         setuptemp = self.copy_setup()
         return self.run(
@@ -46,7 +99,11 @@
     @property
     def dist_packages_dir(self):
         return os.path.join(
-            self.installdir, 'usr', 'lib', 'python3', 'dist-packages')
+            self.installdir, 'usr', 'lib', self.python_version, 'dist-packages')
+
+    @property
+    def python_version(self):
+        return self.run_output(['py3versions', '-i'])
 
     # Takes the setup.py file and puts a couple little gems on the
     # front to make things work better.

=== modified file 'snapcraft/repo.py'
--- snapcraft/repo.py	2015-09-21 15:21:15 +0000
+++ snapcraft/repo.py	2015-09-24 04:06:53 +0000
@@ -14,6 +14,7 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+import apt
 import glob
 import itertools
 import os
@@ -22,18 +23,19 @@
 import urllib
 import urllib.request
 
-import apt
 from xml.etree import ElementTree
 
-_DEFAULT_SOURCES = '''deb http://${mirror}archive.ubuntu.com/ubuntu/ vivid main restricted
-deb http://${mirror}archive.ubuntu.com/ubuntu/ vivid-updates main restricted
-deb http://${mirror}archive.ubuntu.com/ubuntu/ vivid universe
-deb http://${mirror}archive.ubuntu.com/ubuntu/ vivid-updates universe
-deb http://${mirror}archive.ubuntu.com/ubuntu/ vivid multiverse
-deb http://${mirror}archive.ubuntu.com/ubuntu/ vivid-updates multiverse
-deb http://security.ubuntu.com/ubuntu vivid-security main restricted
-deb http://security.ubuntu.com/ubuntu vivid-security universe
-deb http://security.ubuntu.com/ubuntu vivid-security multiverse
+import snapcraft.common
+
+_DEFAULT_SOURCES = '''deb http://${prefix}.ubuntu.com/${suffix}/ ${release} main restricted
+deb http://${prefix}.ubuntu.com/${suffix}/ ${release}-updates main restricted
+deb http://${prefix}.ubuntu.com/${suffix}/ ${release} universe
+deb http://${prefix}.ubuntu.com/${suffix}/ ${release}-updates universe
+deb http://${prefix}.ubuntu.com/${suffix}/ ${release} multiverse
+deb http://${prefix}.ubuntu.com/${suffix}/ ${release}-updates multiverse
+deb http://${security}.ubuntu.com/${suffix} ${release}-security main restricted
+deb http://${security}.ubuntu.com/${suffix} ${release}-security universe
+deb http://${security}.ubuntu.com/${suffix} ${release}-security multiverse
 '''
 _GEOIP_SERVER = "http://geoip.ubuntu.com/lookup"
 
@@ -61,11 +63,16 @@
 class Ubuntu:
 
     def __init__(self, rootdir, recommends=False, sources=_DEFAULT_SOURCES):
-        sources = sources or _DEFAULT_SOURCES
         self.downloaddir = os.path.join(rootdir, 'download')
         self.rootdir = rootdir
-        self.apt_cache = _setup_apt_cache(rootdir, sources)
         self.recommends = recommends
+        sources = sources or _DEFAULT_SOURCES
+        local = False
+        if 'SNAPCRAFT_LOCAL_SOURCES' in os.environ:
+            print('using local sources')
+            sources = _get_local_sources_list()
+            local = True
+        self.apt_cache = _setup_apt_cache(rootdir, sources, local)
 
     def get(self, package_names):
         os.makedirs(self.downloaddir, exist_ok=True)
@@ -78,18 +85,35 @@
             except KeyError:
                 raise PackageNotFoundError(name)
 
+        skipped_essential = []
+        skipped_blacklisted = []
+
+        # unmark some base packages here
+        # note that this will break the consistency check inside apt_cache
+        # (self.apt_cache.broken_count will be > 0)
+        # but that is ok as it was consistent before we excluded
+        # these base package
         for pkg in self.apt_cache:
             # those should be already on each system, it also prevents
             # diving into downloading libc6
             if (pkg.candidate.priority in 'essential' and
                pkg.name not in package_names):
-                print('Skipping priority essential/imporant %s' % pkg.name)
+                skipped_essential.append(pkg.name)
+                pkg.mark_keep()
                 continue
             if (pkg.name in manifest_dep_names and pkg.name not in package_names):
-                print('Skipping blacklisted from manifest package %s' % pkg.name)
+                skipped_blacklisted.append(pkg.name)
+                pkg.mark_keep()
                 continue
-            if pkg.marked_install:
-                pkg.candidate.fetch_binary(destdir=self.downloaddir)
+
+        if skipped_essential:
+            print('Skipping priority essential packages:', skipped_essential)
+        if skipped_blacklisted:
+            print('Skipping blacklisted from manifest packages:', skipped_blacklisted)
+
+        # download the remaining ones with proper progress
+        apt.apt_pkg.config.set("Dir::Cache::Archives", self.downloaddir)
+        self.apt_cache.fetch_archives()
 
     def unpack(self, rootdir):
         pkgs_abs_path = glob.glob(os.path.join(self.downloaddir, '*.deb'))
@@ -114,7 +138,19 @@
         return manifest_dep_names
 
 
-def get_geoip_country_code_prefix():
+def _get_local_sources_list():
+    sources_list = glob.glob('/etc/apt/sources.list.d/*.list')
+    sources_list.append('/etc/apt/sources.list')
+
+    sources = ''
+    for source in sources_list:
+        with open(source) as f:
+            sources += f.read()
+
+    return sources
+
+
+def _get_geoip_country_code_prefix():
     try:
         with urllib.request.urlopen(_GEOIP_SERVER) as f:
             xml_data = f.read()
@@ -122,20 +158,41 @@
         cc = et.find("CountryCode")
         if cc is None:
             return ""
-        return cc.text.lower() + "."
+        return cc.text.lower()
     except (ElementTree.ParseError, urllib.error.URLError):
         pass
-    return ""
-
-
-def _setup_apt_cache(rootdir, sources):
+    return ''
+
+
+def _format_sources_list(sources, arch, release='vivid'):
+    if arch in ('amd64', 'i386'):
+        geoip_prefix = _get_geoip_country_code_prefix()
+        prefix = geoip_prefix + '.archive' if geoip_prefix else 'archive'
+        suffix = 'ubuntu'
+        security = 'security'
+    else:
+        prefix = 'ports'
+        suffix = 'ubuntu-ports'
+        security = 'ports'
+
+    return string.Template(sources).substitute({
+        'prefix': prefix,
+        'release': release,
+        'suffix': suffix,
+        'security': security,
+    })
+
+
+def _setup_apt_cache(rootdir, sources, local=False):
     os.makedirs(os.path.join(rootdir, 'etc', 'apt'), exist_ok=True)
     srcfile = os.path.join(rootdir, 'etc', 'apt', 'sources.list')
+
+    if not local:
+        sources = _format_sources_list(sources, snapcraft.common.get_arch())
+
     with open(srcfile, 'w') as f:
-        mirror_prefix = get_geoip_country_code_prefix()
-        sources_list = string.Template(sources).substitute(
-            {"mirror": mirror_prefix})
-        f.write(sources_list)
+        f.write(sources)
+
     progress = apt.progress.text.AcquireProgress()
     apt_cache = apt.Cache(rootdir=rootdir, memonly=True)
     apt_cache.update(fetch_progress=progress, sources_list=srcfile)

=== modified file 'snapcraft/tests/test_plugin.py'
--- snapcraft/tests/test_plugin.py	2015-09-22 03:08:07 +0000
+++ snapcraft/tests/test_plugin.py	2015-09-24 04:06:53 +0000
@@ -16,7 +16,6 @@
 
 import logging
 import os
-import sys
 import tempfile
 
 import fixtures
@@ -210,19 +209,6 @@
 
         self.assertEqual('test stage mock-part\n', fake_logger.output)
 
-    def test_local_plugins(self):
-        """Ensure local plugins are loaded from parts/plugins"""
-        def mock_import_modules(module_name):
-            # called with the name only and sys.path set
-            self.assertEqual(module_name, "x_mock")
-            self.assertTrue(sys.path[0].endswith("parts/plugins"))
-            return mock_plugin
-        with patch("importlib.import_module", side_effect=mock_import_modules):
-            plugin.PluginHandler(
-                "x-mock", "mock-part", {}, load_config=False, load_code=True)
-        # sys.path is cleaned afterwards
-        self.assertFalse(sys.path[0].endswith("parts/plugins"))
-
     def test_non_local_plugins(self):
         """Ensure regular plugins are loaded from snapcraft only"""
         def mock_import_modules(module_name):

=== modified file 'snapcraft/tests/test_repo.py'
--- snapcraft/tests/test_repo.py	2015-09-21 15:14:29 +0000
+++ snapcraft/tests/test_repo.py	2015-09-24 04:06:53 +0000
@@ -16,6 +16,7 @@
 
 import os
 import tempfile
+import unittest.mock
 
 from snapcraft import repo
 from snapcraft import tests
@@ -23,6 +24,43 @@
 
 class UbuntuTestCase(tests.TestCase):
 
+    @unittest.mock.patch('snapcraft.repo._get_geoip_country_code_prefix')
+    def test_sources_amd64_vivid(self, mock_cc):
+        mock_cc.return_value = 'ar'
+
+        sources_list = repo._format_sources_list(
+            repo._DEFAULT_SOURCES, 'amd64', 'vivid')
+
+        expected_sources_list = '''deb http://ar.archive.ubuntu.com/ubuntu/ vivid main restricted
+deb http://ar.archive.ubuntu.com/ubuntu/ vivid-updates main restricted
+deb http://ar.archive.ubuntu.com/ubuntu/ vivid universe
+deb http://ar.archive.ubuntu.com/ubuntu/ vivid-updates universe
+deb http://ar.archive.ubuntu.com/ubuntu/ vivid multiverse
+deb http://ar.archive.ubuntu.com/ubuntu/ vivid-updates multiverse
+deb http://security.ubuntu.com/ubuntu vivid-security main restricted
+deb http://security.ubuntu.com/ubuntu vivid-security universe
+deb http://security.ubuntu.com/ubuntu vivid-security multiverse
+'''
+        self.assertEqual(sources_list, expected_sources_list)
+
+    @unittest.mock.patch('snapcraft.repo._get_geoip_country_code_prefix')
+    def test_sources_armhf_trusty(self, mock_cc):
+        sources_list = repo._format_sources_list(
+            repo._DEFAULT_SOURCES, 'armhf', 'trusty')
+
+        expected_sources_list = '''deb http://ports.ubuntu.com/ubuntu-ports/ trusty main restricted
+deb http://ports.ubuntu.com/ubuntu-ports/ trusty-updates main restricted
+deb http://ports.ubuntu.com/ubuntu-ports/ trusty universe
+deb http://ports.ubuntu.com/ubuntu-ports/ trusty-updates universe
+deb http://ports.ubuntu.com/ubuntu-ports/ trusty multiverse
+deb http://ports.ubuntu.com/ubuntu-ports/ trusty-updates multiverse
+deb http://ports.ubuntu.com/ubuntu-ports trusty-security main restricted
+deb http://ports.ubuntu.com/ubuntu-ports trusty-security universe
+deb http://ports.ubuntu.com/ubuntu-ports trusty-security multiverse
+'''
+        self.assertEqual(sources_list, expected_sources_list)
+        self.assertFalse(mock_cc.called)
+
     def test_fix_symlinks(self):
         tempdirObj = tempfile.TemporaryDirectory()
         self.addCleanup(tempdirObj.cleanup)

=== modified file 'snapcraft/yaml.py'
--- snapcraft/yaml.py	2015-09-18 18:04:48 +0000
+++ snapcraft/yaml.py	2015-09-24 04:06:53 +0000
@@ -166,14 +166,46 @@
 
     def runtime_env(self, root):
         env = []
-        env.append('PATH="{0}/bin:{0}/usr/bin:$PATH"'.format(root))
-        env.append('LD_LIBRARY_PATH="{0}/lib:{0}/usr/lib:{0}/lib/{1}:{0}/usr/lib/{1}:$LD_LIBRARY_PATH"'.format(root, snapcraft.common.get_arch_triplet()))
+        env.append('PATH="' + ':'.join([
+            '{0}/bin',
+            '{0}/usr/bin',
+            '$PATH'
+        ]).format(root) + '"')
+        env.append('LD_LIBRARY_PATH="' + ':'.join([
+            '{0}/lib',
+            '{0}/usr/lib',
+            '{0}/lib/{1}',
+            '{0}/usr/lib/{1}',
+            '$LD_LIBRARY_PATH'
+        ]).format(root, snapcraft.common.get_arch_triplet()) + '"')
         return env
 
     def build_env(self, root):
         env = []
-        env.append('CFLAGS="-I{0}/include -I{0}/usr/include -I{0}/include/{1} -I{0}/usr/include/{1} $CFLAGS"'.format(root, snapcraft.common.get_arch_triplet()))
-        env.append('LDFLAGS="-L{0}/lib -L{0}/usr/lib -L{0}/lib/{1} -L{0}/usr/lib/{1} $LDFLAGS"'.format(root, snapcraft.common.get_arch_triplet()))
+        env.append('CFLAGS="' + ' '.join([
+            '-I{0}/include',
+            '-I{0}/usr/include',
+            '-I{0}/include/{1}',
+            '-I{0}/usr/include/{1}',
+            '$CFLAGS'
+        ]).format(root, snapcraft.common.get_arch_triplet()) + '"')
+        env.append('LDFLAGS="' + ' '.join([
+            '-L{0}/lib',
+            '-L{0}/usr/lib',
+            '-L{0}/lib/{1}',
+            '-L{0}/usr/lib/{1}',
+            '$LDFLAGS'
+        ]).format(root, snapcraft.common.get_arch_triplet()) + '"')
+        env.append('PKG_CONFIG_SYSROOT_DIR={0}'.format(root))
+        env.append('PKG_CONFIG_PATH=' + ':'.join([
+            '{0}/usr/lib/pkgconfig',
+            '{0}/usr/lib/{1}/pkgconfig',
+            '{0}/usr/share/pkgconfig',
+            '{0}/usr/local/lib/pkgconfig',
+            '{0}/usr/local/lib/{1}/pkgconfig',
+            '{0}/usr/local/share/pkgconfig',
+            '$PKG_CONFIG_PATH'
+        ]).format(root, snapcraft.common.get_arch_triplet()))
         return env
 
     def build_env_for_part(self, part):