Merge ~mpontillo/maas:release-package-distros-and-signature into maas:master

[Mike Pontillo]

Enhancements to package build scripts

 * Allow seleciton of distribution when running release-package script.
 * Use BASH_SOURCE to determine the runtime directory.
 * Update maintiner e-mail address before signing.
 * Update .gitignore for debian/ tree integration.
 * Drive-by fix to use proper argument to select commit.
 * Use exit trap to clean up changelog.
 * Add a script to set up pbuilder-dist for each supported distribution.
 * Add a script to create .debs for each platform, and a Debian repository
   in a format that can be served via HTTP.

80397cf... by Mike Pontillo on 2017-07-17

Fix usage.

8de8b37... by Mike Pontillo on 2017-07-17

Use exit trap to clean up changelog.

799900f... by Mike Pontillo on 2017-07-17

Add a script to set up pbuilder-dist.

095886c... by Mike Pontillo on 2017-07-17

Check for a few more development tools.

1be606e... by Mike Pontillo on 2017-07-17

Add a script to create an HTTP-accessible Debian repository.

[Andres Rodriguez (andreserl)]

lgtm! comment inline though!

review: Approve

edac8ae... by Mike Pontillo on 2017-07-19

Add --keep-changelog option to release-build script.

[Mike Pontillo (mpontillo)]

Thanks for the review.

diff --git a/.gitignore b/.gitignore
index 54fa9ea..8b7f183 100644
--- a/.gitignore
+++ b/.gitignore
@@ -15,6 +15,7 @@
 /.idea/scopes
 /.idea/workspace.xml
 /.installed.cfg
+/.pybuild/
 /.noseids
 /.run
 /.run-e2e
@@ -24,6 +25,8 @@
 /coverage
 /coverage.xml
 /db
+/debian/build/
+/debian/debhelper-build-stamp
 /develop-eggs
 /dist
 /docs/api.rst
diff --git a/maas_2.3.0~alpha1-6157-g52d1d0e.orig.tar.gz b/maas_2.3.0~alpha1-6157-g52d1d0e.orig.tar.gz
new file mode 100644
index 0000000..96d335b
Binary files /dev/null and b/maas_2.3.0~alpha1-6157-g52d1d0e.orig.tar.gz differ
diff --git a/utilities/create-deb-repo b/utilities/create-deb-repo
new file mode 100755
index 0000000..cea33c0
--- /dev/null
+++ b/utilities/create-deb-repo
@@ -0,0 +1,111 @@
+#!/bin/bash -e
+
+# Uses the output of the 'release-build' script to create Debian packages.
+# Also signs the packages, scans them, and prepares an archive that can be
+# hosted with HTTP.
+
+ROOTDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")"/.. && pwd)"
+cd "$ROOTDIR"
+BUILDDIR=$ROOTDIR/build_pkg
+if [ ! -d $BUILDDIR ]; then
+    echo "[!] You must run the 'release-build' script first."
+fi
+PBUILDFOLDER=${PBUILDFOLDER:-$HOME/pbuilder}
+
+usage () {
+    DISTRO_ARGS=$(distro-info --supported | sed 's/^/[--/g' | sed 's/$/]/' | tr '\n' ' ')
+    echo "Usage:"
+    echo "    $0 $DISTRO_ARGS"
+    exit 1
+}
+
+DEFAULT_DISTROS="$(ubuntu-distro-info --supported)"
+PACKAGE_DISTROS=""
+
+for arg in "$@"; do
+    if [ "$arg" = "-h" -o "$arg" = "--help" ]; then
+        usage
+    fi
+    if echo "$arg" | grep -q "^--"; then
+        distro="$(echo "$arg" | sed 's/^--//')"
+        if echo $DEFAULT_DISTROS | grep -q $distro; then
+            PACKAGE_DISTROS="$PACKAGE_DISTROS $distro"
+        fi
+        shift
+    fi
+done
+
+if [ "$PACKAGE_DISTROS" = "" ]; then
+    PACKAGE_DISTROS="$DEFAULT_DISTROS"
+fi
+
+scan_packages () {
+    # Create a repository that can be hosted with HTTP. Based loosely on:
+    #     https://help.ubuntu.com/community/CreateAuthenticatedRepository
+    cd $PBUILDFOLDER/$1_result
+    echo ""
+    echo "MAAS versions in distribution '$1':"
+    ls -1 *.deb | cut -f2 -d_ | sort -u | sed 's/^/    /g'
+    echo ""
+    echo ""
+    dpkg-sig --sign builder *.deb
+    mkdir -p ../archive/dists/$1/main/binary-amd64
+    cp -fl *.deb ../archive/dists/$1/main/binary-amd64
+    cd ../archive
+    dpkg-scanpackages -m dists/$1/main/binary-amd64 | tee dists/$1/main/binary-amd64/Packages
+    cat dists/$1/main/binary-amd64/Packages | gzip --fast > dists/$1/main/binary-amd64/Packages.gz
+    apt-ftparchive release dists/$1 | tee dists/$1/Release
+    rm -f dists/$1/InRelease
+    gpg --digest-algo sha512 --clearsign -o dists/$1/InRelease dists/$1/Release
+    rm -f dists/$1/Release.gpg
+    gpg --digest-algo sha512 -abs -o dists/$1/Release.gpg dists/$1/Release
+}
+
+distro=xenial
+if echo "$PACKAGE_DISTROS" | grep -q "$distro"; then
+    version=16.04
+    cd $BUILDDIR
+    pbuilder-dist $distro build $(ls -1rt *.dsc | grep "~$version" | tail -1)
+    scan_packages $distro
+fi
+
+distro=yakkety
+if echo "$PACKAGE_DISTROS" | grep -q "$distro"; then
+    version=16.10
+    cd $BUILDDIR
+    pbuilder-dist $distro build $(ls -1rt *.dsc | grep "~$version" | tail -1)
+    scan_packages $distro
+fi
+
+distro=zesty
+if echo "$PACKAGE_DISTROS" | grep -q "$distro"; then
+    version=17.04
+    cd $BUILDDIR
+    pbuilder-dist $distro build $(ls -1rt *.dsc | grep "~$version" | tail -1)
+    scan_packages $distro
+fi
+
+distro=artful
+if echo "$PACKAGE_DISTROS" | grep -q "$distro"; then
+    version=17.10
+    cd $BUILDDIR
+    pbuilder-dist $distro build $(ls -1rt *.dsc | grep "~$version" | tail -1)
+    scan_packages $distro
+fi
+
+cd $PBUILDFOLDER/archive
+echo ""
+echo "Public signing key (use with 'apt-key add'):"
+gpg --armor --export "$DEBEMAIL" | tee pubkey
+
+echo ""
+echo "You should be able to use the result as a repository by running:"
+echo "    cd $PBUILDFOLDER/archive && python3 -m http.server"
+echo ""
+echo "Then adding [one of] the following to the target machine's apt sources:"
+for ip in $(ip r | grep -o 'src.*' | awk '{ print $2}'); do
+    echo "    deb http://$ip:8000/ <distro> main"
+done
+echo ""
+echo "... and adding the key as follows:"
+echo "    curl http://<repository-ip>:8000/pubkey | apt-key add -"
diff --git a/utilities/pbuilder-setup b/utilities/pbuilder-setup
new file mode 100755
index 0000000..f34c35d
--- /dev/null
+++ b/utilities/pbuilder-setup
@@ -0,0 +1,48 @@
+#!/bin/bash -e
+
+errors=0
+if [ -z "$DEBEMAIL" ]; then
+    echo '[!] Set your $DEBEMAIL to an e-mail address matching your GPG key.' 1>&2
+    let errors=$errors+1
+fi
+
+if [ -z "$DEBNAME" ]; then
+    echo '[!] Set your $DEBNAME to your full name.' 1>&2
+    let errors=$errors+1
+fi
+
+if [ $errors -gt 0 ]; then
+    exit 1
+fi
+
+MISSING=""
+if [ ! -x /usr/bin/pbuilder-dist ]; then
+    MISSING="$MISSING ubuntu-dev-tools"
+fi
+
+if [ ! -x /usr/bin/dpkg-scanpackages ]; then
+    MISSING="$MISSING dpkg-dev"
+fi
+
+if [ ! -x /usr/bin/dpkg-sig ]; then
+    MISSING="$MISSING dpkg-sig"
+fi
+
+if [ "$MISSING" != "" ]; then
+    sudo apt-get install -yu $MISSING
+fi
+
+
+# MAAS doesn't support building on Trusty, so don't bother with it.
+DISTRIBUTIONS="$(distro-info --supported | grep -v trusty)"
+
+PBUILDFOLDER=${PBUILDFOLDER:-$HOME/pbuilder}
+
+for distro in $DISTRIBUTIONS; do
+    if [ -f $PBUILDFOLDER/$distro-base.tgz ]; then
+        pbuilder-dist $distro update
+    else
+        pbuilder-dist $distro create
+    fi
+done
+
diff --git a/utilities/release-build b/utilities/release-build
index 007627d..36a21f9 100755
--- a/utilities/release-build
+++ b/utilities/release-build
@@ -1,29 +1,63 @@
-#!/bin/sh -e
+#!/bin/bash -e
 
-ROOTDIR=$(pwd)
+ROOTDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")"/.. && pwd)"
+cd "$ROOTDIR"
 BUILDDIR=$ROOTDIR/build_pkg
 if [ ! -d $BUILDDIR ]; then
     mkdir $BUILDDIR
 fi
 
-error() {
-	echo "ERROR: $@" >&2
-	exit 1
-}
-
 usage () {
+    DISTRO_ARGS=$(distro-info --supported | sed 's/^/[--/g' | sed 's/$/]/' | tr '\n' ' ')
     echo "Usage:"
-    echo "    $0 [commit (defaults to HEAD)]"
+    echo "    $0 $DISTRO_ARGS \\"
+    echo "       [--keep-changelog] [commit (defaults to HEAD)]"
     exit 1
 }
 
+DEFAULT_DISTROS="$(ubuntu-distro-info --supported | grep -v trusty)"
+PACKAGE_DISTROS=""
+KEEP_CHANGELOG=0
+
+for arg in "$@"; do
+    if [ "$arg" = "-h" -o "$arg" = "--help" ]; then
+        usage
+    fi
+    if echo "$arg" | grep -q "^--"; then
+        option="$(echo "$arg" | sed 's/^--//')"
+        if [ "$option" = "keep-changelog" ]; then
+            KEEP_CHANGELOG=1
+        fi
+        if echo $DEFAULT_DISTROS | grep -q $option; then
+            PACKAGE_DISTROS="$PACKAGE_DISTROS $option"
+        fi
+        shift
+    fi
+done
+
+if [ "$PACKAGE_DISTROS" = "" ]; then
+    PACKAGE_DISTROS="$DEFAULT_DISTROS"
+fi
+
 if [ -z "$1" ]; then
     echo "WARNING: No commit specified, using HEAD instead"
     COMMIT="HEAD"
 else
-    COMMIT="$2"
+    COMMIT="$1"
 fi
 
+function cleanup {
+    # Revert any unstaged temporary changes to the changelog.
+    cd "$ROOTDIR"
+    if [ $KEEP_CHANGELOG -eq 0 ]; then
+        git checkout -- debian/changelog
+    fi
+}
+trap cleanup EXIT
+
+# Ensure the maintainer e-mail address is set, so that signing works properly.
+dch -a "" --release-heuristic log --nomultimaint
+
 PKG=$(head -n1 debian/changelog | awk '{print $1}')
 MAJOR_VER=$(head -n 1 debian/changelog | sed 's/^.*(//' | sed 's/).*//' | sed 's/-.*//')
 REV_COUNT=$(git rev-list --count $COMMIT)
@@ -49,14 +83,27 @@ cd $BUILDDIR
 tar zxvf ${PKG}_${FULL_VER}.orig.tar.gz
 cd "${PKG}-${FULL_VER}.orig/"
 cp -r $ROOTDIR/debian .
-sed -i "s/) UNRELEASED;/~16.04.1) xenial;/i" debian/changelog
-debuild -S -sa
-sed -i "s/~16.04.1) xenial;/~16.10.1) yakkety;/" debian/changelog
-debuild -S
-sed -i "s/~16.10.1) yakkety;/~17.04.1) zesty;/" debian/changelog
-debuild -S
-sed -i "s/~17.04.1) zesty;/~17.10.1) artful;/" debian/changelog
-debuild -S
-
-cd $ROOTDIR
-sed -i "s/${FULL_VER}-0ubuntu1/${MAJOR_VER}-0ubuntu1/i" debian/changelog
+
+if echo "$PACKAGE_DISTROS" | grep -q "xenial"; then
+    cp $ROOTDIR/debian/changelog debian/changelog
+    sed -i "s/) UNRELEASED;/~16.04.1) xenial;/i" debian/changelog
+    debuild -S -sa
+fi
+
+if echo "$PACKAGE_DISTROS" | grep -q "yakkety"; then
+    cp $ROOTDIR/debian/changelog debian/changelog
+    sed -i "s/) UNRELEASED;/~16.10.1) yakkety;/" debian/changelog
+    debuild -S
+fi
+
+if echo "$PACKAGE_DISTROS" | grep -q "zesty"; then
+    cp $ROOTDIR/debian/changelog debian/changelog
+    sed -i "s/) UNRELEASED;/~17.04.1) zesty;/" debian/changelog
+    debuild -S
+fi
+
+if echo "$PACKAGE_DISTROS" | grep -q "artful"; then
+    cp $ROOTDIR/debian/changelog debian/changelog
+    sed -i "s/) UNRELEASED;/~17.10.1) artful;/" debian/changelog
+    debuild -S
+fi