Merge ~smoser/curtin:cleanup/curtainer-patch-version into curtin:master

[Scott Moser]

curtainer: patch source version from --source.

After adding 'CURTIN_EXE_VERSION' and 'CURTIN_VERSION' in output
we now see nice output for runs from trunk. But when running from
a package via curtainer, we see:
  CURTIN_VMTEST_CURTIN_EXE=.. curtin-vmtest-proposed-a-29 curtin
  CURTIN_VMTEST_CURTIN_EXE_VERSION=18.1-17-gae48e86f-0ubuntu1~17.10.1
  CURTIN_VMTEST_CURTIN_VERSION=18.1

That is confusing as it appears our vmtest version is different
from our exe version.

That is because the patching of the version that we do in debian/rules
only applies to the built binaries, the source still has the un-modified
@@PACKAGED_VERSION@@ in tact.

The change here is patch the source code output by --source
by default, so that running 'curtin version' from that source will
provide the correct output. Jenkins is the only known consumer of this
ouput so it was easier to change the default behavior than have the caller
become aware. To disable this behavior pass '--no-patch-version'

Note that curtin has verified that it downloaded the
version of the source that it is patching in, so this is
known-to-be-correct.

Note, we will have to have the jenkins job start passing '--patch-version' to curtainer.

and we will also probably want to first check if curtainer *has* that option.

./tools/curtainer --help | grep -q -- --patch-version &&
   pverarg="--patch-version" || pverarg=""

./tools/curtainer ubuntu-daily:xenial $pverarg ....

b91e519... by Scott Moser on 2018-05-22

better identify where that .dist file came from

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:6f7f6eaed478f2b623ec68d94e499f279f4e0540
https://jenkins.ubuntu.com/server/job/curtin-ci/949/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/949/console
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/949/console
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/949/console
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/949/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/949/rebuild

review: Needs Fixing (continuous-integration)

[Ryan Harper (raharper)]

review: Approve

[Scott Moser (smoser)]

since our only consumer to my knowledge of '--source' is jenkins...
i thjink its probably sane to just switch the default behavior
and add a '--no-patch-version' option for future consumer.

then we dont need any changes to jenkins jobs.

9ca81ae... by Scott Moser on 2018-05-22

flip the logic.

patch the source by default.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:1af5e654857c70bc762ced0e968b605b077cb99e
https://jenkins.ubuntu.com/server/job/curtin-ci/951/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/951/console
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/951/console
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/951/console
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/951/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/951/rebuild

review: Needs Fixing (continuous-integration)

[Ryan Harper (raharper)]

Yes, that's a great idea.

[Ryan Harper (raharper)]

That works even when we want to run by hand; generally we always want to do that unless we have a specific reason to try cross versions.

[Ryan Harper (raharper)]

This looks fine, the commit description is now incorrect as you inverted the logic.

review: Approve

[Scott Moser (smoser)]

commit description was updated. i'll hit rebuild and with c-i vote, land.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:
https://jenkins.ubuntu.com/server/job/curtin-ci/954/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/954/console
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/954/console
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/954/console
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/954/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/954/rebuild

review: Needs Fixing (continuous-integration)

[Scott Moser (smoser)]

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/curtin/commit/?id=50c0007c

diff --git a/tools/curtainer b/tools/curtainer
index ca698c6..466d719 100755
--- a/tools/curtainer
+++ b/tools/curtainer
@@ -17,10 +17,13 @@ Usage: ${0##*/} [ options ] <image> name
    start a container of image (ubuntu-daily:xenial) and install curtin.
 
    options:
-      --proposed    enable proposed
-      --daily       enable daily curtin archive
-      --source   D  grab the source deb, unpack inside, and copy unpacked
-                    source out to 'D'
+      --no-patch-version  do not patch source dir/curtin/version.py
+                          by default, --source will have version.py updated
+                          with dpkg's version.
+      --proposed          enable proposed
+      --daily             enable daily curtin archive
+      --source   D        grab the source deb, unpack inside, and copy unpacked
+                          source out to 'D'
 EOF
 }
 
@@ -114,7 +117,7 @@ get_source() {
 
 main() {
     local short_opts="hv"
-    local long_opts="help,daily,proposed,source:,verbose"
+    local long_opts="help,daily,no-patch-version,proposed,source:,verbose"
     local getopt_out=""
     getopt_out=$(getopt --name "${0##*/}" \
         --options "${short_opts}" --long "${long_opts}" -- "$@") &&
@@ -123,13 +126,14 @@ main() {
 
     local cur="" next=""
     local proposed=false daily=false src="" name="" maxwait=30
-    local eatmydata="eatmydata" getsource="none"
+    local eatmydata="eatmydata" getsource="none" patch_version=true
 
     while [ $# -ne 0 ]; do
         cur="$1"; next="$2";
         case "$cur" in
             -h|--help) Usage ; exit 0;;
                --source) getsource="$next"; shift;;
+               --no-patch-version) patch_version=false;;
                --proposed) proposed=true;;
                --daily) daily=true;;
             -v|--verbose) VERBOSITY=$((${VERBOSITY}+1));;
@@ -147,6 +151,7 @@ main() {
     if [ "$getsource" != "none" ]; then
         [ ! -e "$getsource" ] || fail "source output '$getsource' exists."
     fi
+    getsource="${getsource%/}"
 
     lxc launch "$src" "$name" || fail "failed lxc launch $src $name"
     CONTAINER=$name
@@ -218,6 +223,16 @@ main() {
         if [ "$src_ver" != "$pkg_ver" ]; then
             fail "source version ($src_ver) != package version ($pkg_ver)"
         fi
+        if "${patch_version}"; then
+            local verfile="$getsource/curtin/version.py"
+            grep -q "@@PACKAGED_VERSION@@" "$verfile" ||
+                fail "failed patching version: " \
+                    "@@PACKAGED_VERSION@@ not found in $verfile"
+            sed -i.curtainer-dist \
+                "s,@@PACKAGED_VERSION@@,${pkg_ver}," "$verfile" ||
+                fail "failed modifying $verfile"
+            debug 1 "patched $verfile pkg version to $pkg_ver."
+        fi
         inside "$name" rm -Rf "$isrcd" ||
             fail "failed removal of extract dir"
         debug 1 "put source for curtin at $src_ver in $getsource"