Merge lp:~smoser/curtin/trunk.install-deps into lp:~curtin-dev/curtin/trunk

[Scott Moser]

document patching images with curtin depends and provide tool

official vmtest should run with images as close to maas images as possible.
Thus, this should not be used for official tests.

However, developer use can patch the images that are used for boot, and
can dramatically decrease install time by installing curtin dependencies.

Example real world improvement even with a local proxy these are the install boot times before and after:
  PreciseTestBasic 66.22 37.79
  TrustyTestBasic 66.74 45.41
  VividTestAllindata 142.34 114.37
  VividTestBasic 76.89 50.03
  VividTestBonding 61.81 36.52
  VividTestLvm 73.47 48.32
  WilyTestBasic 78.65 53.53
  WilyTestBonding 64.48 38.70
  WilyTestLvm 87.24 51.62

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:321
https://server-team-jenkins.canonical.com/job/curtin-ci/46/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/44/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/46/rebuild

review: Needs Fixing (continuous-integration)

[Ryan Harper (raharper)]

Some nits in comments

[lp:~smoser/curtin/trunk.install-deps]

322. By Scott Moser on 2015-12-09

fix english

323. By Scott Moser on 2015-12-09

fix usage

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:323
https://server-team-jenkins.canonical.com/job/curtin-ci/47/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/45/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/47/rebuild

review: Needs Fixing (continuous-integration)

[lp:~smoser/curtin/trunk.install-deps]

324. By Scott Moser on 2015-12-09

fix pep8

325. By Scott Moser on 2015-12-09

remove copied / bad code

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:325
https://server-team-jenkins.canonical.com/job/curtin-ci/48/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/generic-update-mp/46/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/curtin-ci/48/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

Ryan,
 with regard to knowing / determining if this has already been patched before, i dont really know. I just added it as a quick way for developer to improve test speed.

One thign we could potentially do is create the modified image as a .qcow referencing the original.
then something like:
   CURTIN_VMTEST_PATCH_IMAGE=1 make vmtests
could look for a 'root-image.patched.img' file and use it. Then the original would be un-modified, and easily select between the two. the sync image woudl also be made to create the patched image if not present.

The above doesn't grow well if there are multiple different ways you can patch an image, though.

but i do like the qcow backed delta file. performance coudl potentially suffer some, but not likely much.

[Scott Moser (smoser)]

Now that we have the improved syncing, and our mirror creates and knows about vmtest.root-image and vmtest.root-tgz, we could also make it create 'vmtest.root-image-prepped'.

We can make the creation of and use of that 'vmtest.root-iamge.prepped' to be done only if VMTEST_USE_PREPPED_IMAGES=1

The result would be that someone can use that for faster development and then run vmtest without it to get more 'official'.

Unmerged revisions

325. By Scott Moser on 2015-12-09

remove copied / bad code

324. By Scott Moser on 2015-12-09

fix pep8

323. By Scott Moser on 2015-12-09

fix usage

322. By Scott Moser on 2015-12-09

fix english

321. By Scott Moser on 2015-12-09

document patching images with curtin depends and provide tool

official vmtest should run with images as close to maas images as possible.
Thus, this should not be used for official tests.

However, developer use can patch the images that are used for boot, and
can dramatically decrease install time by installing curtin dependencies.

Example real world improvement even with a local proxy:
  # test before after
  PreciseTestBasic 66.22 37.79
  TrustyTestBasic 66.74 45.41
  VividTestAllindata 142.34 114.37
  VividTestBasic 76.89 50.03
  VividTestBonding 61.81 36.52
  VividTestLvm 73.47 48.32
  WilyTestBasic 78.65 53.53
  WilyTestBonding 64.48 38.70
  WilyTestLvm 87.24 51.62

=== modified file 'curtin/commands/main.py'
--- curtin/commands/main.py	2015-11-18 21:29:57 +0000
+++ curtin/commands/main.py	2015-12-09 18:02:39 +0000
@@ -109,7 +109,13 @@
             # bad usage will be reported by the real reporter
             return
 
-    ret = install_deps(verbosity=verbosity)
+    # This is less than perfect, it is here for patching a root image
+    # with 'curtin --install-deps'.  Better would be to provide
+    # 'curtin install-deps' with --no-allow-daemons and then use that.
+    val = os.environ.get("CURTIN_INSTALL_DEPS_DAEMONS", "1")
+    allow_daemons = val.strip().lower() not in ("0", "false")
+
+    ret = install_deps(verbosity=verbosity, allow_daemons=allow_daemons)
 
     if ret != 0 or install_only:
         sys.exit(ret)

=== modified file 'doc/devel/README-vmtest.txt'
--- doc/devel/README-vmtest.txt	2015-12-01 01:27:21 +0000
+++ doc/devel/README-vmtest.txt	2015-12-09 18:02:39 +0000
@@ -94,6 +94,22 @@
     'apt_proxy' environment variable.  If that is not set it will 
     look at the host's apt config and read 'Acquire::HTTP::Proxy'
 
+== Patching root image with deps for faster install ==
+IMAGES_DIR defines the directory where the stream mirror exists.
+the tool install-curtin-deps can patch these so each install does not have
+to do that work.
+The root images are read from IMAGES_DIR which defaults to /srv/images.
+
+  IMAGE_DIR=/srv/images
+  for img in $(find "${IMAGE_DIR" -type f -name "root-image"); do
+     echo "working on $img"
+     if [ '!' -f "$img.dist" ]; then
+        cp --sparse=always "$img" "$img.dist.tmp" &&
+            mv "$img.dist.tmp" "$img.dist" || break;
+     fi
+     ./tools/install-curtin-deps "$img" || break;
+   done
+
 == Environment Variables ==
 Some environment variables affect the running of vmtest
   * apt_proxy: 

=== added file 'tools/install-curtin-deps'
--- tools/install-curtin-deps	1970-01-01 00:00:00 +0000
+++ tools/install-curtin-deps	2015-12-09 18:02:39 +0000
@@ -0,0 +1,183 @@
+#!/bin/bash
+
+VERBOSITY=0
+TEMP_D=""
+WORKING_D=""
+
+error() { echo "$@" 1>&2; }
+fail() { [ $# -eq 0 ] || error "$@"; exit 1; }
+
+Usage() {
+    cat <<EOF
+Usage: ${0##*/} target
+
+Install curtin dependencies into target.  Target can be a path to a root
+filesystem (/your/mounted/root) or a filesystem image (.img).
+
+options:
+   -v | --verbose   increase verbosity
+EOF
+}
+
+bad_Usage() { Usage 1>&2; [ $# -eq 0 ] || error "$@"; return 1; }
+cleanup() {
+    [ -z "${TEMP_D}" -o ! -d "${TEMP_D}" ] || rm -Rf "${TEMP_D}"
+    [ -z "$WORKING_D" -o ! -d "$WORKING_D" ] || rm -Rf "$WORKING_D"
+}
+
+debug() {
+    local level=${1}; shift;
+    [ "${level}" -gt "${VERBOSITY}" ] && return
+    error "${@}"
+}
+
+hascmd() {
+    local cmd="$1" envref="$2" t=""
+    shift 2
+    if [ -n "${envref}" -a -n "${!envref}" ]; then
+        cmd="${!envref}"
+        _RET="$cmd"
+        return 0
+    elif [ -x "$cmd" -a -f "$x" ]; then
+        t=$(readlink -f "$cmd") ||
+           { error "failed to get full path to $cmd"; return 1; }
+        _RET="$t"
+        return
+    elif t=$(which "$cmd" 2>/dev/null); then
+        _RET="$t"
+        return
+    else
+        for t in "$@"; do
+            [ -x "$t" -a -f "$t" ] && t=$(readlink -f "$t") &&
+                _RET="$t" && return
+        done
+    fi
+    return 1
+}
+
+find_apt_proxy() {
+    # pick an apt proxy for the guest
+    local out=""
+
+    # if user set uncommon 'apt_proxy', then trust it
+    [ -n "$apt_proxy" ] && echo "$apt_proxy" && return 0
+
+    # see if the host has an apt proxy configured, and use it
+    if command -v apt-config >/dev/null 2>&1; then
+        out=$(apt-config shell x Acquire::HTTP::Proxy) &&
+        out=$(sh -c 'eval $1 && echo $x' -- "$out") && [ -n "$out" ] &&
+        echo "$out" && return
+    fi
+
+    return 1
+}
+
+main() {
+    local short_opts="hv"
+    local long_opts="help,verbose"
+    local getopt_out=""
+    getopt_out=$(getopt --name "${0##*/}" \
+        --options "${short_opts}" --long "${long_opts}" -- "$@") &&
+        eval set -- "${getopt_out}" ||
+        { bad_Usage; return; }
+
+    local cur="" next="" curtin=""
+    local sudo="" target_in="" proxy=""
+
+    while [ $# -ne 0 ]; do
+        cur="$1"; next="$2";
+        case "$cur" in
+            -h|--help) Usage ; exit 0;;
+            -v|--verbose) VERBOSITY=$((${VERBOSITY}+1)); vflags="${vflags}v";;
+            --) shift; break;;
+        esac
+        shift;
+    done
+
+    [ $# -eq 1 ] ||
+        { bad_Usage "got $# args ($*). Expected only 1"; return 1; }
+    target_in="$1"
+    [ -f "$target_in" -o -d "$target_in" ] ||
+        { bad_Usage "$target_in: not a file or directory"; return; }
+    target=$(readlink -f "$target_in") ||
+        { error "failed to get full path to $target_in"; return 1; }
+
+    trap cleanup EXIT
+
+    [ "$(id -u)" = "0" ] && sudo="" || sudo="sudo"
+    [ -z "$vflags" ] || vflags="-${vflags}"
+
+    hascmd curtin CURTIN ${0%/*}/../bin/curtin ./bin/curtin &&
+        curtin="$_RET" || {
+            error "no curtin in path";
+            return 1;
+        }
+    if [ -d "$target" ]; then
+        debug 1 "using curtin=$curtin"
+        install_deps "$target" "$curtin"
+    else
+        local micb=""
+        hascmd mount-image-callback MOUNT_IMAGE_CALLBACK && micb="$_RET" || {
+            error "no mount-image-callback. install cloud-image-utils.";
+            return 1;
+        }
+
+        $sudo ${apt_proxy:+apt_proxy="$apt_proxy"} \
+            $micb --system-resolvconf --proc "$target" -- \
+            "$0" ${vflags} _MOUNTPOINT_ || {
+                error "failed to apply to target"
+                return 1
+            }
+    fi
+
+}
+
+install_deps() {
+    local rootd="$1" curtin="$2" wdir=""
+    [ -d "$rootd/tmp" ] || { error "$rootd does not have /tmp"; return 1; }
+    debug 2 "installing into $rootd with $curtin"
+    local t chroot="chroot" ret=""
+    t=$(readlink -f "$rootd") ||
+        { error "failed getting full path to $rootd"; return 1; }
+    rootd="$t"
+    if [ "$rootd" = "/" ]; then
+        chroot=""
+    fi
+
+    local proxy=""
+    proxy=$(find_apt_proxy) && [ -n "$proxy" ] &&
+        export http_proxy="$proxy" &&
+        debug 1 "using apt proxy=$proxy"
+
+    exed=$(mktemp -d "$rootd/tmp/${0##*/}.XXXXXX") ||
+        { error "failed to get tmpdir in ${rootd}/tmp"; return 1; }
+    WORKING_D="$exed"
+    # wdir is the path inside the chroot. exed is the path from outside.
+    wdir="${exed#${rootd%/}}"
+
+    $curtin pack -- curtin --install-deps > "$exed/packed" || {
+        error "failed to curtin pack: $curtin pack -- curtin --install-deps";
+        return 1;
+    }
+    chmod 755 "$exed/packed" ||
+        { error "failed to give exe to $exed/packed"; return 1; }
+
+    local fix_update_initramfs=false uix="$rootd/usr/sbin/update-initramfs"
+    if [ -x "$uix" ]; then
+       mv "$uix" "$uix.dist" && fix_update_initramfs=true
+       ln -sf /bin/true "$uix" || { mv "$uix.dist" "$uix" && return 1; }
+    fi
+    CURTIN_INSTALL_DEPS_DAEMONS=false LANG=C $chroot "$rootd" \
+        "$wdir/packed" "--prefix=$wdir/x"
+    ret=$?
+    [ $ret -eq 0 ] || error "failed to run pack inside $rootd";
+    if $fix_update_initramfs; then
+        rm -f "$uix"
+        mv "$uix.dist" "$uix"
+    fi
+    return $ret
+}
+
+
+main "$@"
+# vi: ts=4 expandtab