Merge lp:~smoser/curtin/trunk.use-smtar into lp:~curtin-dev/curtin/trunk

[Scott Moser]

[Ryan Harper (raharper)]

This is missing a unit/functional test of smtar.

review: Needs Fixing

[Scott Moser (smoser)]

After merging https://code.launchpad.net/~smoser/curtin/install-from-url/+merge/277482
we're now installing from a url. That url is gzip compressed, so smtar would be used.

One thing i'd like to do is to be able to run the tests with a install from .tar also.

the reason would be that on a local system, installing http://host.ip/foo.tar does uncompressed transfer through virtio with no decompression cost. while the .tar.gz does compressed transfer at cost of decompress to cpu.

[lp:~smoser/curtin/trunk.use-smtar]

292. By Scott Moser on 2015-11-13

merge from trunk

293. By Scott Moser on 2015-11-18

merge from trunk

294. By Scott Moser on 2015-11-19

precise's 'file' says application/x-gzip not application/gzip.

295. By Scott Moser on 2015-11-19

update dependencies

[Scott Moser (smoser)]

ok. so we're now pushing code through 'smtar' in vmtest. We can definitely add unit tests, but the code path is executed now.

I ran a full test suite successfully through here.

=== modified file 'curtin/commands/extract.py'
--- curtin/commands/extract.py	2015-10-28 20:29:28 +0000
+++ curtin/commands/extract.py	2015-11-19 01:21:54 +0000
@@ -52,10 +52,13 @@
 
 
 def extract_root_tgz_url(source, target):
+    # extract a -root.tar.gz url in the 'target' directory
+    #
+    # Uses smtar to avoid specifying the compression type
     curtin.util.subp(args=['sh', '-cf',
                            ('wget "$1" --progress=dot:mega -O - |'
-                            'tar -C "$2" ' + ' '.join(tar_xattr_opts()) +
-                            ' ' + '-Sxpzf - --numeric-owner'),
+                            'smtar -C "$2" ' + ' '.join(tar_xattr_opts()) +
+                            ' ' + '-Sxpf - --numeric-owner'),
                            '--', source, target])
 
 
 
=== modified file 'curtin/deps/__init__.py'
--- curtin/deps/__init__.py	2015-11-18 16:36:32 +0000
+++ curtin/deps/__init__.py	2015-11-19 01:21:54 +0000
@@ -27,6 +27,7 @@
 
 REQUIRED_EXECUTABLES = [
     # executable in PATH, package
+    ('file', 'file'),
     ('lvcreate', 'lvm2'),
     ('mdadm', 'mdadm'),
     ('mkfs.btrfs', 'btrfs-tools'),
 
=== modified file 'debian/control'
--- debian/control	2015-11-17 18:28:30 +0000
+++ debian/control	2015-11-19 01:21:54 +0000
@@ -27,6 +27,7 @@
 Priority: extra
 Depends: btrfs-tools,
          e2fsprogs,
+         file,
          gdisk,
          lvm2,
          mdadm,
 
=== added file 'helpers/smtar'
--- helpers/smtar	1970-01-01 00:00:00 +0000
+++ helpers/smtar	2015-11-19 01:21:54 +0000
@@ -0,0 +1,116 @@
+#!/bin/sh
+# smtar (smart tar)
+# GNU Tar can only determine the compression type if input is a local file.
+# If input is a pipe, it will not even attempt.
+#
+# This works around that limitation by using 'file' to determine
+# the compression format via a local temp file of BUFLEN (1024) bytes.
+# After determining format, it passes the correct flag to tar.
+#
+# Compression format determination is done with 'file' via use of a temp file
+#
+#  The following are supported:
+#
+#    # compression option provided explicitly: just exec tar
+#    $ ./smtar -tvz < my.tar.gz
+#
+#    # file argument provided, tar can determine: just exec tar
+#    $ ./smtar -tvf my.tar.gz 
+#    $ ./smtar -tvf my.tar
+#
+#    # input from stdin. determine the appropriate compress flag and execute
+#    $ ./smtar -tv < my.tar.bz2
+#    $ ./smtar -tv < my.tar.bz2
+#    $ cat my.tar.xz | ./smtar -tv -f -
+#    $ wget http://some.tar | ./smtar -tv -f -
+#
+#
+TEMPF=""
+BUFLEN="1024"
+cleanup() { [ -z "$TEMPF" ] || rm -f "$TEMPF"; }
+error() { echo "$@" 1>&2; }
+fail() { [ $# -eq 0 ] || error "$@"; exit 1; }
+find_tar_filearg() {
+    # walk through list of args, return the 'file' argument in _RET
+    local cur="" next=""
+    while [ $# -ne 0 ]; do
+        cur="$1"
+        next="$2"
+        case "$cur" in
+            --file=*) _RET=${cur#*=}; return 0;;
+            --file) _RET=$next; return 0;;
+            --*=*) :;;
+            *-f) _RET="$next"; return 0;;
+            --) _RET=""; return 0;;
+        esac
+        shift
+    done
+    return 1
+}
+
+tar_has_compress_opt() {
+    # this isnt perfect, but catch common ways
+    # without fully parsing the args, we risk interpreting
+    #   tar -xvf -J
+    # as bzip2 compression, where in reality its a file name '-J'
+    local cur="" next=""
+    while [ $# -ne 0 ]; do
+        cur="$1"
+        next="$2"
+        case "$cur" in
+            -z|--gzip|--gunzip|--ungzip) return 0;;
+            -j|--bzip2) return 0;;
+            -J|--xz) return 0;;
+            -Z|--compress|--uncompress) return 0;;
+            --) return 1;;
+        esac
+        shift
+    done
+    return 1
+}
+
+# see if we can get out without reading anything
+if [ -t 0 ] || tar_has_compress_opt; then
+    # input is a terminal, or args contain a compress option
+    exec tar "$@"
+fi
+
+# if there was a compression arg in input, then let it be
+find_tar_filearg "$@"
+if ! [ "$_RET" = "/dev/stdin" -o "$_RET" = "-" -o -z "$_RET" ]; then
+   exec "tar" "$@"
+fi
+
+# now we have work to do
+zopt=""
+TEMPF=$(mktemp) || fail "mktemp failed"
+trap cleanup EXIT
+head -c "$BUFLEN" > "$TEMPF" || fail "FAILED: head -c '$BUFLEN'"
+size=$(stat --format="%s" "$TEMPF")
+file_out=$(LANG=C file --mime-type "$TEMPF")
+# my.tar: application/x-tar
+# my.tar.bz2: application/x-bzip2
+# my.tar.gz: application/gzip
+# my.tar.xz: application/x-xz
+# my.tar.Z:  application/x-compress
+if [ $? -eq 0 ]; then
+    case "$file_out" in
+        */x-bzip2|*/bzip2) zopt="--bzip2";;
+        */x-gzip|*/gzip) zopt="--gzip";;
+        */x-xz|*/xz) zopt="--xz";;
+        */x-compress|*/compress) zopt="--compress";;
+        *) zopt="";;
+    esac
+else
+    error "WARN: 'file' failed on input"
+fi
+if [ "$size" -lt "$BUFLEN" ]; then
+    # input was less than BUFLEN chars, so we just exec tar with input from it
+    exec < "$TEMPF"
+    rm -f "$TEMPF"
+    exec tar $zopt "$@"
+else
+    ( cat "$TEMPF" && rm "$TEMPF" && exec cat ) | exec tar $zopt "$@"
+fi
+
+# vi: ts=4 expandtab syntax=sh