Merge lp:~smoser/cloud-utils/trunk.mic-lxd into lp:cloud-utils

[Scott Moser]

mount-image-callback: add lxd container support.

given lxd container on local system 'x1', you can now do:
   sudo mount-image-callback -vv --system-resolvconf \
       lxd:x1 -- chroot _MOUNTPOINT_ apt-get update

using mic to chroot into a lxd container is useful compared to 'lxc-chroot'
or simply chroot /path/to/container/rootfs because:
a.) lxc-chroot puts in own network namespace (meaning network doesnt work without bringing it up).
b.) chroot /path/to/container does not set resolvconf or system mounts
c.) mic knows assumed paths to lxd container
d.) mic uses lxc-usernsexec to join the appropriate user namespace.

Along the way, we add two little tools that can be used:
 uns-switch: user namespace switch ... switch to the user based on some environment variables that mic sets up
 mchroot: mic chroot. reads $MOUNTPOINT.

[Ryan Harper (raharper)]

\o/

Nice!

[Scott Moser (smoser)]

we also have to someway account for changing uid/gid mapping on the files that are modified in the target.
so as this is, it doesnt work.

[Ryan Harper (raharper)]

followed up with LXD folks, and they suggested lxc-usernsexec, here's how I got it working

1) look up "root" user in /etc/subuid

SHIFT=$(awk -F: '/^root/ {print $2":"$3}' /etc/subuid)
echo $SHIFT
165536:65536

Then instead of just chroot, do:

sudo lxc-usernsexec -m b:0:$SHIFT -- chroot ...

[lp:~smoser/cloud-utils/trunk.mic-lxd]

304. By Scott Moser on 2016-09-15

seems functional at this point.

305. By Scott Moser on 2016-09-15

mention lack of usernsexec

306. By Scott Moser on 2016-09-15

white space consistentcy

[Scott Moser (smoser)]

I dont think this is fabulous at the moment, but it scratches a very nice itch

$ lxc init ubuntu-daily:xenial x1
$ sudo mount-image-callback --system-resolvconf lxd:x1 -- mchroot sh -ec '
    echo deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main universe |
       tee /etc/apt/sources.list.d/proposed.list
    apt-get update -qy
    apt-get dist-upgrade -qy'

$ lxc start x1

[Ryan Harper (raharper)]

One more fix; I think the list of umounts is missing being updated, as this leaves them mounted.

% sudo mount-image-callback -v --system-mounts --system-resolvconf lxd:x1 chroot _MOUNTPOINT_ -- /bin/bash
using directory at /var/lib/lxd/containers/x1/rootfs for lxd:x1 (true)
replaced string _MOUNTPOINT_ in arguments arg 1
added helper uns-switch for chroot
replacing /etc/resolvconf
invoking: MOUNTPOINT=/var/lib/lxd/containers/x1/rootfs uns-switch chroot /var/lib/lxd/containers/x1/rootfs /bin/bash
# exit
exit
cmd returned 0. unmounting /var/lib/lxd/containers/x1/rootfs
% grep x1.zfs /proc/mounts
lxd-zfs/containers/x1 /var/lib/lxd/containers/x1.zfs zfs rw,relatime,xattr,noacl 0 0
udev /var/lib/lxd/containers/x1.zfs/rootfs/dev devtmpfs rw,nosuid,relatime,size=3915008k,nr_inodes=978752,mode=755 0 0
proc /var/lib/lxd/containers/x1.zfs/rootfs/proc proc rw,nosuid,nodev,noexec,relatime 0 0
sysfs /var/lib/lxd/containers/x1.zfs/rootfs/sys sysfs rw,nosuid,nodev,noexec,relatime 0 0

[Ryan Harper (raharper)]

I think this fixes that and handles the nbd case where the bind mounts disappear automatically.

http://paste.ubuntu.com/23183820/

[Ryan Harper (raharper)]

OK, this is better:

http://paste.ubuntu.com/23183888/

Only add to the UMOUNTS list if we're doing a lxd ns switch.

[lp:~smoser/cloud-utils/trunk.mic-lxd]

307. By Scott Moser on 2016-09-16

fix unmounting system mounts

go through the mounts that we did backwards when unmounting.
we are still unmounting recursively, but if you do them in
backwards order generally that should be ok.

we still do recursive so that if you did:
 mount-image-callback my.img chroot /
   % mount -t tmpfs /dev/ /dev
   % exit

it would unmount the thing in there.

This may have to change at some point to avoid unmounting things
we did *not* mount (since we are now operating possibly on a directory
that might already have mounts under it).

308. By Scott Moser on 2016-10-04

merge with trunk

309. By Scott Moser on 2016-10-04

move nbd mounting out of main

310. By Scott Moser on 2016-10-05

merge with trunk

311. By Scott Moser on 2016-10-05

merge with trunk

=== modified file 'bin/mount-image-callback'
--- bin/mount-image-callback	2016-10-05 13:15:20 +0000
+++ bin/mount-image-callback	2016-10-05 13:44:01 +0000
@@ -16,6 +16,10 @@
    mount a file to a temporary mount point and then
    invoke the provided cmd with args
 
+   supported 'file' are:
+      lxd:name  : the rootfs for lxd container 'name'
+      file      : any disk format supported by qemu-nbd
+
    the temporary mountpoint will be put in an a environment variable
    named MOUNTPOINT.
 
@@ -81,8 +85,10 @@
 }
 
 do_umounts() {
-	local um="" fails=0 mydir="$PWD/"
-	for um in "$@"; do
+	local um="" fails=0 mydir="$PWD/" mounts="" i=0
+	mounts=( "$@" )
+	for((i=${#mounts[@]}-1;i>=0;i--)); do
+		um=${mounts[$i]}
 		um=$(readlink -f "$um") || {
 			error "WARNING: failed to get full path to '$um'";
 			fails=$(($fails+1))
@@ -140,6 +146,46 @@
 	fi
 }
 
+add_bin() {
+	cat > "$1" || { error "failed to write to $1"; return 1; }
+	chmod 755 "$1" || { error "failed to set perms on $1"; return 1; }
+}
+
+add_helpers() {
+	local d="$1"
+	local umap="$1" gmap="$2"
+	[ -d "$1" ] || mkdir -p "$1"
+	add_bin "$d/uns-switch" <<"EOF" || return 1
+#!/bin/sh
+[ $# -eq 0 ] && set -- ${SHELL:-/bin/bash}
+if [ -z "$MIC_SUBUID" -a -z "$MIC_SUBGID" ]; then
+	exec "$@"
+fi
+exec lxc-usernsexec ${MIC_SUBUID:+-m u:0:${MIC_SUBUID}} \
+	${MIC_SUBGID:+-m g:0:${MIC_SUBGID}} -- "$@"
+EOF
+	add_bin "$d/mchroot" <<"EOF" || return 1
+#!/bin/sh
+exec uns-switch chroot "$MOUNTPOINT" "$@"
+EOF
+	return
+}
+
+get_subid() {
+	local id="$1" file="$2"
+	awk -F: '$1 == id { printf("%s:%s\n", $2, $3); exit(0); }' \
+		id="$id" "$file"
+}
+
+set_subids() {
+	local id="$1" out="" muid="" mgid=""
+	[ -z "$id" ] && return 0
+	muid=$(get_subid "$id" /etc/subuid) &&
+		mgid=$(get_subid "$id" /etc/subgid) &&
+		[ -n "$muid" -a -n "$mgid" ] &&
+		export MIC_SUBGID="$muid" MIC_SUBUID="$mgid" && return
+}
+
 mount_overlay() {
 	local lower="$1" upper="$2" workdir="$3"
 	local olayopts="lowerdir=$lower,upperdir=$upper"
@@ -346,14 +392,38 @@
 		return 1;
 	}
 
+	local uns_switch=false
 	img_in="$1"
 	shift 1
 
-	img=$(readlink -f "$img_in") ||
-		{ error "failed to get full path to $img_in"; return 1; }
-
-	[ -f "$img" ] ||
-		{ error "$img: not a file"; return 1; }
+	case "$img_in" in
+		lxd:*)
+			if [ "$ptnum" != "auto" ]; then
+				error "--ptnum incompatible with directory"
+				return 1;
+			fi
+			command -v lxc-usernsexec >/dev/null 2>&1 || {
+				error "lxd support needs lxc-usernsexec"; 
+				return 1;
+			}
+			img="/var/lib/lxd/containers/${img_in#lxd:}/rootfs"
+			[ -d "$img" ] || {
+				error "${img_in}: no rootfs in $img. Not a container?"
+				return 1;
+			}
+			set_subids lxd || {
+				error "failed reading subids for lxd";
+				return 1;
+			}
+			uns_switch=true
+			;;
+		*)
+			img=$(readlink -f "$img_in") ||
+				{ error "failed to get full path to $img_in"; return 1; }
+			[ -f "$img" ] ||
+				{ error "$img: not a file"; return 1; }
+			;;
+	esac
 
 	[ "$(id -u)" = "0" ] ||
 		{ error "sorry, must be root"; return 1; }
@@ -364,7 +434,6 @@
 	if [ -z "$mp" ]; then
 		mp="${TEMP_D}/mp"
 		mkdir "$mp" || return
-		mp_is_tmp=true
 	else
 		[ -d "$mp" ] ||
 			{ error "mountpoint '$mp': not a directory"; return 1; }
@@ -380,24 +449,17 @@
 		img_mp=$mp
 	fi
 
-	local cmd="" arg="" found=false
-	cmd=( )
-	for arg in "$@"; do
-		if [ "${arg}" = "_MOUNTPOINT_" ]; then
-			debug 1 "replaced string _MOUNTPOINT_ in arguments arg ${#cmd[@]}"
-			arg=$mp
-		fi
-		cmd[${#cmd[@]}]="$arg"
-	done
-
-	if [ "${cmd[0]##*/}" = "bash" -o "${cmd[0]##*/}" = "sh" ] &&
-	   [ ${#cmd[@]} -eq 0 ]; then
-		debug 1 "invoking shell ${cmd[0]}"
-		error "MOUNTPOINT=$mp"
-	fi
-
 	out=""
-	if [ "$ptnum" = "auto" -o "$ptnum" = "0" ] &&
+	if [ -d "$img" ]; then
+		debug 1 "using directory at $img for $img_in"
+		mp=$img
+		if $overlay; then
+			debug 1 "mounting $img to $mp"
+			mount --bind "$img" "$img_mp" ||
+				{ error "failed to mount --bind '$img' '$mp'"; return 1; }
+			UMOUNTS[${#UMOUNTS[@]}]="$mp"
+		fi
+	elif [ "$ptnum" = "auto" -o "$ptnum" = "0" ] &&
 		out=$(set -f; mount -o "loop,$rwmode" $opts "$img" "$img_mp" 2>&1); then
 		debug 1 "mounted simple filesystem image $rwmode in '$img_in'"
 		UMOUNTS[${UMOUNT[@]}]="$img_mp"
@@ -430,8 +492,10 @@
 	for bindmp in $bmounts; do
 		[ -d "$mp${bindmp}" ] || mkdir "$mp${bindmp}" ||
 			{ error "failed mkdir $bindmp in mount"; return 1; }
-		mount --bind "$bindmp" "$mp/${bindmp}" ||
+		mount --bind "$bindmp" "$mp${bindmp}" ||
 			{ error "failed bind mount '$bindmp'"; return 1; }
+		UMOUNTS[${#UMOUNTS[@]}]="$mp${bindmp}"
+		debug 1 "mounted $bindmp to $mp${bindmp}"
 	done
 
 	if ${system_resolvconf}; then
@@ -462,6 +526,19 @@
 		error "MOUNTPOINT=$mp"
 	fi
 
+	if $uns_switch && [ "$1" = "chroot" ]; then
+		debug 1 "added helper uns-switch for chroot"
+		cmd=( uns-switch "${cmd[@]}" )
+	elif $uns_switch; then
+		debug 1 "uns-switch helper can be used for changing user map"
+	fi
+
+	add_helpers "$TEMP_D/bin" "$SUBUID" "$SUBGID" || {
+		error "failed to add helpers to $TEMP_D";
+		return 1;
+	}
+	PATH="$TEMP_D/bin:$PATH"
+
 	local startwd="$PWD"
 	debug 1 "invoking: MOUNTPOINT=$mp" "${cmd[@]}"