Merge lp:~racb/ubuntu/saucy/autopkgtest/lxc into lp:ubuntu/saucy/autopkgtest

[Robie Basak]

Add LXC support: adt-virt-lxc

[Martin Pitt (pitti)]

Hey Robie, many thanks for working on this! That's a nice addition.

Some things which I noticed during first spot-check:

+# autopkgtest is Copyright (C) 2006-2007 Canonical Ltd.

Please add a copyright for you, you wrote that file after all.

127 + lxc_container_name = 'adt-virt-lxc-' + ''.join([random.choice(string.ascii_lowercase) for i in range(6)])
128 + vsp.execute('sudo lxc-start-ephemeral -n %s --keep-data -d -o' % lxc_container_name,[lxc_template], downp=False, outp=True)

I'm not entirely sure what that does, but it looks like lxc_container_name is a temporary file name of some sort? Would it be more adequate to use the standard mkstemp() function, or is that really just a name? In the latter case, how does this make sure that it doesn't use an existing name?

[James Hunt (jamesodhunt)]

Great! This might go some way towards providing the facility outlined on bug 1158391.

[Jean-Baptiste Lallement (jibel)]

Thanks for your work!
I tested the following scenarios:
- i386 / amd64
- test package from dsc and source tree
- unbuilt tree and binary package
- clone mode, as well as options ephemeral and eatmydata

Overall it works fine but:
1. If the base container is not created from an ubuntu-cloud template (i.e debootstrap based), the driver goes to an infinite wait loop. It should fail gracefully in that case
2. The requirement for a cloud template means that the usage of this driver is limited to Ubuntu. Using any type of lxc container would allow to run tests in LXC on Debian for example.
3. This requirement is not documented
4. More generally there is no man page for this driver while there are for the other drivers
5. If eatmydata is not installed into the container and the driver is invoked with option --eatmydata then it crashes with
    lxc-attach: No such file or directory - failed to exec 'eatmydata'
    <VirtSubproc>: failure: (down) mkdir failed (exit status 255)
  * It should either:
    - Fail gracefully,
    - or install eatmydata if not installed (but an implicit installation might be confusing and affect the test)
    - or run without eatmydata (and display a warning)

I think at least 1, 3 and 5 should be fixed

review: Needs Fixing

[Iain Lane (laney)]

→ WIP based on the last comments. Please reset to Needs Review once addressed.

[Robie Basak (racb)]

Thanks for the review, Jean-Baptiste. I've addressed 1, 3, 4 and 5. This driver could in theory work in Debian with Debian's cloud-init. The only real issue is that there's no standard way to detect if an LXC container has finished booting, so I used cloud-init's boot-finished sentinel file.

Specifically:

> 1. If the base container is not created from an ubuntu-cloud template (i.e debootstrap based), the driver goes to an infinite wait loop. It should fail gracefully in that case

Now explicitly checked and fails with a specific error message for this case.

> 2. The requirement for a cloud template means that the usage of this driver is limited to Ubuntu. Using any type of lxc container would allow to run tests in LXC on Debian for example.

As above. Anything with cloud-init should work, and if we can have some generic lxc-wait type mechanism for boot finished detection, we could expand support in the future.

> 3. This requirement is not documented
> 4. More generally there is no man page for this driver while there are for the other drivers

Man page written.

> 5. If eatmydata is not installed into the container and the driver is invoked with option --eatmydata then it crashes with...

I now explicitly check that the container has eatmydata if --eatmydata is specified, and fail with a specific error in that case. I thought that this would be better than printing a warning, as chances are that the developer won't see it (and the developer can trivially work around the problem by not using the --eatmydata option). Like you, I felt that an implicit installation might be confusing.

[Martin Pitt (pitti)]

Thanks for the updates!

I adjusted the code to be PEP8 compliant and fixed the pyflakes errors, and pushed to the Debian git:

http://anonscm.debian.org/gitweb/?p=autopkgtest/autopkgtest.git;a=commitdiff;h=617e6f48

I also created a set of test cases:

http://anonscm.debian.org/gitweb/?p=autopkgtest/autopkgtest.git;a=commitdiff;h=3fd0972a

I'm closing this MP now. It's not formally merged into the ubuntu branch, but we'll sync it in the t-series.

Thanks!

review: Approve

=== modified file 'Makefile'
--- Makefile	2011-07-30 17:57:27 +0000
+++ Makefile	2013-10-16 16:21:23 +0000
@@ -26,6 +26,7 @@
 		virt-subproc/adt-virt-xenlvm \
 		virt-subproc/adt-virt-null \
 		virt-subproc/adt-virt-schroot \
+		virt-subproc/adt-virt-lxc \
 		runner/adt-run \
 		runner/adt-testreport-onepackage \
 		runner/adt-testreport-runloop \
 
=== added file 'virt-subproc/adt-virt-lxc'
--- virt-subproc/adt-virt-lxc	1970-01-01 00:00:00 +0000
+++ virt-subproc/adt-virt-lxc	2013-10-16 16:21:23 +0000
@@ -0,0 +1,217 @@
+#!/usr/bin/python
+#
+# adt-virt-lxc is part of autopkgtest
+# autopkgtest is a tool for testing Debian binary packages
+#
+# autopkgtest is Copyright (C) 2006-2007, 2013 Canonical Ltd.
+#
+# adt-virt-lxc was derived from adt-virt-schroot and modified to suit LXC by
+# Robie Basak <[e-mail]>.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+# See the file CREDITS for a full list of credits information (often
+# installed as /usr/share/doc/autopkgtest/CREDITS).
+
+import sys
+import os
+import string
+import re as regexp
+import grp
+import pwd
+from optparse import OptionParser
+import random
+import subprocess
+import time
+
+LXC_WAIT_RETRY_TIME = 0.2
+
+try: our_base = os.environ['AUTOPKGTEST_BASE']+'/lib'
+except KeyError: our_base = '/usr/share/autopkgtest/python';
+sys.path.insert(1, our_base)
+
+import VirtSubproc as vsp
+
+def pw_uid(exp_name):
+	try:
+		return pwd.getpwnam(exp_name).pw_uid
+	except KeyError:
+		return None
+
+def gr_gid(exp_name):
+	try:
+		return grp.getgrnam(exp_name).gr_gid
+	except KeyError:
+		return None
+
+def match(exp_names, ids, extract_id):
+	for exp_name in [n for n in exp_names.split(',') if n]:
+		if extract_id(exp_name) in ids:
+			return True
+	return False
+
+def parse_args():
+	global lxc_template, debuglevel, eatmydata, ephemeral
+
+	usage = "%prog [<options>] <template>"
+	parser = OptionParser(usage=usage)
+	pa = parser.add_option
+	pe = parser.error
+
+	pa('-d', '--debug', action='store_true', dest='debug');
+	pa('--eatmydata', action='store_true')
+        pa('--ephemeral', action='store_true')
+
+	(opts,args) = parser.parse_args()
+	if len(args) != 1: pe("need exactly one arg, template lxc container name")
+
+	lxc_template = args[0]
+	eatmydata = opts.eatmydata
+        ephemeral = opts.ephemeral
+
+	vsp.debuglevel = opts.debug
+
+def check_for_cloud_init(container, template_name):
+        with open('/dev/null', 'wb') as null:
+            result = subprocess.call(
+                ['sudo', 'lxc-attach', '-n', container, '--', 'dpkg-query', '-W',
+                    'cloud-init'],
+                close_fds=True, stdout=null, stderr=null)
+        if result:
+            raise RuntimeError(
+                ("Container %s does not use cloud-init, and this driver " +
+                    "requires it. Use eg. " +
+                    "\"lxc-create -t ubuntu-cloud ...\" to create your " +
+                    "container instead.")
+                    % repr(template_name)
+            )
+
+def check_for_eatmydata(container, template_name):
+        with open('/dev/null', 'wb') as null:
+            result = subprocess.call(
+                ['sudo', 'lxc-attach', '-n', container, '--', 'sh', '-c',
+                    'command -v eatmydata'],
+                close_fds=True, stdout=null, stderr=null)
+        if result:
+            raise RuntimeError(
+                ("Container %s does not contain eatmydata, which is required "
+                    "for the --eatmydata option."
+                    % repr(template_name))
+            )
+
+def full_path_split(path):
+	head, tail = os.path.split(os.path.normpath(path))
+	if head in ['', os.path.sep, 2 * os.path.sep]:
+		return [head, tail]
+	else:
+		return full_path_split(head) + [tail]
+
+def wait_for_lexists(path):
+        vsp.debug('waiting for %s to exist' % repr(path))
+	while not os.path.lexists(path):
+		time.sleep(LXC_WAIT_RETRY_TIME)
+
+def wait_for_lxc_guest_start(lxc_name, template_name):
+        check_for_cloud_init(lxc_name, template_name)
+        for subdir in ['delta0', 'rootfs']:
+            host_guest_root = '/var/lib/lxc/%s/%s' % (lxc_name, subdir)
+            if os.path.lexists(host_guest_root):
+                break
+        else:
+            raise RuntimeError(
+                "Failed to find host root directory for container %s." %
+                repr(lxc_name)
+            )
+	host_instance_path = os.path.join(
+		host_guest_root, 'var/lib/cloud/instance')
+	wait_for_lexists(host_instance_path)
+	guest_instance_link = os.readlink(host_instance_path)
+	# Due to overlayfs, we can sometimes read the link when it doesn't
+	# officially exist in the guest. Keep waiting in this case. The
+	# "(overlay-whiteout)" string is defined by the overlayfs module.
+	while guest_instance_link == '(overlay-whiteout)': # do not translate
+		time.sleep(LXC_WAIT_RETRY_TIME)
+		guest_instance_link = os.readlink(host_instance_path)
+	host_instance_link = os.path.join(
+		host_guest_root, *full_path_split(guest_instance_link)[1:])
+	host_boot_finished_path = os.path.join(
+		host_instance_link, 'boot-finished')
+	wait_for_lexists(host_boot_finished_path)
+
+def generate_random_lxc_container_name():
+	return ('adt-virt-lxc-' +
+		''.join([random.choice(string.ascii_lowercase) for i in range(6)]))
+
+def get_available_lxc_container_name():
+	'''Return an LXC container name that isn't already taken.
+
+	There is a race condition between returning this name and creation of
+	the container. Ideally lxc-start-ephemeral would generate a name in a
+	race free way and return the name used in machine-readable way, so that
+	this function would not be necessary. See LP: #1197754.
+	'''
+
+	candidate = generate_random_lxc_container_name()
+	while os.path.lexists('/var/lib/lxc/%s' % candidate):
+		candidate = generate_random_lxc_container_name()
+	return candidate
+
+def hook_open():
+	global sessid, downtmp, lxc_container_name
+	lxc_container_name = get_available_lxc_container_name()
+        if ephemeral:
+            vsp.execute('sudo lxc-start-ephemeral -n %s -k -d -o' % lxc_container_name,[lxc_template], downp=False, outp=True)
+        else:
+            vsp.execute('sudo lxc-clone -n %s -o' % lxc_container_name,[lxc_template], downp=False, outp=True)
+            vsp.execute('sudo lxc-start -n %s -d' % lxc_container_name,[], downp=False, outp=True)
+        try:
+            vsp.debug('waiting for lxc guest start')
+            wait_for_lxc_guest_start(lxc_container_name, lxc_template)
+            vsp.debug('lxc guest started')
+            vsp.down = ['sudo', '-E', 'lxc-attach','-n',lxc_container_name,'--']
+            if eatmydata:
+                    check_for_eatmydata(lxc_container_name, lxc_template)
+                    vsp.down.extend(['eatmydata', '--'])
+            vsp.downkind = 'auxverb'
+        except:
+            # Clean up on failure
+            vsp.execute('sudo lxc-stop -n',[lxc_container_name], downp=False, outp=True)
+            vsp.execute('sudo lxc-destroy -n',[lxc_container_name], downp=False, outp=True)
+            raise
+
+def hook_downtmp():
+	downtmp = '/tmp/adt-downtmp'
+	vsp.execute('mkdir %s' % downtmp, downp=True)
+	return downtmp
+
+def hook_revert():
+	vsp.downtmp_remove()
+	hook_cleanup()
+	hook_open()
+
+def hook_cleanup():
+	global sessid, downtmp
+	vsp.downtmp_remove()
+	vsp.execute('sudo lxc-stop -n',[lxc_container_name], downp=False, outp=True)
+	vsp.execute('sudo lxc-destroy -n',[lxc_container_name], downp=False, outp=True)
+
+def hook_forked_inchild():
+	pass
+
+def hook_capabilities():
+	return ['revert', 'revert-full-system', 'root-on-testbed', 'suggested-normal-user=ubuntu']
+
+parse_args()
+vsp.main()
 
=== added file 'virt-subproc/adt-virt-lxc.1'
--- virt-subproc/adt-virt-lxc.1	1970-01-01 00:00:00 +0000
+++ virt-subproc/adt-virt-lxc.1	2013-10-16 16:21:23 +0000
@@ -0,0 +1,124 @@
+.TH adt\-virt-lxc 1 2013 "Linux Programmer's Manual"
+.SH NAME
+adt\-virt\-lxc \- autopkgtest virtualisation server using LXC
+.SH SYNOPSYS
+.B adt\-virt\-lxc
+.RB [ \-\-d | \-\-debug ]
+.RB [ \-\-eatmydata ]
+.RB [ \-\-ephemeral ]
+.I lxc\-template
+.SH DESCRIPTION
+.B adt-virt-lxc
+provides an autopkgtest virtualisation server using LXC. It adapts the raw
+functionality provided by the
+.BR lxc- *
+tools for use by autopkgtest.
+
+Normally
+.B adt-virt-lxc
+will be invoked by
+.BR adt-run .
+
+.SH REQUIREMENTS
+.B adt-virt-lxc
+assumes that you have already prepared a suitable LXC template container. The
+template must be Debian-based and use cloud-init. \fBdebootstrap\fR(8)-based
+containers such as those created with the \fIdebian\fR or \fIubuntu\fR LXC
+templates will not work.
+
+On Ubuntu, you can automatically download and install a suitable template using
+the command:
+
+.RS
+.EX
+lxc-create -t ubuntu-cloud -n adt -- -r $(distro-info --devel) -d daily
+.EE
+.RE
+
+The supplied LXC template must be clean. The \fBcloud-init\fR
+\fI/var/cloud/instance\fR symlink should not exist, and the subsequent
+\fI/var/lib/cloud/instance/boot-finished\fR should also not exist. If you start
+your template container in order to modify it, make sure you remove these
+artifacts when you have finished making your modifications.
+
+
+.SH OPTIONS
+.TP
+.B --eatmydata
+This option is intended for test developer use. It enables the use of
+\fBeatmydata\fR(1) inside the container on all commands in order to improve
+performance. This requires
+.BR eatmydata (1)
+to already be installed inside the supplied template, which you will need to do
+manually.
+
+However, since eatmydata is not part of the dep8 specification and may affect
+the test
+environment, it is not recommended for use in automation.
+.TP
+.B --ephemeral
+This option makes use of lxc-start-ephemeral to create temporary containers,
+instead of the default behaviour of cloning your containers. This can be used
+to dramatically improve performance, although you may see issues due to the
+overlayfs filesystem as used by lxc-start-ephemeral not being completely
+transparent. To avoid issues, you should not use this option in automation.
+.TP
+.BR \-d " | " \-\-debug
+Enables debugging output.
+.SH INPUT, OUTPUT AND EXIT STATUS
+The behaviour of
+.B adt-virt-lxc
+is as described by the AutomatedTesting virtualisation regime
+specification.
+
+.SH NOTES
+
+\fBadt-run\fR assumes it can use \fBfakeroot\fR(1) to gain root for the
+\fIbinary\fR target in a package build. If \fBfakeroot\fR(1) isn't available in
+your template container, you should use \fBadt-run\fR's \fB\-\-gain\-root\fR
+option to specify what to use instead. On Ubuntu, you can use
+\fB\-\-gain\-root=sudo\fR since this is included in Ubuntu cloud images
+configured to permit root access without a password for the default user. See
+the example below.
+
+\fBadt-run\fR does not run \fBapt-get update\fR at the start of a package
+build, which can cause a build failure if you are using a development release
+template. You will need to run \fBapt-get update\fR in the template yourself as
+required, and remove the boot artifacts as discussed in \fBREQUIREMENTS\fR
+before trying again.
+
+.SH EXAMPLE
+
+On Ubuntu, download and install a suitable template and call it \fIadt\fR:
+
+.RS
+.EX
+lxc-create -t ubuntu-cloud -n \fIadt\fR -- -r $(distro-info --devel) -d daily
+.EE
+.RE
+
+Run tests against \fIhello_2.8\-4.dsc\fR, using the LXC template \fIadt\fR,
+and with an ephemeral container for speed:
+
+.RS
+.EX
+adt-run --gain-root=sudo \fIhello_2.8\-4.dsc\fR --- adt-virt-lxc --ephemeral \fIadt\fR
+.EE
+.RE
+
+.SH SEE ALSO
+\fBadt-run\fR(1),
+\fBadt\-virt-null\fR(1),
+\fBadt\-virt-chroot\fR(1),
+\fBeatmydata\fR(1),
+\fB/usr/share/doc/autopkgtest/\fR.
+
+.SH AUTHORS AND COPYRIGHT
+.B adt-virt-lxc
+was written by Robie Basak <[e-mail]>.
+
+This manpage is part of autopkgtest, a tool for testing Debian binary
+packages.  autopkgtest is Copyright (C) 2006-2013 Canonical Ltd and others.
+
+See \fB/usr/share/doc/autopkgtest/CREDITS\fR for the list of
+contributors and full copying conditions.