Merge lp:~gary/lpsetup/paralleltweaks into lp:lpsetup

[Gary Poster]

add the tweaks necessary for bug 1013921 and bug 1014916

This branch adds the tweaks necessary for bug 1013921 and bug 1014916.

 * For 1013921, make lp-setup-lxc-test include -vvv when calling bin/test so that testr gets test timings and can better balance tests.
 * For 1014916, only when the container OS is Lucid (not necessary for Precise), do the following:
   * Change /var/lib/lxc/lptests/rootfs/etc/init/networking.conf to not wait for udev (that is, use start on (local-filesystems))
   * in the host remove the container's /etc/init/udevtrigger.conf (rm /var/lib/lxc/lptests/rootfs/etc/init/udevtrigger.conf

See the relevant bugs for the reasoning, if desired.

Given our lack of integration tests, I manually tested this on ec2 and it seems to be fine.

Thank you

[Gary Poster (gary)]

Ah right: I also changed the application of the fix for bug 974584 to only happen for lucid, as described by the preexisting comment.

review: Approve

[Gary Poster (gary)]

Oops, didn't mean to approve.

[Brad Crittenden (bac)]

This branch looks good. Thanks for the fixes Gary.

review: Approve (code)

[Launchpad QA Bot (lpqabot)]

The attempt to merge lp:~gary/lpsetup/paralleltweaks into lp:lpsetup failed. Below is the output from the failed tests.

nose.plugins.cover: ERROR: Coverage not available: unable to import coverage module
EEEEE...................EE
======================================================================
ERROR: Failure: ImportError (No module named shelltoolbox)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/tests/subcommands/test_get.py", line 9, in <module>
    from lpsetup import handlers
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/handlers.py", line 21, in <module>
    from shelltoolbox import (
ImportError: No module named shelltoolbox

======================================================================
ERROR: Failure: ImportError (No module named shelltoolbox)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/tests/subcommands/test_inithost.py", line 9, in <module>
    from lpsetup import handlers
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/handlers.py", line 21, in <module>
    from shelltoolbox import (
ImportError: No module named shelltoolbox

======================================================================
ERROR: Failure: ImportError (No module named shelltoolbox)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/python2.7/dist-packages/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/tests/subcommands/test_install.py", line 9, in <module>
    from lpsetup import handlers
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/handlers.py", line 21, in <module>
    from shelltoolbox import (
ImportError: No module named shelltoolbox

======================================================================
ERROR: Failure: ImportError (No module named shelltoolbox)
-----------------------------------------------------...

[Launchpad QA Bot (lpqabot)]

The attempt to merge lp:~gary/lpsetup/paralleltweaks into lp:lpsetup failed. Below is the output from the failed tests.

nose.plugins.cover: ERROR: Coverage not available: unable to import coverage module
......................................................................EEE..
======================================================================
ERROR: test_idempotent (lpsetup.tests.test_utils.ScrubberTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/tests/test_utils.py", line 234, in test_idempotent
    self.lxc_pattern, get_running_containers())
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/utils.py", line 208, in __init__
    self.ignored_containers = list(ignored_containers)
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/utils.py", line 116, in get_running_containers
    containers = run('lxc-ls').split()
  File "/usr/lib/python2.7/dist-packages/shelltoolbox/__init__.py", line 418, in run
    close_fds=kwargs.pop('close_fds', True), **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

======================================================================
ERROR: test_scrub (lpsetup.tests.test_utils.ScrubberTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/tests/test_utils.py", line 218, in test_scrub
    self.lxc_pattern, get_running_containers())
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/utils.py", line 208, in __init__
    self.ignored_containers = list(ignored_containers)
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/utils.py", line 116, in get_running_containers
    containers = run('lxc-ls').split()
  File "/usr/lib/python2.7/dist-packages/shelltoolbox/__init__.py", line 418, in run
    close_fds=kwargs.pop('close_fds', True), **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

======================================================================
ERROR: test_scrub_no_dir (lpsetup.tests.test_utils.ScrubberTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/tests/test_utils.py", line 228, in test_scrub_no_dir
    self.lxc_pattern, get_running_containers())
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/utils.py", line 208, in __init__
    self.ignored_containers = list(ignored_containers)
  File "/home/tarmac/repos/lpsetup/trunk/lpsetup/utils.py", line 116, in get_running_containers
    containers = run('lxc-ls').split()
  File "/usr/lib/python2.7/dist-packages/shelltoolbox/__init__.py", line 418, in run
    close_fds=kwargs.pop('close_fds', True), **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 679, in __in...

[Brad Crittenden (bac)]

Gary sorry I have taken your MP hostage as I work through these tarmac issues.

[Gary Poster (gary)]

:-) yay, it worked! Thank you

=== modified file 'lpsetup/subcommands/lxcinstall.py'
--- lpsetup/subcommands/lxcinstall.py	2012-06-27 09:10:56 +0000
+++ lpsetup/subcommands/lxcinstall.py	2012-06-27 21:21:20 +0000
@@ -134,13 +134,14 @@
     dst = get_container_path(lxc_name, '/root/.ssh/')
     mkdirs(dst)
     shutil.copy(user_authorized_keys, dst)
-    # XXX 2012-04-18 frankban bug=974584:
-    #     Add a line to the container's fstab to be able to create semaphores
-    #     in lxc. This workaround needs to be removed once the lxc bug is
-    #     resolved for lucid containers too.
-    file_append(
-        '/var/lib/lxc/{0}/fstab'.format(lxc_name),
-        'none dev/shm tmpfs defaults 0 0\n')
+    if lxc_os == 'lucid':
+        # XXX 2012-04-18 frankban bug=974584:
+        # Add a line to the container's fstab to be able to create semaphores
+        # in lxc. This workaround needs to be removed once the lxc bug is
+        # resolved for lucid containers too.
+        file_append(
+            '/var/lib/lxc/{0}/fstab'.format(lxc_name),
+            'none dev/shm tmpfs defaults 0 0\n')
 
 
 def start_lxc(lxc_name):
@@ -179,6 +180,17 @@
     ssh(lxc_name,
         'ln -s /usr/lib/graphviz /rootfs/usr/lib/graphviz',
         key=ssh_key_path)
+    # XXX gary 2012-06-26 bug=1014916: this fixes a bug in Lucid LXC
+    # containers.
+    if lxc_os == 'lucid':
+        # Change /var/lib/lxc/lptests/rootfs/etc/init/networking.conf to not
+        # wait for udev (that is, use start on (local-filesystems)).
+        udevtrigger = get_container_path(
+            lxc_name, '/etc/init/udevtrigger.conf')
+        call('mv', udevtrigger, udevtrigger + '.orig')
+        networking = get_container_path(lxc_name, '/etc/init/networking.conf')
+        call('mv', networking, networking + '.orig')
+        render_to_file('networking.conf', {}, networking)
 
 
 def setup_launchpad_lxc(

=== modified file 'lpsetup/templates/lp-setup-lxc-test'
--- lpsetup/templates/lp-setup-lxc-test	2012-06-12 15:25:13 +0000
+++ lpsetup/templates/lp-setup-lxc-test	2012-06-27 21:21:20 +0000
@@ -20,4 +20,4 @@
 lxc-start-ephemeral -u {user} -S '{ssh_key_path}' -o {lxc_name} -- \
     "env -u LANG xvfb-run --error-file=/var/tmp/xvfb-errors.log \
     --server-args='-screen 0 1024x768x24' \
-    -a $PWD/bin/test --shuffle --subunit $@"
+    -a $PWD/bin/test -vvv --shuffle --subunit $@"

=== added file 'lpsetup/templates/networking.conf'
--- lpsetup/templates/networking.conf	1970-01-01 00:00:00 +0000
+++ lpsetup/templates/networking.conf	2012-06-27 21:21:20 +0000
@@ -0,0 +1,14 @@
+# networking - configure virtual network devices
+#
+# This task causes virtual network devices that do not have an associated
+# kernel object to be started on boot.
+
+description     "configure virtual network devices"
+
+start on (local-filesystems)
+
+task
+
+pre-start exec mkdir -p /var/run/network
+
+exec ifup -a