Merge lp:~nuclearbob/utah/arm into lp:utah

[Max Brustkern]

This branch adds support for provisioning panda boards in a bamboo-feeder setup, such as the magners lab. Notable additions include:
bamboofeeder.py: provides the BambooFeederMachine class
power.py: provides power commands for non-cobbler physical machines using an arbitrary command or fence_cdu.
run_test_bamboo_feeder.py: script to run the new provisioning method. Also incorporated into run_utah_test.py if '-a arm' is specifried.
Notable changes include:
provisioning.py: Machine class now supports cleanup functions. Subclasses other than BambooFeederMachine have not been updated to use them, but I intend to propose a later merge for that. Image handling has been slightly altered to support ARM images, which are not ISOs, and thus currently will not work with the ISO class. The ISO class could be updated for this later, or an alternate solution developed. Default preseed support has moved into Machine from CustomVM.
sqlite.py: ManualCobblerSQLiteInventory class now supports other machine types. It should be renamed, and 'cargs' (cobbler arguments) should be genericized, possibly into machineinfo. I intend to propose a later merge for that as well.
libvirtvm.py: CustomVM now uses the default preseed support in Machine.

There may be more changes I'm forgetting, so I'll check over that and comment once I have the diffs. I merged the latest dev branch before this proposal, so new things should be incorporated.

[lp:~nuclearbob/utah/arm]

731. By Max Brustkern on 2012-10-19

Removed duplicate class definition

[Max Brustkern (nuclearbob)]

Other changes:
u-boot-tools added to Suggests. cobbler and vm-tools demoted there as well, since they're only needed for specific integrations. Maybe at some point those integrations should be separate packages. Getting the main test runner script to respond to that and output good, useful error would be a chunk of work, but probably a good thing to do in the long run.
test running scripts were updated to pass through the clean parameter, based on the --no-destroy option.
powertimeout in config is now 15 by default to support the pandas better.
pxedir and wwwdir are added to config to control file locations. These could be used for further pxe integration later.
OrderedSet and HashableDict were added to facilitate cleanup functionality.
Machine class got proper destroy and depcheck methods to support Super calls from subclasses.
depcheck is used in BambooFeederMachine to make sure u-boot-tools is available. I think CobblerMachine and VMToolsVM should move to a similar system in a future merge.

[Max Brustkern (nuclearbob)]

More about cleanup:
Machine subclasses can now run cleanfile, cleanfunction, and cleancommand. When cleanup is invoked, typically as part of the Machine class's new destroy function, which is also called by the default destructor, those OrderedSets are iterated over and executed. OrderedSets are used to prevent duplicate registrations. First, everything in cleanfiles is removed. Links and files are set to be writeable and unlinked. Directories are recursively set to be writeable and recursively unlinked, similar to how they were handled in the CobblerMachine cleanup, which should eventually be altered to use this routine. Functions in cleanfunctions are run, which any arguments passed in, and a timeout of 60 seconds. This could be altered or removed in the future. Finally, comamnds in cleancommands are run through the machine's _runargs method.

[Joe Talbott (joetalbott)]

line 10: one too many 'o's in vm-tools
753: 'clearn' should probably be 'clean'.

Other than those looks good to me.

review: Needs Fixing

[lp:~nuclearbob/utah/arm]

732. By Max Brustkern on 2012-10-25

vm-toools had tooo many ooos

733. By Max Brustkern on 2012-10-25

clearn -> clean

[Max Brustkern (nuclearbob)]

Fixed those.

review: Needs Resubmitting

[Joe Talbott (joetalbott)]

Awesome. Go for it.

review: Approve

=== modified file 'debian/control'
--- debian/control	2012-10-05 20:22:37 +0000
+++ debian/control	2012-10-25 20:09:20 +0000
@@ -17,7 +17,8 @@
          python-apt, python-libvirt,
          python-netifaces, python-paramiko, python-psutil,
          utah-client (=${binary:Version})
-Recommends: cobbler, kvm, vm-tools
+Recommends: kvm
+Suggests: cobbler, u-boot-tools, vm-tools
 Description: Ubuntu Test Automation Harness
  Automation framework for testing in Ubuntu
 
 
=== modified file 'examples/run_install_test.py'
--- examples/run_install_test.py	2012-10-04 16:11:18 +0000
+++ examples/run_install_test.py	2012-10-25 20:09:20 +0000
@@ -90,12 +90,13 @@
     try:
         inventory = TinySQLiteInventory()
         machine = inventory.request(CustomVM,
-                arch=args.arch, boot=args.boot, debug=args.debug,
-                diskbus=args.diskbus, disksizes=args.gigabytes,
-                dlpercentincrement=10, emulator=args.emulator,
-                image=args.image, initrd=args.initrd, installtype=args.type,
-                kernel=args.kernel, new=True, preseed=args.preseed,
-                rewrite=args.rewrite, series=args.series, xml=args.xml)
+                arch=args.arch, boot=args.boot, clean=(not args.no_destroy), 
+                debug=args.debug, diskbus=args.diskbus,
+                disksizes=args.gigabytes, dlpercentincrement=10,
+                emulator=args.emulator, image=args.image, initrd=args.initrd,
+                installtype=args.type, kernel=args.kernel, new=True,
+                preseed=args.preseed, rewrite=args.rewrite,
+                series=args.series, xml=args.xml)
         exitstatus, locallogs = run_tests(args, machine)
 
     except UTAHException as error:
 
=== added file 'examples/run_test_bamboo_feeder.py'
--- examples/run_test_bamboo_feeder.py	1970-01-01 00:00:00 +0000
+++ examples/run_test_bamboo_feeder.py	2012-10-25 20:09:20 +0000
@@ -0,0 +1,104 @@
+#!/usr/bin/python
+
+import argparse
+import os
+import sys
+
+from utah.exceptions import UTAHException
+from utah.group import check_user_group, print_group_error_message
+from utah.provisioning.baremetal.bamboofeeder import BambooFeederMachine
+from utah.provisioning.inventory.sqlite import ManualCobblerSQLiteInventory
+from utah.run import run_tests
+from utah.url import url_argument
+
+
+def get_parser():
+    parser = argparse.ArgumentParser(
+            description=('Provision a pandaboard in a bamboo-feeder setup '
+                         'and run one or more UTAH runlists there.'),
+            epilog=("For example:\n"
+                    "Provision a machine using a precise server image "
+                    "with i386 architecture and run the two given runlists\n"
+                    "\t%(prog)s -s precise -t server -a i386 \\\n"
+                    "\t\t/usr/share/utah/client/examples/master.run \\\n"
+                    "\t\t'http://people.canonical.com/~max/max_test.run'"),
+            formatter_class=argparse.RawDescriptionHelpFormatter)
+    parser.add_argument('runlists', metavar='runlist', nargs='+',
+                        help='URLs of runlist files to run')
+    parser.add_argument('-s', '--series', metavar='SERIES',
+                        choices=('hardy', 'lucid', 'natty',
+                                 'oneiric', 'precise', 'quantal'),
+                        help='Series to use for VM creation (%(choices)s)')
+    parser.add_argument('-t', '--type', metavar='TYPE',
+                        choices=('desktop', 'server', 'mini', 'alternate'),
+                        help=('Install type to use for VM creation '
+                              '(%(choices)s)'))
+    parser.add_argument('-a', '--arch', metavar='ARCH',
+                        choices=('i386', 'amd64'),
+                        help=('Architecture to use for VM creation '
+                              '(%(choices)s)'))
+    parser.add_argument('--name', help='Name of machine to provision')
+    parser.add_argument('-i', '--image', type=url_argument,
+                        help='Image/ISO file to use for installation')
+    parser.add_argument('-p', '--preseed', type=url_argument,
+                        help='Preseed file to use for installation')
+    parser.add_argument('-n', '--no-destroy', action='store_true',
+                        help='Preserve VM after tests have run')
+    parser.add_argument('-d', '--debug', action='store_true',
+                        help='Enable debug logging')
+    parser.add_argument('-j', '--json', action='store_true',
+                        help='Enable json logging (default is YAML)')
+    return parser
+
+
+def run_test_bamboo_feeder(args=None):
+    if not check_user_group():
+        print_group_error_message(__file__)
+        sys.exit(3)
+
+    if args is None:
+        args = get_parser().parse_args()
+
+    locallogs = []
+    exitstatus = 0
+    machine = None
+
+    try:
+        inventory = ManualCobblerSQLiteInventory(
+                db=os.path.join('~', '.utah-bamboofeeder-inventory'),
+                lockfile=os.path.join('~', '.utah-bamboofeeder-lock'))
+        kw = {}
+        for arg in ['image', 'preseed']:
+            value = vars(args)[arg]
+            if value is not None:
+                kw.update([(arg, value)])
+        machine = inventory.request(machinetype=BambooFeederMachine,
+                                    clean=(not args.no_destroy),
+                                    debug=args.debug, new=True,
+                                    dlpercentincrement=10, **kw)
+        exitstatus, locallogs = run_tests(args, machine)
+
+    except UTAHException as error:
+        mesg = 'Exception: ' + str(error)
+        try:
+            machine.logger.error(mesg)
+        except (AttributeError, NameError):
+            sys.stderr.write(mesg)
+        exitstatus = 2
+
+    finally:
+        if machine is not None:
+            try:
+                machine.destroy()
+            except UTAHException as error:
+                sys.stderr.write('Failed to destroy machine: ' + str(error))
+
+        if len(locallogs) != 0:
+            print('Test logs copied to the following files:')
+            print("\t" + "\n\t".join(locallogs))
+
+    sys.exit(exitstatus)
+
+
+if __name__ == '__main__':
+    run_test_bamboo_feeder()
 
=== modified file 'examples/run_test_cobbler.py'
--- examples/run_test_cobbler.py	2012-10-04 16:11:18 +0000
+++ examples/run_test_cobbler.py	2012-10-25 20:09:20 +0000
@@ -76,7 +76,8 @@
             value = vars(args)[arg]
             if value is not None:
                 kw.update([(arg, value)])
-        machine = inventory.request(debug=args.debug, new=True,
+        machine = inventory.request(clean=(not args.no_destroy),
+                                    debug=args.debug, new=True,
                                     dlpercentincrement=10, **kw)
         exitstatus, locallogs = run_tests(args, machine)
 
 
=== modified file 'examples/run_test_vm.py'
--- examples/run_test_vm.py	2012-09-05 12:06:10 +0000
+++ examples/run_test_vm.py	2012-10-25 20:09:20 +0000
@@ -64,7 +64,8 @@
                 kw.update([(arg, value)])
         if args.type is not None:
             kw.update([('installtype', args.type)])
-        machine = inventory.request(debug=args.debug, new=True,
+        machine = inventory.request(clean=(not args.no_destroy),
+                                    debug=args.debug, new=True,
                                     dlpercentincrement=10, **kw)
         exitstatus, locallogs = run_tests(args, machine)
 
 
=== modified file 'examples/run_utah_tests.py'
--- examples/run_utah_tests.py	2012-10-04 16:11:18 +0000
+++ examples/run_utah_tests.py	2012-10-25 20:09:20 +0000
@@ -88,24 +88,21 @@
         args = get_parser().parse_args()
 
     if args.arch is not None and 'arm' in args.arch:
-        sys.stderr.write("ARM support is not included in this release.\n")
-        sys.stderr.write("Please check the roadmap, "
-                         "as it will be included in a future version.\n")
-        sys.exit(4)
-
-    function = run_test_vm
-
-    for option in [args.boot, args.emulator, args.image, args.initrd,
-                   args.kernel, args.preseed, args.xml]:
-        if option is not None:
-            from run_install_test import run_install_test
-            function = run_install_test
-            break
-
-    if args.machinetype == 'physical':
+        from run_test_bamboo_feeder import run_test_bamboo_feeder
+        function = run_test_bamboo_feeder
+    elif args.machinetype == 'physical':
         from run_test_cobbler import run_test_cobbler
         function = run_test_cobbler
+    else:
+        function = run_test_vm
 
+        for option in [args.boot, args.emulator, args.image, args.initrd,
+                       args.kernel, args.preseed, args.xml]:
+            if option is not None:
+                from run_install_test import run_install_test
+                function = run_install_test
+                break
+    
     function(args=args)
 
 if __name__ == '__main__':
 
=== modified file 'utah/config.py' (properties changed: +x to -x)
--- utah/config.py	2012-10-16 10:46:10 +0000
+++ utah/config.py	2012-10-25 20:09:20 +0000
@@ -46,12 +46,14 @@
     nfscommand=['sudo', os.path.join('/', 'etc', 'init.d', 'nfs-kernel-server'), 'reload'],
     nfsconfigfile=os.path.join('/', 'etc', 'exports'),
     nfsoptions='*(ro,async,no_root_squash,no_subtree_check)',
-    powertimeout=5,
+    powertimeout=15,
     preseed=None,
+    pxedir=os.path.join('/', 'var', 'lib', 'tftpboot', 'pxelinux.cfg'),
     qemupath='qemu:///system',
     rewrite='all',
     series='precise',
     template=None,
+    wwwdir=os.path.join('/', 'var', 'www', 'utah'),
 )
 
 # These depend on the local user/path, and need to be filtered out
 
=== added file 'utah/orderedcollections.py'
--- utah/orderedcollections.py	1970-01-01 00:00:00 +0000
+++ utah/orderedcollections.py	2012-10-25 20:09:20 +0000
@@ -0,0 +1,75 @@
+"""
+Provide additional collections to support cleanup functions.
+"""
+
+
+import collections
+
+KEY, PREV, NEXT = range(3)
+
+
+class OrderedSet(collections.MutableSet):
+
+    def __init__(self, iterable=None):
+        self.end = end = [] 
+        end += [None, end, end]         # sentinel node for doubly linked list
+        self.map = {}                   # key --> [key, prev, next]
+        if iterable is not None:
+            self |= iterable
+
+    def __len__(self):
+        return len(self.map)
+
+    def __contains__(self, key):
+        return key in self.map
+
+    def add(self, key):
+        if key not in self.map:
+            end = self.end
+            curr = end[PREV]
+            curr[NEXT] = end[PREV] = self.map[key] = [key, curr, end]
+
+    def discard(self, key):
+        if key in self.map:        
+            key, prev, next = self.map.pop(key)
+            prev[NEXT] = next
+            next[PREV] = prev
+
+    def __iter__(self):
+        end = self.end
+        curr = end[NEXT]
+        while curr is not end:
+            yield curr[KEY]
+            curr = curr[NEXT]
+
+    def __reversed__(self):
+        end = self.end
+        curr = end[PREV]
+        while curr is not end:
+            yield curr[KEY]
+            curr = curr[PREV]
+
+    def pop(self, last=True):
+        if not self:
+            raise KeyError('set is empty')
+        key = next(reversed(self)) if last else next(iter(self))
+        self.discard(key)
+        return key
+
+    def __repr__(self):
+        if not self:
+            return '%s()' % (self.__class__.__name__,)
+        return '%s(%r)' % (self.__class__.__name__, list(self))
+
+    def __eq__(self, other):
+        if isinstance(other, OrderedSet):
+            return len(self) == len(other) and list(self) == list(other)
+        return set(self) == set(other)
+
+    def __del__(self):
+        self.clear()                    # remove circular references
+
+
+class HashableDict(dict):
+    def __hash__(self):
+        return hash(tuple(sorted(self.items())))
 
=== added file 'utah/provisioning/baremetal/bamboofeeder.py'
--- utah/provisioning/baremetal/bamboofeeder.py	1970-01-01 00:00:00 +0000
+++ utah/provisioning/baremetal/bamboofeeder.py	2012-10-25 20:09:20 +0000
@@ -0,0 +1,304 @@
+"""
+Support bare-metal provisioning of bamboo-feeder-based systems.
+"""
+
+import netifaces
+import os
+import pipes
+import shutil
+import subprocess
+import tempfile
+
+from utah import config
+from utah.provisioning.provisioning import (
+        CustomInstallMixin,
+        Machine,
+        SSHMixin,
+        )
+from utah.provisioning.baremetal.power import PowerMixin
+from utah.provisioning.baremetal.exceptions import UTAHBMProvisioningException
+from utah.retry import retry
+
+
+class BambooFeederMachine(CustomInstallMixin, SSHMixin, PowerMixin, Machine):
+    """
+    Provide a class to provision an ARM board from a bamboo-feeder setup.
+    """
+    def __init__(self, boot=None, cargs=None, inventory=None,
+                 name=None, powercmd=None, preboot=None, *args, **kw):
+        # TODO: change the name of cargs here and elsewhere
+        # TODO: respect rewrite setting
+        if name is None:
+            raise UTAHBMProvisioningException(
+                    'Machine name reqired for bamboo-feeder machine')
+        if powercmd is not None:
+            self.powercmd = powercmd
+        elif cargs is not None:
+            self.power = {}
+            for item in cargs:
+                if 'power' in item:
+                    self.power[item] = cargs[item]
+        else:
+            raise UTAHBMProvisioningException(
+                    'No power control information specified')
+        try:
+            self.macaddress = cargs['mac-address']
+        except AttributeError:
+            raise UTAHBMProvisioningException(
+                    'No MAC address specified')
+        self.inventory = inventory
+        super(BambooFeederMachine, self).__init__(*args, name=name, **kw)
+        if self.inventory is not None:
+            self.cleanfunction(self.inventory.release, machine=self)
+        self._depcheck()
+        if self.image is None:
+            raise UTAHBMProvisioningException(
+                    'Image file required for bamboo-feeder installation')
+        try:
+            iface = config.wwwiface
+        except AttributeError:
+            iface = config.bridge
+        self.ip = netifaces.ifaddresses(iface)[netifaces.AF_INET][0]['addr']
+        self.logger.debug('Configuring for ' + iface + ' with IP ' + self.ip)
+        self._cmdlinesetup()
+        imageurl = 'http://' + self.ip + '/utah/' + self.name + '.img'
+        preenvurl = 'http://' + self.ip + '/utah/' + self.name + '.preEnv'
+        if preboot is None:
+            self.preboot = ('console=ttyO2,115200n8 imageurl={imageurl} '
+                            'bootcfg={preenvurl}'.format(
+                            imageurl=imageurl, preenvurl=preenvurl))
+        else:
+            # TODO: maybe make this automatically add needed options
+            self.preboot = preboot
+        self.logger.debug('Preboot setup:')
+        self.logger.debug(self.preboot)
+        self.logger.debug('BambooFeederMachine init finished')
+
+    def _prepareimage(self):
+        """
+        Make a copy of the image shared via wwa so we can edit it.
+        """
+        self.logger.info('Making copy of install image')
+        self.wwwimage = os.path.join(config.wwwdir, self.name + '.img')
+        self.logger.debug('Copying ' + self.image + ' to ' + self.wwwimage)
+        self.cleanfile(self.wwwimage)
+        shutil.copyfile(self.image, self.wwwimage)
+        return self.wwwimage
+
+    def _mountimage(self):
+        """
+        Mount an ARM boot image so we can edit it.
+        """
+        self.logger.info('Mounting install image')
+        self.imagedir = os.path.join(self.tmpdir, 'image.d')
+        if not os.path.isdir(self.imagedir):
+            os.makedirs(self.imagedir)
+        self.sector = int(subprocess.check_output(['file', 
+                self.wwwimage]
+                ).strip().split(';')[1].split(',')[3].split()[1])
+        self.logger.debug('Image start sector is ' + str(self.sector))
+        self.cleanfunction(self._umountimage)
+        self.logger.debug('Mounting ' + self.wwwimage + ' at ' +
+                          self.imagedir)
+        self._runargs(['sudo', 'mount', self.wwwimage, self.imagedir,
+                       '-o', 'loop', '-o', 'offset=' + str(self.sector*512),
+                       '-o', 'uid=' + config.user])
+
+    def _setupconsole(self):
+        """
+        Setup the install to use a serial console."
+        """
+        self.logger.info('Setting up the install to use the serial console')
+        preenvfile = os.path.join(self.imagedir, 'preEnv.txt')
+        serialpreenvfile = os.path.join(self.imagedir, 'preEnv.txt-serial')
+        self.logger.debug('Copying ' + serialpreenvfile + ' to ' + preenvfile)
+        shutil.copyfile(serialpreenvfile, preenvfile)
+
+    def _unpackinitrd(self):
+        """
+        Unpack the uInitrd file into a directory so we can modify it.
+        """
+        self.logger.info('Unpacking uInitrd')
+        if not os.path.isdir(os.path.join(self.tmpdir, 'initrd.d')):
+            os.makedirs(os.path.join(self.tmpdir, 'initrd.d'))
+        os.chdir(os.path.join(self.tmpdir, 'initrd.d'))
+        filesize = os.path.getsize(self.initrd)
+        for line in subprocess.check_output(
+                ['mkimage', '-l', self.initrd]).splitlines():
+            if 'Data Size:' in line:
+                datasize = int(line.split()[2])
+                break
+        headersize = filesize - datasize
+        self.logger.debug('uInitrd header size is ' + str(headersize))
+        pipe = pipes.Template()
+        pipe.prepend('dd if=$IN bs=1 skip=' + str(headersize), 'f-')
+        pipe.append('gunzip', '--')
+        pipe.append('cpio -ivd 2>/dev/null', '-.')
+        # TODO: suppress dd output
+        pipe.copy(self.initrd, '/dev/null')
+        
+    def _repackinitrd(self):
+        """
+        Repack the uInitrd file from the initrd.d directory.
+        """
+        self.logger.info('Repacking uInitrd')
+        os.chdir(os.path.join(self.tmpdir, 'initrd.d'))
+        pipe = pipes.Template()
+        pipe.prepend('find .', '.-')
+        pipe.append('cpio --quiet -o -H newc', '--')
+        pipe.append('gzip -9fc', '--')
+        initrd = os.path.join(self.tmpdir, 'initrd.gz')
+        self.logger.debug('Repacking compressed initrd')
+        pipe.copy('/dev/null', initrd)
+        self.logger.debug('Creating uInitrd with mkimage')
+        self._runargs(['mkimage',
+                               '-A', 'arm',
+                               '-O', 'linux',
+                               '-T', 'ramdisk',
+                               '-C', 'gzip',
+                               '-a', '0',
+                               '-e', '0',
+                               '-n', 'initramfs',
+                               '-d', initrd,
+                               self.initrd])
+
+    def _umountimage(self):
+        """
+        Unmount the image after we're done with it."
+        """
+        self.logger.info('Unmounting image')
+        self._runargs(['sudo', 'umount', self.imagedir])
+
+    def _cmdlinesetup(self, boot=None):
+        """
+        Add the needed options to the command line for an automatic install.
+        """
+        if boot is None:
+            boot = ''
+        super(BambooFeederMachine, self)._cmdlinesetup(boot=boot)
+        # TODO: minimize these
+        for option in ('auto', 'ro', 'text'):
+            if option not in self.cmdline:
+                self.logger.info('Adding boot option: ' + option)
+                self.cmdline += ' ' + option
+        for parameter in (('cdrom-detect/try_usb', 'true'),
+                          ('console', 'ttyO2,115200n8'),
+                          ('country', 'US'),
+                          ('hostname', self.name),
+                          ('language', 'en'),
+                          ('locale', 'en_US'),
+                          ('loghost', self.ip),
+                          ('log_port', '10514'),
+                          ('netcfg/choose_interface', 'auto'),
+                          ('url', 'http://' + self.ip + '/utah/' + self.name +
+                           '.cfg'),
+                         ):
+            if parameter[0] not in self.cmdline:
+                self.logger.info('Adding boot option: ' + '='.join(parameter))
+                self.cmdline += ' ' + '='.join(parameter)
+        self.cmdline.strip()
+
+    def _configurepxe(self):
+        """
+        Setup PXE configuration to boot remote image.
+        """
+        # TODO: Maybe move this into pxe.py
+        # TODO: look into cutting out the middleman/
+        # booting straight into the installer? (maybe nfs?)
+        self.logger.info('Configuring PXE')
+        pxeconfig = """default utah-bamboofeeder
+prompt 0
+timeout 3
+
+label utah/bamboofeeder
+kernel panda/uImage
+append {preboot}
+initrd panda/uInitrd
+""".format(preboot=self.preboot)
+        tmppxefile = os.path.join(self.tmpdir, 'pxe')
+        open(tmppxefile, 'w').write(pxeconfig)
+        self.logger.debug('PXE info written to ' + tmppxefile)
+        pxefile = os.path.join(config.pxedir,
+                '01-' + self.macaddress.replace(':', '-'))
+        self.cleancommand(('sudo', 'rm', '-f', pxefile))
+        self.logger.debug('Copying ' + tmppxefile + ' to ' + pxefile)
+        self._runargs(['sudo', 'cp', tmppxefile, pxefile])
+        preenvfile = os.path.join(config.wwwdir, self.name + '.preEnv')
+        # TODO: sort this out with self.boot
+        # figure out which one should be what and customizable
+        self.preenv = 'bootargs=' + self.cmdline
+        self.logger.debug('Preenv setup:')
+        self.logger.debug(self.preenv)
+        self.cleanfile(preenvfile)
+        self.logger.debug('Writing preenv setup to ' + preenvfile)
+        open(preenvfile, 'w').write(self.preenv)
+        
+    def _create(self):
+        """
+        Install the OS on the system.
+        """
+        # TODO: more checks and exceptions for failures
+        self.logger.info('Preparing system install')
+        self.tmpdir = tempfile.mkdtemp(prefix='/tmp/' + self.name + '_')
+        self.cleanfile(self.tmpdir)
+        self.logger.debug('Using ' + self.tmpdir + ' as temp dir')
+        os.chdir(self.tmpdir)
+        self._prepareimage()
+        self._mountimage()
+        self._setupconsole()
+        self.initrd = os.path.join(self.imagedir, 'uInitrd')
+        self.logger.debug('uInitrd is ' + self.initrd)
+        self._unpackinitrd()
+        self._setuplatecommand()
+        # TODO: check if this is still needed
+        if self.installtype == 'desktop':
+            self.logger.info('Configuring latecommand for desktop')
+            myfile = open(os.path.join(self.tmpdir, 'initrd.d',
+                          'utah-latecommand'), 'a')
+            myfile.write(
+"""
+chroot /target sh -c 'sed "/eth[0-9]/d" -i /etc/network/interfaces'
+cp /etc/resolv.conf /target/etc/resolv.conf
+""")
+            myfile.close()
+        self._setuppreseed()
+        self.logger.debug('Copying preseed to download location')
+        preseedfile = os.path.join(config.wwwdir, self.name + '.cfg')
+        self.cleanfile(preseedfile)
+        shutil.copyfile(os.path.join(self.tmpdir, 'initrd.d', 'preseed.cfg'),
+                        preseedfile)
+        self._repackinitrd()
+        self._configurepxe()
+        self._umountimage()
+        self.restart()
+        self.logger.info('System installing')
+        self.logger.info('Checking every ' + str(config.checktimeout) +
+                         ' seconds until system is installed')
+        self.logger.info('Will time out after ' + str(config.installtimeout)
+                         + ' seconds')
+        retry(self.sshcheck, config.checktimeout, logmethod=self.logger.info)
+
+        self.provisioned = True
+        self.active = True
+        # TODO: Make this a method that this and CobblerMachine can call
+        uuid_check_command = ('[ "{uuid}" == "$(cat /etc/utah/uuid)" ]'
+                              .format(uuid=self.uuid))
+        if self.run(uuid_check_command)[0] != 0:
+            self.provisioned = False
+            raise UTAHBMProvisioningException(
+                    'Installed UUID differs from Machine UUID; '
+                    'installation probably failed. '
+                    'Try restarting cobbler or running cobbler sync')
+        self.logger.info('System installed')
+        self.cleanfunction(self.run, ('dd', 'bs=512k', 'count=10',
+                'if=/dev/zero', 'of=/dev/mmcblk0'), root=True)
+        return True
+        
+    def _depcheck(self):
+        """
+        Check for dependencies that are in Recommends or Suggests.
+        """
+        super(BambooFeederMachine, self)._depcheck()
+        cmd = ['which', 'mkimage']
+        if self._runargs(cmd) != 0:
+            raise UTAHBMProvisioningException('u-boot-tools not installed')
 
=== added file 'utah/provisioning/baremetal/power.py'
--- utah/provisioning/baremetal/power.py	1970-01-01 00:00:00 +0000
+++ utah/provisioning/baremetal/power.py	2012-10-25 20:09:20 +0000
@@ -0,0 +1,60 @@
+"""
+Provide functions for a machine to interface with a power management system.
+Currently only the Sentry CDU in use in the Canonical labs is supported.
+"""
+
+import time
+
+from utah import config
+from utah.provisioning.baremetal.exceptions import UTAHBMProvisioningException
+
+
+class PowerMixin(object):
+    """
+    Provide power cycle commands for the Sentry CDU.
+    """
+    def powercommand(self):
+        try:
+            cmd = self.powercmd
+        except AttributeError:
+            try:
+                if self.power['power-type'] == 'sentryswitch_cdu':
+                    cmd = ['/usr/sbin/fence_cdu',
+                           '-a', self.power['power-address'],
+                           '-n', self.power['power-id'],
+                           '-l', self.power['power-user'],
+                           '-p', self.power['power-pass'],
+                           '-o']
+                else:
+                    raise UTAHProvisioningException('Power type ' +
+                                                    self.power['power-type'] +
+                                                    ' not supported')
+            except AttributeError:
+                raise UTAHBMProvisioningException('Power commands not defined'
+                                                  ' for this machine')
+        return cmd
+
+    def _start(self):
+        """
+        Use the machine's powercommand to start the machine.
+        """
+        self.logger.debug('Starting system')
+        self._runargs(self.powercommand() + ['on'])
+
+    def stop(self):
+        """
+        Use the machine's powercommand to stop the machine.
+        """
+        self.logger.debug('Stopping system')
+        self._runargs(self.powercommand() + ['off'])
+
+    def restart(self):
+        """
+        Stop the machine, wait powertimeout, then start it.
+        """
+        self.logger.info('Restarting system')
+        self.stop()
+        self.logger.debug('Waiting ' + str(config.powertimeout) + ' seconds')
+        time.sleep(config.powertimeout)
+        self._start()
+
 
=== modified file 'utah/provisioning/inventory/sqlite.py'
--- utah/provisioning/inventory/sqlite.py	2012-08-22 15:20:25 +0000
+++ utah/provisioning/inventory/sqlite.py	2012-10-25 20:09:20 +0000
@@ -83,6 +83,7 @@
     All columns other than machineid, name, and state are assumed to be
     arguments for cobbler system creation.
     """
+    # TODO: rename this
     def __init__(self, db='~/.utah-cobbler-inventory',
                  lockfile='~/.utah-cobbler-lock', *args, **kw):
         db = os.path.expanduser(db)
@@ -99,7 +100,7 @@
             raise UTAHProvisioningInventoryException('No machines in database')
         self.machines = []
 
-    def request(self, name=None, *args, **kw):
+    def request(self, machinetype=CobblerMachine, name=None, *args, **kw):
         query = 'SELECT * FROM machines'
         queryvars = []
         if name is not None:
@@ -121,7 +122,7 @@
                             "WHERE machineid=? AND state='available'",
                             [machineid]).rowcount
                     if update == 1:
-                        machine = CobblerMachine(*args, cargs=cargs,
+                        machine = machinetype(*args, cargs=cargs,
                                 inventory=self, name=name, **kw)
                         self.machines.append(machine)
                         return machine
 
=== modified file 'utah/provisioning/provisioning.py'
--- utah/provisioning/provisioning.py	2012-10-16 10:46:10 +0000
+++ utah/provisioning/provisioning.py	2012-10-25 20:09:20 +0000
@@ -19,7 +19,12 @@
 import apt.cache
 from glob import glob
 
+from utah.commandstr import commandstr
 from utah.iso import ISO
+from utah.orderedcollections import (
+        HashableDict,
+        OrderedSet,
+        )
 from utah.preseed import Preseed
 from utah.provisioning.exceptions import UTAHProvisioningException
 from utah.retry import retry
@@ -37,16 +42,17 @@
     A fully implemented subclass will provide all public methods except read,
     I.E.:
 
-    * provisioncheck, activecheck, getclientdeb, installclient
+    * provisioncheck, activecheck, getclientdeb, installclient, destroy
       (all these can be used from Machine or replaced.)
-    * destroy, stop, uploadfiles, downloadfiles, run
+    * stop, uploadfiles, downloadfiles, run
       (these must be implemented separately.)
     """
-    def __init__(self, arch=None, debug=False, directory=None, image=None,
-                 dlpercentincrement=1, initrd=None, installtype=None,
-                 kernel=None, machineid=None, machineuuid=None, name=None,
-                 new=False, prefix='utah', preseed=None, rewrite=None,
-                 series=None, template=None, xml=None):
+    def __init__(self, arch=None, clean=None, debug=False, directory=None,
+                 image=None, dlpercentincrement=1, initrd=None,
+                 installtype=None, kernel=None, machineid=None,
+                 machineuuid=None, name=None, new=False, prefix='utah',
+                 preseed=None, rewrite=None, series=None, template=None,
+                 xml=None):
         """
         Initialize the object representing the machine.
         One of these groups of arguments should be included:
@@ -60,6 +66,7 @@
                 reinstall a specific machine.
         The subclass is responsible for provisioning with these arguments.
         Other arguments:
+            clean: Enable cleanup functions.
             debug: Enable debug logging.
             directory: Where the machine's files go.  Should be persistent for
                 VMs, temporary is fine for installation-only files, like those
@@ -99,6 +106,15 @@
         self.series = series
         self.template = template
 
+        if clean is None:
+            self.clean = True
+        else:
+            self.clean = clean
+
+        self.cleanfiles = OrderedSet()
+        self.cleanfunctions = OrderedSet()
+        self.cleancommands = OrderedSet()
+
         self._namesetup(name)
         self._dirsetup(directory)
 
@@ -116,14 +132,21 @@
         self.active = False
         self._loggersetup()
 
+        if preseed is None:
+            preseed = '/etc/utah/default-preseed.cfg'
+
+        fileargs = ['preseed', 'xml', 'kernel', 'initrd']
+
         if image is None:
             self.image = None
-        else:
+        elif image.endswith('.iso'):
             self.image = ISO(image,
                     dlpercentincrement=self.dlpercentincrement,
                     logger=self.logger)
+        else:
+            fileargs.append('image')
 
-        for item in ['preseed', 'xml', 'kernel', 'initrd']:
+        for item in fileargs:
             # Ensure every file/url type argument is available locally
             arg = locals()[item]
             if arg is None:
@@ -159,6 +182,7 @@
         """
         Setup the machine's directory, automatically or using a specified one.
         """
+        # TODO: Move this to vm
         if directory is None:
             self.directory = os.path.join(config.vmpath, self.name)
         else:
@@ -340,7 +364,15 @@
         Should generally not be called directly outside of the class;
         provisioncheck() or activecheck() should be used.
         """
-        self._unimplemented(sys._getframe().f_code.co_name)
+        if not self.new:
+            try:
+                self.logger.debug('Trying to load existing machine')
+                self._load()
+                self.provisioned = True
+                return self.provisioned
+            except Exception as err:
+                self.logger.debug('Failed to load machine: ' + str(err))
+        self._create()
 
     def _create(self):
         """
@@ -364,7 +396,8 @@
         should be powered off if remote power management is available.
         Now returns False by default to work with multiple inheritance.
         """
-        return False
+        self.__del__()
+        del self
 
     def _load(self):
         """
@@ -470,11 +503,95 @@
             line = p.stdout.readline().strip()
             self.logger.debug(line)
         return p.returncode
+        
+    def _depcheck(self):
+        """
+        Check for dependencies that are in Recommends or Suggests.
+        None exist for the main class, but we implement it here to allow
+        super() to work.
+        """
+        pass
 
     def _unimplemented(self, method):
         raise UTAHProvisioningException(self.__class__.__name__ +
               ' attempted to call the ' + method +
               ' method of the base Machine class, which is not implemented')
+              
+    def cleanup(self):
+        """
+        Clean up temporary files and tear down other setup (VM, etc.).
+        If a subset is specified, only that subset will be cleaned.
+        This can be used to clean up install files after the install has started.
+        Files are removed first.
+        Functions are run second.
+        Commands are run third.
+        """
+        if self.clean:
+            self.logger.debug('Running cleanup')
+            for path in tuple(self.cleanfiles):
+                if os.path.islink(path):
+                    self.logger.debug('Removing link ' + path)
+                    os.unlink(path)
+                elif os.path.isfile(path):
+                    self.logger.debug('Changing permissions of '+ path)
+                    os.chmod(path, 0664)
+                    self.logger.debug('Removing file ' + path)
+                    os.unlink(path)
+                elif os.path.isdir(path):
+                # Cribbed from http://svn.python.org
+                # /projects/python/trunk/Mac/BuildScript/build-installer.py
+                    for dirpath, dirnames, filenames in os.walk(path):
+                        for name in (dirnames + filenames):
+                            absolute_name = os.path.join(dirpath, name)
+                            if not os.path.islink(absolute_name):
+                                self.logger.debug('Changing permissions of '
+                                                  + absolute_name)
+                                os.chmod(absolute_name, 0775)
+                    self.logger.debug('Recursively Removing directory '
+                                      + path)
+                    shutil.rmtree(path)
+                else:
+                    self.logger.debug(path + ' is not a link, file, or '
+                                      'directory; not removing')
+                self.cleanfiles.remove(path)
+            for function in tuple(self.cleanfunctions):
+                timeout, command, args, kw = function
+                self.logger.debug('Running: ' +
+                                  commandstr(command, *args, **kw))
+                utah.timeout.timeout(timeout, command, *args, **kw)
+                self.cleanfunctions.remove(function)
+            for command in tuple(self.cleancommands):
+                self._runargs(command)
+                self.cleancommands.remove(command)
+        else:
+            self.logger.debug('Not cleaning up')
+
+    def cleanfile(self, path):
+        """
+        Register a link, file, or directory to be cleaned later.
+        Links will be unlinked.
+        Files will have permissions changed to 664 and unlinked.
+        Directories will recursively have permissions changed to 775 (except
+        for links contained within) and then be recursively removed.
+        """
+        self.cleanfiles.add(path)
+
+    def cleanfunction(self, function, *args, **kw):
+        """
+        Register a function to be run on cleanup.
+        Functions are run through utah.timeout.timeout.
+        """
+        self.cleanfunctions.add((60, function, args, HashableDict(kw)))
+
+    def cleancommand(self, cmd):
+        """
+        Register a command to be run on cleanup.
+        Commands are run through Machine._runargs.
+        """
+        self.cleancommands.add(cmd)
+
+    def __del__(self):
+        self.cleanup()
 
 
 class SSHMixin(object):
@@ -1010,6 +1127,7 @@
         else:
             self.cmdline = boot
         self.boottimeout = config.boottimeout
+        # TODO: Refactor this into lists like BambooFeederMachine
         if self.rewrite == 'all':
             self.logger.info('Adding needed command line options')
             if self.installtype == 'desktop':
 
=== modified file 'utah/provisioning/vm/libvirtvm.py'
--- utah/provisioning/vm/libvirtvm.py	2012-10-16 10:46:10 +0000
+++ utah/provisioning/vm/libvirtvm.py	2012-10-25 20:09:20 +0000
@@ -349,8 +349,7 @@
     def __init__(self, arch=None, boot=None, diskbus=None, disksizes=None,
                  emulator=None, image=None, initrd=None, installtype=None,
                  kernel=None, machineid=None, macs=None, name=None,
-                 prefix='utah', preseed=None, series=None, xml=None,
-                 *args, **kw):
+                 prefix='utah', series=None, xml=None, *args, **kw):
         # Make sure that no other virtualization solutions are running
         # TODO: see if this is needed for qemu or just kvm
         process_checker = ProcessChecker()
@@ -373,15 +372,12 @@
             name = '-'.join([str(prefix), str(machineid)])
         else:
             autoname = False
-        if preseed is None:
-            preseed = '/etc/utah/default-preseed.cfg'
         if xml is None:
             xml = '/etc/utah/default-vm.xml'
         super(CustomVM, self).__init__(arch=arch, image=image, initrd=initrd,
                                        installtype=installtype, kernel=kernel,
                                        machineid=machineid, name=name,
-                                       preseed=preseed, series=series, xml=xml,
-                                       *args, **kw)
+                                       series=series, xml=xml, *args, **kw)
         # TODO: do a better job of separating installation
         # into _create rather than __init__
         if self.image is None: