Merge lp:~nuclearbob/utah/consolidate-scripts into lp:utah

[Max Brustkern]

This branch removes the scripts other than run_utah_tests.py and consolidates their functionality into run_utah_tests.py

It relies on the cleanup-cleanup branch to ensure that explicit destruction calls for machines are not necessary.

Principally, it moves command line handling back in run_utah_tests.py. The script is responsible for parsing the command line options and getting a Machine object (from an Inventory in all cases except --skip-provisioning) and then it hands that off to functions of run.py to run the tests and returns results.

Some try/catch stuff is simplified as well, since the number of possible paths should be reduced. I also cleaned up the handling of command line options/config options/defaults in a few cases, and moved some things around between the different types of Machine classes so that the arguments passed to the different Machine constructors could be more straightforward.

[Javier Collado (javier.collado)]

Documentation needs to be updated. In particular, the part in
`docs/source/conf.py` that takes care of the manpage generation of the scripts
that are now gone.

Aside from that, I think it would be a good idea to refactor the code in the
big if/elif/else to still have a function for each different provisioning case.

review: Needs Fixing

[Andy Doan (doanac)]

should we keep the original scripts and just print an error, so people don't think dpkg failed or something?

[Andy Doan (doanac)]

i agree about breaking out the if/else stuff.

Also it might be worth using context managers for the machine/inventory API's now that its in one spot. It could get rid of the destroy/del stuff?

[Max Brustkern (nuclearbob)]

I'd like to see different provisioning methods, whether those are functions or not, that basically only consist of getting the inventory and machine objects. Everything else I'd like to see handled the same way, either in run_utah_test.py or run.py. I'll work on cleaning up the documentation, and I think if we can land this proposal:
https://code.launchpad.net/~doanac/utah/run-cleanups/+merge/159879
That'll put me in better shape to make the changes we've talked about in this branch.

[Max Brustkern (nuclearbob)]

The documentation should be updated now. I'm pondering the best place for machine getting functions. Right now, each machine type only works with a single inventory type. Would it make sense to provide functions at the level of a Machine subclass or at an Inventory level that would parse some arguments and request a machine? Or maybe we should be pulling the Machine-specific stuff out of args to pass that in? I'm not sure.

[Javier Collado (javier.collado)]

There still a few .rst files under docs/source/man that should be removed.

If you try to build the documentation you'll see some error messages because
those .rst files include the .txt file that was generated from the script help
output to generate the man page.

Regarding what should be the way to get a machine, I think a function or a
class method in the inventory module would be a good fit with how the code
works now.

review: Needs Fixing

[Andy Doan (doanac)]

On 04/25/2013 12:32 PM, Max Brustkern wrote:

> https://code.launchpad.net/~nuclearbob/utah/consolidate-scripts/+merge/160963

> === modified file 'examples/run_utah_tests.py'

> def get_parser():

looking these two snippets
1)
> + parser.add_argument('-a', '--arch', metavar='ARCH',
> + choices=('i386', 'amd64', 'arm'),
> + help=('Architecture to use for VM creation '
> + '(%(choices)s)'))

2)
> + if args.arch:
> + arch = args.arch
> + else:
> + arch = config.arch

You could just set the default value in argparse to config.arch and save
4 lines of code.

> + if args.machinetype:
> + machinetype = args.machinetype
> + else:
> machinetype = config.machinetype
> - else:
> - machinetype = args.machinetype

you could probably do the same with this as well by assigning the
default in the argparse.

> + else:
> + kw = {'clean': (not args.no_destroy),
> + 'new': True,
> + }
> + for arg in ['arch',
> + 'boot',
> + 'debug',
> + 'image',
> + 'initrd',
> + 'kernel',
> + 'name',
> + 'preseed',
> + 'rewrite',
> + 'series',
> + ]:
> + kw[arg] = getattr(args, arg)

this can be done explicityline and save two lines of code doing
something like:

  kw['arch'] = args.arch

and i think it might look more readable.

> + kw['installtype'] = args.type
> + if machinetype == 'physical':
> + if arch and 'arm' in arch:
> + inventory = ManualBaremetalSQLiteInventory(
> + db=os.path.join('~', '.utah-bamboofeeder-inventory'),
> + lockfile=os.path.join('~', '.utah-bamboofeeder-lock'))
> + kw['machinetype'] = BambooFeederMachine
> + else:
> + inventory = ManualBaremetalSQLiteInventory()
> + kw['machinetype'] = CobblerMachine
> + else:
> + inventory = TinySQLiteInventory()
> + for arg in ['diskbus', 'emulator', 'xml']:
> + kw[arg] = getattr(args, arg)
> + kw['disksizes'] = args.gigabytes
> + machine = inventory.request(**kw)

also - i think I'd break the whole block of "if args.skip_provisioning"
to here. into a utility function called something like:

  def _get_machine(args)

> if __name__ == '__main__':
> try:
> - try:
> - if isinstance(config.jobtimeout, int):
> - timeout(config.jobtimeout, run_utah_tests)
> - else:
> - run_utah_tests()
> - except AttributeError:
> - run_utah_tests()
> - finally:
> - cleanup.run()
> + exitstatus = timeout(getattr(config, 'jobtimeout', None),
> + run_utah_tests)

why the "getattr" - can't we ensure this will always be available by
providing a default in config.py?

> === modified file 'utah/provisioning/provisioning.py'

> - self...

[Max Brustkern (nuclearbob)]

> On 04/25/2013 12:32 PM, Max Brustkern wrote:
>
> > https://code.launchpad.net/~nuclearbob/utah/consolidate-
> scripts/+merge/160963
>
> > === modified file 'examples/run_utah_tests.py'
>
> > def get_parser():
>
> looking these two snippets
> 1)
> > + parser.add_argument('-a', '--arch', metavar='ARCH',
> > + choices=('i386', 'amd64', 'arm'),
> > + help=('Architecture to use for VM creation '
> > + '(%(choices)s)'))
>
> 2)
> > + if args.arch:
> > + arch = args.arch
> > + else:
> > + arch = config.arch
>
> You could just set the default value in argparse to config.arch and save
> 4 lines of code.
>
There used to be some reason that didn't work, but I can't think of why now, so I'll try it.
> > + if args.machinetype:
> > + machinetype = args.machinetype
> > + else:
> > machinetype = config.machinetype
> > - else:
> > - machinetype = args.machinetype
>
> you could probably do the same with this as well by assigning the
> default in the argparse.
>
Ditto.
> > + else:
> > + kw = {'clean': (not args.no_destroy),
> > + 'new': True,
> > + }
> > + for arg in ['arch',
> > + 'boot',
> > + 'debug',
> > + 'image',
> > + 'initrd',
> > + 'kernel',
> > + 'name',
> > + 'preseed',
> > + 'rewrite',
> > + 'series',
> > + ]:
> > + kw[arg] = getattr(args, arg)
>
> this can be done explicityline and save two lines of code doing
> something like:
>
> kw['arch'] = args.arch
>
> and i think it might look more readable.
>
I'm interested in what everybody else thinks of this, since I find the repetition very unpleasant to look at. Every time I see it, I think "we could just do that to a whole list of things and then add things to the list and I'll never make an error copying and pasting the code."
> > + kw['installtype'] = args.type
> > + if machinetype == 'physical':
> > + if arch and 'arm' in arch:
> > + inventory = ManualBaremetalSQLiteInventory(
> > + db=os.path.join('~', '.utah-bamboofeeder-inventory'),
> > + lockfile=os.path.join('~', '.utah-bamboofeeder-lock'))
> > + kw['machinetype'] = BambooFeederMachine
> > + else:
> > + inventory = ManualBaremetalSQLiteInventory()
> > + kw['machinetype'] = CobblerMachine
> > + else:
> > + inventory = TinySQLiteInventory()
> > + for arg in ['diskbus', 'emulator', 'xml']:
> > + kw[arg] = getattr(args, arg)
> > + kw['disksizes'] = args.gigabytes
> > + machine = inventory.request(**kw)
>
> also - i think I'd break the whole block of "if args.skip_provisioning"
> to here. into a utility function called something like:
>
> def _get_machine(args)
>
That seems reasonable. I didn't do that initially because I didn't think we'd be using that function a...

[Andy Doan (doanac)]

On 04/25/2013 05:39 PM, Max Brustkern wrote:
>>> + else:
>>> > >+ kw = {'clean': (not args.no_destroy),
>>> > >+ 'new': True,
>>> > >+ }
>>> > >+ for arg in ['arch',
>>> > >+ 'boot',
>>> > >+ 'debug',
>>> > >+ 'image',
>>> > >+ 'initrd',
>>> > >+ 'kernel',
>>> > >+ 'name',
>>> > >+ 'preseed',
>>> > >+ 'rewrite',
>>> > >+ 'series',
>>> > >+ ]:
>>> > >+ kw[arg] = getattr(args, arg)
>> >
>> >this can be done explicityline and save two lines of code doing
>> >something like:
>> >
>> > kw['arch'] = args.arch
>> >
>> >and i think it might look more readable.
>> >
> I'm interested in what everybody else thinks of this, since I find the repetition very unpleasant to look at. Every time I see it, I think "we could just do that to a whole list of things and then add things to the list and I'll never make an error copying and pasting the code."

Two things:

1) fair enough - this is mostly subjective so we should decide as a team

2) I think this is actually more of a symptom of a larger problem.
Having a constructor that takes in 18 parameters is a bit extreme. Its
possible what I'm going to say is beyond the scope of this MP, but think
about the two cases we are using the "attr" hacks:

a) to set parameters in the call to the constructor
b) to take the parameters from the constructor and make them member
variables.

Now also remember the fact that we default things not provided to args
with a value from config.py.

I can't help but wonder if we shouldn't just pass "args" to the
constructor instead. I don't really like the idea of using "args", but
it feels cleaner than insane constructors.

Doing that would clean up my complain about run_utah_tests.py. You could
go extreme and then just assign self.args = args in Machine.__init__.
That feels a little crazy - and would probably turn into a fairly
big/error-prone refactor.

[Andy Doan (doanac)]

On 04/25/2013 05:39 PM, Max Brustkern wrote:
>>> === modified file 'utah/run.py'
>> >
>>> > >- try:
>>> > >- shutil.copyfile(machine.finalpreseed, p)
>>> > >- except (IOError, shutil.Error) as err:
>>> > >- logging.warning('Failed to copy preseed file: %s', err)
>>> > >+ if machine.hasattr('finalpreseed'):
>> >
>> >do we really need hasattr instead of just doing machine.finalpreseed?
> I don't like explicitly checking for ProvisionedMachine, which is what we did before. I don't think run.py should know or care about what Machine subclass it's using. We could just do a try and catch the AttributeError, but that seems like it could open us up to other problems.

I didn't like the ProvisionedMachine check either and I see your point
now. I say keep it as is.

[Max Brustkern (nuclearbob)]

> On 04/25/2013 05:39 PM, Max Brustkern wrote:
> ...
> Two things:
>
> 1) fair enough - this is mostly subjective so we should decide as a team
>
Looks like your idea is preferred, so I implemented that.
> 2) I think this is actually more of a symptom of a larger problem.
> Having a constructor that takes in 18 parameters is a bit extreme. Its
> possible what I'm going to say is beyond the scope of this MP, but think
> about the two cases we are using the "attr" hacks:
>
> a) to set parameters in the call to the constructor
> b) to take the parameters from the constructor and make them member
> variables.
>
> Now also remember the fact that we default things not provided to args
> with a value from config.py.
>
> I can't help but wonder if we shouldn't just pass "args" to the
> constructor instead. I don't really like the idea of using "args", but
> it feels cleaner than insane constructors.
>
> Doing that would clean up my complain about run_utah_tests.py. You could
> go extreme and then just assign self.args = args in Machine.__init__.
> That feels a little crazy - and would probably turn into a fairly
> big/error-prone refactor.
I think this larger problem is better solved by separating installation, and probably coming up with a completely new interface. I think as part of that sorting out which arguments are used for installation, machine management, and test running might serve us well and allow us to pass those around in ways that make more sense.

[Andy Doan (doanac)]

On 04/26/2013 03:02 PM, Max Brustkern wrote:
> I think this larger problem is better solved by separating installation, and probably coming up with a completely new interface. I think as part of that sorting out which arguments are used for installation, machine management, and test running might serve us well and allow us to pass those around in ways that make more sense.

+1 - I hesitated to say that because its even more work :)

[Javier Collado (javier.collado)]

There are a few lines that have to be removed in:
debian/utah.manpages

to avoid generating manual pages for the scripts that have been removed.

review: Needs Fixing

[lp:~nuclearbob/utah/consolidate-scripts]

878. By Javier Collado on 2013-04-30

Merged changes to fix DefaultValidator and updated schemas to draftv4 (LP: #1165175)

Source branch: lp:~javier.collado/utah/bug1165175

879. By Javier Collado on 2013-04-30

Merged changes to refactor cleanup for the machine class

Source branch: lp:~nuclearbob/utah/cleanup-cleanup

[Javier Collado (javier.collado)]

The exception handling in run_utah_tests.py has been updated so that the cleanup is executed after all exceptions are handled. The way it was written before, exceptions were handled just after the cleanup to make sure that error messages were printed at the bottom of the log file so that it was easy to know the error cause without having to read the log backwards and skip all the cleanup. I think I prefer that unless there's some other thing I'm missing. Please let me know your thoughts on this.

Aside from this, I see a few new default values have been added to command line parameters. Please consider adding those values to the help string via %(default)s, so that the user knows that values when passing -h/--help to run_utah_tests.py.

[Max Brustkern (nuclearbob)]

You have a good point about the finally for the cleanup. Does this look better to you:
if __name__ == '__main__':
    try:
        try:
            exitstatus = timeout(config.jobtimeout, run_utah_tests)
        finally:
            cleanup.run()
    except UTAHTimeout as exception:
        sys.stderr.write('UTAH timeout: {}\n'.format(exception.message))
        sys.exit(ReturnCodes.TIMEOUT_ERROR)
    except UTAHException as error:
        sys.stderr.write('UTAH exception: {}\n'.format(error))
        sys.exit(ReturnCodes.UTAH_EXCEPTION_ERROR)
    except Exception:
        sys.stderr.write('Unhandled error in UTAH:\n{}\n'
                         .format(''.join(format_exception(*sys.exc_info()))))
        sys.exit(ReturnCodes.UNHANDLED_ERROR)
    sys.exit(exitstatus)

As far as the default values, do you think I should add all of them, or just the ones that aren't inherited from config? It seems weird to have a default potentially dependent on environment variables, but since that's how we're processing things it makes sense.

[Javier Collado (javier.collado)]

Yes, that would look good to me.

Regarding default values, I was thinking about the ones that were newly added, but it's true that to be consistent lot of others default could be added as well. Maybe we should skip this for now, since I understand is out of the scope of this merge request.

[lp:~nuclearbob/utah/consolidate-scripts]

918. By Max Brustkern on 2013-05-01

Moved cleanup before exception handling

[Javier Collado (javier.collado)]

When passing the -i/--image argument through the command line, I see that utah is trying to download the mini image instead of using the one that was passed as an argument.

review: Needs Fixing

[lp:~nuclearbob/utah/consolidate-scripts]

919. By Max Brustkern on 2013-05-01

Updated help strings

[Javier Collado (javier.collado)]

Example output:
$ PYTHONPATH=. examples/run_utah_tests.py -i ~/download/distro/raring/raring-server-i386.iso utah/client/examples/pass.run -d -n
2013-05-01 11:02:30,082 root INFO: UTAH version: dev-r881~raring
2013-05-01 11:02:30,083 root DEBUG: Executing SQL statement: CREATE TABLE IF NOT EXISTS machines(machineid INTEGER PRIMARY KEY, state TEXT), [
]
2013-05-01 11:02:30,083 root DEBUG: Executing SQL statement: INSERT INTO machines (state) VALUES ('provisioned'), []
2013-05-01 11:02:30,259 utah-22 INFO: Attempting to retrieve raring-mini-amd64.iso
2013-05-01 11:02:30,266 utah-22 INFO: Download attempt 0
2013-05-01 11:02:30,284 utah-22 DEBUG: Server md5 is 69c919c499068b1681303130093f21cc
2013-05-01 11:02:30,284 utah-22 INFO: Attempting to download http://archive.ubuntu.com/ubuntu/dists/raring/main/installer-amd64/current/images
/./netboot/mini.iso
2013-05-01 11:02:30,320 utah-22 INFO: File 0% downloaded
2013-05-01 11:02:30,321 utah-22 DEBUG: 0 read, 0% of 36700160 total
2013-05-01 11:02:30,324 utah-22 DEBUG: 8192 read, 0% of 36700160 total

[Max Brustkern (nuclearbob)]

I moved cleanup and updated the help strings. I'll sort out the image thing, thanks for finding that.

[lp:~nuclearbob/utah/consolidate-scripts]

920. By Max Brustkern on 2013-05-01

Fixed image handling typos

[Max Brustkern (nuclearbob)]

Fixed some dumb typos on the image thing.

review: Needs Resubmitting

[Javier Collado (javier.collado)]

When trying to provision a VM, I got the following traceback:

Unhandled error in UTAH:
Traceback (most recent call last):
  File "examples/run_utah_tests.py", line 216, in <module>
    exitstatus = timeout(config.jobtimeout, run_utah_tests)
  File "utah/timeout.py", line 65, in timeout
    return command(*args, **kw)
  File "examples/run_utah_tests.py", line 205, in run_utah_tests
    exitstatus, locallogs = run_tests(args, _get_machine(args))
  File "examples/run_utah_tests.py", line 187, in _get_machine
    machine = inventory.request(**kw)
  File "utah/provisioning/vm/vm.py", line 570, in request
    return machinetype(machineid=machineid, *args, **kw)
  File "utah/provisioning/vm/vm.py", line 173, in __init__
    self._namesetup()
  File "utah/provisioning/provisioning.py", line 208, in _namesetup
    self.name = self._makename()
  File "utah/provisioning/provisioning.py", line 238, in _makename
    machineid = str(self.machineid)
AttributeError: 'CustomVM' object has no attribute 'machineid'

review: Needs Fixing

[lp:~nuclearbob/utah/consolidate-scripts]

921. By Max Brustkern on 2013-05-01

Fixed machineid not being set

922. By Max Brustkern on 2013-05-01

Fixed failure to chmod preseed when it doesn't exist, and fixed self tests

[Javier Collado (javier.collado)]

Working fine for me now. Thanks.

review: Approve

=== modified file 'debian/changelog'
--- debian/changelog	2013-05-01 17:37:58 +0000
+++ debian/changelog	2013-05-01 21:24:29 +0000
@@ -14,9 +14,20 @@
   * Make rsyslog timeout error message clearer (LP: #1169846)
   * Write to log UTAH version as soon as possible (LP: #1152216)
   * Fixed help output to use just one runlist (LP: #1112597)
+<<<<<<< TREE
   * Setup rc.local to use output passed via command line (LP: #1155615)
+=======
+  * Combined current provisioning functions into one script and removed
+    old scripts
+  * Set permissions when copying preseed so overwrite works (LP: #1130901)
+  * Enabled reading power command from config (LP: #1153735)
+>>>>>>> MERGE-SOURCE
 
+<<<<<<< TREE
  -- Max Brustkern <[e-mail]>  Fri, 26 Apr 2013 13:58:25 -0400
+=======
+ -- Max Brustkern <[e-mail]>  Thu, 25 Apr 2013 11:02:23 -0400
+>>>>>>> MERGE-SOURCE
 
 utah (0.10ubuntu1) UNRELEASED; urgency=low
 

=== modified file 'debian/utah.manpages'
--- debian/utah.manpages	2012-08-31 15:36:31 +0000
+++ debian/utah.manpages	2013-05-01 21:24:29 +0000
@@ -1,4 +1,1 @@
-docs/build/man/run_install_test.py.1
-docs/build/man/run_test_cobbler.py.1
-docs/build/man/run_test_vm.py.1
 docs/build/man/run_utah_tests.py.1

=== modified file 'docs/source/conf.py'
--- docs/source/conf.py	2013-04-18 10:46:00 +0000
+++ docs/source/conf.py	2013-05-01 21:24:29 +0000
@@ -148,8 +148,7 @@
 
     return description, options, epilog
 
-module_names = ('client', 'run_install_test', 'run_test_cobbler',
-                'run_test_vm', 'run_utah_tests')
+module_names = ('client', 'run_utah_tests')
 for module_name in module_names:
     module = __import__(module_name)
     parser = module.get_parser()
@@ -367,13 +366,6 @@
 man_pages = [
     ('man/utah', 'utah', u'UTAH client test runner',
      [u'Canonical Ltd'], 1),
-    ('man/run_install_test.py', 'run_install_test.py',
-     u'UTAH server test runner (provisioning)', [u'Canonical Ltd'], 1),
-    ('man/run_test_cobbler.py', 'run_test_cobbler.py',
-     u'UTAH server test runner (physical hardware)', [u'Canonical Ltd'], 1),
-    ('man/run_test_vm.py', 'run_test_vm.py',
-     u'UTAH server test runner (virtual hardware)',
-     [u'Canonical Ltd'], 1),
     ('man/run_utah_tests.py', 'run_utah_tests.py',
      u'UTAH server test runner (any hardware)', [u'Canonical Ltd'], 1),
 ]

=== removed file 'docs/source/man/run_install_test.py.rst'
--- docs/source/man/run_install_test.py.rst	2012-09-05 11:30:16 +0000
+++ docs/source/man/run_install_test.py.rst	1970-01-01 00:00:00 +0000
@@ -1,26 +0,0 @@
-:orphan:
-
-run_install_test.py manual page
-===============================
-
-Synopsis
---------
-**run_install_test.py** [options] [runlist ...]
-
-Description
------------
-.. include:: run_install_test_description.txt
-
-Options
--------
-.. include:: run_install_test_options.txt
-
-Examples
---------
-.. include:: run_install_test_epilog.txt
-
-See also
---------
-:manpage:`run_test_cobbler.py(1)`
-:manpage:`run_test_vm.py(1)`
-:manpage:`run_utah_tests.py(1)`

=== removed file 'docs/source/man/run_test_cobbler.py.rst'
--- docs/source/man/run_test_cobbler.py.rst	2012-09-05 11:30:16 +0000
+++ docs/source/man/run_test_cobbler.py.rst	1970-01-01 00:00:00 +0000
@@ -1,26 +0,0 @@
-:orphan:
-
-run_test_cobbler.py manual page
-===============================
-
-Synopsis
---------
-**run_test_cobbler.py** [options] [runlist ...]
-
-Description
------------
-.. include:: run_test_cobbler_description.txt
-
-Options
--------
-.. include:: run_test_cobbler_options.txt
-
-Examples
---------
-.. include:: run_test_cobbler_epilog.txt
-
-See also
---------
-:manpage:`run_install_test.py(1)`
-:manpage:`run_test_vm.py(1)`
-:manpage:`run_utah_tests.py(1)`

=== removed file 'docs/source/man/run_test_vm.py.rst'
--- docs/source/man/run_test_vm.py.rst	2012-09-05 11:30:16 +0000
+++ docs/source/man/run_test_vm.py.rst	1970-01-01 00:00:00 +0000
@@ -1,26 +0,0 @@
-:orphan:
-
-run_test_vm.py manual page
-==========================
-
-Synopsis
---------
-**run_test_vm.py** [options] [runlist ...]
-
-Description
------------
-.. include:: run_test_vm_description.txt
-
-Options
--------
-.. include:: run_test_vm_options.txt
-
-Examples
---------
-.. include:: run_test_vm_epilog.txt
-
-See also
---------
-:manpage:`run_install_test.py(1)`
-:manpage:`run_test_cobbler.py(1)`
-:manpage:`run_utah_tests.py(1)`

=== removed file 'examples/run_install_test.py'
--- examples/run_install_test.py	2013-04-18 14:15:12 +0000
+++ examples/run_install_test.py	1970-01-01 00:00:00 +0000
@@ -1,114 +0,0 @@
-#!/usr/bin/env python
-
-# Ubuntu Testing Automation Harness
-# Copyright 2012 Canonical Ltd.
-
-# This program is free software: you can redistribute it and/or modify it
-# under the terms of the GNU General Public License version 3, as published
-# by the Free Software Foundation.
-
-# This program is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranties of
-# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
-# PURPOSE.  See the GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License along
-# with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-"""Create a VM and run a test."""
-
-
-import argparse
-import logging
-import sys
-
-from utah.cleanup import cleanup
-from utah.exceptions import UTAHException
-from utah.group import check_user_group, print_group_error_message
-from utah.provisioning.vm.vm import CustomVM, TinySQLiteInventory
-from utah.run import (
-    common_arguments,
-    custom_arguments,
-    file_arguments,
-    run_tests,
-    virtual_arguments,
-    configure_logging,
-    ReturnCodes,
-)
-
-
-def get_parser():
-    parser = argparse.ArgumentParser(
-        description=('Provision a machine '
-                     'and run one or more UTAH runlists there.'),
-        epilog=("For example:\n"
-                "Provision a VM using a precise server image "
-                "with i386 architecture and run the two given runlists\n"
-                "\t%(prog)s -s precise -t server -a i386 \\\n"
-                "\t\t/usr/share/utah/client/examples/master.run \\\n"
-                "\t\t'http://people.canonical.com/~max/max_test.run'"),
-        formatter_class=argparse.RawDescriptionHelpFormatter)
-    parser = common_arguments(parser)
-    parser = custom_arguments(parser)
-    parser = file_arguments(parser)
-    parser = virtual_arguments(parser)
-    return parser
-
-
-def run_install_test(args=None):
-    if args is None:
-        args = get_parser().parse_args()
-
-    if not check_user_group():
-        print_group_error_message(__file__)
-        sys.exit(ReturnCodes.GROUP_ERROR)
-
-    locallogs = []
-    exitstatus = ReturnCodes.SUCCESS
-    machine = None
-
-    configure_logging(args.debug)
-
-    try:
-        inventory = TinySQLiteInventory()
-        machine = inventory.request(
-            CustomVM,
-            arch=args.arch, boot=args.boot, clean=(not args.no_destroy),
-            debug=args.debug, diskbus=args.diskbus,
-            disksizes=args.gigabytes, dlpercentincrement=10,
-            emulator=args.emulator, image=args.image, initrd=args.initrd,
-            installtype=args.type, kernel=args.kernel, new=True,
-            preseed=args.preseed, rewrite=args.rewrite,
-            series=args.series, xml=args.xml)
-        exitstatus, locallogs = run_tests(args, machine)
-    finally:
-        if not args.no_destroy and machine is not None:
-            try:
-                machine.destroy()
-            except UTAHException as error:
-                sys.stderr.write('Failed to destroy machine: ' + str(error))
-            finally:
-                try:
-                    inventory.destroy(machine.machineid)
-                except UTAHException as error:
-                    sys.stderr.write('Failed to update inventory: '
-                                     + str(error))
-                finally:
-                    del machine
-        if len(locallogs) != 0:
-            print('Test logs copied to the following files:')
-            print("\t" + "\n\t".join(locallogs))
-
-    sys.exit(exitstatus)
-
-
-if __name__ == '__main__':
-    logging.warning('This script is deprecated; please use run_utah_tests.py')
-    logging.warning('The same command line arguments should work there, i.e.:')
-    argv = list(sys.argv)
-    argv[0] = 'run_utah_tests.py'
-    logging.warning(' '.join(argv))
-    try:
-        run_install_test()
-    finally:
-        cleanup.run()

=== removed file 'examples/run_test_bamboo_feeder.py'
--- examples/run_test_bamboo_feeder.py	2013-04-10 15:34:49 +0000
+++ examples/run_test_bamboo_feeder.py	1970-01-01 00:00:00 +0000
@@ -1,119 +0,0 @@
-#!/usr/bin/env python
-
-# Ubuntu Testing Automation Harness
-# Copyright 2012 Canonical Ltd.
-
-# This program is free software: you can redistribute it and/or modify it
-# under the terms of the GNU General Public License version 3, as published
-# by the Free Software Foundation.
-
-# This program is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranties of
-# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
-# PURPOSE.  See the GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License along
-# with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-"""Provision a panda board in a bamboo-feeder setup and run a test."""
-
-
-import argparse
-import os
-import sys
-import logging
-
-from utah.cleanup import cleanup
-from utah.exceptions import UTAHException
-from utah.group import check_user_group, print_group_error_message
-from utah.provisioning.baremetal.bamboofeeder import BambooFeederMachine
-from utah.provisioning.baremetal.inventory import \
-    ManualBaremetalSQLiteInventory
-from utah.run import (
-    common_arguments,
-    custom_arguments,
-    name_argument,
-    run_tests,
-    configure_logging,
-    ReturnCodes,
-)
-
-
-def get_parser():
-    parser = argparse.ArgumentParser(
-        description=('Provision a pandaboard in a bamboo-feeder setup '
-                     'and run one or more UTAH runlists there.'),
-        epilog=("For example:\n"
-                "Provision a machine using a precise server image "
-                "with i386 architecture and run the two given runlists\n"
-                "\t%(prog)s -s precise -t server -a i386 \\\n"
-                "\t\t/usr/share/utah/client/examples/master.run \\\n"
-                "\t\t'http://people.canonical.com/~max/max_test.run'"),
-        formatter_class=argparse.RawDescriptionHelpFormatter)
-    parser = common_arguments(parser)
-    parser = custom_arguments(parser)
-    parser = name_argument(parser)
-    return parser
-
-
-def run_test_bamboo_feeder(args=None):
-    if args is None:
-        args = get_parser().parse_args()
-
-    if not check_user_group():
-        print_group_error_message(__file__)
-        sys.exit(ReturnCodes.GROUP_ERROR)
-
-    locallogs = []
-    exitstatus = ReturnCodes.SUCCESS
-    machine = None
-
-    configure_logging(args.debug)
-
-    try:
-        inventory = ManualBaremetalSQLiteInventory(
-            db=os.path.join('~', '.utah-bamboofeeder-inventory'),
-            lockfile=os.path.join('~', '.utah-bamboofeeder-lock'))
-        kw = {}
-        for arg in ['boot', 'image', 'preseed']:
-            value = getattr(args, arg)
-            if value is not None:
-                kw[arg] = value
-        machine = inventory.request(machinetype=BambooFeederMachine,
-                                    clean=(not args.no_destroy),
-                                    debug=args.debug, new=True,
-                                    dlpercentincrement=10, **kw)
-        exitstatus, locallogs = run_tests(args, machine)
-
-    except UTAHException as error:
-        mesg = 'Exception: ' + str(error)
-        try:
-            logging.error(mesg)
-        except (AttributeError, NameError):
-            sys.stderr.write(mesg)
-        exitstatus = ReturnCodes.UTAH_EXCEPTION_ERROR
-    except Exception:
-        logging.exception('Unhandled error in UTAH')
-        exitstatus = ReturnCodes.UNHANDLED_ERROR
-    finally:
-        if machine is not None:
-            try:
-                machine.destroy()
-            except UTAHException as error:
-                sys.stderr.write('Failed to destroy machine: ' + str(error))
-
-        if len(locallogs) != 0:
-            print('Test logs copied to the following files:')
-            print("\t" + "\n\t".join(locallogs))
-
-    sys.exit(exitstatus)
-
-
-if __name__ == '__main__':
-    logging.warning('This script is thought to be broken')
-    logging.warning('If it is working, please report it to the UTAH team via '
-                    '[e-mail]')
-    try:
-        run_test_bamboo_feeder()
-    finally:
-        cleanup.run()

=== removed file 'examples/run_test_cobbler.py'
--- examples/run_test_cobbler.py	2013-04-10 15:34:49 +0000
+++ examples/run_test_cobbler.py	1970-01-01 00:00:00 +0000
@@ -1,127 +0,0 @@
-#!/usr/bin/env python
-
-# Ubuntu Testing Automation Harness
-# Copyright 2012 Canonical Ltd.
-
-# This program is free software: you can redistribute it and/or modify it
-# under the terms of the GNU General Public License version 3, as published
-# by the Free Software Foundation.
-
-# This program is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranties of
-# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
-# PURPOSE.  See the GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License along
-# with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-"""Provision a machine with cobbler and run a test."""
-
-
-import argparse
-import sys
-import logging
-
-from utah.cleanup import cleanup
-from utah.exceptions import UTAHException
-from utah.group import check_user_group, print_group_error_message
-from utah.provisioning.baremetal.cobbler import CobblerMachine
-from utah.provisioning.baremetal.inventory import \
-    ManualBaremetalSQLiteInventory
-from utah.run import (
-    common_arguments,
-    custom_arguments,
-    name_argument,
-    run_tests,
-    configure_logging,
-    ReturnCodes,
-)
-
-
-def get_parser():
-    parser = argparse.ArgumentParser(
-        description=('Provision a machine using cobbler '
-                     'and run one or more UTAH runlists there.'),
-        epilog=("For example:\n"
-                "Provision a machine using a precise server image "
-                "with i386 architecture and run the two given runlists\n"
-                "\t%(prog)s -s precise -t server -a i386 \\\n"
-                "\t\t/usr/share/utah/client/examples/master.run \\\n"
-                "\t\t'http://people.canonical.com/~max/max_test.run'"),
-        formatter_class=argparse.RawDescriptionHelpFormatter)
-    parser = common_arguments(parser)
-    parser = custom_arguments(parser)
-    parser = name_argument(parser)
-    return parser
-
-
-def run_test_cobbler(args=None):
-    if args is None:
-        args = get_parser().parse_args()
-
-    if not check_user_group():
-        print_group_error_message(__file__)
-        sys.exit(ReturnCodes.GROUP_ERROR)
-
-    locallogs = []
-    exitstatus = ReturnCodes.SUCCESS
-    machine = None
-
-    configure_logging(args.debug)
-
-    try:
-        inventory = ManualBaremetalSQLiteInventory()
-        kw = {}
-        for arg in ['arch',
-                    'boot',
-                    'image',
-                    'preseed',
-                    'rewrite',
-                    'series',
-                    ]:
-            value = getattr(args, arg)
-            if value is not None:
-                kw[arg] = value
-        if getattr(args, 'type') is not None:
-            kw['installtype'] = args.type
-        machine = inventory.request(CobblerMachine,
-                                    clean=(not args.no_destroy),
-                                    debug=args.debug, dlpercentincrement=10,
-                                    name=args.name, new=True, **kw)
-        exitstatus, locallogs = run_tests(args, machine)
-
-    except UTAHException as error:
-        mesg = 'Exception: ' + str(error)
-        try:
-            logging.error(mesg)
-        except (AttributeError, NameError):
-            sys.stderr.write(mesg)
-        exitstatus = ReturnCodes.UTAH_EXCEPTION_ERROR
-    except Exception:
-        logging.exception('Unhandled error in UTAH')
-        exitstatus = ReturnCodes.UNHANDLED_ERROR
-    finally:
-        if machine is not None:
-            try:
-                machine.destroy()
-            except UTAHException as error:
-                sys.stderr.write('Failed to destroy machine: ' + str(error))
-
-        if len(locallogs) != 0:
-            print('Test logs copied to the following files:')
-            print("\t" + "\n\t".join(locallogs))
-
-    sys.exit(exitstatus)
-
-
-if __name__ == '__main__':
-    logging.warning('This script is deprecated; please use run_utah_tests.py')
-    logging.warning('Add the -m physical argument to your existing list, i.e.:')
-    argv = list(sys.argv)
-    argv[0] = 'run_utah_tests.py'
-    argv.extend(['-m', 'physical'])
-    logging.warning(' '.join(argv))
-    try:
-        run_test_cobbler()
-    finally:
-        cleanup.run()

=== removed file 'examples/run_test_vm.py'
--- examples/run_test_vm.py	2013-04-10 15:34:49 +0000
+++ examples/run_test_vm.py	1970-01-01 00:00:00 +0000
@@ -1,115 +0,0 @@
-#!/usr/bin/env python
-
-# Ubuntu Testing Automation Harness
-# Copyright 2012 Canonical Ltd.
-
-# This program is free software: you can redistribute it and/or modify it
-# under the terms of the GNU General Public License version 3, as published
-# by the Free Software Foundation.
-
-# This program is distributed in the hope that it will be useful, but
-# WITHOUT ANY WARRANTY; without even the implied warranties of
-# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
-# PURPOSE.  See the GNU General Public License for more details.
-
-# You should have received a copy of the GNU General Public License along
-# with this program.  If not, see <http://www.gnu.org/licenses/>.
-
-"""Create a VM and run a test."""
-
-
-import argparse
-import logging
-import sys
-
-from utah.cleanup import cleanup
-from utah.exceptions import UTAHException
-from utah.group import check_user_group, print_group_error_message
-from utah.provisioning.vm.vm import TinySQLiteInventory
-from utah.run import (
-    common_arguments,
-    run_tests,
-    configure_logging,
-    ReturnCodes,
-)
-
-
-def get_parser():
-    parser = argparse.ArgumentParser(
-        description=('Create a virtual machine '
-                     'and run a UTAH runlist there.'),
-        epilog=("For example:\n"
-                "Provision a VM using a precise server image "
-                "with i386 architecture and run the two given runlists\n"
-                "\t%(prog)s -s precise -t server -a i386 \\\n"
-                "\t\t/usr/share/utah/client/examples/master.run \\\n"
-                "\t\t'http://people.canonical.com/~max/max_test.run'"),
-        formatter_class=argparse.RawDescriptionHelpFormatter)
-    return common_arguments(parser)
-
-
-def run_test_vm(args=None):
-    if args is None:
-        args = get_parser().parse_args()
-
-    if not check_user_group():
-        print_group_error_message(__file__)
-        sys.exit(ReturnCodes.GROUP_ERROR)
-
-    locallogs = []
-    exitstatus = ReturnCodes.SUCCESS
-    machine = None
-
-    configure_logging(args.debug)
-
-    try:
-        inventory = TinySQLiteInventory()
-        kw = {}
-        for arg in ['arch', 'series']:
-            value = getattr(args, arg)
-            if value is not None:
-                kw[arg] = value
-        if args.type is not None:
-            kw.update([('installtype', args.type)])
-        machine = inventory.request(clean=(not args.no_destroy),
-                                    debug=args.debug, new=True,
-                                    dlpercentincrement=10, **kw)
-        exitstatus, locallogs = run_tests(args, machine)
-
-    except UTAHException as error:
-        sys.stderr.write('Exception: ' + str(error))
-        exitstatus = ReturnCodes.UTAH_EXCEPTION_ERROR
-    except Exception:
-        logging.exception('Unhandled error in UTAH')
-        exitstatus = ReturnCodes.UNHANDLED_ERROR
-    finally:
-        if not args.no_destroy and machine is not None:
-            try:
-                machine.destroy()
-            except UTAHException as error:
-                sys.stderr.write('Failed to destroy machine: ' + str(error))
-            finally:
-                try:
-                    inventory.destroy(machine.machineid)
-                except UTAHException as error:
-                    sys.stderr.write('Failed to update inventory: '
-                                     + str(error))
-                finally:
-                    del machine
-        if len(locallogs) != 0:
-            print('Test logs copied to the following files:')
-            print("\t" + "\n\t".join(locallogs))
-
-        sys.exit(exitstatus)
-
-
-if __name__ == '__main__':
-    logging.warning('This script is deprecated; please use run_utah_tests.py')
-    logging.warning('The same command line arguments should work there, i.e.:')
-    argv = list(sys.argv)
-    argv[0] = 'run_utah_tests.py'
-    logging.warning(' '.join(argv))
-    try:
-        run_test_vm()
-    finally:
-        cleanup.run()

=== modified file 'examples/run_utah_tests.py'
--- examples/run_utah_tests.py	2013-04-25 12:29:33 +0000
+++ examples/run_utah_tests.py	2013-05-01 21:24:29 +0000
@@ -20,27 +20,31 @@
 
 import argparse
 import logging
+import os
 import sys
+
 from traceback import format_exception
 
 import utah
+
 from utah import config
 from utah.cleanup import cleanup
+from utah.exceptions import UTAHException
 from utah.group import check_user_group, print_group_error_message
+from utah.provisioning.baremetal.bamboofeeder import BambooFeederMachine
+from utah.provisioning.baremetal.cobbler import CobblerMachine
+from utah.provisioning.baremetal.inventory import \
+    ManualBaremetalSQLiteInventory
+from utah.provisioning.ssh import ProvisionedMachine
+from utah.provisioning.vm.vm import TinySQLiteInventory
 from utah.run import (
-    common_arguments,
-    custom_arguments,
-    file_arguments,
-    name_argument,
-    virtual_arguments,
     configure_logging,
+    master_runlist_argument,
     run_tests,
     ReturnCodes,
 )
 from utah.timeout import timeout, UTAHTimeout
-from run_install_test import run_install_test
-from utah.provisioning.ssh import ProvisionedMachine
-from utah.exceptions import UTAHException
+from utah.url import url_argument
 
 
 def get_parser():
@@ -57,75 +61,159 @@
         formatter_class=argparse.RawDescriptionHelpFormatter)
     parser.add_argument('-m', '--machinetype', metavar='MACHINETYPE',
                         choices=('physical', 'virtual'),
-                        help='Type of machine to provision (%(choices)s)')
+                        default=config.machinetype,
+                        help='Type of machine to provision (%(choices)s) '
+                             '(Default is %(default)s)')
     parser.add_argument('-v', '--variant',
+                        default=config.variant,
                         help='Variant of architecture, i.e., armel, armhf')
     parser.add_argument('--skip-provisioning', action='store_true',
-                        help=('Reuse a system that is already provisioned '
-                              '(name argument must be passed)'))
-    parser = common_arguments(parser)
-    parser = custom_arguments(parser)
-    parser = file_arguments(parser)
-    parser = name_argument(parser)
-    parser = virtual_arguments(parser)
+                        help='Reuse a system that is already provisioned '
+                             '(name argument must be passed)')
+    parser.add_argument('runlist', metavar='runlist',
+                        type=master_runlist_argument,
+                        help='URLs of runlist files to run')
+    parser.add_argument('-s', '--series', metavar='SERIES',
+                        choices=config.serieschoices,
+                        default=config.series,
+                        help='Series to use for installation (%(choices)s) '
+                             '(Default is %(default)s)')
+    parser.add_argument('-t', '--type', metavar='TYPE',
+                        choices=('desktop', 'server', 'mini', 'alternate'),
+                        default=config.installtype,
+                        help='Install type to use for installation '
+                             '(%(choices)s) (Default is %(default)s)')
+    parser.add_argument('-a', '--arch', metavar='ARCH',
+                        choices=('i386', 'amd64', 'arm'),
+                        default=config.arch,
+                        help='Architecture to use for installation '
+                             '(%(choices)s) (Default is %(default)s)')
+    parser.add_argument('-n', '--no-destroy', action='store_true',
+                        help='Preserve VM after tests have run')
+    parser.add_argument('-d', '--debug', action='store_true',
+                        help='Enable debug logging')
+    parser.add_argument('-j', '--json', action='store_true',
+                        help='Enable json logging (default is YAML)')
+    parser.add_argument('-f', '--files', action='append',
+                        help='File or directory to copy from test system ')
+    parser.add_argument('-o', '--outdir',
+                        help=('Directory to store locally copied files '
+                              '(Default is {}/machine-name)'
+                              .format(config.logpath)))
+    parser.add_argument('--dumplogs', action='store_true',
+                        help='Write client output logs to standard out')
+    parser.add_argument('--outputpreseed', action='store_true',
+                        help='Copy preseed to logs directory and list as '
+                        'log file in output')
+    parser.add_argument('-i', '--image', type=url_argument,
+                        default=config.image,
+                        help='Image/ISO file to use for installation')
+    parser.add_argument('-p', '--preseed', type=url_argument,
+                        default=config.preseed,
+                        help='Preseed file to use for installation')
+    parser.add_argument('-b', '--boot',
+                        default=config.boot,
+                        help='Boot arguments for initial installation')
+    parser.add_argument('--rewrite',
+                        choices=('all', 'minimal', 'casperonly', 'none'),
+                        default=config.rewrite,
+                        help='Set level of automatic configuration rewriting '
+                             '(Default is %(default)s)')
+    parser.add_argument('-k', '--kernel', type=url_argument,
+                        default=config.kernel,
+                        help='Kernel file to use for installation')
+    parser.add_argument('-r', '--initrd', type=url_argument,
+                        default=config.initrd,
+                        help='InitRD file to use for installation')
+    parser.add_argument('--name',
+                        default=config.name,
+                        help='Name of machine to provision')
+    parser.add_argument('-e', '--emulator',
+                        default=config.emulator,
+                        help='Emulator to use (kvm and qemu are supported, '
+                             'kvm will be favored if available)')
+    parser.add_argument('-x', '--xml', type=url_argument,
+                        default=config.xml,
+                        help='XML VM definition file '
+                             '(Default is %(default)s)')
+    parser.add_argument('-g', '--gigabytes', action='append',
+                        default=config.disksizes,
+                        help='Size in gigabytes of virtual disk, '
+                             'specify more than once for multiple disks '
+                             '(Default is %(default)s)')
+    parser.add_argument('--diskbus', metavar='DISKBUS',
+                        choices=('virtio', 'sata', 'ide'),
+                        default=config.diskbus,
+                        help='Disk bus to use for customvm installation '
+                             '(%(choices)s) (Default is %(default)s)')
     return parser
 
 
+def _get_machine(args):
+    if args.skip_provisioning:
+        # TBD: Inventory should be used to verify machine
+        # is not running other tests
+        machine = ProvisionedMachine(name=args.name)
+    else:
+        kw = {'clean': (not args.no_destroy),
+              'new': True,
+              }
+        kw['arch'] = args.arch
+        kw['boot'] = args.boot
+        kw['debug'] = args.debug
+        kw['image'] = args.image
+        kw['initrd'] = args.initrd
+        kw['kernel'] = args.kernel
+        kw['name'] = args.name
+        kw['preseed'] = args.preseed
+        kw['rewrite'] = args.rewrite
+        kw['series'] = args.series
+        kw['xml'] = args.xml
+        kw['installtype'] = args.type
+        if args.machinetype == 'physical':
+            if 'arm' in args.arch:
+                inventory = ManualBaremetalSQLiteInventory(
+                    db=os.path.join('~', '.utah-bamboofeeder-inventory'),
+                    lockfile=os.path.join('~', '.utah-bamboofeeder-lock'))
+                kw['machinetype'] = BambooFeederMachine
+            else:
+                inventory = ManualBaremetalSQLiteInventory()
+                kw['machinetype'] = CobblerMachine
+        else:
+            inventory = TinySQLiteInventory()
+            kw['diskbus'] = args.diskbus
+            kw['emulator'] = args.emulator
+            kw['disksizes'] = args.gigabytes
+        machine = inventory.request(**kw)
+    return machine
+
+
 def run_utah_tests(args=None):
     if args is None:
         args = get_parser().parse_args()
 
     if not check_user_group():
         print_group_error_message(__file__)
-        sys.exit(ReturnCodes.GROUP_ERROR)
-
-    if args.machinetype is None:
-        machinetype = config.machinetype
-    else:
-        machinetype = args.machinetype
+        return ReturnCodes.GROUP_ERROR
 
     configure_logging(args.debug)
     logging.info('UTAH version: %s', utah.__version__)
 
-    # Default is now CustomVM
-    function = run_install_test
-
-    if args.skip_provisioning:
-        def run_provisioned_tests(args):
-            """Run test cases in a provisioned machine."""
-            locallogs = []
-            try:
-                # TBD: Inventory should be used to verify machine
-                # is not running other tests
-                machine = ProvisionedMachine(name=args.name)
-                exitstatus, locallogs = run_tests(args, machine)
-            finally:
-                if len(locallogs) != 0:
-                    print('Test logs copied to the following files:')
-                    print("\t" + "\n\t".join(locallogs))
-            sys.exit(ReturnCodes.SUCCESS)
-
-        function = run_provisioned_tests
-    if args.arch is not None and 'arm' in args.arch:
-        # If arch is arm, use BambooFeederMachine
-        from run_test_bamboo_feeder import run_test_bamboo_feeder
-        function = run_test_bamboo_feeder
-    elif machinetype == 'physical':
-        # If machinetype is physical but arch isn't arm, use CobblerMachine
-        from run_test_cobbler import run_test_cobbler
-        function = run_test_cobbler
-
-    function(args=args)
+    exitstatus = ReturnCodes.SUCCESS
+    locallogs = []
+
+    exitstatus, locallogs = run_tests(args, _get_machine(args))
+    if len(locallogs) > 0:
+        print('Test logs copied to the following files:')
+        print("\t" + "\n\t".join(locallogs))
+
+    return(exitstatus)
+
 
 if __name__ == '__main__':
     try:
         try:
-            if isinstance(config.jobtimeout, int):
-                timeout(config.jobtimeout, run_utah_tests)
-            else:
-                run_utah_tests()
-        except AttributeError:
-            run_utah_tests()
+            exitstatus = timeout(config.jobtimeout, run_utah_tests)
         finally:
             cleanup.run()
     except UTAHTimeout as exception:
@@ -138,3 +226,4 @@
         sys.stderr.write('Unhandled error in UTAH:\n{}\n'
                          .format(''.join(format_exception(*sys.exc_info()))))
         sys.exit(ReturnCodes.UNHANDLED_ERROR)
+    sys.exit(exitstatus)

=== modified file 'tests/test_run.py'
--- tests/test_run.py	2013-04-23 17:17:44 +0000
+++ tests/test_run.py	2013-05-01 21:24:29 +0000
@@ -134,10 +134,9 @@
         logs = []
         args = Mock()
         args.outputpreseed = True
-
         machine = Mock()
         machine.configure_mock(name='name', finalpreseed='/bad file name')
-        _copy_preseed(machine, machine, logs)
+        _copy_preseed(machine, args, logs)
         self.assertEqual(0, len(logs))
         self.assertTrue(warning.called)
         call_args, _call_kwargs = warning.call_args
@@ -148,8 +147,10 @@
     def test_copy_preseed_good(self, copyfile):
         """Ensure _copy_preseed works for good data."""
         logs = []
+        args = Mock()
+        args.outputpreseed = True
         machine = Mock()
         machine.configure_mock(name='name', finalpreseed='/bin/true')
-        _copy_preseed(machine, machine, logs)
+        _copy_preseed(machine, args, logs)
         self.assertEqual(1, len(logs))
         self.assertTrue(copyfile.called)

=== modified file 'utah/config.py'
--- utah/config.py	2013-04-04 14:55:25 +0000
+++ utah/config.py	2013-05-01 21:24:29 +0000
@@ -60,6 +60,8 @@
     boottimeout=90,
     # Time to wait between checking if system is available over ssh
     checktimeout=15,
+    # Default setting for whether to clean up after a test run
+    clean=True,
     # Default log level to print to the console
     consoleloglevel=logging.WARNING,
     # Default debug setting
@@ -67,7 +69,7 @@
     # Command to download test images
     dlcommand='dl-ubuntu-test-iso',
     # Percentage increment at which to report download updates
-    dlpercentincrement=1,
+    dlpercentincrement=10,
     # Number of times to retry image download
     dlretries=10,
     # Default diskbus for VMs
@@ -93,14 +95,24 @@
     installtype='mini',
     # Directory to store local images
     isodir='/var/cache/utah/iso',
+    # Overall timeout for run_utah_tests
+    jobtimeout=None,
     # Default kernel for installation
     kernel=None,
     # Default location of log directory
     logpath=os.path.join('/', 'var', 'log', 'utah'),
+    # Directory to hold non-temporary files (mostly for VM)
+    machinedir=None,
     # Default machine ID
     machineid=None,
+    # Machine information object for physical machines
+    machineinfo=None,
     # Default machine type
     machinetype='virtual',
+    # UUID for machine
+    machineuuid=None,
+    # List of mac addresses for VM
+    macs=[],
     # Default machine name
     name=None,
     # Default setting of installing a new machine vs. using an existing one
@@ -117,14 +129,22 @@
     outputpreseed=False,
     # Directory where utah client and other needed packages reside
     packagedir=os.path.join('/', 'usr', 'share', 'utah'),
+    # Command to use to power cycle machine
+    powercmd=None,
     # Time to wait between power off and power on for physical systems
     powertimeout=15,
+    # Preboot for bamboo-feeder panda boards
+    preboot=None,
+    # Prefix for machine names
+    prefix='utah',
     # Default preseed
     preseed=os.path.join('/', 'etc', 'utah', 'default-preseed.cfg'),
     # Location of PXE configuration files
     pxedir=os.path.join('/', 'var', 'lib', 'tftpboot', 'pxelinux.cfg'),
     # libvirt URL for virtual machine creation
     qemupath='qemu:///system',
+    # default timeout between retry attempts
+    retry_timeout=3,
     # Default setting for configuration rewriting
     rewrite='all',
     # Time to wait for client process to finish in run.py
@@ -140,17 +160,19 @@
         'quantal',
         'raring',
     ],
+    # sqlite database connection timeout
+    # a value higher than the default is used to avoid db locking problems
+    sqlite3_db_connection_timeout=30,
     # Default machine template
     template=None,
+    # location of the jinja2 templates used by the provisionig module
+    template_dir=os.path.join('/', 'usr', 'share', 'utah', 'templates'),
     # Directory to hold web-accessible files
     wwwdir=os.path.join('/', 'var', 'www', 'utah'),
+    # Default architecture variant (armel, armhd, non-pae, etc.)
+    variant=None,
     # Default VM XML file
     xml=os.path.join('/', 'etc', 'utah', 'default-vm.xml'),
-    # sqlite database connection timeout
-    # a value higher than the default is used to avoid db locking problems
-    sqlite3_db_connection_timeout=30,
-    # location of the jinja2 templates used by the provisionig module
-    template_dir=os.path.join('/', 'usr', 'share', 'utah', 'templates'),
 
     install_steps=[
         {
@@ -208,9 +230,6 @@
             'timeout': 180,
         }
     ],
-
-    # default timeout between retry attempts
-    retry_timeout=3,
 )
 
 # These depend on the local user/path, and need to be filtered out

=== modified file 'utah/provisioning/baremetal/bamboofeeder.py'
--- utah/provisioning/baremetal/bamboofeeder.py	2013-04-24 14:27:37 +0000
+++ utah/provisioning/baremetal/bamboofeeder.py	2013-05-01 21:24:29 +0000
@@ -40,8 +40,8 @@
 
     # TODO: raise more exceptions if ProcessRunner fails
     # maybe get some easy way to do that like failok
-    def __init__(self, machineinfo=None, name=None, preboot=None, *args,
-                 **kw):
+    def __init__(self, machineinfo=config.machineinfo, name=config.name,
+                 preboot=config.preboot, *args, **kw):
         # TODO: respect rewrite setting
         if name is None:
             raise UTAHBMProvisioningException(
@@ -55,23 +55,20 @@
                                                   name=name, **kw)
         if self.inventory is not None:
             self.cleanfunction(self.inventory.release, machine=self)
-        self._depcheck()
         if self.image is None:
             raise UTAHBMProvisioningException(
                 'Image file required for bamboo-feeder installation')
+        self._depcheck()
         self.ip = self._ipaddr(config.wwwiface)
         self.logger.debug('Configuring for %s with IP %s',
                           config.wwwiface, self.ip)
         self._cmdlinesetup()
         imageurl = 'http://{}/utah/{}.img'.format(self.ip, self.name)
         preenvurl = 'http://{}/utah/{}.preEnv'.format(self.ip, self.name)
-        if preboot is None:
-            self.preboot = ('console=ttyO2,115200n8 imageurl={imageurl} '
-                            'bootcfg={preenvurl}'.format(
-                            imageurl=imageurl, preenvurl=preenvurl))
-        else:
-            # TODO: maybe make this automatically add needed options
-            self.preboot = preboot
+        self.preboot = (preboot or
+                        'console=ttyO2,115200n8 imageurl={imageurl} '
+                        'bootcfg={preenvurl}'
+                        .format(imageurl=imageurl, preenvurl=preenvurl))
         self.logger.debug('Preboot setup:')
         self.logger.debug(self.preboot)
         self.logger.debug('BambooFeederMachine init finished')

=== modified file 'utah/provisioning/baremetal/cobbler.py'
--- utah/provisioning/baremetal/cobbler.py	2013-04-24 14:27:37 +0000
+++ utah/provisioning/baremetal/cobbler.py	2013-05-01 21:24:29 +0000
@@ -39,7 +39,8 @@
 
     # TODO: raise more exceptions if ProcessRunner fails
     # maybe get some easy way to do that like failok
-    def __init__(self, machineinfo=None,  name=None, *args, **kw):
+    def __init__(self, machineinfo=config.machineinfo, name=config.name,
+                 *args, **kw):
         # TODO: support for reusing existing machines
         if name is None:
             raise UTAHBMProvisioningException(
@@ -49,7 +50,6 @@
                 'No cobbler arguments given for machine creation')
         else:
             self.machineinfo = machineinfo
-        self.power = {}
         super(CobblerMachine, self).__init__(*args, machineinfo=machineinfo,
                                              name=name, **kw)
         if self.inventory is not None:

=== modified file 'utah/provisioning/baremetal/power.py'
--- utah/provisioning/baremetal/power.py	2013-04-04 13:19:16 +0000
+++ utah/provisioning/baremetal/power.py	2013-05-01 21:24:29 +0000
@@ -33,15 +33,15 @@
 
     """
 
-    def __init__(self, machineinfo=None, powercmd=None, *args, **kw):
+    def __init__(self, machineinfo=config.machineinfo,
+                 powercmd=config.powercmd, *args, **kw):
         """Store power control info for later use."""
         if machineinfo is not None:
             self.power = {}
             for item in machineinfo:
                 if 'power' in item:
                     self.power[item] = machineinfo[item]
-        if powercmd is not None:
-            self.powercmd = powercmd
+        self.powercmd = powercmd
         super(PowerMixin, self).__init__(*args, **kw)
 
     def powercommand(self):

=== modified file 'utah/provisioning/provisioning.py'
--- utah/provisioning/provisioning.py	2013-04-24 14:27:37 +0000
+++ utah/provisioning/provisioning.py	2013-05-01 21:24:29 +0000
@@ -64,12 +64,15 @@
 
     """
 
-    def __init__(self, arch=None, boot=None, clean=True, debug=False,
-                 directory=None, image=None, dlpercentincrement=1,
-                 initrd=None, installtype=None, inventory=None, kernel=None,
-                 machineid=None, machineuuid=None, name=None, new=False,
-                 prefix='utah', preseed=None, rewrite=None, series=None,
-                 template=None, xml=None):
+    def __init__(self, arch=config.arch, boot=config.boot, clean=True,
+                 debug=False, image=config.image,
+                 dlpercentincrement=config.dlpercentincrement,
+                 initrd=config.initrd, installtype=config.installtype,
+                 inventory=None, kernel=config.kernel,
+                 machineid=config.machineid, machineuuid=config.machineuuid,
+                 name=config.name, new=False, preseed=config.preseed,
+                 rewrite=config.rewrite, series=config.series,
+                 template=config.template, xml=config.xml):
         """Initialize the object representing the machine.
 
         One of these groups of arguments should be included:
@@ -85,9 +88,6 @@
         Other arguments:
             clean: Enable cleanup functions.
             debug: Enable debug logging.
-            directory: Where the machine's files go.  Should be persistent for
-                VMs, temporary is fine for installation-only files, like those
-                used to install a physical machine using another method.
             dlpercentincrement: How often to log download updates to INFO
                 (All updates logged to DEBUG)
             inventory: Inventory object managing the machine; used for cleanup
@@ -98,7 +98,6 @@
                 supplied.
             new: Request a new machine (or a reinstall if a specific machine
                 was requested.)
-            prefix: prefix for automatically named machines.
             rewrite: How much to alter supplied preseed and xml files.
                 all:         everything we need for an automated install
                 minimal:     insert latecommand into preseed
@@ -117,45 +116,27 @@
         """
         # TODO: Make this work right with super at some point.
         # TODO: Consider a global temp file creator, maybe as part of install.
+        # TODO: Make everything use config.dlpercentincrement instead of
+        # passing it around all the time
+        self.arch = arch
+        self.boot = boot
+        self.clean = clean
         self.debug = debug
         self.dlpercentincrement = dlpercentincrement
+        self.installtype = installtype
         self.inventory = inventory
         self.machineid = machineid
         self.new = new
-        self.prefix = prefix
+        self.rewrite = rewrite
+        self.series = series
         self.template = template
+        self.uuid = uuid
 
         self.boottimeout = config.boottimeout
 
-        if clean is None:
-            self.clean = True
-        else:
-            self.clean = clean
-
         # TODO: Move namesetup into vm
         self._namesetup(name)
 
-        if arch is None:
-            self.arch = config.arch
-        else:
-            self.arch = arch
-        if boot is None:
-            self.boot = config.boot
-        else:
-            self.boot = boot
-        if installtype is None:
-            self.installtype = config.installtype
-        else:
-            self.installtype = installtype
-        if rewrite is None:
-            self.rewrite = config.rewrite
-        else:
-            self.rewrite = rewrite
-        if series is None:
-            self.series = config.series
-        else:
-            self.series = series
-
         if machineuuid is None:
             self.uuid = str(uuid.uuid4())
         else:
@@ -165,20 +146,18 @@
         self.active = False
         self._loggersetup()
 
-        if preseed is None:
-            preseed = config.preseed
-
         fileargs = ['initrd', 'kernel', 'preseed', 'xml']
 
         if image is None:
-            image = config.image
-        if image is None or image.endswith('.iso'):
             self.image = ISO(arch=arch,
-                             dlpercentincrement=self.dlpercentincrement,
-                             image=image,
+                             dlpercentincrement=dlpercentincrement,
                              installtype=installtype,
                              logger=self.logger,
                              series=series)
+        elif image.endswith('.iso'):
+            self.image = ISO(dlpercentincrement=dlpercentincrement,
+                             image=image,
+                             logger=self.logger)
         else:
             fileargs.append('image')
 
@@ -186,8 +165,6 @@
             # Ensure every file/url type argument is available locally
             arg = locals()[item]
             if arg is None:
-                arg = getattr(config, item)
-            if arg is None:
                 setattr(self, item, None)
             else:
                 if arg.startswith('~'):
@@ -199,6 +176,7 @@
 
                 self.percent = 0
                 self.logger.info('Preparing %s: %s', item, path)
+                # TODO: implement download utility function and use it here
                 filename = urllib.urlretrieve(path,
                                               reporthook=self.dldisplay)[0]
                 setattr(self, item, filename)

=== modified file 'utah/provisioning/vm/vm.py'
--- utah/provisioning/vm/vm.py	2013-04-24 14:31:11 +0000
+++ utah/provisioning/vm/vm.py	2013-05-01 21:24:29 +0000
@@ -134,9 +134,10 @@
     """Install a VM from an image using libvirt direct kernel booting."""
 
     # TODO: probably remove parameters from the private methods
-    def __init__(self, directory=None, diskbus=None, disksizes=None,
-                 emulator=None, machineid=None, macs=None, name=None,
-                 prefix='utah', *args, **kw):
+    def __init__(self, directory=config.machinedir, diskbus=config.diskbus,
+                 disksizes=config.disksizes, emulator=config.emulator,
+                 machineid=config.machineid, macs=config.macs,
+                 name=config.name, prefix=config.prefix, *args, **kw):
         # Make sure that no other virtualization solutions are running
         # TODO: see if this is needed for qemu or just kvm
         process_checker = ProcessChecker()
@@ -146,16 +147,11 @@
                 message = process_checker.get_error_message(app)
                 raise UTAHVMProvisioningException(message)
 
-        if diskbus is None:
-            self.diskbus = config.diskbus
-        else:
-            self.diskbus = diskbus
-        if disksizes is None:
-            disksizes = config.disksizes
-        if disksizes is None:
-            self.disksizes = [8]
-        else:
-            self.disksizes = disksizes
+        self.diskbus = diskbus
+        self.disksizes = disksizes
+        self.emulator = emulator
+        self.macs = macs
+        self.prefix = prefix
         self.disks = []
         if name is None:
             autoname = True
@@ -178,34 +174,21 @@
             self._loggerunsetup()
             self._loggersetup()
             self._cmdlinesetup()
-        if emulator is None:
-            emulator = config.emulator
-        if emulator is None:
+        if self.emulator is None:
             if self._supportsdomaintype('kvm'):
                 self.logger.info(
                     'Setting type to kvm since it is in libvirt capabilities')
-                self.domaintype = 'kvm'
+                self.emulator = 'kvm'
             elif self._supportsdomaintype('qemu'):
                 self.logger.info(
                     'Setting type to qemu since it is in libvirt capabilities')
-                self.domaintype = 'qemu'
+                self.emulator = 'qemu'
             else:
                 raise UTAHVMProvisioningException(
                     'kvm and qemu not supported in libvirt capabilities; '
                     'please make sure qemu and/or kvm are installed '
                     'and libvirt is configured correctly')
-        else:
-            self.domaintype = emulator
-        if self.domaintype == 'qemu':
-            self.logger.debug('Raising boot timeout for qemu domain')
-            self.boottimeout *= 4
-        if macs is None:
-            macs = []
-        self.macs = macs
-        if directory is None:
-            self.directory = os.path.join(config.vmpath, self.name)
-        else:
-            self.directory = directory
+        self.directory = (directory or os.path.join(config.vmpath, self.name))
         self.cleanfile(self.directory)
         self.logger.info('Checking if machine directory {} exists'
                          .format(self.directory))
@@ -298,7 +281,7 @@
         xmlt.find('uuid').text = self.uuid
         self.logger.debug(
             'Setting type to qemu in case no hardware virtualization present')
-        xmlt.getroot().set('type', self.domaintype)
+        xmlt.getroot().set('type', self.emulator)
         ose = xmlt.find('os')
         if self.arch == ('i386'):
             ose.find('type').set('arch', 'i686')

=== modified file 'utah/run.py'
--- utah/run.py	2013-04-19 19:27:58 +0000
+++ utah/run.py	2013-05-01 21:24:29 +0000
@@ -33,22 +33,21 @@
     parse_yaml_file,
     ReturnCodes as ClientReturnCodes,
 )
-from utah.client.runner import Runner
 from utah.client.exceptions import (
     YAMLEmptyFile,
     YAMLParsingError,
 )
+from utah.client.runner import Runner
 from utah.exceptions import UTAHException
 from utah.retry import retry
 from utah.timeout import timeout
 from utah.url import url_argument
-from utah.provisioning.ssh import ProvisionedMachine
 
 
 # Return codes for the server
 class ReturnCodes:
 
-    """Provide standard return codes for run\_ scripts."""
+    r"""Provide standard return codes for run\_ scripts."""
 
     SUCCESS = 0  # No problems found
     UTAH_EXCEPTION_ERROR = 1  # UTAH exception caught
@@ -131,114 +130,8 @@
     return filename
 
 
-def common_arguments(parser):
-    """Centralize command line arguments for all run\_ scripts.
-
-    :returns: argparse parser with arguments added
-    :rtype: obj
-
-    """
-    parser.add_argument('runlist', metavar='runlist',
-                        type=master_runlist_argument,
-                        help='URLs of runlist files to run')
-    parser.add_argument('-s', '--series', metavar='SERIES',
-                        choices=config.serieschoices,
-                        help='Series to use for VM creation (%(choices)s)')
-    parser.add_argument('-t', '--type', metavar='TYPE',
-                        choices=('desktop', 'server', 'mini', 'alternate'),
-                        help=('Install type to use for VM creation '
-                              '(%(choices)s)'))
-    parser.add_argument('-a', '--arch', metavar='ARCH',
-                        choices=('i386', 'amd64'),
-                        help=('Architecture to use for VM creation '
-                              '(%(choices)s)'))
-    parser.add_argument('-n', '--no-destroy', action='store_true',
-                        help='Preserve VM after tests have run')
-    parser.add_argument('-d', '--debug', action='store_true',
-                        help='Enable debug logging')
-    parser.add_argument('-j', '--json', action='store_true',
-                        help='Enable json logging (default is YAML)')
-    parser.add_argument('-f', '--files', action='append',
-                        help='File or directory to copy from test system ')
-    parser.add_argument('-o', '--outdir',
-                        help=('Directory to store locally copied files '
-                              '(Default is {}/machine-name)'
-                              .format(config.logpath)))
-    parser.add_argument('--dumplogs', action='store_true',
-                        help='Write client output logs to standard out')
-    parser.add_argument('--outputpreseed', action='store_true',
-                        help='Copy preseed to logs directory and list as '
-                        'log file in output')
-    return parser
-
-
-def custom_arguments(parser):
-    """Centralize arguments for installing from an image.
-
-    :returns: argparse parser with arguments added
-    :rtype: obj
-
-    """
-    parser.add_argument('-i', '--image', type=url_argument,
-                        help='Image/ISO file to use for installation')
-    parser.add_argument('-p', '--preseed', type=url_argument,
-                        help='Preseed file to use for installation')
-    parser.add_argument('-b', '--boot',
-                        help='Boot arguments for initial installation')
-    parser.add_argument('--rewrite', choices=('all', 'minimal', 'casperonly',
-                        'none'), help='Enable or disable automatic '
-                        'configuration rewriting')
-    return parser
-
-
-def file_arguments(parser):
-    """Centralize arguments for custom kernel and initrd support.
-
-    :returns: argparse parser with arguments added
-    :rtype: obj
-
-    """
-    parser.add_argument('-k', '--kernel', type=url_argument,
-                        help='Kernel file to use for installation')
-    parser.add_argument('-r', '--initrd', type=url_argument,
-                        help='InitRD file to use for installation')
-    return parser
-
-
-def name_argument(parser):
-    """Centralize arguments for named machines.
-
-    :returns: argparse parser with arguments added
-    :rtype: obj
-
-    """
-    parser.add_argument('--name', help='Name of machine to provision')
-    return parser
-
-
-def virtual_arguments(parser):
-    """Centralize arguments for virtual machines.
-
-    :returns: argparse parser with arguments added
-    :rtype: obj
-
-    """
-    parser.add_argument('-e', '--emulator',
-                        help=('Emulator to use (kvm and qemu are supported, '
-                              'kvm will be favored if available)'))
-    parser.add_argument('-x', '--xml', type=url_argument,
-                        help='XML VM definition file')
-    parser.add_argument('-g', '--gigabytes', action='append',
-                        help=('Size in gigabytes of virtual disk, '
-                              'specify more than once for multiple disks'))
-    parser.add_argument('--diskbus', metavar='DISKBUS',
-                        choices=('virtio', 'sata', 'ide'),
-                        help=('Disk bus to use for customvm installation '
-                              '(%(choices)s)'))
-    return parser
-
-
 def _get_runlist(url):
+    # TODO: Make something that this and utah.iso._get_resource can both use
     try:
         return urllib.urlretrieve(url)[0]
     except urllib.ContentTooShortError as e:
@@ -329,21 +222,25 @@
     """Copy preseed to locallogs.
 
     If we are provisioning a system, we can optionally copy its preseed along
-    with other locallogs from the job. Systems already provisioned will not
+    with other locallogs from the job.
 
     """
-    if (not isinstance(machine, ProvisionedMachine) and
-            (args.outputpreseed or config.outputpreseed)):
+    if args.outputpreseed or config.outputpreseed:
         if args.outputpreseed:
             logging.debug('Capturing preseed due to command line option')
 
         p = os.path.join(config.logpath, '{}-preseed.cfg'.format(machine.name))
-        try:
-            shutil.copyfile(machine.finalpreseed, p)
-        except (IOError, shutil.Error) as err:
-            logging.warning('Failed to copy preseed file: %s', err)
+        if machine.hasattr('finalpreseed'):
+            try:
+                if os.path.isfile(p):
+                    os.chmod(p, 0664)
+                shutil.copyfile(machine.finalpreseed, p)
+            except (IOError, OSError, shutil.Error) as err:
+                logging.warning('Failed to copy preseed file: %s', err)
+            else:
+                locallogs.append(p)
         else:
-            locallogs.append(p)
+            logging.debug('Machine has no preseed to capture')
 
 
 def run_tests(args, machine):
@@ -391,10 +288,7 @@
     :rtype: list(str)
 
     """
-    if args.outdir is None:
-        outdir = os.path.join(config.logpath, machine.name)
-    else:
-        outdir = args.outdir
+    outdir = (args.outdir or os.path.join(config.logpath, machine.name))
     if not os.path.isdir(outdir):
         try:
             os.makedirs(outdir)
@@ -507,7 +401,7 @@
 
     """
     logging.info('Checking if UTAH client is finished')
-    machine.sshcheck()
+    machine.activecheck()
     try:
         exitstatus = machine.run('/usr/share/utah/client/utah-done.py',
                                  quiet=True)[0]