Merge lp:~doanac/utah/server-cleanups into lp:utah

[Andy Doan]

this shouldn't change any functionality. however, it cleans some things up so that a new feature I'm working on will be easier/cleaner to merge.

[Max Brustkern (nuclearbob)]

I see two things that look like they'll work differently.

If you try to upload files and some of them don't exist, the exception will be raised before uploading any of them instead of after. That seems like it's probably fine.

Also, the handling of exceptions related to weird provisioning possibilities for Libvirt goes away. I think that makes the class slightly less robust for situations that we're not seeing at all right now. I wonder if we will see them in the future. Probably not. Seems reasonable, but I haven't tested anything yet. If you have, we're probably in good shape.

[Andy Doan (doanac)]

On 04/18/2013 03:50 PM, Max Brustkern wrote:
> I see two things that look like they'll work differently.
>
> If you try to upload files and some of them don't exist, the exception will be raised before uploading any of them instead of after. That seems like it's probably fine.

yeah, the results from the run don't change, so it seemed harmless.

> Also, the handling of exceptions related to weird provisioning possibilities for Libvirt goes away. I think that makes the class slightly less robust for situations that we're not seeing at all right now. I wonder if we will see them in the future. Probably not. Seems reasonable, but I haven't tested anything yet. If you have, we're probably in good shape.

the main difference was something I thought didn't make sense. In the
original code, you could specify !new, and if _load failed, we'd call
_create and try and correct your mistake for you. In the new code, that
would fail. alternatively it did a similar logic for new=True. I found
the logic confusing, not sure if it has practical uses?

[Max Brustkern (nuclearbob)]

I think it served some purposes when I was testing weird gymnastics of the original implementation regarding non-automatically-generated names, but I don't think it's really doing anything useful for us now.

[Javier Collado (javier.collado)]

The changes look good and I have been able to run the pass runlist
successfully. Please find below a few comments:

- tests/test_template.py
I think `_tmpdir` should be set in a `setupClass` method rather than in defined
as a global variable.

- utah/template.py
  - _add_backslash_filter
    `:rtype:` should be `string` and `:returns:` a description of what is
    returned.

  - as_buff
    The backslash filter could be added to `as_buff` at import time like this:
    def as_buff(template, **kw):
        <code>

    as_buff._env = ...

    This way `as_buff` would be easier to read and the code would take care
    only of the rendering, not the initialization.

flake8 output:
$ find . -name '*.py' | xargs flake8
./tests/test_debs.py:18:1: F401 'apt' imported but unused

pep257 output:
$ find . -name '*.py' | xargs pep257
test_debs.py:38:4: PEP257 Exported definitions should have docstrings.
test_debs.py:56:4: PEP257 Exported definitions should have docstrings.

[lp:~doanac/utah/server-cleanups]

872. By Andy Doan on 2013-04-22

improve test_template logic as per javier's review

873. By Andy Doan on 2013-04-22

template.py initialize improvement as per javier

874. By Andy Doan on 2013-04-22

clean up to test_debs.py as per javier

[Andy Doan (doanac)]

I think I've addressed Javier's issues now.

[Javier Collado (javier.collado)]

Thanks for the update. A few more comments:

- Test cases
The test cases are broken. It looks like the idea of initializing the template
environment at import time wasn't good for testing. The problem is the update
to `config.template_dir` in `tests/common.py` happens after
`template.as_buff._env` is already set.

I managed to make it work using something like this in `tests/common.py`:
from utah import template

...

def setUp():
    ....
    config.template_dir = ...
    reload(template)

However, I don't like much this trick, so I'll let you decide if you prefer
this or revert to the original code in which the environment was created when
the first template was rendered.

- Documentation
  - Please add ``utah.template`` to ``reference.rst``, so that it's included in
    the documentation.

  - Note that backslash characters must be escaped in the
    `_add_backslash_filter` documentation string to be properly rendered.
    Anyway, this not really an issue since private methods aren present in the
    documentation by default.

[lp:~doanac/utah/server-cleanups]

875. By Andy Doan on 2013-04-23

add new template module

876. By Andy Doan on 2013-04-23

fix broken test case

The test cases are broken. It looks like the idea of initializing the template
environment at import time wasn't good for testing. The problem is the update
to `config.template_dir` in `tests/common.py` happens after
`template.as_buff._env` is already set.

[Andy Doan (doanac)]

> However, I don't like much this trick, so I'll let you decide if you prefer
> this or revert to the original code in which the environment was created when
> the first template was rendered.

yeah - i think my first way is probably cleaner. last two revno's 875 and 876 should fix.

[Javier Collado (javier.collado)]

Thanks for all the updates. I think the changes are ready to be merged.

There's a small warning when building the documentation, but I'll fix it now
before merging:

server-cleanups/docs/source/reference.rst:60: WARNING: Title underline too short.

``utah.template``
----------------

review: Approve

[Javier Collado (javier.collado)]

I needed to remove 'tests/__init__.py' since it was causing packaging problems.

If it's needed to have that file, we can address the issue in a separate merge
request.

=== modified file 'docs/source/reference.rst'
--- docs/source/reference.rst	2013-03-29 17:00:50 +0000
+++ docs/source/reference.rst	2013-04-23 17:57:25 +0000
@@ -56,6 +56,12 @@
 .. automodule:: utah.timeout
    :members:
 
+``utah.template``
+----------------
+
+.. automodule:: utah.template
+   :members:
+
 ``utah.url``
 ----------------
 

=== added file 'tests/__init__.py'
--- tests/__init__.py	1970-01-01 00:00:00 +0000
+++ tests/__init__.py	2013-04-23 17:57:25 +0000
@@ -0,0 +1,16 @@
+# Ubuntu Testing Automation Harness
+# Copyright 2013 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""tests for UTAH server code"""

=== added file 'tests/common.py'
--- tests/common.py	1970-01-01 00:00:00 +0000
+++ tests/common.py	2013-04-23 17:57:25 +0000
@@ -0,0 +1,33 @@
+# Ubuntu Testing Automation Harness
+# Copyright 2013 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Provide common functions and content for self tests."""
+
+import shutil
+import tempfile
+
+from utah import config
+
+
+def setUp():
+    """setup function called for the duration of the test run."""
+
+    # we need a consitent template across all the runs
+    config.template_dir = tempfile.mkdtemp(prefix='utah-templates')
+
+
+def tearDown():
+    """Clean up after ourselves."""
+    shutil.rmtree(config.template_dir)

=== added file 'tests/test_debs.py'
--- tests/test_debs.py	1970-01-01 00:00:00 +0000
+++ tests/test_debs.py	2013-04-23 17:57:25 +0000
@@ -0,0 +1,89 @@
+# Ubuntu Testing Automation Harness
+# Copyright 2013 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Unit tests for the utah.provisioning.debs module."""
+
+import os
+import shutil
+import tempfile
+import unittest
+
+from mock import patch
+
+from utah import config
+from utah.provisioning.exceptions import UTAHProvisioningException
+from utah.provisioning.debs import (
+    _schema_deb,
+    get_client_debs,
+)
+
+
+class TestDebs(unittest.TestCase):
+
+    """Tests the functions of the utah.provisioning.debs module."""
+
+    def setUp(self):
+        """Set up a temp directory for test data."""
+        self.pkgdir = config.packagedir
+        self.tmpdir = tempfile.mkdtemp(prefix='utah_debs')
+        config.packagedir = self.tmpdir
+
+        ver = 'FOO'
+
+        class dummy(object):
+            def __init__(self):
+                self.installedVersion = ver
+        self.cache = {'utah': dummy()}
+
+        for pkg in ['common', 'client']:
+            fname = 'utah-{}_{}_all.deb'.format(pkg, ver)
+            pkg = os.path.join(self.tmpdir, fname)
+            with open(pkg, 'w') as f:
+                f.write('')
+
+    def tearDown(self):
+        """Clean up stuff from the setup function."""
+        config.packagedir = self.pkgdir
+        shutil.rmtree(self.tmpdir)
+
+    def _setup_schema(self, oldest, latest):
+        pkgfmt = 'python-jsonschema_0.{}-0~ppa1_all.deb'
+
+        for x in xrange(oldest, latest + 1):
+            fname = pkgfmt.format(x)
+            with open(os.path.join(self.tmpdir, fname), 'w') as f:
+                f.write('')
+        return pkgfmt.format(latest)
+
+    def test_missing_schema(self):
+        """Ensure exception is raised when missing schema deb."""
+        self.assertRaises(UTAHProvisioningException, _schema_deb)
+
+    def test_schema_deb(self):
+        """Check that we return the latest schema deb."""
+
+        latest = self._setup_schema(1, 3)
+
+        #ensure we got the latest version of the schema
+        deb = _schema_deb()
+        self.assertEqual(latest, os.path.basename(deb))
+
+    @patch('apt.cache.Cache')
+    def test_client_debs(self, cache):
+        """Ensure get_client_debs API works."""
+
+        cache.side_effect = [self.cache, self.cache]
+        self._setup_schema(2, 4)
+        self.assertEqual(len(get_client_debs()), 3)

=== modified file 'tests/test_rsyslog.py'
--- tests/test_rsyslog.py	2013-04-22 11:14:38 +0000
+++ tests/test_rsyslog.py	2013-04-23 17:57:25 +0000
@@ -131,8 +131,9 @@
         r = RSyslog("utah-test", '/tmp')
         threading.Thread(target=self.producer, args=(r.port, messages)).start()
 
-        def booted_cb():
+        def booted_cb(match):
             self.test_future_booted = True
+            self.assertEqual(match.string.rstrip(), steps[1]['message'])
 
         self.test_future_booted = False
         r.wait_for_install(steps, booted_cb)
@@ -169,11 +170,13 @@
         r = RSyslog("utah-test", '/tmp')
         threading.Thread(target=self.producer, args=(r.port, messages)).start()
 
-        def booted_cb():
+        def booted_cb(match):
             self.test_callbacks_booted = True
+            self.assertEqual(match.string.rstrip(), messages[1])
 
-        def blah_cb():
+        def blah_cb(match):
             self.test_callbacks_blah = True
+            self.assertEqual(match.string.rstrip(), messages[-1])
 
         self.test_callbacks_booted = self.test_callbacks_blah = False
         callbacks = {

=== added file 'tests/test_ssh.py'
--- tests/test_ssh.py	1970-01-01 00:00:00 +0000
+++ tests/test_ssh.py	2013-04-23 17:57:25 +0000
@@ -0,0 +1,33 @@
+# Ubuntu Testing Automation Harness
+# Copyright 2013 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Simple tests for the SSHMixin Object."""
+
+import unittest
+
+from utah.provisioning.exceptions import UTAHProvisioningException
+from utah.provisioning.ssh import SSHMixin
+
+
+class TestSSHMixin(unittest.TestCase):
+
+    """Performs testing of our SSHMixin class."""
+
+    def test_missing_files(self):
+        """Make sure we exit *before* trying to provision if missing files."""
+        ssh = SSHMixin()
+        fname = '/this does not exist'
+        with self.assertRaisesRegexp(UTAHProvisioningException, fname):
+            ssh.uploadfiles(fname, '/tmp/')

=== added file 'tests/test_template.py'
--- tests/test_template.py	1970-01-01 00:00:00 +0000
+++ tests/test_template.py	2013-04-23 17:57:25 +0000
@@ -0,0 +1,50 @@
+# Ubuntu Testing Automation Harness
+# Copyright 2013 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Module to test our template utility."""
+
+import os
+import unittest
+
+from utah import config
+from utah import template
+
+from tests.common import (  # NOQA
+    setUp,     # Used by nosetests
+    tearDown,  # Used by nosetests
+)
+
+
+class TestTemplate(unittest.TestCase):
+
+    """Unit test class for our template utility."""
+
+    def setUp(self):
+        """Executed before each test_ method is called."""
+        with open(os.path.join(config.template_dir, 'tmp.jinja2'), 'w') as f:
+            f.write('line1\nfoo={{foo}}')
+
+    def test_as_buff(self):
+        """minimal test to make sure the template api works."""
+        buff = template.as_buff('tmp.jinja2', foo='bar')
+        self.assertEqual(buff, 'line1\nfoo=bar')
+
+    def test_write(self):
+        """minimal test to make sure the file write api works."""
+        self.setUp()
+        f = os.path.join(config.template_dir, 'tmp.txt')
+        template.write('tmp.jinja2', f, foo='bar')
+        with open(f, 'r') as f:
+            self.assertEqual(f.read(), 'line1\nfoo=bar')

=== modified file 'utah/provisioning/baremetal/bamboofeeder.py'
--- utah/provisioning/baremetal/bamboofeeder.py	2013-04-04 18:11:06 +0000
+++ utah/provisioning/baremetal/bamboofeeder.py	2013-04-23 17:57:25 +0000
@@ -284,7 +284,6 @@
         retry(self.sshcheck, logmethod=self.logger.info,
               retry_timeout=config.checktimeout)
 
-        self.provisioned = True
         self.active = True
         self._uuid_check()
         self.logger.info('System installed')

=== modified file 'utah/provisioning/baremetal/cobbler.py'
--- utah/provisioning/baremetal/cobbler.py	2013-04-10 15:26:26 +0000
+++ utah/provisioning/baremetal/cobbler.py	2013-04-23 17:57:25 +0000
@@ -84,8 +84,6 @@
         if self.name not in machines:
             raise UTAHBMProvisioningException(
                 'No machine named {} exists in cobbler'.format(self.name))
-        else:
-            self.provisioned = True
 
     def _create(self):
         """Install the OS on the machine."""
@@ -218,12 +216,11 @@
         if self.installtype == 'desktop':
             self._removenfs()
 
-        self.provisioned = True
         self.active = True
         self._uuid_check()
         self.logger.info('System installed')
 
-    def _disable_netboot(self):
+    def _disable_netboot(self, match):
         self.logger.info('install seems to have booted, disabling netboot')
         self._cobble(['system', 'edit', '--name={}'.format(self.name),
                       '--netboot-enabled=N'])

=== added file 'utah/provisioning/debs.py'
--- utah/provisioning/debs.py	1970-01-01 00:00:00 +0000
+++ utah/provisioning/debs.py	2013-04-23 17:57:25 +0000
@@ -0,0 +1,59 @@
+# Ubuntu Testing Automation Harness
+# Copyright 2012 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Module provides paths to the utah client debs needed for install."""
+
+import apt.cache
+import os
+
+from glob import glob
+
+from utah import config
+from utah.provisioning.exceptions import UTAHProvisioningException
+
+
+def _utah_deb(deb):
+    utah_version = apt.cache.Cache()['utah'].installedVersion
+    if utah_version is None:
+        raise UTAHProvisioningException("UTAH package isn't installed")
+
+    basename = ('utah-{}_{}_all.deb'.format(deb, utah_version))
+    path = os.path.join(config.packagedir, basename)
+    if not os.path.isfile(path):
+        raise UTAHProvisioningException(
+            'UTAH {} binary package file not found in {}'.format(path, deb))
+    return path
+
+
+def _schema_deb():
+    debpath = config.packagedir
+    deb_file_glob = os.path.join(debpath, 'python-jsonschema_*_all.deb')
+    deb_files = glob(deb_file_glob)
+    if not deb_files:
+        msg = 'python-jsonschema package file not found in {}'.format(debpath)
+        raise UTAHProvisioningException(msg)
+    deb_files.sort()
+    return deb_files[-1]
+
+
+def get_client_debs():
+    """Return paths to all .debs required to install the utah-client.
+
+    :returns: Local paths to deb files
+    :rtype: list
+    :raises UTAHProvisioningException: When package(s) can't be found
+
+    """
+    return [_schema_deb(), _utah_deb('common'), _utah_deb('client')]

=== modified file 'utah/provisioning/provisioning.py'
--- utah/provisioning/provisioning.py	2013-04-19 11:42:45 +0000
+++ utah/provisioning/provisioning.py	2013-04-23 17:57:25 +0000
@@ -38,10 +38,12 @@
 import utah.timeout
 
 from utah import config
+from utah import template
 from utah.cleanup import cleanup
 from utah.process import ProcessRunner
 from utah.iso import ISO
 from utah.preseed import Preseed
+from utah.provisioning.debs import get_client_debs
 from utah.provisioning.rsyslog import RSyslog
 from utah.provisioning.exceptions import UTAHProvisioningException
 from utah.retry import retry
@@ -206,17 +208,6 @@
                                   path, getattr(self, item))
 
         self.finalpreseed = self.preseed
-
-        self.template_env = \
-            Environment(loader=FileSystemLoader(config.template_dir))
-
-        def add_backslash_filter(value):
-            """Append backslash character to each line"""
-            # TODO: consider moving this elsewhere
-            return '\n'.join(['{}\\'.format(line)
-                              for line in value.splitlines()])
-        self.template_env.filters['add_backslash'] = add_backslash_filter
-
         self.logger.debug('Machine init finished')
 
     @property
@@ -368,6 +359,7 @@
         utah.timeout.timeout(timeout, retry, self.pingcheck,
                              logmethod=logmethod, retry_timeout=checktimeout)
 
+<<<<<<< TREE
     def getutahdeb(self, deb):
         """Return path of utah deb files.
 
@@ -417,6 +409,8 @@
         deb_file = deb_files[0]
         return deb_file
 
+=======
+>>>>>>> MERGE-SOURCE
     def installclient(self):
         """Install the required packages on the machine.
 
@@ -427,9 +421,7 @@
         """
         self.logger.info('Installing client deb on machine')
         tmppath = os.path.normpath('/tmp')
-        debs = [self.getjsonschemadeb(), self.getutahdeb('common'),
-                self.getutahdeb('client')]
-        for deb in debs:
+        for deb in get_client_debs():
             try:
                 self.uploadfiles([deb], tmppath)
             except UTAHProvisioningException as err:
@@ -446,11 +438,9 @@
                 except AttributeError:
                     raise err
 
-            remote_deb = os.path.join(tmppath, os.path.basename(deb))
-            template = self.template_env.get_template(
-                'install-deb-command.jinja2')
-            install_command = template.render(deb=remote_deb)
-            returncode, _stdout, stderr = self.run(install_command, root=True)
+            deb = os.path.join(tmppath, os.path.basename(deb))
+            cmd = template.as_buff('install-deb-command.jinja2', deb=deb)
+            returncode, _stdout, stderr = self.run(cmd, root=True)
             if (returncode != 0 or
                     re.search(r'script returned error exit status \d+',
                               stderr)):
@@ -468,20 +458,16 @@
         provisioncheck() or activecheck() should be used.
 
         """
-        # TODO: make sure provisioned is set consistently by
-        # provisioncheck, _provision, and _load
-        # i.e., right now CobblerMachine has a _load method that sets it
-        # VMs have a separate _provision that does set it and a _load that
-        # by necessity doesn't
-        # Do we really need this function separate from provisioncheck?
-        # Let's discuss this one
         if not self.new:
             try:
                 self.logger.debug('Trying to load existing machine')
                 self._load()
+                self.provisioned = True
+                return
             except Exception as err:
                 self.logger.debug('Failed to load machine: %s', str(err))
         self._create()
+        self.provisioned = True
 
     def _create(self):
         # TODO: discuss separation of this vs. start when separating Install
@@ -814,23 +800,19 @@
                         os.path.join(tmpdir, 'initrd.d', 'utah-ssh-key'))
 
         self.logger.info('Creating latecommand scripts')
-        template = self.template_env.get_template('utah-latecommand.jinja2')
-        latecommand = template.render(user=config.user,
-                                      uuid=self.uuid,
-                                      log_file='/target/var/log/utah-install')
         filename = os.path.join(tmpdir, 'initrd.d', 'utah-latecommand')
-        with open(filename, 'w') as f:
-            f.write(latecommand)
+        template.write('utah-latecommand.jinja2',
+                       filename,
+                       user=config.user,
+                       uuid=self.uuid,
+                       log_file='/target/var/log/utah-install')
 
-        template = self.template_env.get_template(
-            'utah-latecommand-in-target.jinja2')
-        latecommand = template.render(
-            packages=config.installpackages,
-            log_file='/var/log/utah-install')
         filename = os.path.join(tmpdir, 'initrd.d',
                                 'utah-latecommand-in-target')
-        with open(filename, 'w') as f:
-            f.write(latecommand)
+        template.write('utah-latecommand-in-target.jinja2',
+                       filename,
+                       packages=config.installpackages,
+                       log_file='/var/log/utah-install')
 
     def _setuppreseed(self, tmpdir=None):
         """Rewrite the preseed to automate installation and access."""
@@ -884,12 +866,12 @@
             question.prepend('ubiquity ubiquity/summary note')
             question.prepend('ubiquity ubiquity/reboot boolean true')
 
-        template = self.template_env.get_template('latecommand-wrapper.jinja2')
         filename = os.path.join(tmpdir, 'initrd.d', 'latecommand-wrapper')
         target_log_file = '/target{}'.format(log_file)
-        with open(filename, 'w') as f:
-            f.write(template.render(latecommand=question.value.text,
-                                    log_file=target_log_file))
+        template.write('latecommand-wrapper.jinja2',
+                       filename,
+                       latecommand=question.value.text,
+                       log_file=target_log_file)
         question.value = (
             'sh latecommand-wrapper || '
             'logger -s -t utah "Late command failure detected" '
@@ -956,14 +938,12 @@
         self.logger.info('Inserting preseed into casper')
         if tmpdir is None:
             tmpdir = self.tmpdir
-        template = self.template_env.get_template(
-            'casper-preseed-script.jinja2')
-        preseedscript = template.render()
+
         casper_dir = os.path.join(tmpdir, 'initrd.d', 'scripts',
                                   'casper-bottom')
+
         filename = os.path.join(casper_dir, 'utah')
-        with open(filename, 'w') as f:
-            f.write(preseedscript)
+        template.write('casper-preseed-script.jinja2', filename)
         os.chmod(filename, 0755)
 
         orderfilename = os.path.join(casper_dir, 'ORDER')
@@ -1009,7 +989,6 @@
                              "-f /var/log/syslog \n")
 
         self.logger.info('Creating rsyslog config file')
-        template = self.template_env.get_template('50-utahdefault.conf.jinja2')
         conffilename = os.path.join(tmpdir, 'initrd.d', '50-utahdefault.conf')
         with open(conffilename, 'w') as f:
             if self.rsyslog.port:
@@ -1018,7 +997,7 @@
             else:
                 self.logger.debug('setting up logging to go to serial console')
                 dest = '|/dev/ttyS0'
-            f.write(template.render(dest=dest))
+            f.write(template.as_buff('50-utahdefault.conf.jinja2', dest=dest))
 
     def _repackinitrd(self, tmpdir=None):
         """Pack an initrd from our directory.

=== modified file 'utah/provisioning/rsyslog.py'
--- utah/provisioning/rsyslog.py	2013-04-22 10:59:26 +0000
+++ utah/provisioning/rsyslog.py	2013-04-23 17:57:25 +0000
@@ -132,7 +132,11 @@
         # i.e. LP#1100386
         try:
             self._wait_for_steps(steps, logfile, callbacks)
+<<<<<<< TREE
         except UTAHTimeout:
+=======
+        except UTAHException:
+>>>>>>> MERGE-SOURCE
             self.logger.warning('Timed out waiting for boot, but continuing')
 
     def _wait_for_steps(self, steps, logfile, callbacks):
@@ -152,6 +156,7 @@
                 future_pats = self._future_patterns(steps, x)
                 pattern.extend(future_pats)
                 self.logger.info('Waiting %ds for: %s', timeout, message)
+<<<<<<< TREE
                 match = self._wait_for(f, pattern, message, timeout)
                 if match is None:
                     remaining_messages = [step['message']
@@ -166,24 +171,28 @@
                     self.logger.error(log_message)
                     raise UTAHTimeout(log_message)
                 if match in fail_pattern:
+=======
+                pattern, match = self._wait_for(f, pattern, message, timeout)
+                if pattern in fail_pattern:
+>>>>>>> MERGE-SOURCE
                     raise UTAHException('Failure pattern found: {}'
-                                        .format(match))
-                if match in future_pats:
+                                        .format(pattern))
+                if pattern in future_pats:
                     msg = 'Expected pattern missed, matched future pattern: %s'
                     self.logger.warn(msg, match)
-                    x = self._fast_forward(steps, match, callbacks)
+                    x = self._fast_forward(steps, pattern, match, callbacks)
                 else:
                     self.logger.info('Matched pattern %r for %r message',
-                                     match, message)
+                                     pattern, message)
 
-                self._do_callback(steps[x], callbacks)
+                self._do_callback(steps[x], callbacks, match)
                 x += 1
 
     @staticmethod
-    def _do_callback(step, callbacks):
+    def _do_callback(step, callbacks, match):
         for name, func in callbacks.iteritems():
             if func and step.get(name, False):
-                func()
+                func(match)
 
     @staticmethod
     def _future_patterns(steps, index):
@@ -198,7 +207,7 @@
         return patterns
 
     @staticmethod
-    def _fast_forward(steps, pattern, callbacks):
+    def _fast_forward(steps, pattern, match, callbacks):
         """Figure out what should be the next step.
 
         Look through each item in the steps array to find the index of the
@@ -211,7 +220,7 @@
         x = 0
         while x < len(steps):
             # make sure we don't skip a callback
-            RSyslog._do_callback(steps[x], callbacks)
+            RSyslog._do_callback(steps[x], callbacks, match)
 
             patterns = steps[x]['pattern']
             if not isinstance(patterns, list):
@@ -232,8 +241,19 @@
                     sys.stderr.write(data)
                 writer.write(data)
                 for pat in pats:
+<<<<<<< TREE
                     if pat.match(data):
                         return pat.pattern
+=======
+                    match = pat.match(data)
+                    if match:
+                        return (pat.pattern, match)
+        log_message = ('Timeout ({}) occurred for {!r} message'
+                       .format(timeout, message))
+        # Log error message to write the timestamp when the timeout happened
+        self.logger.error(log_message)
+        raise UTAHException(log_message)
+>>>>>>> MERGE-SOURCE
 
     def _read_udp(self):
         data = None

=== modified file 'utah/provisioning/ssh.py'
--- utah/provisioning/ssh.py	2013-04-18 14:27:33 +0000
+++ utah/provisioning/ssh.py	2013-04-23 17:57:25 +0000
@@ -166,6 +166,18 @@
 
         return retval, ''.join(stdout_lines), ''.join(stderr_lines)
 
+    @staticmethod
+    def _check_files(files):
+        failed = []
+        for f in files:
+            if not os.path.isfile(f):
+                failed.append(f)
+        if len(failed):
+            msg = 'Files do not exist: {}'.format(' '.join(failed))
+            err = UTAHProvisioningException(msg)
+            err.files = failed
+            raise err
+
     def uploadfiles(self, files, target=os.path.normpath('/tmp/')):
         """Copy a file or list of files to a target directory on the machine.
 
@@ -178,8 +190,9 @@
         if isinstance(files, basestring):
             files = [files]
 
+        self._check_files(files)
+
         self.activecheck()
-        failed = []
         sftp_client = None
         try:
             self.ssh_client.connect(self.name,
@@ -187,15 +200,12 @@
                                     key_filename=config.sshprivatekey)
             sftp_client = self.ssh_client.open_sftp()
             for localpath in files:
-                if os.path.isfile(localpath):
-                    self.ssh_logger.info('Uploading {} from the host '
-                                         'to {} on the machine'
-                                         .format(localpath, target))
-                    remotepath = os.path.join(target,
-                                              os.path.basename(localpath))
-                    sftp_client.put(localpath, remotepath)
-                else:
-                    failed.append(localpath)
+                self.ssh_logger.info(
+                    'Uploading %s from the host to %s on the machine',
+                    localpath,
+                    target)
+                remotepath = os.path.join(target, os.path.basename(localpath))
+                sftp_client.put(localpath, remotepath)
         except paramiko.BadHostKeyException as err:
             raise UTAHProvisioningException(
                 'Host key exception encountered; '
@@ -207,11 +217,6 @@
         finally:
             if sftp_client:
                 sftp_client.close()
-            if len(failed) > 0:
-                err = UTAHProvisioningException('Files do not exist: {}'
-                                                .format(' '.join(failed)))
-                err.files = failed
-                raise err
 
     def downloadfiles(self, files, target=os.path.normpath('/tmp/')):
         """Copy a file or list of files from the machine to a local target.
@@ -382,6 +387,9 @@
         # No cleanup needed for systems that are already provisioned
         self.clean = False
 
+        self.active = False
+        self.provisioned = True
+
         # System is expected to be available already, so there's no need to
         # wait before trying to connect through ssh
         self.check_timeout = 3
@@ -391,9 +399,6 @@
         if installtype is None:
             self.installtype = config.installtype
 
-        self.template_env = \
-            Environment(loader=FileSystemLoader(config.template_dir))
-
     def activecheck(self):
         """Check if machine is active.
 

=== modified file 'utah/provisioning/vm/vm.py'
--- utah/provisioning/vm/vm.py	2013-04-04 17:17:21 +0000
+++ utah/provisioning/vm/vm.py	2013-04-23 17:57:25 +0000
@@ -71,41 +71,6 @@
         self.vm = self.lv.lookupByName(self.name)
         self.logger.info('VM loaded')
 
-    def _provision(self):
-        """Make an existing VM available using libvirt to look up the VM."""
-        self.logger.info('Provisioning VM')
-        if self.new:
-            self.logger.debug('New VM requested')
-            try:
-                self._load()
-                self.logger.error('VM already exists')
-                raise UTAHVMProvisioningException(
-                    'New VM requested, but {} already exists'
-                    .format(self.name))
-            except libvirt.libvirtError as err:
-                if err.get_error_code() == 42:
-                    self._create()
-                else:
-                    raise err
-
-        try:
-            self._load()
-        except libvirt.libvirtError as err:
-            if err.get_error_code() == 42:
-                self.logger.debug('Lookup failed')
-                try:
-                    self._create()
-                    self._load()
-                except UTAHVMProvisioningException as error:
-                    self.logger.error('VM lookup failed')
-                    raise UTAHVMProvisioningException(
-                        'Cannot find VM named {}: {}'
-                        .format(self.name, str(error)))
-            else:
-                raise err
-        self.provisioned = True
-        self.logger.info('VM provisioned')
-
     def activecheck(self):
         """Verify the machine is provisioned, then start it if needed. """
         self.logger.debug('Checking if VM is active')

=== added file 'utah/template.py'
--- utah/template.py	1970-01-01 00:00:00 +0000
+++ utah/template.py	2013-04-23 17:57:25 +0000
@@ -0,0 +1,53 @@
+# Ubuntu Testing Automation Harness
+# Copyright 2013 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU General Public License version 3, as published
+# by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License along
+# with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Provide easy accessors to the Jinja2 template library"""
+
+import os
+
+from jinja2 import Environment, FileSystemLoader
+
+from utah import config
+
+
+def _add_backslash_filter(value):
+    r"""Append backslash character to each line.
+
+    :returns: an version of the string where \\'s have been to each \\n
+    :rtype: string
+
+    """
+    return '\n'.join(['{}\\'.format(line) for line in value.splitlines()])
+
+
+def as_buff(template, **kw):
+    """Return the rendered template as a string."""
+    if not getattr(as_buff, '_env', None):
+        paths = [
+            config.template_dir,
+            os.path.join(os.path.dirname(__file__), '../templates'),
+        ]
+        as_buff._env = \
+            Environment(loader=FileSystemLoader(paths))
+        as_buff._env.filters['add_backslash'] = _add_backslash_filter
+    template = as_buff._env.get_template(template)
+    return template.render(kw)
+
+
+def write(template, path, **kw):
+    """Render the given template as a file."""
+    content = as_buff(template, **kw)
+    with open(path, 'w') as f:
+        f.write(content)