Merge lp:~javier.collado/utah/readable-preseed into lp:utah

[Javier Collado]

The changes in this branch are intended to improve the readability of the
generated preseed once it's rewritten.

To do this, it replaces the templates that were include in the code with other
ones in separate files using jinja2 template library. Some of the files aren't
really templates because they don't have any substitution variable, but I've
moved them to a file to keep a better separatation between the code and the
data that is used in the provisioned machines. I could replace the template
rendering for them with just a file copy if you find this doesn't make much
sense.

An example of a fragment of a the preseed for a desktop would be as follows:

ubiquity ubiquity/success_command string \
    sh utah-latecommand; \
    in-target sh -c '\
        while (fuser /var/lib/dpkg/lock \
                     /var/cache/apt/archives/lock \
                     /var/lib/apt/lists/lock \
               >/dev/null 2>&1); \
        do \
            echo "Waiting for dpkg lock to become available"; \
            sleep 10; \
        done; \
        apt-get install -y --force-yes openssh-server >>/var/log/utah-install 2>&1; \
        while (fuser /var/lib/dpkg/lock \
                     /var/cache/apt/archives/lock \
                     /var/lib/apt/lists/lock \
               >/dev/null 2>&1); \
        do \
            echo "Waiting for dpkg lock to become available"; \
            sleep 10; \
        done; \
        apt-get install -y --force-yes gdebi >>/var/log/utah-install 2>&1; \
        sed -i -e '"'"'/^exit 0$/i\
            while (fuser /var/lib/dpkg/lock \
                         /var/cache/apt/archives/lock \
                         /var/lib/apt/lists/lock \
                   >/dev/null 2>&1); \
            do \
                echo "Waiting for dpkg lock to become available"; \
                sleep 10; \
            done; \
            apt-get install -y --force-yes openssh-server >>/var/log/utah-install 2>&1; \
            while (fuser /var/lib/dpkg/lock \
                         /var/cache/apt/archives/lock \
                         /var/lib/apt/lists/lock \
                   >/dev/null 2>&1); \
            do \
                echo "Waiting for dpkg lock to become available"; \
                sleep 10; \
            done; \
            apt-get install -y --force-yes gdebi >>/var/log/utah-install 2>&1; \
        '"'"'\
        /etc/rc.local;\
        '; [ "$?" -eq "0" ]

Note that this was all in a single line and now it's possible to take a look at
it if it's needed to troubleshoot a problem.

I've tested the changes with {desktop,server} {i386,amd64} images and didn't
fine any problem.

[Max Brustkern (nuclearbob)]

This looks pretty great. I think it would be awesome if we also templated the default preseed, libvirt XML file, and maybe even stuff like the kernel command line options and the commands we pass to cobbler, but all of those would involve more work to keep them compatible with user input. I think this is a really good step in the direction of separation of code from configuration.

As far as the actual code, it all looks reasonable to me. If you've tested it, it should be fine. If I have some time, I'll try to test it as well.

Finally, if we merge this before the rsyslog stuff, we'll need to make sure that branch gets properly updated to take these changes into account, and vice versa.

[Andy Doan (doanac)]

> === added directory 'templates'
> === added file 'templates/50-utahdefault.conf.jinja2'
> --- templates/50-utahdefault.conf.jinja2 1970-01-01 00:00:00 +0000
> +++ templates/50-utahdefault.conf.jinja2 2013-02-20 19:07:27 +0000
> @@ -0,0 +1,18 @@
> +# Default rules for rsyslog.
> +# Minimal file provided for utah installation logging
> +# For more information see rsyslog.conf(5) and /etc/rsyslog.conf
> +auth,authpriv.* /var/log/auth.log
> +*.*;auth,authpriv.none -/dev/ttyS0
> +*.*;auth,authpriv.none -/var/log/syslog
> +kern.* -/var/log/kern.log
> +mail.* -/var/log/mail.log
> +mail.err /var/log/mail.err
> +news.crit /var/log/news/news.crit
> +news.err /var/log/news/news.err
> +news.notice -/var/log/news/news.notice
> +*.emerg *
> +daemon.*;mail.*;\
> + news.err;\
> + *.=debug;*.=info;\
> + *.=notice;*.=warn |/dev/xconsole

you can delete these last 4 lines. They just cause a warning in the
target. I did them in my syslog-install branch.

[Andy Doan (doanac)]

On 02/20, Max Brustkern wrote:
> This looks pretty great. I think it would be awesome if we also templated the default preseed, libvirt XML file, and maybe even stuff like the kernel command line options and the commands we pass to cobbler, but all of those would involve more work to keep them compatible with user input. I think this is a really good step in the direction of separation of code from configuration.
>
> As far as the actual code, it all looks reasonable to me. If you've tested it, it should be fine. If I have some time, I'll try to test it as well.
>
> Finally, if we merge this before the rsyslog stuff, we'll need to make sure that branch gets properly updated to take these changes into account, and vice versa.

Its okay if you want to merge this first. I'm taking Max's branch and
mine and rebasing them. With the new rsylogd support I'm adding its
become sufficiently different that dragging the original history around
isn't going to be helpful.

[lp:~javier.collado/utah/readable-preseed]

831. By Javier Collado on 2013-02-21

Removed lines that were causing a warning (following Andy's comment)

[Javier Collado (javier.collado)]

Both of you seem to agree with the changes, so I'm merging them.

review: Approve

=== modified file 'debian/utah.install'
--- debian/utah.install	2012-12-05 16:15:01 +0000
+++ debian/utah.install	2013-02-21 08:48:18 +0000
@@ -5,3 +5,4 @@
 utah-client_*_all.deb usr/share/utah
 utah-common_*_all.deb usr/share/utah
 python-jsonschema_*_all.deb usr/share/utah
+templates usr/share/utah

=== added directory 'templates'
=== added file 'templates/50-utahdefault.conf.jinja2'
--- templates/50-utahdefault.conf.jinja2	1970-01-01 00:00:00 +0000
+++ templates/50-utahdefault.conf.jinja2	2013-02-21 08:48:18 +0000
@@ -0,0 +1,13 @@
+#  Default rules for rsyslog.
+#  Minimal file provided for utah installation logging
+#  For more information see rsyslog.conf(5) and /etc/rsyslog.conf
+auth,authpriv.*			/var/log/auth.log
+*.*;auth,authpriv.none		-/dev/ttyS0
+*.*;auth,authpriv.none		-/var/log/syslog
+kern.*				-/var/log/kern.log
+mail.*				-/var/log/mail.log
+mail.err			/var/log/mail.err
+news.crit			/var/log/news/news.crit
+news.err			/var/log/news/news.err
+news.notice			-/var/log/news/news.notice
+*.emerg				*

=== added file 'templates/casper-preseed-script.jinja2'
--- templates/casper-preseed-script.jinja2	1970-01-01 00:00:00 +0000
+++ templates/casper-preseed-script.jinja2	2013-02-21 08:48:18 +0000
@@ -0,0 +1,4 @@
+#!/bin/sh
+
+cp preseed.cfg utah-ssh-key utah-latecommand /root
+cp 50-utahdefault.conf /root/etc/rsyslog.d

=== added file 'templates/check-locks-command.jinja2'
--- templates/check-locks-command.jinja2	1970-01-01 00:00:00 +0000
+++ templates/check-locks-command.jinja2	2013-02-21 08:48:18 +0000
@@ -0,0 +1,9 @@
+{# Check lock files before package installation -#}
+while (fuser /var/lib/dpkg/lock \
+             /var/cache/apt/archives/lock \
+             /var/lib/apt/lists/lock \
+       >/dev/null 2>&1); \
+do \
+    echo "Waiting for dpkg lock to become available"; \
+    sleep 10; \
+done; \

=== added file 'templates/install-deb-command.jinja2'
--- templates/install-deb-command.jinja2	1970-01-01 00:00:00 +0000
+++ templates/install-deb-command.jinja2	2013-02-21 08:48:18 +0000
@@ -0,0 +1,2 @@
+{% include "check-locks-command.jinja2" %}
+DEBIAN_FRONTEND=noninteractive gdebi -n -q {{deb}}

=== added file 'templates/preseed-install-commands.jinja2'
--- templates/preseed-install-commands.jinja2	1970-01-01 00:00:00 +0000
+++ templates/preseed-install-commands.jinja2	2013-02-21 08:48:18 +0000
@@ -0,0 +1,5 @@
+{% for package_name in package_names -%}
+{% include "check-locks-command.jinja2" %}
+{# Install package -#}
+apt-get install -y --force-yes {{package_name}} >>{{log_file}} 2>&1; \
+{% endfor %}

=== added file 'templates/preseed-postinstall-commands.jinja2'
--- templates/preseed-postinstall-commands.jinja2	1970-01-01 00:00:00 +0000
+++ templates/preseed-postinstall-commands.jinja2	2013-02-21 08:48:18 +0000
@@ -0,0 +1,4 @@
+sed -i -e '/^exit 0$/i\
+    {{install_commands | indent}}
+'\
+/etc/rc.local;

=== added file 'templates/preseed-success-command.jinja2'
--- templates/preseed-success-command.jinja2	1970-01-01 00:00:00 +0000
+++ templates/preseed-success-command.jinja2	2013-02-21 08:48:18 +0000
@@ -0,0 +1,8 @@
+{# Note that since the chroot command is passed to sh within single quotes, the
+single quotes inside the chroot command have to be escaped using '"'"'.
+For more information, please refer to:
+http://stackoverflow.com/q/1250079/183066 -#}
+\
+    in-target sh -c '\
+        {{commands | replace('\'', '\'"\'"\'') | indent(8)}}\
+        '; 

=== added file 'templates/utah-latecommand.jinja2'
--- templates/utah-latecommand.jinja2	1970-01-01 00:00:00 +0000
+++ templates/utah-latecommand.jinja2	2013-02-21 08:48:18 +0000
@@ -0,0 +1,18 @@
+#!/bin/sh
+
+ROOTDIR=/target/root/.ssh
+USERDIR=/target/home/{{user}}/.ssh
+
+for DIR in $ROOTDIR $USERDIR
+do
+  mkdir -p $DIR
+  cp utah-ssh-key $DIR/authorized_keys
+  chmod 700 $DIR
+done
+
+chown -R root:root $ROOTDIR
+chown -R 1000:1000 $USERDIR
+
+mkdir -p /target/etc/utah
+echo "{{uuid}}" > /target/etc/utah/uuid
+echo "{{user}}" >> /target/etc/utah/users

=== modified file 'utah/config.py'
--- utah/config.py	2013-01-24 15:56:24 +0000
+++ utah/config.py	2013-02-21 08:48:18 +0000
@@ -140,6 +140,8 @@
     # sqlite database connection timeout
     # a value higher than the default is used to avoid db locking problems
     sqlite3_db_connection_timeout=30,
+    # location of the jinja2 templates used by the provisionig module
+    template_dir=os.path.join('/', 'usr', 'share', 'utah', 'templates')
 )
 
 # These depend on the local user/path, and need to be filtered out

=== modified file 'utah/provisioning/provisioning.py'
--- utah/provisioning/provisioning.py	2013-02-13 15:43:27 +0000
+++ utah/provisioning/provisioning.py	2013-02-21 08:48:18 +0000
@@ -33,6 +33,8 @@
 
 from glob import glob
 
+from jinja2 import Environment, FileSystemLoader
+
 import utah.timeout
 
 from utah import config
@@ -62,16 +64,6 @@
     * stop, uploadfiles, downloadfiles, run
       (these must be implemented separately.)
     """
-    # Used in installation commands to make sure
-    # files aren't locked
-    CHECK_LOCK_COMMAND = (
-        'while (fuser /var/lib/dpkg/lock /var/cache/apt/archives/lock '
-        '/var/lib/apt/lists/lock >/dev/null 2>&1); '
-        'do '
-        'echo "Waiting for dpkg lock to become available"; '
-        'sleep 10; '
-        'done; ')
-
     def __init__(self, arch=None, boot=None, clean=None, debug=False,
                  directory=None, image=None, dlpercentincrement=1,
                  initrd=None, installtype=None, kernel=None, machineid=None,
@@ -214,6 +206,9 @@
 
         self.finalpreseed = self.preseed
 
+        self.template_env = \
+            Environment(loader=FileSystemLoader(config.template_dir))
+
         self.logger.debug('Machine init finished')
 
     def _namesetup(self, name=None):
@@ -401,11 +396,9 @@
                     raise err
 
             remote_deb = os.path.join(tmppath, os.path.basename(deb))
-            install_command = (
-                '{}'
-                'DEBIAN_FRONTEND=noninteractive gdebi -n -q {}'
-                .format(self.CHECK_LOCK_COMMAND,
-                        remote_deb))
+            template = self.template_env.get_template(
+                'install-deb-command.jinja2')
+            install_command = template.render(deb=remote_deb)
             returncode, _stdout, stderr = self.run(install_command, root=True)
             if (returncode != 0 or
                     re.search(r'script returned error exit status \d+',
@@ -787,28 +780,12 @@
                         os.path.join(tmpdir, 'initrd.d', 'utah-ssh-key'))
 
         self.logger.info('Creating latecommand script')
+        template = self.template_env.get_template('utah-latecommand.jinja2')
+        latecommand = template.render(user=config.user,
+                                      uuid=self.uuid)
         filename = os.path.join(tmpdir, 'initrd.d', 'utah-latecommand')
-        latecommand = open(filename, 'w')
-        latecommand.write("""#!/bin/sh
-
-ROOTDIR=/target/root/.ssh
-USERDIR=/target/home/{user}/.ssh
-
-for DIR in $ROOTDIR $USERDIR
-do
-  mkdir -p $DIR
-  cp utah-ssh-key $DIR/authorized_keys
-  chmod 700 $DIR
-done
-
-chown -R root:root $ROOTDIR
-chown -R 1000:1000 $USERDIR
-
-mkdir -p /target/etc/utah
-echo "{uuid}" > /target/etc/utah/uuid
-echo "{user}" >> /target/etc/utah/users
-""".format(user=config.user, uuid=self.uuid))
-        latecommand.close()
+        with open(filename, 'w') as f:
+            f.write(latecommand)
 
     def _setuppreseed(self, tmpdir=None):
         """
@@ -864,29 +841,26 @@
             question.qname = 'ubiquity/success_command'
             log_file = '/var/log/utah-install'
 
-            install_commands = (
-                ' '.join(['{}'
-                          'apt-get install -y --force-yes {} >>{} 2>&1; '
-                          .format(self.CHECK_LOCK_COMMAND, pkg_name, log_file)
-                          for pkg_name in config.installpackages]))
-            postinstall_commands = (
-                'sed -i -e \'/^exit 0$/i{}\' /etc/rc.local;'
-                .format(install_commands)
-            )
-            chroot_command = ' '.join([install_commands, postinstall_commands])
-            # Note that since the chroot command is passed to sh within single
-            # quotes, the single quotes inside the chroot command have to be
-            # escaped using '"'"'
-            # For more information, please refer to:
-            # http://stackoverflow.com/q/1250079/183066
-            question.value.prepend(
-                'in-target sh -c \'{}\'; '
-                .format(chroot_command.replace('\'', '\'"\'"\'')))
+            template = self.template_env.get_template(
+                'preseed-install-commands.jinja2')
+            install_commands = template.render(
+                package_names=config.installpackages,
+                log_file=log_file)
+            template = self.template_env.get_template(
+                'preseed-postinstall-commands.jinja2')
+            postinstall_commands = template.render(
+                install_commands=install_commands)
+            template = self.template_env.get_template(
+                'preseed-success-command.jinja2')
+            chroot_command = template.render(
+                commands=''.join([install_commands,
+                                  postinstall_commands]))
+            question.value.prepend(chroot_command)
             question.prepend("ubiquity ubiquity/summary note")
             question.prepend("ubiquity ubiquity/reboot boolean true")
         else:
             self.logger.info('Prepending latecommand')
-            question.value.prepend('sh utah-latecommand ; ')
+            question.value.prepend('\\\n    sh utah-latecommand; ')
 
     def _rewrite_success_command(self, preseed):
         """
@@ -895,7 +869,7 @@
         """
         self.logger.info('Prepending success_command')
         question = preseed['ubiquity/success_command']
-        question.value.prepend('sh utah-latecommand ; ')
+        question.value.prepend('\\\n    sh utah-latecommand; ')
 
     def _rewrite_pkgsel_include(self, preseed):
         """
@@ -934,30 +908,22 @@
         self.logger.info('Inserting preseed into casper')
         if tmpdir is None:
             tmpdir = self.tmpdir
-        preseedscript = os.path.join(tmpdir, 'initrd.d', 'scripts',
-                                     'casper-bottom', 'utah')
-        open(preseedscript, 'w').write("""#!/bin/sh
-
-cp preseed.cfg utah-ssh-key utah-latecommand /root
-cp 50-utahdefault.conf /root/etc/rsyslog.d
-""")
-        os.chmod(preseedscript, 0755)
+        template = self.template_env.get_template(
+            'casper-preseed-script.jinja2')
+        preseedscript = template.render()
         casper_dir = os.path.join(tmpdir, 'initrd.d', 'scripts',
                                   'casper-bottom')
-        tmpfilename = os.path.join(casper_dir, 'ORDER.tmp')
-        tmpfile = open(tmpfilename, 'w')
+        filename = os.path.join(casper_dir, 'utah')
+        with open(filename, 'w') as f:
+            f.write(preseedscript)
+        os.chmod(filename, 0755)
+
         orderfilename = os.path.join(casper_dir, 'ORDER')
-        orderfile = open(orderfilename, 'r')
-        tmpfile.write("""/scripts/casper-bottom/utah
-[ -e /conf/param.conf ] && . /conf/param.conf
-""")
-        tmpfile.flush()
-        for line in orderfile:
-            tmpfile.write(line)
-            tmpfile.flush()
-        tmpfile.close()
-        orderfile.close()
-        os.rename(tmpfilename, orderfilename)
+        self._runargs(['sed', '-i',
+                       '1i/scripts/casper-bottom/'
+                       'utah\\n'
+                       '[ -e /conf/param.conf ] && . /conf/param.conf',
+                       orderfilename])
 
     def _setuplogging(self, tmpdir=None):
         """
@@ -976,27 +942,12 @@
                              "-f /var/log/syslog \n")
         else:
             self.logger.info('No inittab found, creating rsyslog config file')
+            template = self.template_env.get_template(
+                '50-utahdefault.conf.jinja2')
             conffilename = os.path.join(tmpdir, 'initrd.d',
                                         '50-utahdefault.conf')
-            open(conffilename, 'w').write("""
-#  Default rules for rsyslog.
-#  Minimal file provided for utah installation logging
-#  For more information see rsyslog.conf(5) and /etc/rsyslog.conf
-auth,authpriv.*			/var/log/auth.log
-*.*;auth,authpriv.none		-/dev/ttyS0
-*.*;auth,authpriv.none		-/var/log/syslog
-kern.*				-/var/log/kern.log
-mail.*				-/var/log/mail.log
-mail.err			/var/log/mail.err
-news.crit			/var/log/news/news.crit
-news.err			/var/log/news/news.err
-news.notice			-/var/log/news/news.notice
-*.emerg				*
-daemon.*;mail.*;\
-	news.err;\
-	*.=debug;*.=info;\
-	*.=notice;*.=warn	|/dev/xconsole
-""")
+            with open(conffilename, 'w') as f:
+                f.write(template.render())
 
     def _repackinitrd(self, tmpdir=None):
         """