Merge lp:~nuclearbob/utah/internal-jsonschema into lp:utah

[Max Brustkern]

To address https://bugs.launchpad.net/utah/+bug/1042895, this downloads a python-jsonschema deb from the stable PPA during postinstall, then installs that before installing the client deb.

[Javier Collado (javier.collado)]

I don't really like much the idea of downloading the jsonschema package from the PPA when the server is installed. However, it's true that it makes things easier when the VM has connection problems. What happens when the machine where the server is installed has connection problems is something I haven't tested.

Anyway, I have some suggestions:

- Drop code to add the PPA to the client in the latecommand (isn't going to be used after all)

- Update "debian/control" for "utah-client" to have "python-jsonschema (> 0.2)". This will prevent the client from being installed when the expected package version isn't available.

- Replace `os.listdir`:

  for myfile in os.listdir(debpath):
      if (myfile.startswith('python-jsonschema_') and
          myfile.endswith('_all.deb')):
          debpath = os.path.join(debpath, myfile)
          break
  debpath = os.path.normpath(debpath)
  if not os.path.isfile(debpath):
      raise UTAHProvisioningException('python-jsonschema package file '
                                      'not found in {path}\n'
                                      .format(path=debpath))
  return debpath

with `glob.glob` :

  from glob import glob

  deb_file_glob = os.path.join(debpath, 'python-jsonschema_*_all.deb')
  deb_files = glob(deb_file_glob)
  if not deb_files:
      raise UTAHProvisioningException('python-jsonschema package file '
                                      'not found in {path}\n'
                                      .format(path=debpath))
  deb_file = deb_files[0]
  return deb_file

review: Needs Fixing

[Max Brustkern (nuclearbob)]

I think we've already determined we don't want to carry the jsonschema code in utah, so we have a few other options for how to get it:
- Add the PPA to the client system
  If we don't have internet access from the client, this will fail.
- Carry a jsonschema deb file in the utah tree
  This is probably a worse idea than carrying the source, so I don't think we should consider it unless we rule out all the other options.
- Download the jsonschema deb from the PPA during the package build.
  I don't know how to make this work on the buildd right now. I'm not sure if we can.
- Use a recipe that inserts a branch containing the jsonschema code into the utah build tree, and build a deb during package build time.
  If we have the jsconschema code in a branch in launchpad, this seems doable. I like this option the best, but there are probably drawbacks to it that I'm not thinking of.

Let me know what you think, or if you can think of other options I've left out, and we can figure something out.

[Javier Collado (javier.collado)]

@Max

According to this link:
https://help.launchpad.net/Packaging/PPA/BuildingASourcePackage#Dependencies
it should be possible to add a "Build-Depends" section against a "python-jsonschema" version found in a PPA.

After that, it should be possible to get the sources for "python-jsonschema", build it and include as part of the "utah" package itself. This way if the package build succeeds, then we now that the .deb for jsonschema will be available without any risk of a postinstall script failure.

What do you think about this? If you like the idea, I can create a new branch and give it a try.

[Max Brustkern (nuclearbob)]

I like this idea, but I'm not sure if the buildd's have access to PPAs. If you want to give it a try, go for it. If it works, it's definitely my favorite idea so far.

[Javier Collado (javier.collado)]

I've spent some time on some changes based on your branch to get the package from the PPA at build time instead of at installation time:
lp:~javier.collado/utah/bug1042895

In my machine, it worked fine and now I'm waiting for a build in my PPA to make sure that the build dependency can be retrieved correctly in the launchpad environment. If that succeeds, then I'll create a merge proposal.

[Max Brustkern (nuclearbob)]

I'll be interested to see how that goes. I've had success building locally with similar setups in the past, but I'm concerned about the process working on a buildd. I hope it does, and then we can finally get this checked off.

=== modified file 'debian/control'
--- debian/control	2012-08-14 16:55:16 +0000
+++ debian/control	2012-08-29 15:51:18 +0000
@@ -11,7 +11,7 @@
 Package: utah
 Architecture: all
 Depends: ${misc:Depends}, ${python:Depends},
-         bzr, bsdtar, git, libvirt-bin, lzma,
+         bzr, bsdtar, git, libvirt-bin, lzma, dpkg-dev,
          python-apt, python-libvirt, python-jsonschema,
          python-netifaces, python-paramiko, python-psutil, python-yaml,
          utah-client (=${binary:Version})

=== modified file 'debian/utah.postinst'
--- debian/utah.postinst	2012-08-14 15:09:17 +0000
+++ debian/utah.postinst	2012-08-29 15:51:18 +0000
@@ -1,5 +1,15 @@
 #!/bin/bash -e
 
+function getjsonschema
+{
+    cd /usr/share/utah
+    PPA="http://ppa.launchpad.net/utah/stable/ubuntu"
+    ARCH=$(dpkg-architecture -qDEB_HOST_ARCH)
+    SERIES=$(lsb_release -sc)
+    wget "$PPA/$(wget "$PPA/dists/$SERIES/main/binary-$ARCH/Packages" -O - 2>/dev/null | grep "python-jsonschema_.*_all.deb" | awk '{print $2}')"
+    cd - > /dev/null
+}
+
 function configure
 {
     if [ -f "/etc/utah/dns" ]
@@ -92,6 +102,8 @@
         usermod -a -G utah $USER
       done
     fi
+
+    getjsonschema
 }
 
 case $1 in

=== added file 'debian/utah.prerm'
--- debian/utah.prerm	1970-01-01 00:00:00 +0000
+++ debian/utah.prerm	2012-08-29 15:51:18 +0000
@@ -0,0 +1,19 @@
+#!/bin/bash -e
+
+function cleanup
+{
+    rm -f /usr/share/utah/python-jsonschema_*_all.deb
+}
+
+case "$1" in
+    remove|purge|abort-install)
+        cleanup
+        ;;
+    upgrade|failed-upgrade|abort-upgrade)
+        ;;
+    *)
+        echo "$0: didn't understand being called with '$1'" 1>&2
+        exit 0
+        ;;
+esac
+#DEBHELPER#

=== modified file 'utah/provisioning/provisioning.py'
--- utah/provisioning/provisioning.py	2012-08-24 18:26:16 +0000
+++ utah/provisioning/provisioning.py	2012-08-29 15:51:18 +0000
@@ -255,6 +255,25 @@
         self.logger.debug('Client deb path is ' + debpath)
         return debpath
 
+    def getjsonschemadeb(self):
+        """
+        Return the path of the .deb file for the python-jsonschema installed
+        as part of the main package.
+        """
+        debpath = os.path.join('/usr', 'share', 'utah')
+        for myfile in os.listdir(debpath):
+            if (myfile.startswith('python-jsonschema_') and
+                myfile.endswith('_all.deb')):
+                debpath = os.path.join(debpath, myfile)
+                break
+        debpath = os.path.normpath(debpath)
+        if not os.path.isfile(debpath):
+            raise UTAHProvisioningException('python-jsonschema package file '
+                                            'not found in {path}\n'
+                                            .format(path=debpath))
+        self.logger.debug('python-jsconschema deb path is ' + debpath)
+        return debpath
+
     def installclient(self):
         """
         Install the utah-client package on the
@@ -262,29 +281,30 @@
         """
         self.logger.info('Installing client deb on machine')
         tmppath = os.path.normpath('/tmp')
-        clientdeb = self.getclientdeb()
-        try:
-            self.uploadfiles([clientdeb], tmppath)
-        except UTAHProvisioningException as err:
+        debs = [self.getjsonschemadeb(), self.getclientdeb()]
+        for deb in debs:
             try:
-                for myfile in err.files:
-                    if 'utah-client' in myfile:
-                        raise UTAHProvisioningException('Client file not '
-                                + 'found; UTAH was probably updated '
-                                + 'during the run.  Please try again.',
-                                retry=True)
-                raise err
-            except AttributeError:
-                raise err
+                self.uploadfiles([deb], tmppath)
+            except UTAHProvisioningException as err:
+                try:
+                    for myfile in err.files:
+                        if deb in myfile:
+                            raise UTAHProvisioningException('File not '
+                                    'found: ' + deb + '; UTAH was probably '
+                                    'updated during the run. '
+                                    'Please try again.', retry=True)
+                    raise err
+                except AttributeError:
+                    raise err
 
-        remote_clientdeb = os.path.join(tmppath, os.path.basename(clientdeb))
-        install_command = ('DEBIAN_FRONTEND=noninteractive '
-                           'gdebi -n -q {}'
-                           .format(remote_clientdeb))
-        returncode, stdout, stderr = self.run(install_command, root=True)
-        if (returncode != 0
-            or re.search(r'script returned error exit status \d+', stderr)):
-            raise UTAHProvisioningException('Failed to install client\n')
+            remote_deb = os.path.join(tmppath, os.path.basename(deb))
+            install_command = ('DEBIAN_FRONTEND=noninteractive '
+                               'gdebi -n -q {}'
+                               .format(remote_deb))
+            returncode, stdout, stderr = self.run(install_command, root=True)
+            if (returncode != 0
+                or re.search(r'script returned error exit status \d+', stderr)):
+                raise UTAHProvisioningException('Failed to install client\n')
 
     def _provision(self):
         """