Merge lp:~javier.collado/utah/bug1042895 into lp:utah

[Javier Collado]

This branch adds a few changes on top of lp:~nuclearbob/utah/internal-jsonschema to get the python-jsonschema package into the utah package at build time instead of at installation time. This means that when utah package is installed, the server can install the client in a machine that doesn't have access to the PPA where the python-jsonschema is available.

For more information, have a look at the discussion in this merge proposal:
https://code.launchpad.net/~nuclearbob/utah/internal-jsonschema/+merge/121891

For a log of of a successful build, have a look at this link (explanation below):
https://launchpadlibrarian.net/118421199/buildlog_ubuntu-precise-i386.utah_0.4ubuntu70-rev698~precise1~ppa2_BUILDING.txt.gz

The build above happened in my PPA (ppa:javier.collado/ppa) with a dependency to the utah stable ppa (ppa:utah/stable). In the log it can be seen that the PPA is added to the sources:
RUN: /usr/share/launchpad-buildd/slavebin/override-sources-list ['override-sources-list', '17aa1ff55999741047f67cee641dfd0183a08439', 'deb http://ppa.launchpad.net/javier.collado/ppa/ubuntu precise main', 'deb http://ppa.launchpad.net/utah/stable/ubuntu precise main', 'deb http://ftpmaster.internal/ubuntu precise main restricted universe multiverse', 'deb http://ftpmaster.internal/ubuntu precise-security main restricted universe multiverse', 'deb http://ftpmaster.internal/ubuntu precise-updates main restricted universe multiverse']
Overriding sources.list in build-17aa1ff55999741047f67cee641dfd0183a08439

Later, the build-depends in the packages is retrieved:
Build-Depends: debhelper (>= 7.0.50~), devscripts, python-all, python-sphinx, python-jsonschema (>= 0.5~)

and the right version of the package is installed:
Setting up python-jsonschema (0.5-0~ppa1) ...

When the package builds, the Makefile commands get the binary package to make it available for the packaging:
apt-get download python-jsonschema
Get:1 Downloading python-jsonschema 0.5-0~ppa1 [10.3 kB]
Fetched 10.3 kB in 0s (0 B/s)

For the utah PPAs, since the package is available in both of them, the configuration to add a PPA dependency won't be needed as the PPA in which the package is build seems to be added automatically to the sources, so there shouldn't be any problem in building the package in any of them.

[Max Brustkern (nuclearbob)]

I'm still reviewing this, but on lines 41-43 of the merge, a number of dependencies are added to utah which are also dependencies of utah-client (bzr, git, python-yaml.) I think we had a previous merge where we determined that dependencies of utah-client don't also need to be dependencies of utah, since utah depends on utah-client. I'm guessing this was branched off before that happened. If it would be helpful to find the old merge, let me know, and I can do that.

[Max Brustkern (nuclearbob)]

From what I understand, the recommended syntax in the .install file in the debian directory does not have a leading slash on absolute paths so that different build environments can put the file in a correct relative path if necessary. If the package built correctly, it's not a big deal, but for consistency it might be good to remove the slash from debian/utah.install on line 56 of the merge.

[Max Brustkern (nuclearbob)]

Finally, is the prerm script necessary? Shouldn't package removal take care of that file? If we do need that prerm, is there a reason we need it only for the jsonschema deb and not for the utah-client deb?

I like the rest of it. Let me know what you think about those points, and we can get those tests working again.

review: Needs Information

[lp:~javier.collado/utah/bug1042895]

699. By Javier Collado on 2012-10-05

Reverted utah dependencies as they are in the dev branch

700. By Javier Collado on 2012-10-05

Removed initial slash from path

701. By Javier Collado on 2012-10-05

Added missing comma

702. By Javier Collado on 2012-10-05

Removed prerm script

The .deb file is already removed by apt, so there's no need to remove it manually

[Javier Collado (javier.collado)]

@Max

Thanks for your review because those were three good catches.

1. Dependencies
I've removed them to match what we currently have in the dev branch. dpkg-dev was in the merge, but isn't in the dev branch. Do you know if we're actually using anything from there?

2. Absolute path in .install files
I've never read the official documentation, but I've always seen relative paths as you said.

3. prerm script
That's correct, the prerm script is no longer needed. Just for my own curiosity I tried to removed the package and didn't get any error. Anyway, without the script everything works exactly the same way.

[Max Brustkern (nuclearbob)]

1. I think at one point I was trying to use dpkg-architecture to grab the arch, but we get that from uname now, so I don't think dpkg-dev is needed. A quick look through the sources shows us only using the dpkg command itself, and not any of the dpkg-* stuff in dpkg-dev, so I'm going to go ahead and merge without that. Maybe we can get those jobs working today!

review: Approve

=== modified file 'Makefile'
--- Makefile	2012-09-05 07:43:32 +0000
+++ Makefile	2012-10-05 14:20:31 +0000
@@ -1,5 +1,8 @@
 PYC_PATTERN?=*.pyc
-all: client conf/utah/bootspeed-preseed.cfg conf/utah/bridged-network-vm.xml conf/utah/config manpages
+all: python-jsonschema client conf/utah/bootspeed-preseed.cfg conf/utah/bridged-network-vm.xml conf/utah/config manpages
+
+python-jsonschema:
+	apt-get download python-jsonschema
 
 client:
 	[ -d "client" ] || mkdir client
@@ -30,4 +33,4 @@
 	rm -f utah-*
 	cd docs && make clean
 
-.PHONY: client clean manpages
+.PHONY: python-jsonschema client clean manpages

=== modified file 'debian/control'
--- debian/control	2012-09-14 11:11:38 +0000
+++ debian/control	2012-10-05 14:20:31 +0000
@@ -3,7 +3,8 @@
 X-Python-Version: >= 2.5
 Priority: optional
 Maintainer: Max Brustkern <[e-mail]>
-Build-Depends: debhelper (>= 7.0.50~), devscripts, python-all, python-sphinx
+Build-Depends: debhelper (>= 7.0.50~), devscripts,
+               python-all, python-sphinx, python-jsonschema (>= 0.5~)
 Standards-Version: 3.9.3
 Homepage: https://code.launchpad.net/utah
 Vcs-Bzr: https://code.launchpad.net/utah

=== modified file 'debian/utah.install'
--- debian/utah.install	2012-09-26 15:20:13 +0000
+++ debian/utah.install	2012-10-05 14:20:31 +0000
@@ -2,4 +2,4 @@
 conf/utah etc
 examples usr/share/utah
 utah-client_*_all.deb usr/share/utah
-
+python-jsonschema_*_all.deb usr/share/utah

=== modified file 'utah/provisioning/provisioning.py'
--- utah/provisioning/provisioning.py	2012-10-03 10:50:29 +0000
+++ utah/provisioning/provisioning.py	2012-10-05 14:20:31 +0000
@@ -17,6 +17,7 @@
 import paramiko
 import re
 import apt.cache
+from glob import glob
 
 from utah.iso import ISO
 from utah.preseed import Preseed
@@ -256,6 +257,21 @@
         self.logger.debug('Client deb path is ' + debpath)
         return debpath
 
+    def getjsonschemadeb(self):
+        """
+        Return the path of the .deb file for the python-jsonschema installed
+        as part of the main package.
+        """
+        debpath = os.path.join('/usr', 'share', 'utah')
+        deb_file_glob = os.path.join(debpath, 'python-jsonschema_*_all.deb')
+        deb_files = glob(deb_file_glob)
+        if not deb_files:
+            raise UTAHProvisioningException('python-jsonschema package file '
+                                            'not found in {path}\n'
+                                            .format(path=debpath))
+        deb_file = deb_files[0]
+        return deb_file
+
     def installclient(self):
         """
         Install the utah-client package on the
@@ -263,36 +279,37 @@
         """
         self.logger.info('Installing client deb on machine')
         tmppath = os.path.normpath('/tmp')
-        clientdeb = self.getclientdeb()
-        try:
-            self.uploadfiles([clientdeb], tmppath)
-        except UTAHProvisioningException as err:
+        debs = [self.getjsonschemadeb(), self.getclientdeb()]
+        for deb in debs:
             try:
-                for myfile in err.files:
-                    if 'utah-client' in myfile:
-                        raise UTAHProvisioningException('Client file not '
-                                + 'found; UTAH was probably updated '
-                                + 'during the run.  Please try again.',
-                                retry=True)
-                raise err
-            except AttributeError:
-                raise err
-        if self.run('export DEBIAN_FRONTEND=noninteractive ; while '
-                 + '(fuser /var/lib/dpkg/lock >/dev/null 2>&1) ; do echo '
-                 + '"Waiting for dpkg lock to become available" ; sleep 10 ; '
-                 + 'done ; dpkg -i ' + os.path.join(tmppath,
-                 os.path.split(self.getclientdeb())[1])
-                 + ' || apt-get -y -f install', root=True) != 0:
-            raise UTAHProvisioningException('Failed to install client')
+                self.uploadfiles([deb], tmppath)
+            except UTAHProvisioningException as err:
+                try:
+                    for myfile in err.files:
+                        if deb in myfile:
+                            raise UTAHProvisioningException('File not '
+                                    'found: ' + deb + '; UTAH was probably '
+                                    'updated during the run. '
+                                    'Please try again.', retry=True)
+                    raise err
+                except AttributeError:
+                    raise err
 
-        remote_clientdeb = os.path.join(tmppath, os.path.basename(clientdeb))
-        install_command = ('DEBIAN_FRONTEND=noninteractive '
-                           'gdebi -n -q {}'
-                           .format(remote_clientdeb))
-        returncode, stdout, stderr = self.run(install_command, root=True)
-        if (returncode != 0
-            or re.search(r'script returned error exit status \d+', stderr)):
-            raise UTAHProvisioningException('Failed to install client\n')
+            remote_deb = os.path.join(tmppath, os.path.basename(deb))
+            install_command = (
+                    'DEBIAN_FRONTEND=noninteractive; '
+                    'while (fuser /var/lib/dpkg/lock >/dev/null 2>&1); '
+                    'do '
+                    'echo "Waiting for dpkg lock to become available"; '
+                    'sleep 10; '
+                    'done ; '
+                    'gdebi -n -q {}'
+                    .format(remote_deb))
+            returncode, stdout, stderr = self.run(install_command, root=True)
+            if (returncode != 0 or
+                    re.search(r'script returned error exit status \d+',
+                              stderr)):
+                raise UTAHProvisioningException('Failed to install client\n')
 
     def _provision(self):
         """
@@ -757,9 +774,6 @@
 mkdir -p /target/etc/utah
 echo "{uuid}" > /target/etc/utah/uuid
 echo "{user}" >> /target/etc/utah/users
-
-# Add stable PPA to get latest python-jsonschema version
-chroot /target sh -c 'add-apt-repository -y ppa:utah/stable; apt-get update'
 """.format(user=config.user, uuid=self.uuid))
         latecommand.close()