Merge lp:~danilo/launchpad/bug-580345-devel into lp:launchpad

[Данило Шеган]

Add more logging to generate-translation-templates scripts that run on the build slaves.

= Bug 580345 =

At the moment, we have very terse build job debug output. Considering how build farm (or at least translation jobs on it) works in such a way that we lose the log after a job completes, we should at least have an option of better output during build time (which we can track on launchpad.net/builders).

After this branch lands, we need to prepare another canonical-buildd package and roll it out to build slaves (only LaMont can do it).

== Tests ==

 bin/test -vvct translationtemplates

== QA ==

QAd on dogfood (rubidium slave) by following https://dev.launchpad.net/Translations/GenerateTemplatesOnTestServers

[Henning Eggers (henninge)]

Thanks for adding this!

<henninge> danilos: wasn't there some problem with 'debug_exec', too?
<danilos> henninge, it's fixed (original branch was against production-devel, but that doesn't make any difference for build slaves)
<danilos> henninge, I had a bashism ("function debug_func {" instead of "debug_func()" which works everywhere)
* deryck hat die Verbindung getrennt (Quit: Leaving)
<henninge> danilos: ah yes, I see.
<henninge> danilos: the python --version is pretty useless, though.
<henninge> danilos: you should use the python in the chroot. This script is still running outside of it.
<danilos> henninge, I know, but can't do that in the shell script :)
<henninge> not?
<danilos> henninge, well, chroot would first have to be set-up and that happens only at the end
<henninge> danilos: but python works from almost anywhere.
<henninge> let me try that ... ;)
<henninge> danilos: yup, works. Just do "$BUILD_CHROOT/usr/bin/python --version"
<danilos> henninge, sure

review: Approve (code)

=== modified file 'lib/canonical/buildd/generate-translation-templates'
--- lib/canonical/buildd/generate-translation-templates	2010-03-12 08:32:39 +0000
+++ lib/canonical/buildd/generate-translation-templates	2010-05-19 15:34:37 +0000
@@ -36,21 +36,31 @@
 BUILD_CHROOT="$HOME/build-$BUILDID/chroot-autobuild"
 USER=$(whoami)
 
+# Debug output.
+echo "Running as $USER for build $BUILDID on $BRANCH_URL."
+echo "Results expected in $RESULT_NAME."
+
 BUILDD_PACKAGE=canonical/buildd
 POTTERY=$BUILDD_PACKAGE/pottery
 # The script should be smarter about detecting the python version.
 PYMODULES=/usr/lib/pymodules/python2.6
+echo -n "Default Python in the chroot is: "
+$BUILD_CHROOT/usr/bin/python --version
 
 GENERATE_SCRIPT=$PYMODULES/$POTTERY/generate_translation_templates.py
 
+debug_exec() {
+   echo "Executing '$1'..."
+   $1 || echo "Got error $? from '$1'."
+}
+
 # Copy pottery files to chroot.
-$SUDO $MKDIR -p $BUILD_CHROOT$PYMODULES/$BUILDD_PACKAGE
-$SUDO $TOUCH $BUILD_CHROOT$PYMODULES/canonical/__init__.py
-$SUDO $TOUCH $BUILD_CHROOT$PYMODULES/canonical/buildd/__init__.py
-$SUDO $CP -r $BUILDD_HOME/$POTTERY $BUILD_CHROOT$PYMODULES/$BUILDD_PACKAGE
-$SUDO $CHMOD -R go+rX $BUILD_CHROOT$PYMODULES/canonical
-$SUDO $CHMOD 755 $BUILD_CHROOT$GENERATE_SCRIPT
-
+debug_exec "$SUDO $MKDIR -vp $BUILD_CHROOT$PYMODULES/$BUILDD_PACKAGE"
+debug_exec "$SUDO $TOUCH $BUILD_CHROOT$PYMODULES/canonical/__init__.py"
+debug_exec "$SUDO $TOUCH $BUILD_CHROOT$PYMODULES/canonical/buildd/__init__.py"
+debug_exec "$SUDO $CP -vr $BUILDD_HOME/$POTTERY $BUILD_CHROOT$PYMODULES/$BUILDD_PACKAGE"
+debug_exec "$SUDO $CHMOD -v -R go+rX $BUILD_CHROOT$PYMODULES/canonical"
+debug_exec "$SUDO $CHMOD -v 755 $BUILD_CHROOT$GENERATE_SCRIPT"
 
 # Enter chroot, switch back to unprivileged user, execute the generate script.
 $SUDO $CHROOT $BUILD_CHROOT \
 
=== modified file 'lib/canonical/buildd/pottery/generate_translation_templates.py'
--- lib/canonical/buildd/pottery/generate_translation_templates.py	2010-03-18 13:27:47 +0000
+++ lib/canonical/buildd/pottery/generate_translation_templates.py	2010-05-19 15:34:37 +0000
@@ -8,6 +8,8 @@
 import sys
 import tarfile
 
+import logging
+
 from bzrlib.branch import Branch
 from bzrlib.export import export
 
@@ -31,15 +33,27 @@
         self.work_dir = work_dir
         self.branch_spec = branch_spec
         self.result_name = result_name
+        self.logger = self._setupLogger()
+
+    def _setupLogger(self):
+        """Sets up and returns a logger."""
+        logger = logging.getLogger("generate-templates")
+        logger.setLevel(logging.DEBUG)
+        ch = logging.StreamHandler()
+        ch.setLevel(logging.DEBUG)
+        logger.addHandler(ch)
+        return logger
 
     def _getBranch(self):
         """Set `self.branch_dir`, and check out branch if needed."""
         if ':' in self.branch_spec:
             # This is a branch URL.  Check out the branch.
             self.branch_dir = os.path.join(self.work_dir, 'source-tree')
+            self.logger.info("Getting remote branch %s..." % self.branch_spec)
             self._checkout(self.branch_spec)
         else:
             # This is a local filesystem path.  Use the branch in-place.
+            self.logger.info("Using local branch %s..." % self.branch_spec)
             self.branch_dir = self.branch_spec
 
     def _checkout(self, branch_url):
@@ -48,24 +62,33 @@
         The branch is checked out to the location specified by
         `self.branch_dir`.
         """
+        self.logger.info("Opening branch %s..." % branch_url)
         branch = Branch.open(branch_url)
+        self.logger.info("Getting branch revision tree...")
         rev_tree = branch.basis_tree()
+        self.logger.info("Exporting branch to %s..." % self.branch_dir)
         export(rev_tree, self.branch_dir)
+        self.logger.info("Exporting branch done.")
 
     def _makeTarball(self, files):
         """Put the given files into a tarball in the working directory."""
         tarname = os.path.join(self.work_dir, self.result_name)
+        self.logger.info("Making tarball with templates in %s..." % tarname)
         tarball = tarfile.open(tarname, 'w|gz')
         files = [name for name in files if not name.endswith('/')]
         for path in files:
             full_path = os.path.join(self.branch_dir, path)
+            self.logger.info("Adding template %s..." % full_path)
             tarball.add(full_path, path)
         tarball.close()
+        self.logger.info("Tarball generated.")
 
     def generate(self):
         """Do It.  Generate templates."""
+        self.logger.info("Generating templates for %s." % self.branch_spec)
         self._getBranch()
         pots = intltool.generate_pots(self.branch_dir)
+        self.logger.info("Generated %d templates." % len(pots))
         if len(pots) > 0:
             self._makeTarball(pots)
         return 0