Merge lp:~javier.collado/ubuntu-test-runlists/jenkins-smoketest-setup-logpath into lp:~canonical-ci-engineering/ubuntu-test-runlists/jenkins-smoketest-setup

[Javier Collado]

This branch updates the script that generates the smoke testing jobs to include
`logpath` in the configuration file that is created on the fly. This way,
syslog files are written to the workspace and there isn't any reuse of files
from `/var/log/utah` to avoid problems such as the one in bug1160696.

To verify the changes work fine, I've created a few temporary jobs in the
private jenkins instance whose name is the same as the smoke testing ones with
the `jcollado-` prefix. They have been working correctly for the last three
daily images, so I think the changes are ready to be merged.

[Max Brustkern (nuclearbob)]

It looks like this won't grab other log files mentioned by the output of run_utah_tests, such as a preseed file, or any files fetched with the --files argument. I guess we could use --outdir to make sure all the fetched files end up in the right place. I'm not sure if the preseed file will automatically use outdir or not, but maybe it should.

[Javier Collado (javier.collado)]

@Max

Thanks for pointing that out. I'll review where those files will be stored if requested.

[Javier Collado (javier.collado)]

I've run a test with `--outputpreseed --files /etc/lsb_release` and both of
them were stored in the log directory after the test execution.

Looking at the code, I see that for the preseed `config.logpath` is always used
to figure out were to write the file. Regarding the `--files` option, I see
that config.logpath is used by default unless `--outdir` is specified.

In my opinion the `outdir` should be used also for the preseed if defined to
store that file in some other location because that preseed isn't really a log
file.

Anyway, for the main concern about this merge request, I think we're safe. For
the future, defining an `outdir` in the configuration file generated on the fly
and updating UTAH code to use it for the preseed would be nice to separate log
files and data files.

[Max Brustkern (nuclearbob)]

I think your suggestions for future improvements are good, and your test results indicate we should be ready to merge.

review: Approve

=== modified file 'setup-jobs.py'
--- setup-jobs.py	2013-03-27 11:28:47 +0000
+++ setup-jobs.py	2013-04-02 11:40:20 +0000
@@ -284,11 +284,6 @@
 
                 command = command.format(arch=arch)
                 jobtype = 'script' if scripts_url else 'runlist'
-                jobname = '-'.join((args.series, installtype, arch,
-                                    'smoke', name))
-                if args.prefix:
-                    jobname = '{}-{}'.format(args.prefix, jobname)
-                logging.info('Processing job ' + jobname)
 
                 template = environment.get_template('smoke.xml.jinja')
                 smokexml = template.render(name=name,

=== modified file 'templates/smoke.xml.jinja'
--- templates/smoke.xml.jinja	2013-03-27 11:02:34 +0000
+++ templates/smoke.xml.jinja	2013-04-02 11:40:20 +0000
@@ -23,14 +23,18 @@
 UTAH_CONFIG_DIR=${WORKSPACE}/config
 mkdir -p ${UTAH_CONFIG_DIR}
 UTAH_CONFIG_FILE=${UTAH_CONFIG_DIR}/utah.cfg
-UTAH_LOG_FILE=${WORKSPACE}/utah-server.log
-UTAH_DEBUGLOG_FILE=${WORKSPACE}/utah-server-debug.log
-UTAH_SSH_LOG_FILE=${WORKSPACE}/utah-server-ssh.log
+LOG_DIR=${WORKSPACE}/log
+mkdir -p ${LOG_DIR}
+chmod a+w ${LOG_DIR}
+UTAH_LOG_FILE=${LOG_DIR}/utah-server.log
+UTAH_DEBUGLOG_FILE=${LOG_DIR}/utah-server-debug.log
+UTAH_SSH_LOG_FILE=${LOG_DIR}/utah-server-ssh.log
 cat <<EOF > ${UTAH_CONFIG_FILE}
 {
     "debuglog": "${UTAH_DEBUGLOG_FILE}",
     "logfile": "${UTAH_LOG_FILE}",
-    "ssh_logfile": "${UTAH_SSH_LOG_FILE}"
+    "ssh_logfile": "${UTAH_SSH_LOG_FILE}",
+    "logpath": "${LOG_DIR}"
 }
 EOF
 for file in $UTAH_LOG_FILE $UTAH_DEBUGLOG_FILE $UTAH_SSH_LOG_FILE; do
@@ -38,18 +42,15 @@
     chmod a+w ${file}
 done
 
-sudo -u utah -i UTAH_CONFIG_DIR=${UTAH_CONFIG_DIR} {{command}} -x /etc/utah/bridged-network-vm.xml > utah-server-stdout.log
+sudo -u utah -i UTAH_CONFIG_DIR=${UTAH_CONFIG_DIR} {{command}} -x /etc/utah/bridged-network-vm.xml > ${LOG_DIR}/utah-server-stdout.log
 RETCODE=$?
 
-# Gather logs
-# Log files from the client are in the server stdout prefixed with a tab character
-CLIENT_LOG_FILES=$(grep -P "^\t/" utah-server-stdout.log | sed 's/^\t//' | xargs)
-mkdir clientlogs
-cp ${CLIENT_LOG_FILES} clientlogs
-MACHINE_NAME=$(grep "^Running on machine:" utah-server-stdout.log | sed 's/^Running on machine: //')
-SYSLOG_FILE=${MACHINE_NAME}.syslog.log
-sudo cp /var/log/utah/${SYSLOG_FILE} .
-sudo chmod a+r ${SYSLOG_FILE}
+# Make sure syslog file is readable to add it as artifact
+[ -n "$(find ${LOG_DIR} -name '*.syslog.log')" ] && sudo -u utah chmod a+r ${LOG_DIR}/*.syslog.log
+
+# Move yaml file directory to the workspace since it's not a log file, but a
+# results files
+[ -n "$(find ${LOG_DIR} -name '*.yaml')" ] && mv ${LOG_DIR}/*.yaml ${WORKSPACE}
 
 exit $RETCODE
         </command>