Merge lp:~pwlars/ci-core-jenkins/set-version-description into lp:ci-core-jenkins

[Paul Larson]

Set the build description to the full version ID, including device tarball and rootfs version

This should get us to a full image ID that can be used by the dashboard. I suspect some other places are going to have to change too though, or else the image id won't match. We should probably not be testing against channels with the release name in the name, but rather the alias "ubuntu-core/devel-proposed" for example. This matches what we were asked to do on the touch image testing, because it ensures continuous results for a channel. The downside (for us) is that the revision numbers are not guaranteed to be sequential. So when the "w" release opens, they will start over at 1.

We need the full image version for other reasons too though, as the image_id itself does not give the complete picture of the component versions that went into it.

[Para Siva (psivaa)]

Thanks for the description setting. I've just commented on the image_id that's being passed to the result fetcher job.

review: Needs Information

[Paul Larson (pwlars)]

Right, that's what I meant by needing to change the image id in other places. I think that either 1. we need to separate out the short version id from the full description, or 2. (if 1 is not enough) use the full version spec everywhere.

[Paul Larson (pwlars)]

This extracts just the short id before calling fetch-latest-result

review: Needs Resubmitting

[Para Siva (psivaa)]

An inline comment about extracting SHORT_IMAGE_ID

review: Needs Fixing

[Francis Ginther (fginther)]

Piling on with another comment.

Otherwise looks good.

review: Needs Fixing

[Paul Larson (pwlars)]

Good catch, thanks for the review. Fixed in r12

review: Needs Resubmitting

[Para Siva (psivaa)]

Thanks for the work and the fixes, Paul. +1

review: Approve

[Francis Ginther (fginther)]

Approve

review: Approve

[Ubuntu CI Bot (uci-bot)]

The attempt to merge lp:~pwlars/ci-core-jenkins/set-version-description into lp:ci-core-jenkins failed. Below is the output from the failed tests.

Using base prefix '/usr'
New python executable in /tmp/venv-ci-core-jenkins_rt0y8it/bin/python3
Also creating executable in /tmp/venv-ci-core-jenkins_rt0y8it/bin/python
Installing setuptools, pip...done.
Running virtualenv with interpreter /usr/bin/python3
Ignoring indexes: https://pypi.python.org/simple/
Downloading/unpacking requests==2.5.3 (from -r requirements.txt (line 1))
Installing collected packages: requests
Successfully installed requests
Cleaning up...
Downloading/unpacking testtools==1.7.1 (from -r test_requirements.txt (line 1))
Downloading/unpacking extras (from testtools==1.7.1->-r test_requirements.txt (line 1))
  Downloading extras-0.0.3.tar.gz
  Running setup.py (path:/tmp/venv-ci-core-jenkins_rt0y8it/build/extras/setup.py) egg_info for package extras

Downloading/unpacking python-mimeparse (from testtools==1.7.1->-r test_requirements.txt (line 1))
  Downloading python-mimeparse-0.1.4.tar.gz
  Running setup.py (path:/tmp/venv-ci-core-jenkins_rt0y8it/build/python-mimeparse/setup.py) egg_info for package python-mimeparse

Downloading/unpacking unittest2>=1.0.0 (from testtools==1.7.1->-r test_requirements.txt (line 1))
Downloading/unpacking traceback2 (from testtools==1.7.1->-r test_requirements.txt (line 1))
  Downloading traceback2-1.4.0-py2.py3-none-any.whl
Downloading/unpacking six>=1.4 (from unittest2>=1.0.0->testtools==1.7.1->-r test_requirements.txt (line 1))
  Downloading six-1.9.0-py2.py3-none-any.whl
Downloading/unpacking argparse (from unittest2>=1.0.0->testtools==1.7.1->-r test_requirements.txt (line 1))
  Downloading argparse-1.3.0-py2.py3-none-any.whl
Downloading/unpacking linecache2 (from traceback2->testtools==1.7.1->-r test_requirements.txt (line 1))
  Downloading linecache2-1.0.0-py2.py3-none-any.whl
Installing collected packages: testtools, extras, python-mimeparse, unittest2, traceback2, six, argparse, linecache2
*** Error compiling '/tmp/venv-ci-core-jenkins_rt0y8it/build/testtools/testtools/_compat2x.py'...
  File "/tmp/venv-ci-core-jenkins_rt0y8it/build/testtools/testtools/_compat2x.py", line 16
    raise exc_class, exc_obj, exc_tb
                   ^
SyntaxError: invalid syntax

  Running setup.py install for extras

  Running setup.py install for python-mimeparse

Successfully installed testtools extras python-mimeparse unittest2 traceback2 six argparse linecache2
Cleaning up...
running test
running egg_info
creating workers.egg-info
writing top-level names to workers.egg-info/top_level.txt
writing dependency_links to workers.egg-info/dependency_links.txt
writing workers.egg-info/PKG-INFO
writing manifest file 'workers.egg-info/SOURCES.txt'
reading manifest file 'workers.egg-info/SOURCES.txt'
writing manifest file 'workers.egg-info/SOURCES.txt'
running build_ext
INFO: Executing: bzr branch lp:~canonical-ci-engineering/ci-core-jenkins/pip-cache /tmp/tmp_kmk768mpip-cache-/cache
INFO: Executing: virtualenv -p python3 /tmp/venv-ci-core-jenkins_rt0y8it
INFO: Executing: /tmp/venv-ci-core-jenkins_rt0y8it/bin/pip install --no-index --find-links=/tmp...

[lp:~pwlars/ci-core-jenkins/set-version-description]

13. By Paul Larson on 2015-04-10

fix tests

[Paul Larson (pwlars)]

Testcase fixed

review: Needs Resubmitting

[Francis Ginther (fginther)]

Tests pass now, approve.

review: Approve

[Para Siva (psivaa)]

approving again.

review: Approve

=== modified file 'jenkins/templates/core-result-fetcher.xml.jinja2'
--- jenkins/templates/core-result-fetcher.xml.jinja2	2015-04-10 10:19:24 +0000
+++ jenkins/templates/core-result-fetcher.xml.jinja2	2015-04-10 14:00:57 +0000
@@ -38,7 +38,8 @@
 
 set -e
 
-python3 workers/fetch-latest-result.py -i $IMAGE_ID --channel {{channel}} -d {{device}}
+SHORT_IMAGE_ID=`echo $IMAGE_ID |sed 's/:.*//'`
+python3 workers/fetch-latest-result.py -i $SHORT_IMAGE_ID --channel {{channel}} -d {{device}}
 
 echo "=== Converting the adt results to UTAH format ==="
 rm -rf adt_utah_converter
@@ -60,6 +61,12 @@
       <publishFailedBuilds>true</publishFailedBuilds>
       <postActions class="vector"/>
     </hudson.plugins.build__publisher.BuildPublisher>
+    <hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
+        <regexp/>
+        <regexpForFailed/>
+        <description>$IMAGE_ID</description>
+        <setForMatrix>false</setForMatrix>
+    </hudson.plugins.descriptionsetter.DescriptionSetterPublisher>
   </publishers>
   <buildWrappers/>
 </project>

=== modified file 'workers/get_latest_image_version/__init__.py'
--- workers/get_latest_image_version/__init__.py	2015-03-31 18:21:53 +0000
+++ workers/get_latest_image_version/__init__.py	2015-04-10 14:00:57 +0000
@@ -18,12 +18,19 @@
 
 import argparse
 import configparser
+import re
 import requests
 
 
 def _read_index_json_file(url, response=requests.get):
     r = response(url)
-    return r.json()['images'][-1]['version']
+    detail = r.json()['images'][-1]['version_detail']
+    pattern = 'ubuntu=([0-9\.]+),raw-device=([0-9\.]+),version=([0-9\.]+)'
+    versions = re.search(pattern, detail).groups()
+    return "{version_id}:{device_version}:{rootfs_version}".format(
+        version_id=versions[2],
+        device_version=versions[1],
+        rootfs_version=versions[0])
 
 
 def main():

=== modified file 'workers/tests/test_get_image_version.py'
--- workers/tests/test_get_image_version.py	2015-04-01 19:04:45 +0000
+++ workers/tests/test_get_image_version.py	2015-04-10 14:00:57 +0000
@@ -36,14 +36,16 @@
             "files": [],
             "type": "full",
             "version": 303,
-            "version_detail": "fake 1"
+            "version_detail":
+                "ubuntu=20150331.1,raw-device=20150331.1,version=303"
         },
         {
             "description": "fake 2",
             "files": [],
             "type": "full",
             "version": 304,
-            "version_detail": "fake 2"
+            "version_detail":
+                "ubuntu=20150331.2,raw-device=20150331.1,version=304"
         },
         {
             "base": 306,
@@ -51,7 +53,8 @@
             "files": [],
             "type": "delta",
             "version": 306,
-            "version_detail": "fake 3"
+            "version_detail":
+                "ubuntu=20150401.1,raw-device=20150401.1,version=306"
         },
     ]
 }
@@ -65,4 +68,4 @@
             return fakeResponse()
 
         ret = _read_index_json_file(url, get_response)
-        self.assertEqual(ret, 306)
+        self.assertEqual(ret, "306:20150401.1:20150401.1")