Merge lp:~vila/ubuntu-ci-services-itself/1287955-flavors-config-option into lp:ubuntu-ci-services-itself

[Vincent Ladeuil]

Provide a config option to select the nova flavor for the testbed.

This adds a 'tr_flavors' config option in unit_config that describes the various flavors that the testbed should try to create a nova instance.

This defaults to 'm1.small' for canonistack and 'standard.small' for hpcloud.

While I removed a FIXME doing so, I filed 2 additional bugs:
- bug #1288622 local testing requires floating-ip on hpcloud
- bug #1288617 testbed nova instance flavor selection

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:315
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~vila/ubuntu-ci-services-itself/1287955-flavors-config-option/+merge/209619/+edit-commit-message

http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/304/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/304/rebuild

review: Needs Fixing (continuous-integration)

[Chris Johnston (cjohnston)]

According to LP:

Diff against target: 134 lines (+50/-16) 3 files modified (has conflicts)
Text conflict in test_runner/tstrun/testbed.py

review: Needs Fixing

[lp:~vila/ubuntu-ci-services-itself/1287955-flavors-config-option]

316. By Vincent Ladeuil on 2014-03-06

Merge trunk resolving conflicts

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:316
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/307/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/307/rebuild

review: Approve (continuous-integration)

[Francis Ginther (fginther)]

Looks reasonable.

review: Approve

=== modified file 'juju-deployer/configs/unit_config.yaml.tmpl'
--- juju-deployer/configs/unit_config.yaml.tmpl	2014-03-03 20:43:25 +0000
+++ juju-deployer/configs/unit_config.yaml.tmpl	2014-03-06 13:59:27 +0000
@@ -20,3 +20,6 @@
 base_image: "http://cloud-images.ubuntu.com/releases/13.10/release-20131015/ubuntu-13.10-server-cloudimg-amd64-disk1.img"
 series: saucy
 master_ppa: $CI_MASTER_PPA
+tr_flavors:
+    - m1.small
+    - standard.small

=== modified file 'test_runner/tstrun/run_worker.py'
--- test_runner/tstrun/run_worker.py	2014-03-05 11:20:09 +0000
+++ test_runner/tstrun/run_worker.py	2014-03-06 13:59:27 +0000
@@ -42,6 +42,8 @@
     package_list = params['package_list']
     amqp_utils.progress_update(progress_queue, params)
 
+    conf = tstrun.get_auth_config()
+
     def status_cb(msg):
         amqp_utils.progress_update(progress_queue, {'message': msg})
 
@@ -50,12 +52,10 @@
     results = {}
     try:
         test_bed = None
-        store = data_store.create_for_ticket(ticket, tstrun.get_auth_config())
-        # FIXME: We need either a config option or a parameter for the instance
-        # flavor -- vila 2014-01-29
-        flavor = 'm1.small'
+        store = data_store.create_for_ticket(ticket, conf)
+        flavors = conf['tr_flavors']
         test_bed = testbed.TestBed('testbed-{}'.format(progress_queue),
-                                   flavor, image_id, status_cb)
+                                   flavors, image_id, status_cb)
         test_bed.setup()
 
         # FIXME: There is a potentially confusing failure mode here: by reusing

=== modified file 'test_runner/tstrun/testbed.py'
--- test_runner/tstrun/testbed.py	2014-03-05 11:20:19 +0000
+++ test_runner/tstrun/testbed.py	2014-03-06 13:59:27 +0000
@@ -21,7 +21,7 @@
 import time
 import yaml
 
-
+from novaclient import exceptions
 from novaclient.v1_1 import client
 
 import tstrun
@@ -112,12 +112,12 @@
 
 class TestBed(object):
 
-    def __init__(self, name, flavor, image, status_cb, auth_conf=None):
+    def __init__(self, name, flavors, image, status_cb, auth_conf=None):
         if auth_conf is None:
             auth_conf = tstrun.get_auth_config()
         self.auth_conf = auth_conf
         self.name = name
-        self.flavor = flavor
+        self.flavors = flavors
         self.image = image
         self.status_cb = status_cb
         self.instance = None
@@ -171,14 +171,26 @@
         self.cloud_init_completion_message = ud.cloud_config['final_message']
         self.user_data = ud.dump()
 
+    def find_flavor(self):
+        for flavor in self.flavors:
+            try:
+                return self.nova.flavors.find(name=flavor)
+            except exceptions.NotFound:
+                pass
+        # MISSINGTEST: some cloud doesn't provide one of our expected flavors
+        # -- vila 2014-01-06
+        raise NovaClientException(
+            'None of {} can be found'.format(self.flavors))
+
     def setup(self):
-        # MISSINGTEST: flavors.find can raise HTTP 503 ? -- vila 2014-01-30
-        flavor = self.nova.flavors.find(name=self.flavor)
+        flavor = self.find_flavor()
         image = self.nova.images.find(name=self.image)
         self.create_user_data()
         self.instance = self.nova.servers.create(
             name=self.name, flavor=flavor, image=image,
             userdata=self.user_data, keypair=self.nova_ssh_key_name)
+        # FIXME: This times out for hpcloud because it requires a floating ip
+        # for the instance see http://pad.lv/1288622 -- vila 2014-03-06
         self.wait_for_ip()
         self.wait_for_cloud_init()
 
@@ -296,12 +308,23 @@
 # 'auth_testbed_sshkey' can be defined there if you don't use ~/.ssh/id_rsa.
 
 def test_print_ip():
-    # The following image can be used if needed for precise testing
-    # precise_image = ('ubuntu-released/ubuntu-precise-12.04'
-    #                 '-amd64-server-20131205-disk1.img')
-    saucy_image = ('ubuntu-released/ubuntu-saucy-13.10'
-                   '-amd64-server-20140212-disk1.img')
-    test_bed = TestBed('vila1', 'm1.small', saucy_image,
+    conf = tstrun.get_auth_config()
+    # The following images are "known" to work when they are added while
+    # testing manually. In general, they need to be updated because they
+    # disappear to be replaced by newer ones. They also vary depending on
+    # whether canonistack or hpcloud is used.
+    test_images = dict(
+        cs_precise=('ubuntu-released/ubuntu-precise-12.04'
+                    '-amd64-server-20131205-disk1.img'),
+        cs_saucy=('ubuntu-released/ubuntu-saucy-13.10'
+                  '-amd64-server-20140212-disk1.img'),
+        hp_precise=('Ubuntu Server 12.04.4 LTS (amd64 20140227)'
+                    ' - CI Engineering'),
+        hp_saucy=('Ubuntu Server 13.10 (amd64 20131030) - Partner Image'),
+    )
+    test_bed = TestBed('localtest',
+                       conf['tr_flavors'],
+                       test_images['hp_saucy'],
                        lambda x: None)
     test_bed.setup()
     print 'cloud-init user-data: {}'.format(test_bed.user_data)