Merge lp:~vila/uci-engine/prod-britney into lp:uci-engine

[Vincent Ladeuil]

Running on uci-engine during validation

Work in progress carrying the currently tested delta on uci-britney@bootstack.

Mostly to get a better picture of what still needs to land on lp:uci-engine.

[lp:~vila/uci-engine/prod-britney]

914. By Vincent Ladeuil on 2014-11-01

The ~300 failures to setup the testbed were caused by using an illegal hostname inherited from the nova instance name.

915. By Vincent Ladeuil on 2014-11-01

The test runner check_worker.py in-production test can fail during high
load.

This is ironic as basically it means: It if fails it's broken. And
this is a bit hard to desambiguate from: it fails when something
*else*[1] is broken.

I.e. it's a false positive and creates noise.

Alternatively, a higher level controller could have another way to
measure the load and knowing that the load is high, swallow that failure
by cleaning the queue silently.

[1]: Optionally but more importantly O_o

916. By Vincent Ladeuil on 2014-11-03

Document the trick used to create a vivid image from an utopic one.

917. By Vincent Ladeuil on 2014-11-03

Hack procress_result to work around testbed not properly setup (including wrong host name) issues.

918. By Vincent Ladeuil on 2014-11-03

Merge britney resolving conflicts

919. By Vincent Ladeuil on 2014-11-05

Add autodep8 as a depedency for the test runner for vivid packages that relies on it.

920. By Vincent Ladeuil on 2014-11-05

Decode file content or we can't encode them back.

921. By Vincent Ladeuil on 2014-11-07

Hack swift retries around container deletion.

922. By Vincent Ladeuil on 2014-11-07

Add FIXME about cloud-init earsing apt sources.list from adt-setup-vm and the currently tried workaround.

923. By Vincent Ladeuil on 2014-11-13

We do have a public IP for rabbit now.

924. By Vincent Ladeuil on 2014-11-13

Make it easier to reuse the test for the britney use case.

925. By Vincent Ladeuil on 2014-11-13

Merge trunk, resolving conflicts

926. By Vincent Ladeuil on 2014-11-14

Merge trunk

927. By Vincent Ladeuil on 2014-11-14

Disable the test in-production until it supports running on a loaded engine.

928. By Vincent Ladeuil on 2014-11-15

Whether we have an public IP or not, we still need to expose rabbit.

Despite the fact that in the uci-britney deployment a specific security group has been defined to give access to snakefruit with the same rules that juju create when exposing a service, the juju group is still required.

929. By Vincent Ladeuil on 2014-11-17

One bug leading to OOM.

930. By Vincent Ladeuil on 2014-11-17

merge trunk

Unmerged revisions

930. By Vincent Ladeuil on 2014-11-17

merge trunk

929. By Vincent Ladeuil on 2014-11-17

One bug leading to OOM.

928. By Vincent Ladeuil on 2014-11-15

Whether we have an public IP or not, we still need to expose rabbit.

Despite the fact that in the uci-britney deployment a specific security group has been defined to give access to snakefruit with the same rules that juju create when exposing a service, the juju group is still required.

927. By Vincent Ladeuil on 2014-11-14

Disable the test in-production until it supports running on a loaded engine.

926. By Vincent Ladeuil on 2014-11-14

Merge trunk

925. By Vincent Ladeuil on 2014-11-13

Merge trunk, resolving conflicts

924. By Vincent Ladeuil on 2014-11-13

Make it easier to reuse the test for the britney use case.

923. By Vincent Ladeuil on 2014-11-13

We do have a public IP for rabbit now.

922. By Vincent Ladeuil on 2014-11-07

Add FIXME about cloud-init earsing apt sources.list from adt-setup-vm and the currently tried workaround.

921. By Vincent Ladeuil on 2014-11-07

Hack swift retries around container deletion.

=== modified file 'britney_proxy/britney/process_requests.py'
--- britney_proxy/britney/process_requests.py	2014-11-04 09:05:06 +0000
+++ britney_proxy/britney/process_requests.py	2014-11-17 13:01:51 +0000
@@ -64,7 +64,7 @@
         # debug purposes).
         output_queue = params.get('output_queue', self.output_queue)
         report_queue = params.get('report_queue', None)
-        params = dict(ticket_id=str(uuid.uuid4()),
+        params = dict(ticket_id='britney-{}'.format(str(uuid.uuid4())),
                       progress_trigger=output_queue,
                       series=series,
                       architecture=architecture,
 
=== modified file 'britney_proxy/britney/process_results.py'
--- britney_proxy/britney/process_results.py	2014-10-30 17:30:49 +0000
+++ britney_proxy/britney/process_results.py	2014-11-17 13:01:51 +0000
@@ -21,6 +21,7 @@
 import logging
 import os
 import sys
+import time
 
 
 from britney import queues
@@ -97,7 +98,19 @@
         # logging an exception should be enough. -- vila 2014-10-02
         value = self.tr_data_store.get_file(name)
         target = self.mapper.get_artifact_name(name)
-        self.britney_data_store.put_file(target, value, 'text/plain')
+        return self.britney_data_store.put_file(target, value, 'text/plain')
+
+    def delete_tr_data_store(self):
+        # Last sleep is 0.0 so we don't wait after the last attempt
+        for sleep in (5.0, 10.0, 60.0, 120.0, 0.0):
+            try:
+                self.tr_data_store.delete(recursive=True)
+                # We're done
+                return
+            except data_store.DataStoreException:
+                logging.exception('Cannot delete, wait {}s'.format(sleep))
+                # Try again
+                time.sleep(sleep)
 
     def handle_result(self, params):
         # Copy the produced results from the test request swift container into
@@ -116,17 +129,21 @@
             # uci-engine testbed hostnames are meaningless, so we just use a
             # generic name here.
             'uci-testbed')
+        # FIXME: This can fail if the testbed setup fails as the files are not
+        # produced in this case. But that shouldn't matter if we just copy all
+        # the files produced by adt-run. -- vila 2014-10-31
         self.copy_artifact('{}.log'.format(package))
-        self.copy_artifact('{}.britney.results'.format(package))
+        results_path = '{}.britney.results'.format(package)
+        self.copy_artifact(results_path)
+        results_value = self.tr_data_store.get_file(results_path)
         # The test runner swift container has been processed and can die
-        self.tr_data_store.delete(recursive=True)
+        self.delete_tr_data_store()
         report_queue = request.get('report_queue', None)
         if report_queue is not None:
             # Publish a message to report where the results have been stored
             queue = queues.PublisherQueue(report_queue)
             msg = dict(result_at=self.mapper.dir_name,
-                       britney_result=self.tr_data_store.get_file(
-                           '{}.britney.results'.format(package)))
+                       britney_result=results_value)
             queue.publish(msg)
 
     def handle(self, params):
 
=== modified file 'ci-utils/ci_utils/amqp_worker.py'
--- ci-utils/ci_utils/amqp_worker.py	2014-10-29 16:43:39 +0000
+++ ci-utils/ci_utils/amqp_worker.py	2014-11-17 13:01:51 +0000
@@ -159,6 +159,8 @@
             '[%(asctime)s] %(name)s:%(levelname)s:%(message)s')
         logstream.setFormatter(formatter)
         logstream.setLevel(logging.INFO)
+        # FIXME: This is called for each request and never remove leading to
+        # OOM for long running workers -- vila 2014-11-17
         log.addHandler(logstream)
         return log
 
 
=== modified file 'juju-deployer/test-runner.yaml.tmpl'
--- juju-deployer/test-runner.yaml.tmpl	2014-11-04 12:59:25 +0000
+++ juju-deployer/test-runner.yaml.tmpl	2014-11-17 13:01:51 +0000
@@ -8,7 +8,7 @@
                 main: ./run-python ./test_runner/tstrun/run_worker.py
                 current_code: ${CI_PAYLOAD_URL}
                 available_code: ${CI_PAYLOAD_URL}
-                packages: "python-requests python-novaclient python-swiftclient python-glanceclient python-uci-vms autopkgtest haveged python-subunit python-testtools python-lazr.enum python-kombu"
+                packages: "python-requests python-novaclient python-swiftclient python-glanceclient python-uci-vms autodep8 autopkgtest haveged python-subunit python-testtools python-lazr.enum python-kombu"
                 unit-config: include-base64://configs/unit_config.yaml
                 uid: ubuntu
                 gid: ubuntu
@@ -19,7 +19,8 @@
                     - null
                     - null
                 cron_cmd: ./run-python ./test_runner/bin/check_worker.py
-                cron_schedule: "0 */2 * * *"
+# FIXME: This create too much noise for now -- vila 2014-11-14
+#                cron_schedule: "0 */2 * * *"
                 nagios_context: ci-airline-staging
                 nagios_check_health_params: -t 7800 test_runner.health
         ci-airline-rabbit:
 
=== modified file 'test_runner/bin/check_worker.py'
--- test_runner/bin/check_worker.py	2014-11-04 09:05:06 +0000
+++ test_runner/bin/check_worker.py	2014-11-17 13:01:51 +0000
@@ -38,8 +38,8 @@
 # FIXME: This is wrong by design: it could happen (and it did in real life)
 # that no worker is available for 7 minutes. This leads to a timeout and the
 # progress queue staying alive without anybody subscribing to it. When the
-# request is processed later, it left the expected 6 messages hanging around in
-# the queue... -- vila 2014-10-31
+# request is processed later, it leaves up to the expected 6 messages hanging
+# around in the queue... -- vila 2014-10-31
 def timeout_handler(sig_num, frame):
     raise Timeout('No worker responding after {} seconds'.format(TIMEOUT))
 
@@ -80,7 +80,7 @@
 
 
 def check_worker():
-    queue = 'testrun-test-{}'.format(uuid.uuid4())
+    queue = 'health-check-test-runner-{}'.format(uuid.uuid4())
     series = 'trusty'
     architecture = 'amd64'
     image_id = image_store.uci_image_name('cloudimg', series, architecture)
@@ -93,6 +93,7 @@
         'ppa_list': [],
         'progress_trigger': queue,
     }
+    logging.info('Sending {}'.format(params))
     amqp_utils.send(
         amqp_utils.TEST_RUNNER_QUEUE, json.dumps(params), True)
     try:
 
=== modified file 'test_runner/tstrun/run_test.py'
--- test_runner/tstrun/run_test.py	2014-10-17 12:28:14 +0000
+++ test_runner/tstrun/run_test.py	2014-11-17 13:01:51 +0000
@@ -87,11 +87,11 @@
     out_path = os.path.join('results', '{}-stdout'.format(name))
     if os.path.exists(out_path):
         with open(out_path) as f:
-            details['stdout'] = content.text_content(f.read())
+            details['stdout'] = content.text_content(f.read().decode('utf8'))
     err_path = os.path.join('results', '{}-stderr'.format(name))
     if os.path.exists(err_path):
         with open(err_path) as f:
-            details['stderr'] = content.text_content(f.read())
+            details['stderr'] = content.text_content(f.read().decode('utf8'))
     return details
 
 
 
=== modified file 'test_runner/tstrun/run_worker.py'
--- test_runner/tstrun/run_worker.py	2014-11-04 15:16:50 +0000
+++ test_runner/tstrun/run_worker.py	2014-11-17 13:01:51 +0000
@@ -154,7 +154,15 @@
             self.status_cb('Setting up the testbed for ticket {}'.format(
                 self.ticket_id))
             auth_conf = unit_config.get_auth_config()
-            tb_name = '{}-testbed-{}'.format(package, self.ticket_id)
+            # We can't use arbitrary host names which rules out using the
+            # package name and the ticket id. At least mentioning series/arch
+            # provides some hint about what is running there...
+            # FIXME: Investigate whether we can set some nova attributes for
+            # debug/monitoring purposes -- vila 2014-10-31
+            # FIXME: Separate nova name from host name so we can guarantee
+            # using a proper hostname while still having an informative nova
+            # name. -- vila 2014-11-01.
+            tb_name = 'uci-testbed-{}-{}'.format(series, architecture)
             conf = testbed.vms_config_from_auth_config(tb_name, auth_conf)
             conf.set('vm.image', image_id)
             conf.set('vm.release', series)
 
=== modified file 'test_runner/tstrun/testbed.py'
--- test_runner/tstrun/testbed.py	2014-10-17 12:28:22 +0000
+++ test_runner/tstrun/testbed.py	2014-11-17 13:01:51 +0000
@@ -307,6 +307,9 @@
         self.ensure_ssh_works()
         ppas = self.conf.get('vm.ppas')
         if ppas:
+            # FIXME: britney set 'apt_preserve_sources_list: true' in
+            # /etc/cloud/cloud.cfg, make sure we don't break that or put the
+            # same into user-data -- vila 2014-11-07
             cmd = ['sudo', 'add-apt-repository']
             if self.conf.get('vm.release') > 'precise':
                 cmd.append('--enable-source')
 
=== modified file 'test_runner/tstrun/tests/test_testbed.py'
--- test_runner/tstrun/tests/test_testbed.py	2014-11-04 12:07:12 +0000
+++ test_runner/tstrun/tests/test_testbed.py	2014-11-17 13:01:51 +0000
@@ -341,7 +341,6 @@
 
     @tests.log_on_failure()
     def test_create_usable_testbed(self, logger):
-        self.conf.set('vm.release', 'trusty')
         self.conf.set('vm.image', self.get_image_id())
         tb = testbed.TestBed(self.conf, logger)
         self.addCleanup(tb.teardown)
 
=== modified file 'test_runner/tstrun/tests/test_worker.py'
--- test_runner/tstrun/tests/test_worker.py	2014-11-04 12:07:12 +0000
+++ test_runner/tstrun/tests/test_worker.py	2014-11-17 13:01:51 +0000
@@ -138,7 +138,8 @@
         # series
         ([('precise', dict(series='precise', result='skip')),
           ('trusty', dict(series='trusty', result='pass')),
-          ('utopic', dict(series='utopic', result='pass'))]),
+          ('utopic', dict(series='utopic', result='pass')),
+          ('vivid', dict(series='vivid', result='pass')),]),
         # architectures
         ([('amd64', dict(arch='amd64')), ('i386', dict(arch='i386'))]))
 
@@ -163,8 +164,8 @@
         # ticket, only the master ppa is relevant.
         return [self.conf.get('master_ppa')]
 
-    def get_image_id(self):
-        return image_store.uci_image_name('cloudimg', self.series, self.arch)
+    def get_image_id(self, prefix):
+        return image_store.uci_image_name(prefix, self.series, self.arch)
 
     def assertTestRun(self, logger, params):
         worker = run_worker.TestRunnerWorker(self.ds_factory)
@@ -181,8 +182,9 @@
         params = dict(ticket_id=ticket_id, progress_trigger='progress',
                       series=self.series,
                       architecture=self.arch,
-                      image_id=self.get_image_id(),
+                      image_id=self.get_image_id('cloudimg'),
                       ppa_list=self.get_ppa_list(),
+#                      adt_opts=['--apt-upgrade', '--apt-pocket=proposed'],
                       package_list=[package])
         returns = self.assertTestRun(logger, params)
         (retcode, results) = returns[0]