Merge lp:~vila/uci-engine/britney-results into lp:uci-engine

[Vincent Ladeuil]

britney results processing.

This implement the britney result queue handling and comes with an end-to-end test that covers:
- britney request published to the queue,
- request translated and forwarded to the test runner,
- test runner processing the test request,
- britney test result acquired from the test runner and put in the britney swift container

I.e. the uci-engine part for supporting britney is done \o/

Many (small) bits are still missing and several FIXMEs introduced here already correspond to planned tasks.

The test runner results copy into the britney swift container may (ok, will ;) need adjustments but the heavy wiring work is done and further MPs will be smaller.

I'll better document how this is designed in another MP, here is a rough sketch:
- the 'britney.requests' queue is handled by the ci-airline-for-britney-requests service and forward translated requests to the test runner,
- the test runner does its job and send progress/completion messages to the 'britney.results' queue
- the 'britney.results' queue is handled by the ci-airline-for-britney-results service which copy the test results into the britney swift container which has a specific layou.

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:826
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1511/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1511/rebuild

review: Approve (continuous-integration)

[Evan (ev)]

Looks good. Comments inline, but nothing that cannot be fixed with a follow-up.

review: Approve

[Vincent Ladeuil (vila)]

Replied inline, yes, follow ups are planned to address the FIXMEs.

And thanks again for the lightbulb about dead letter ;)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:827
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1514/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1514/rebuild

review: Approve (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:828
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1515/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1515/rebuild

review: Approve (continuous-integration)

[Ubuntu CI Bot (uci-bot)]

The attempt to merge lp:~vila/uci-engine/britney-results into lp:uci-engine failed. Below is the output from the failed tests.

Running cm...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
uploading webui-content.tgz to swift
Updating source dependencies...
Checking juju status
Private PPAs: disabled
Preparing local branch upload...
Uploading local branch, fingerprint efec3cf7de7c720c561628535b5fef0c01543849
Building charm: lander
Building charm: wsgi-app
Building charm: rabbitmq-worker
Building charm: chroot-builder
Building charm: key-secret-subordinate
Building charm: webui
Building charm: system-image-server
Installing keys from bzr+ssh://bazaar.launchpad.net/~ci-engineering-private/+junk/ci-airline-dev-keys/
Running juju-deployer -v -c /tmp/tmpY4DSzB/deployer/branch-source-builder.yaml -c /tmp/tmpY4DSzB/deployer/britney-proxy.yaml -c /tmp/tmpY4DSzB/deployer/gatekeeper.yaml -c /tmp/tmpY4DSzB/deployer/image-builder.yaml -c /tmp/tmpY4DSzB/deployer/lander.yaml -c /tmp/tmpY4DSzB/deployer/nf-stats-service.yaml -c /tmp/tmpY4DSzB/deployer/ppa-creator.yaml -c /tmp/tmpY4DSzB/deployer/publisher.yaml -c /tmp/tmpY4DSzB/deployer/relations.yaml -c /tmp/tmpY4DSzB/deployer/test-runner.yaml -c /tmp/tmpY4DSzB/deployer/ticket-system.yaml -c /tmp/tmpY4DSzB/deployer/validator.yaml -c /tmp/tmpY4DSzB/deployer/webui.yaml ci-airline
Tests running...
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testConnectFailed ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testProcessQueue ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testRunForever ... OK (0.102 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testSent ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp.TestProgressTrigger.testProgress ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testCancel ... OK (0.105 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoQueue ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoTicket ... OK (0.005 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageCalledProcessError ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageFail ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageKilled ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageSimple ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageUnexpected ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testSaveLastRun ... OK (0.009 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCBRuns ... OK (0.021 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCanCancel ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_invalid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_valid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_abspath ... OK...

[Vincent Ladeuil (vila)]

======================================================================
ERROR: tests.test_cli.CliTest.test_create_ticket
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tarmac/branch.aE0WBg/bin/../tests/test_cli.py", line 70, in test_create_ticket
    num_tickets = self.get_tickets_count()
  File "/tmp/tarmac/branch.aE0WBg/bin/../tests/test_cli.py", line 47, in get_tickets_count
    response.raise_for_status()
  File "/dev/shm/venv-DbmfiI/local/lib/python2.7/site-packages/requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 500 Server Error: INTERNAL SERVER ERROR
======================================================================
ERROR: tests.test_cli.CliTest.test_update_ticket_attributes
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tarmac/branch.aE0WBg/bin/../tests/test_cli.py", line 93, in test_update_ticket_attributes
    num_tickets = self.get_tickets_count()
  File "/tmp/tarmac/branch.aE0WBg/bin/../tests/test_cli.py", line 47, in get_tickets_count
    response.raise_for_status()
  File "/dev/shm/venv-DbmfiI/local/lib/python2.7/site-packages/requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 500 Server Error: INTERNAL SERVER ERROR
======================================================================
ERROR: tests.test_cli.CliTest.test_update_ticket_new_sources
----------------------------------------------------------------------
Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/mock.py", line 1201, in patched
    return func(*args, **keywargs)
  File "/tmp/tarmac/branch.aE0WBg/bin/../tests/test_cli.py", line 118, in test_update_ticket_new_sources
    num_tickets = self.get_tickets_count()
  File "/tmp/tarmac/branch.aE0WBg/bin/../tests/test_cli.py", line 47, in get_tickets_count
    response.raise_for_status()
  File "/dev/shm/venv-DbmfiI/local/lib/python2.7/site-packages/requests/models.py", line 773, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
HTTPError: 500 Server Error: INTERNAL SERVER ERROR
======================================================================
FAIL: tests.test_integration.IntegrationTest.test_create_ticket_via_cli
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tarmac/branch.aE0WBg/bin/../tests/test_integration.py", line 86, in test_create_ticket_via_cli
    ticket_id = self.create_new_ticket(ticket_title, ci_url)
  File "/tmp/tarmac/branch.aE0WBg/bin/../tests/test_integration.py", line 74, in create_new_ticket
    0, proc.returncode, '{} failed:\n{}'.format(args, out))
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 508, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: ('python', '/tmp/tarmac/branch.aE0WBg/tests/../cli/ubuntu-ci', '-u', 'http://10.0.3.253', 'create_ticket', '-t', 'Bazinga!', '-d'...

[Ubuntu CI Bot (uci-bot)]

The attempt to merge lp:~vila/uci-engine/britney-results into lp:uci-engine failed. Below is the output from the failed tests.

Running cm...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
uploading webui-content.tgz to swift
Updating source dependencies...
2014-10-03 12:33:34 INFO juju.cmd supercommand.go:37 running jujud [1.20.8.1-precise-amd64 gc]
2014-10-03 12:33:34 DEBUG juju.agent agent.go:377 read agent config, format "1.18"
2014-10-03 12:33:34 INFO juju.jujud unit.go:78 unit agent unit-ci-airline-ts-postgres-0 start (1.20.8.1-precise-amd64 [gc])
2014-10-03 12:33:34 INFO juju.worker runner.go:260 start "api"
2014-10-03 12:33:34 INFO juju.state.api apiclient.go:242 dialing "wss://10.0.3.1:17070/"
2014-10-03 12:33:34 INFO juju.state.api apiclient.go:176 connection established to "wss://10.0.3.1:17070/"
2014-10-03 12:33:35 INFO juju.state.api apiclient.go:242 dialing "wss://10.0.3.1:17070/"
2014-10-03 12:33:35 INFO juju.state.api apiclient.go:176 connection established to "wss://10.0.3.1:17070/"
2014-10-03 12:33:37 INFO juju.state.api apiclient.go:242 dialing "wss://10.0.3.1:17070/"
2014-10-03 12:33:37 INFO juju.state.api apiclient.go:176 connection established to "wss://10.0.3.1:17070/"
2014-10-03 12:33:38 INFO juju.worker runner.go:260 start "upgrader"
2014-10-03 12:33:38 INFO juju.worker runner.go:260 start "logger"
2014-10-03 12:33:38 DEBUG juju.worker.logger logger.go:35 initial log config: "<root>=DEBUG"
2014-10-03 12:33:38 INFO juju.worker runner.go:260 start "uniter"
2014-10-03 12:33:38 DEBUG juju.worker.logger logger.go:60 logger setup
2014-10-03 12:33:38 INFO juju.worker runner.go:260 start "apiaddressupdater"
2014-10-03 12:33:38 INFO juju.worker runner.go:260 start "rsyslog"
2014-10-03 12:33:38 DEBUG juju.worker.rsyslog worker.go:75 starting rsyslog worker mode 1 for "unit-ci-airline-ts-postgres-0" "tarmac-local"
2014-10-03 12:33:38 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "<root>=DEBUG" to "<root>=WARNING;unit=DEBUG"
2014-10-03 12:33:59 INFO juju-log Installing ['python-psycopg2', 'python-jinja2'] with options: ['--option=Dpkg::Options::=--force-confold']
2014-10-03 12:34:29 INFO install Reading package lists...
2014-10-03 12:34:31 INFO install Building dependency tree...
2014-10-03 12:34:31 INFO install Reading state information...
2014-10-03 12:34:32 INFO install The following extra packages will be installed:
2014-10-03 12:34:32 INFO install libpq5 python-egenix-mxdatetime python-egenix-mxtools python-markupsafe
2014-10-03 12:34:32 INFO install Suggested packages:
2014-10-03 12:34:32 INFO install python-egenix-mxdatetime-dbg python-egenix-mxdatetime-doc
2014-10-03 12:34:32 INFO install python-egenix-mxtools-dbg python-egenix-mxtools-doc python-jinja2-doc
2014-10-03 12:34:32 INFO install python-psycopg2-doc
2014-10-03 12:34:32 INFO install The following NEW packages will be installed:
2014-10-03 12:34:32 INFO install libpq5 python-egenix-mxdatetime python-egenix-mxtools python-jinja2
2014-10-03 12:34:32 INFO install ...

[Martin Pitt (pitti)]

Thanks! This already happened (national holiday yesterday), but I still have some comments and questions.

review: Needs Fixing

[Vincent Ladeuil (vila)]

Replied inline for completing the discussions on IRC.

[Vincent Ladeuil (vila)]

Yet Another Uncommit Comment. This one was added during the review discussion :-/

=== modified file 'britney_proxy/britney/process_requests.py'
--- britney_proxy/britney/process_requests.py	2014-09-25 14:19:29 +0000
+++ britney_proxy/britney/process_requests.py	2014-10-03 08:44:06 +0000
@@ -18,9 +18,11 @@
 
 import json
 import logging
+import sys
 import uuid
 
 
+from britney import queues
 from ci_utils import (
     amqp_utils,
     dump_stack,
@@ -69,39 +71,29 @@
         series = params['series']
         architecture = params['architecture']
         package = params['package']
+        # The output queue can be overridden for a given request (for tests and
+        # debug purposes).
+        output_queue = params.get('output_queue', self.output_queue)
+        report_queue = params.get('report_queue', None)
         # Only amd64 architecture is supported for now, everything else is
         # dropped on the floor as we can't send feedback to the user at this
-        # point ( the data flow is: britney.requests -> test_runner ->
+        # point (the data flow is: britney.requests -> test_runner ->
         # britney.results). See
         # https://app.asana.com/0/8312550429058/12699749556376 'reporting test
         # requests errors'
         if architecture == 'amd64':
             params = dict(ticket_id=str(uuid.uuid1()),
-                          progress_trigger=self.output_queue,
+                          progress_trigger=output_queue,
                           series=series,
                           image_id=self.get_image_id(series),
                           ppa_list=[],
                           package_list=[package])
+            if report_queue is not None:
+                params['report_queue'] = report_queue
             self.test_runner_queue.publish(params)
 
 
-class TestRunnerQueue(object):
-    """A proxy to the queue libray.
-
-    This implementation targets cit_utils/amqp_utils but can be replaced for
-    tests.
-
-    """
-
-    def __init__(self, queue_name):
-        self.queue_name = queue_name
-
-    def publish(self, msg):
-        amqp_utils.send(self.queue_name, json.dumps(msg),
-                        raise_errors=True)
-
-
-class BritneyQueue(object):
+class BritneyRequestsQueue(object):
     """The queue receiving the britney test requests.
 
     The requests are forwarded to the request handler.
@@ -132,17 +124,12 @@
 
 if __name__ == '__main__':
     dump_stack.install_stack_dump_signal()
+    logging.root.setLevel(logging.INFO)
     logging.info('Waiting for messages. ^C to exit.')
-    tr_queue = TestRunnerQueue(amqp_utils.TEST_RUNNER_QUEUE)
+    tr_queue = queues.PublisherQueue(amqp_utils.TEST_RUNNER_QUEUE)
     request_handler = BritneyRequestHandler(tr_queue, 'britney.results')
     # Build the britney requests queue subscribing to the rabbit queue and
     # proving the handler that will forward to the test runner queue.
-    queue = BritneyQueue('britney.requests', request_handler)
+    queue = BritneyRequestsQueue('britney.requests', request_handler)
     queue.consume()
-
-
-# MISSINGTEST: Integration running a simple test (precise/amd64/libpng) under
-# ./tests/test_britney.py (will require a dedicated fixture to not run on
-# tarmac :-/ (which raises the issue of running all deployments on tarmac))
-# 'process_results.py' only needs to implement a sink: just consume the
-# messages without touching them.
+    sys.exit(0)

=== modified file 'britney_proxy/britney/process_results.py'
--- britney_proxy/britney/process_results.py	2014-09-24 14:44:55 +0000
+++ britney_proxy/britney/process_results.py	2014-10-03 08:44:06 +0000
@@ -15,19 +15,167 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 from __future__ import unicode_literals
 
+
+import datetime
+import json
 import logging
+import os
 import sys
-import time
-
-
+
+
+from britney import queues
 from ci_utils import (
+    amqp_utils,
+    data_store,
     dump_stack,
+    unit_config,
 )
 
 
+class BritneyTestRunnerArtifactMapper(object):
+
+    def __init__(self, release, architecture, package, time_stamp, host_name):
+        if package.startswith('lib'):
+            prefix = package[0:4]
+        else:
+            prefix = package[0]
+        self.dir_name = os.path.join(release, architecture, prefix, package,
+                                     '{}_{}'.format(time_stamp, host_name))
+
+    def get_artifact_name(self, name):
+        return os.path.join(self.dir_name, name)
+
+
+# FIXME: We do store *all* britney test results in a single swift
+# container. This won't scale but we'll revisit the layout when we get a better
+# understanding of the britney needs (and numbers !) -- vila 2014-10-03
+class BritneyDataStore(data_store.DataStore):
+
+    def file_name(self, filename):
+        # Allow paths to be used as object names in the container by overriding
+        # the filtering in the base class.
+        return filename
+
+
+class BritneyResultHandler(object):
+    """Process a britney test request.
+
+    This is fed by messages received on a queue or directly for tests by
+    calling the ``handle()`` method.
+    """
+
+    def __init__(self, britney_data_store, _tr_data_store=None,
+                 _time_stamp=None):
+        """Britney results handler.
+
+        :param britney_data_store: The data store where the test results are
+            stored for britney.
+
+        :param _tr_data_store: The data store containing the test results.
+            This is intended to be used by tests, the normal use case is to
+            retrieve the test runner data store from the request parameter in
+            the received message.
+
+        :param _time_stamp: The date and time the test result is
+            processed. This is intended to be used by tests and default to
+            utcnow().
+
+        """
+        self.britney_data_store = britney_data_store
+        self.tr_data_store = _tr_data_store
+        self._time_stamp = _time_stamp
+
+    def handle_progress(self, params):
+        # We just ignore the progress messages
+        pass
+
+    def copy_artifact(self, name):
+        # FIXME: If swift becomes unavailable we can't acquire the test results
+        # from the test runner swift container nor can we upload them to the
+        # britney container. It's quite unlikely though that swift becomes
+        # unavailable *after* the test runner processed the request so just
+        # logging an exception should be enough. -- vila 2014-10-02
+        value = self.tr_data_store.get_file(name)
+        target = self.mapper.get_artifact_name(name)
+        self.britney_data_store.put_file(target, value, 'text/plain')
+
+    def handle_result(self, params):
+        # Copy the produced results from the test request swift container into
+        # the britney swift container.
+        request = params['request']
+        ticket_id = request['ticket_id']
+        series = request['series']
+        # FIXME: Needs to be supported by the test runner -- vila 2014-10-01
+        architecture = 'amd64'
+        if self.tr_data_store is None:
+            self.tr_data_store = data_store.create_for_ticket(
+                ticket_id, unit_config.get_auth_config())
+        time_stamp = self._time_stamp
+        if time_stamp is None:
+            time_stamp = datetime.datetime.utcnow().strftime('%Y%m%d_%H%M%S')
+        package = request['package_list'][0]
+        self.mapper = BritneyTestRunnerArtifactMapper(
+            series, architecture, package, time_stamp,
+            # uci-engine testbed hostnames are meaningless, so we just use a
+            # generic name here.
+            'uci-testbed')
+        self.copy_artifact('{}.log'.format(package))
+        # FIXME: We could delete the test runner swift container at that point
+        # to reduce cluttering swift -- vila 2014-10-02
+        report_queue = request.get('report_queue', None)
+        if report_queue is not None:
+            # Publish a message to report where the results have been stored
+            queue = queues.PublisherQueue(report_queue)
+            msg = dict(result_at=self.mapper.dir_name)
+            queue.publish(msg)
+
+    def handle(self, params):
+        state = params['state']
+        if state == 'STATUS':
+            self.handle_progress(params)
+        elif state == 'COMPLETED':
+            self.handle_result(params)
+        else:
+            # This indicates a programming error in the test runner producing
+            # an unexpected message
+            logging.error('Unable to handle {}'.format(params))
+            # FIXME: properly testing the above will require a logger object
+            # for the handler that can be overriden by tests -- vila 2014-09-30
+
+
+class BritneyResultsQueue(object):
+    """The queue receiving the test runner results for the britney requests."""
+
+    def __init__(self, queue_name, request_handler):
+        """Britney test results queue.
+
+        :param queue_name: The queue name where the test results are received.
+
+        :param request_handler: The handler that processes the results.
+        """
+        self.queue_name = queue_name
+        self.request_handler = request_handler
+
+    def consume(self):
+        conf = amqp_utils.get_config()
+        # Process all messages and delete the queue when done
+        amqp_utils.process_queue(conf, self.queue_name, self.handle_request,
+                                 delete=True)
+
+    def handle_request(self, msg):
+        body = json.loads(msg.body)
+        ret = self.request_handler.handle(body)
+        msg.channel.basic_ack(msg.delivery_tag)
+        return ret
+
+
 if __name__ == '__main__':
     dump_stack.install_stack_dump_signal()
+    logging.root.setLevel(logging.INFO)
     logging.info('Waiting for messages. ^C to exit.')
-    while True:
-        time.sleep(10)
+    britney_data_store = BritneyDataStore(
+        'britney.results', unit_config.get_auth_config(), public=True)
+    request_handler = BritneyResultHandler(britney_data_store)
+    queue = BritneyResultsQueue('britney.results', request_handler)
+    queue.consume()
     sys.exit(0)

=== added file 'britney_proxy/britney/queues.py'
--- britney_proxy/britney/queues.py	1970-01-01 00:00:00 +0000
+++ britney_proxy/britney/queues.py	2014-10-03 08:44:06 +0000
@@ -0,0 +1,34 @@
+# Ubuntu CI Engine
+# Copyright 2014 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import json
+
+
+from ci_utils import amqp_utils
+
+
+class PublisherQueue(object):
+    """A proxy to the queue library.
+
+    This implementation targets ci_utils/amqp_utils but can be replaced for
+    tests.
+    """
+
+    def __init__(self, queue_name):
+        self.queue_name = queue_name
+
+    def publish(self, msg):
+        amqp_utils.send(self.queue_name, json.dumps(msg),
+                        raise_errors=True)

=== added file 'britney_proxy/britney/tests/test_process_results.py'
--- britney_proxy/britney/tests/test_process_results.py	1970-01-01 00:00:00 +0000
+++ britney_proxy/britney/tests/test_process_results.py	2014-10-03 08:44:06 +0000
@@ -0,0 +1,94 @@
+# Ubuntu CI Engine
+# Copyright 2014 Canonical Ltd.
+
+# This program is free software: you can redistribute it and/or modify it
+# under the terms of the GNU Affero General Public License version 3, as
+# published by the Free Software Foundation.
+
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranties of
+# MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+# PURPOSE.  See the GNU Affero General Public License for more details.
+
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import unittest
+
+
+from ucitests import fixtures as uci_fixtures
+
+
+from britney import process_results
+from ci_utils.testing import (
+    features,
+    fixtures,
+)
+
+
+class TestBritneyTestRunnerArtifactMapper(unittest.TestCase):
+
+    def setUp(self):
+        super(TestBritneyTestRunnerArtifactMapper, self).setUp()
+        # Set some defaults to make the tests less verbose
+        self.release = 'utopic'
+        self.architecture = 'amd64'
+        self.time_stamp = '20141001_000000'
+        self.host_name = 'localhost'
+
+    def get_mapper(self, package):
+        mapper = process_results.BritneyTestRunnerArtifactMapper(
+            self.release, self.architecture, package, self.time_stamp,
+            self.host_name)
+        return mapper
+
+    def test_package_name(self):
+        mapper = self.get_mapper('hello')
+        self.assertEquals(
+            'utopic/amd64/h/hello/20141001_000000_localhost/hello.log',
+            mapper.get_artifact_name('hello.log'))
+
+    def test_lib_package_name(self):
+        mapper = self.get_mapper('libpng')
+        self.assertEquals(
+            'utopic/amd64/libp/libpng/20141001_000000_localhost/libpng.log',
+            mapper.get_artifact_name('libpng.log'))
+
+
+# FIXME: This works around https://app.asana.com/0/8499154990155/14496314215756
+# 'unit_config as a fixture for tests that requires it' -- vila 2014-09-25
[e-mail](features.nova_creds)
+class TestBritneyRequestHandler(unittest.TestCase):
+
+    def setUp(self):
+        super(TestBritneyRequestHandler, self).setUp()
+        uci_fixtures.set_uniq_cwd(self)
+        self.tr_ds_path = os.path.join(self.uniq_dir, 'tr_data_store')
+        self.tr_data_store = fixtures.FakeDataStore(self.tr_ds_path)
+        self.britney_ds_path = os.path.join(self.uniq_dir,
+                                            'britney_data_store')
+        self.britney_data_store = fixtures.FakeDataStore(self.britney_ds_path)
+        self.handler = process_results.BritneyResultHandler(
+            self.britney_data_store,
+            _tr_data_store=self.tr_data_store,
+            _time_stamp='20141001_000000')
+
+    def test_status_message(self):
+        params = dict(state='STATUS')
+        # Since the handler does nothing, this is mostly a smoke test
+        self.handler.handle(params)
+
+    def test_completed_message(self):
+        request = dict(ticket_id='not-used', series='utopic',
+                       package_list=['fake'])
+        fake_content = 'fake.log content\n'
+        fake_url = self.tr_data_store.put_file('fake.log', fake_content,
+                                               'text/plain')
+        params = dict(state='COMPLETED', request=request,
+                      artifacts=[dict(name='fake.log', reference=fake_url,
+                                      type='LOGS')])
+        self.handler.handle(params)
+        britney_name = self.handler.mapper.get_artifact_name('fake.log')
+        actual_content = self.britney_data_store.get_file(britney_name)
+        self.assertEquals(fake_content, actual_content)

=== modified file 'ci-utils/ci_utils/testing/fixtures.py'
--- ci-utils/ci_utils/testing/fixtures.py	2014-07-03 14:43:54 +0000
+++ ci-utils/ci_utils/testing/fixtures.py	2014-10-03 08:44:06 +0000
@@ -56,6 +56,10 @@
         return sorted(os.listdir(self.container_id))
 
     def put_file(self, filename, contents, content_type=None):
+        if os.sep in filename:
+            base_dir = os.path.join(self.container_id,
+                                    os.path.dirname(filename))
+            os.makedirs(base_dir)
         full_path = os.path.join(self.container_id, filename)
         with open(full_path, 'w') as f:
             f.write(contents)

=== modified file 'ci-utils/ci_utils/tests/test_fixtures.py'
--- ci-utils/ci_utils/tests/test_fixtures.py	2014-05-06 15:05:18 +0000
+++ ci-utils/ci_utils/tests/test_fixtures.py	2014-10-03 08:44:06 +0000
@@ -43,6 +43,11 @@
         self.assertEqual(['foo'], self.ds.list_files())
         self.assertEqual('foo content\n', self.ds.get_file('foo'))
 
+    def test_put_file_with_slashes(self):
+        url = self.ds.put_file('foo/bar/baz', 'baz content\n')
+        self.assertTrue(url.endswith(os.path.join(self.ds_path,
+                                                  'foo/bar/baz')))
+
     def test_list_files_empty(self):
         self.assertEqual([], self.ds.list_files())
 

=== modified file 'juju-deployer/britney-proxy.yaml.tmpl'
--- juju-deployer/britney-proxy.yaml.tmpl	2014-10-01 08:55:19 +0000
+++ juju-deployer/britney-proxy.yaml.tmpl	2014-10-03 08:44:06 +0000
@@ -10,9 +10,10 @@
 
 # Until we do: https://app.asana.com/0/8312550429058/16178663570516 'minimal
 # images for britney', we use the ones defined in ci_utils/testing/features
-# which requires ucitests
+# which requires ucitests, python-swiftclient and python-novaclient (and
+# indirectly mock and gnupg :-/)
 
-                packages: "python-uci-tests"
+                packages: "python-uci-tests python-swiftclient python-novaclient python-mock python-gnupg"
                 unit-config: include-base64://configs/unit_config.yaml
                 uid: ubuntu
                 gid: ubuntu
@@ -26,6 +27,7 @@
                 main: ./run-python ./britney_proxy/britney/process_results.py
                 current_code: ${CI_PAYLOAD_URL}
                 available_code: ${CI_PAYLOAD_URL}
+                packages: "python-swiftclient"
                 unit-config: include-base64://configs/unit_config.yaml
                 uid: ubuntu
                 gid: ubuntu

=== modified file 'test_runner/tstrun/run_worker.py'
--- test_runner/tstrun/run_worker.py	2014-09-23 14:54:03 +0000
+++ test_runner/tstrun/run_worker.py	2014-10-03 08:44:06 +0000
@@ -232,7 +232,10 @@
         package_list = params['package_list']
         data_store = self._create_data_store(params['ticket_id'])
 
-        results = dict(artifacts=[])
+        # The request parameters are forwarded to the rabbit worker processing
+        # the status/result queue to enable unidirectional communication
+        # between the requester and the ultimate receiver.
+        results = dict(artifacts=[], request=params)
 
         # The tests will succeed unless they fail ;)
         notify = amqp_utils.progress_completed

=== modified file 'tests/test_britney.py'
--- tests/test_britney.py	2014-09-24 16:44:08 +0000
+++ tests/test_britney.py	2014-10-03 08:44:06 +0000
@@ -15,6 +15,7 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 import json
+import os
 import unittest
 
 
@@ -27,6 +28,9 @@
 )
 
 
+from britney import process_results
+
+
 class TestProcessRequests(deployers.DeployerTest):
 
     def setUp(self):
@@ -50,12 +54,12 @@
         else:
             self.test_images = features.canonistack_test_images
 
-    def drain_queue(self, queue_name):
-        self.messages = []
+    def drain_tr_queue(self, queue_name):
+        messages = []
 
         def on_message(msg):
             body = json.loads(msg.body)
-            self.messages.append(body)
+            messages.append(body)
             msg.channel.basic_ack(msg.delivery_tag)
             if body.get('exit'):
                 # this will kick us out of the run-forever worker loop, relying
@@ -65,7 +69,7 @@
         conf = amqp_utils.get_config()
         # Process all messages and delete the queue when done
         amqp_utils.process_queue(conf, queue_name, on_message, delete=True)
-        return self.messages
+        return messages
 
     def assertStatusMsg(self, expected, msg):
         self.assertEqual('STATUS', msg['state'])
@@ -76,22 +80,21 @@
     def get_image_id(self, series):
         return self.test_images[series]
 
-    def test_process_test_request(self):
+    def test_tr_process_test_request(self):
         # Exercise the britney API: feed a test request on the britney requests
         # queue and ensures the test runner receive a translated request and
-        # process it properly.
-
-        # Until we have a proper implementation for the britney results queue,
-        # this will hijack the queue here.
-        progress_queue_name = 'britney.results'
+        # process it properly. We force a specific result queue to avoid
+        # involving the britney results service.
+        result_queue_name = 'britney.test_tr_process_test_request'
         series = 'precise'
         architecture = 'amd64'
         params = dict(series=series,
                       architecture=architecture,
-                      package='libpng')
+                      package='libpng',
+                      output_queue=result_queue_name)
         amqp_utils.send('britney.requests', json.dumps(params),
                         raise_errors=True)
-        msgs = self.drain_queue(progress_queue_name)
+        msgs = self.drain_tr_queue(result_queue_name)
         assertions.assertLength(self, 6, msgs)
         # The test runner received the translated request
         tr_params = msgs[0]
@@ -111,6 +114,48 @@
         self.assertEqual('FAILED', final['result'])
         self.assertEqual('COMPLETED', final['state'])
 
+    def drain_report_queue(self, queue_name):
+
+        messages = []
+
+        def on_message(msg):
+            # We expect a single message here so we just ack it and leave the
+            # loop
+            body = json.loads(msg.body)
+            messages.append(body)
+            msg.channel.basic_ack(msg.delivery_tag)
+            msg.channel.basic_cancel(msg.consumer_tag)
+
+        conf = amqp_utils.get_config()
+        # Process all messages and delete the queue when done
+        amqp_utils.process_queue(conf, queue_name, on_message, delete=True)
+        return messages
+
+    def test_process_test_and_result_request(self):
+        # Exercise the britney API: feed a test request on the britney requests
+        # queue and check that the britney result is processed.
+        report_queue_name = 'test-britney.results'
+        series = 'precise'
+        architecture = 'amd64'
+        params = dict(series=series,
+                      architecture=architecture,
+                      package='libpng',
+                      report_queue=report_queue_name)
+        amqp_utils.send('britney.requests', json.dumps(params),
+                        raise_errors=True)
+        msgs = self.drain_report_queue(report_queue_name)
+        # The results are in the britney swift container
+        assertions.assertLength(self, 1, msgs)
+        # Check the swift container content
+        dir_name = msgs[0]['result_at']
+        log_file_name = os.path.join(dir_name, 'libpng.log')
+        # We delay the import so it won't fail when the 'unit_config' file
+        # is not available
+        from ci_utils import unit_config
+        britney_data_store = process_results.BritneyDataStore(
+            'britney.results', unit_config.get_auth_config(), public=True)
+        self.assertIn(log_file_name, britney_data_store.list_files())
+
 
 if __name__ == "__main__":
     unittest.main()