Merge lp:~joetalbott/uci-engine/split_infra_failures into lp:uci-engine

[Joe Talbott]

amqp_worker - Treat DataStoreExceptions as infrastructure failures.

amqp_worker - Treat DataStoreExceptions as infrastructure failures.

[Larry E Works (larry-e-works)]

LGTM; tested successfully.

review: Approve

[Francis Ginther (fginther)]

When this task was originally discussed, I understood that we wanted to focus on only the test_runner first and so we would need to subclass and override this code just for the test_runner. However, there is much here that I think makes sense and I think the approach should apply to all AMQPWorker components.

The test_runner component does define a couple of exception classes that represent infrastructure failures (for example, TestBedException), but I think the approach of generically treating all non-KeyboardException exceptions as infrastructure failures should work (see inline comment). The test_runner can still subclass this if it needs to refine the handling of infrastructure failures.

review: Needs Fixing

[Vincent Ladeuil (vila)]

Good job !

I think that's the right approach (especially introducing progress_errored !).

I agree with Francis remarks overall.

I'll add that we don't want to introduce a coupling between amqp_worker and data_store so I'd rather override in the test runner which knows about data_store.

https://code.launchpad.net/~vila/uci-engine/nova-failure-as-testbed-exception/+merge/245831 is attempting to land right now and we now also want to catch TestBedException.

As Francis said, we don't want to list all possible exceptions in amqp_worker and TestBedException is definitely specific to the test runner only.

[Vincent Ladeuil (vila)]

> I'll add that we don't want to introduce a coupling between amqp_worker and
> data_store so I'd rather override in the test runner which knows about
> data_store.

Wrong, amqp_worker already import data_store.

> TestBedException is definitely specific to the test runner only.

But that is still valid.

[Vincent Ladeuil (vila)]

I was vaguely considering allowing daughter classes to define their exceptions but that's not needed at that point.

review: Approve

[Francis Ginther (fginther)]

Looks good! I also like the introduction of progress_errored(), just forgot to mention it earlier.

review: Approve

[Paul Larson (pwlars)]

I like this, and have a dumb question about it because I haven't had a chance to try it for myself yet. If you run a ticket through with this and get an infrastructure failure, is this enough so that it actually shows up in the ticket completion status to the user as "error" rather than "failure"? If so, even better, but I wonder if as a later task we should actually explain the difference on the UI somehow. +1 from m e

review: Approve

[Ubuntu CI Bot (uci-bot)]

The attempt to merge lp:~joetalbott/uci-engine/split_infra_failures into lp:uci-engine failed. Below is the output from the failed tests.

Checking juju status
Private PPAs: disabled
Preparing local branch upload...
Uploading local branch, fingerprint 19728412d2e1dd2f0cbf9586e1070c1bde990b6b
Installing keys from bzr+ssh://bazaar.launchpad.net/~ci-engineering-private/+junk/ci-airline-dev-keys/
Running juju-deployer -v -c /tmp/tmpHRJjww/deployer/relations.yaml -c /tmp/tmpHRJjww/deployer/services.yaml ci-airline
Tests running...
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testConnectFailed ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testProcessQueue ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testRunForever ... OK (0.102 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testSent ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestProgressTrigger.testProgress ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testCancel ... OK (0.105 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoQueue ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoTicket ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageCalledProcessError ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageDataStoreException ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageFail ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageKilled ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageSimple ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageUnexpected ... OK (0.006 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testSaveLastRun ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCBRuns ... OK (0.021 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCanCancel ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_invalid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_valid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_abspath ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_basename ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_relpath ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_empty_single_config ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_multiple_config ... OK (0.006 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_single_config ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_features.TestJujuControlBucket.test_no_control_bucket ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_features.TestJujuGetEnvsFromYaml.test_broken_environments_file ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_features.TestJujuGetEnvsFromYaml.test_inconsistent_environments_file ... OK (0.003 secs)
ci-utils.ci_utils.tests.t...

[Ubuntu CI Bot (uci-bot)]

The attempt to merge lp:~joetalbott/uci-engine/split_infra_failures into lp:uci-engine failed. Below is the output from the failed tests.

2015-01-09 10:12:22 INFO juju.cmd supercommand.go:37 running jujud [1.20.14.1-precise-amd64 gc]
2015-01-09 10:12:22 DEBUG juju.agent agent.go:377 read agent config, format "1.18"
2015-01-09 10:12:22 INFO juju.jujud unit.go:78 unit agent unit-ci-airline-ts-django-0 start (1.20.14.1-precise-amd64 [gc])
2015-01-09 10:12:22 INFO juju.worker runner.go:260 start "api"
2015-01-09 10:12:22 INFO juju.state.api apiclient.go:242 dialing "wss://10.0.3.1:17070/"
2015-01-09 10:12:22 INFO juju.state.api apiclient.go:176 connection established to "wss://10.0.3.1:17070/"
2015-01-09 10:12:22 INFO juju.state.api apiclient.go:242 dialing "wss://10.0.3.1:17070/"
2015-01-09 10:12:22 INFO juju.state.api apiclient.go:176 connection established to "wss://10.0.3.1:17070/"
2015-01-09 10:12:23 INFO juju.state.api apiclient.go:242 dialing "wss://10.0.3.1:17070/"
2015-01-09 10:12:23 INFO juju.state.api apiclient.go:176 connection established to "wss://10.0.3.1:17070/"
2015-01-09 10:12:23 INFO juju.worker runner.go:260 start "upgrader"
2015-01-09 10:12:23 INFO juju.worker runner.go:260 start "logger"
2015-01-09 10:12:23 DEBUG juju.worker.logger logger.go:35 initial log config: "<root>=DEBUG"
2015-01-09 10:12:23 INFO juju.worker runner.go:260 start "uniter"
2015-01-09 10:12:23 DEBUG juju.worker.logger logger.go:60 logger setup
2015-01-09 10:12:23 INFO juju.worker runner.go:260 start "apiaddressupdater"
2015-01-09 10:12:23 INFO juju.worker runner.go:260 start "rsyslog"
2015-01-09 10:12:23 DEBUG juju.worker.rsyslog worker.go:75 starting rsyslog worker mode 1 for "unit-ci-airline-ts-django-0" "tarmac-local"
2015-01-09 10:12:23 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "<root>=DEBUG" to "<root>=WARNING;unit=DEBUG"
2015-01-09 10:12:38 INFO juju-log Making dir /srv/ci-airline-ts-django/code/ root:root 555
2015-01-09 10:12:39 INFO juju-log Adding dependencies.
2015-01-09 10:12:39 INFO juju-log installing apt packages...
2015-01-09 10:12:40 INFO config-changed gpg: keyring `/tmp/tmp1rpXVF/secring.gpg' created
2015-01-09 10:12:40 INFO config-changed gpg: keyring `/tmp/tmp1rpXVF/pubring.gpg' created
2015-01-09 10:12:40 INFO config-changed gpg: requesting key 6A8DFC40 from hkp server keyserver.ubuntu.com
2015-01-09 10:12:40 INFO config-changed gpg: /tmp/tmp1rpXVF/trustdb.gpg: trustdb created
2015-01-09 10:12:40 INFO config-changed gpg: key 6A8DFC40: public key "Launchpad PPA for Canonical CI Engineering" imported
2015-01-09 10:12:40 INFO config-changed gpg: Total number processed: 1
2015-01-09 10:12:40 INFO config-changed gpg: imported: 1 (RSA: 1)
2015-01-09 10:12:41 INFO config-changed OK
2015-01-09 10:12:44 INFO config-changed Get:1 http://ppa.launchpad.net precise Release.gpg [316 B]
2015-01-09 10:12:44 INFO config-changed Get:2 http://ubuntu-cloud.archive.canonical.com precise-updates/cloud-tools Release.gpg [543 B]
2015-01-09 10:12:46 INFO config-changed Hit http://archive.ubuntu.com precise Release.gpg
2015-01-09 10:12:46 INFO config-changed Get:3 http://ppa.launchpad.net ...

[Vincent Ladeuil (vila)]

Merging https://code.launchpad.net/~joetalbott/uci-engine/split_infra_failures into https://code.launchpad.net/~canonical-ci-engineering/uci-engine/trunk failed: Test command "/usr/bin/env python bin/called-by-tarmac.py" failed.
Firing tarmac_post_merge hook
:32:45 INFO pgsql-relation-joined TCP/IP connections on port 5432?
2015-01-09 10:32:45 INFO pgsql-relation-joined Syncing...
2015-01-09 10:32:45 INFO juju-log pgsql:12: migration command failed, retrying...
2015-01-09 10:32:51 INFO pgsql-relation-joined OperationalError: could not connect to server: Connection refused
2015-01-09 10:32:51 INFO pgsql-relation-joined Is the server running on host "10.0.3.134" and accepting
2015-01-09 10:32:51 INFO pgsql-relation-joined TCP/IP connections on port 5432?
2015-01-09 10:32:51 INFO pgsql-relation-joined Syncing...
2015-01-09 10:32:51 INFO juju-log pgsql:12: migration command failed, retrying...
2015-01-09 10:33:00 INFO juju-log pgsql:12: migration command failed
2015-01-09 10:33:00 INFO pgsql-relation-joined Traceback (most recent call last):
2015-01-09 10:33:00 INFO pgsql-relation-joined File "/var/lib/juju/agents/unit-ci-airline-ts-django-0/charm/hooks/pgsql-relation-joined", line 624, in <module>
2015-01-09 10:33:00 INFO pgsql-relation-joined hooks.execute(sys.argv)
2015-01-09 10:33:00 INFO pgsql-relation-joined File "/var/lib/juju/agents/unit-ci-airline-ts-django-0/charm/hooks/charmhelpers/core/hookenv.py", line 480, in execute
2015-01-09 10:33:00 INFO pgsql-relation-joined self._hooks[hook_name]()
2015-01-09 10:33:00 INFO pgsql-relation-joined File "/var/lib/juju/agents/unit-ci-airline-ts-django-0/charm/hooks/pgsql-relation-joined", line 561, in pgsql_relation_joined_changed
2015-01-09 10:33:00 INFO pgsql-relation-joined _db_migrate()
2015-01-09 10:33:00 INFO pgsql-relation-joined File "/var/lib/juju/agents/unit-ci-airline-ts-django-0/charm/hooks/pgsql-relation-joined", line 453, in _db_migrate
2015-01-09 10:33:00 INFO pgsql-relation-joined raise Exception("Migration failed")
2015-01-09 10:33:00 INFO pgsql-relation-joined Exception: Migration failed
2015-01-09 10:33:00 ERROR juju.worker.uniter uniter.go:486 hook failed: exit status 1
C

[Ubuntu CI Bot (uci-bot)]

The attempt to merge lp:~joetalbott/uci-engine/split_infra_failures into lp:uci-engine failed. Below is the output from the failed tests.

Checking juju status
Private PPAs: disabled
Preparing local branch upload...
Uploading local branch, fingerprint 7aa345427040a370c9ba2359863bb19c86ef0b17
Installing keys from bzr+ssh://bazaar.launchpad.net/~ci-engineering-private/+junk/ci-airline-dev-keys/
Running juju-deployer -v -c /tmp/tmpP7YcJC/deployer/relations.yaml -c /tmp/tmpP7YcJC/deployer/services.yaml ci-airline
Tests running...
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testConnectFailed ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testProcessQueue ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testRunForever ... OK (0.102 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testSent ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp.TestProgressTrigger.testProgress ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testCancel ... OK (0.107 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoQueue ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoTicket ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageCalledProcessError ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageDataStoreException ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageFail ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageKilled ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageSimple ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageUnexpected ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testSaveLastRun ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCBRuns ... OK (0.021 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCanCancel ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_invalid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_valid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_abspath ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_basename ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_relpath ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_empty_single_config ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_multiple_config ... OK (0.006 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_single_config ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_features.TestJujuControlBucket.test_no_control_bucket ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_features.TestJujuGetEnvsFromYaml.test_broken_environments_file ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_features.TestJujuGetEnvsFromYaml.test_inconsistent_environments_file ... OK (0.003 secs)
ci-utils.ci_utils.tests.t...

[Vincent Ladeuil (vila)]

======================================================================
FAIL: tests.test_webui.WebUITest.test_edit_ticket_with_invalid_mps
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tarmac/branch.rvSApj/bin/../tests/test_webui.py", line 490, in test_edit_ticket_with_invalid_mps
    wait_for(get_element_by_css, 'a.yui3-pagview-link-page-active')
  File "/dev/shm/venv-grSLE6/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 145, in inner
    return func(*args, **kwargs)
  File "/dev/shm/venv-grSLE6/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 852, in wait_for
    return _wait_for(condition, False, _TIMEOUT, _POLL, *args, **kwargs)
  File "/dev/shm/venv-grSLE6/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 816, in _wait_for
    _raise(error)
  File "/dev/shm/venv-grSLE6/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 118, in _raise
    raise AssertionError(msg)
AssertionError: Timed out waiting for: get_element_by_css
Error during wait: Could not identify element: 0 elements found

[Ursula Junque (ursinha)]

Test failures are valid, not just tarmac hiccups (as I supposed). Celso has a branch to fix that: (MP: https://code.launchpad.net/~cprov/uci-engine/webui-native-tickets/+merge/245953)

=== modified file 'ci-utils/ci_utils/amqp_utils.py'
--- ci-utils/ci_utils/amqp_utils.py	2014-10-23 12:09:49 +0000
+++ ci-utils/ci_utils/amqp_utils.py	2015-01-08 14:43:53 +0000
@@ -211,6 +211,15 @@
     return send(progress_trigger, json.dumps(data))
 
 
+def progress_errored(progress_trigger, data=None):
+    if data is None:
+        data = {}
+    data['state'] = 'COMPLETED'
+    data['result'] = 'ERROR'
+    data['exit'] = True
+    return send(progress_trigger, json.dumps(data))
+
+
 def progress_failed(progress_trigger, data=None):
     if data is None:
         data = {}

=== modified file 'ci-utils/ci_utils/amqp_worker.py'
--- ci-utils/ci_utils/amqp_worker.py	2014-10-29 16:43:39 +0000
+++ ci-utils/ci_utils/amqp_worker.py	2015-01-08 14:43:53 +0000
@@ -213,12 +213,18 @@
         return t
 
     def _handle_exception(self, e, store, trigger, logger, ret):
+        error = False
         try:
             err = 'Unexpected exception occurred'
             if isinstance(e, KeyboardInterrupt):
                 err = 'Worker terminated'
             elif isinstance(e, subprocess.CalledProcessError) and logger:
                 err = 'error running {}: {}'.format(e.cmd, e.output)
+                error = True
+            # All exceptions except those that get whitelisted here are
+            # considered infrastructure failures.
+            else:
+                error = True
             ret['error_message'] = err
             ret['traceback'] = traceback.format_exc()
             if logger:
@@ -226,7 +232,10 @@
             self._store_worker_log(store, logger, ret)
         finally:
             if trigger:
-                amqp_utils.progress_failed(trigger, ret)
+                if error:
+                    amqp_utils.progress_errored(trigger, ret)
+                else:
+                    amqp_utils.progress_failed(trigger, ret)
 
     def _on_message(self, msg):
         log.info('on_message: %s', msg.body)

=== modified file 'ci-utils/ci_utils/tests/test_amqp_worker.py'
--- ci-utils/ci_utils/tests/test_amqp_worker.py	2014-10-07 10:04:01 +0000
+++ ci-utils/ci_utils/tests/test_amqp_worker.py	2015-01-08 14:43:53 +0000
@@ -21,7 +21,10 @@
 
 import mock
 
-from ci_utils import amqp_worker
+from ci_utils import (
+    amqp_worker,
+    data_store,
+)
 
 
 class _worker(amqp_worker.AMQPWorker):
@@ -111,6 +114,31 @@
         self.assertEqual('queue-name.output.log',
                          retval['artifacts'][0]['name'])
 
+    def _assert_infrastructure_failure(self, err, worker, msg, amqp_utils):
+        #self.assertEqual(1, amqp_utils.progress_update.call_count)
+        msg.channel.basic_ack.assert_called_once_with('foo')
+        self.assertEqual(0, amqp_utils.progress_failed.call_count)
+        self.assertEqual(1, amqp_utils.progress_errored.call_count)
+        self.assertEqual(
+            'queue-name', amqp_utils.progress_errored.call_args[0][0])
+        self.assertEqual(1, worker.store.put_file.call_count)
+
+    @mock.patch('ci_utils.amqp_worker.amqp_utils')
+    def testOnMessageDataStoreException(self, amqp_utils):
+        '''Test on message logic for an unexpected failure'''
+        msg = mock.Mock()
+        msg.delivery_tag = 'foo'
+        msg.body = json.dumps({
+            'ticket_id': 1,
+            'progress_trigger': 'queue-name',
+        })
+        err = 'this is the worker error'
+        worker = _worker(None, None, data_store.DataStoreException, err)
+        worker._on_message(msg)
+        self._assert_infrastructure_failure(err, worker, msg, amqp_utils)
+        self.assertIn(
+            err, amqp_utils.progress_errored.call_args[0][1]['traceback'])
+
     @mock.patch('ci_utils.amqp_worker.amqp_utils')
     def testOnMessageUnexpected(self, amqp_utils):
         '''Test on message logic for an unexpected failure'''
@@ -123,14 +151,9 @@
         err = 'this is the worker error'
         worker = _worker(None, None, RuntimeError, err)
         worker._on_message(msg)
-        self.assertEqual(1, amqp_utils.progress_update.call_count)
-        msg.channel.basic_ack.assert_called_once_with('foo')
-        self.assertEqual(1, amqp_utils.progress_failed.call_count)
-        self.assertEqual(
-            'queue-name', amqp_utils.progress_failed.call_args[0][0])
+        self._assert_infrastructure_failure(err, worker, msg, amqp_utils)
         self.assertIn(
-            err, amqp_utils.progress_failed.call_args[0][1]['traceback'])
-        self.assertEqual(1, worker.store.put_file.call_count)
+            err, amqp_utils.progress_errored.call_args[0][1]['traceback'])
 
     @mock.patch('ci_utils.amqp_worker.amqp_utils')
     def testOnMessageCalledProcessError(self, amqp_utils):
@@ -154,15 +177,11 @@
 
         worker = _calledprocess_worker(None, None, None, None)
         worker._on_message(msg)
-
-        self.assertEqual(1, amqp_utils.progress_update.call_count)
-        msg.channel.basic_ack.assert_called_once_with('foo')
-        self.assertEqual(1, amqp_utils.progress_failed.call_count)
-        self.assertEqual(
-            'queue-name', amqp_utils.progress_failed.call_args[0][0])
+        self._assert_infrastructure_failure(err, worker, msg, amqp_utils)
         self.assertIn('CalledProcessError',
-                      amqp_utils.progress_failed.call_args[0][1]['traceback'])
-        err = amqp_utils.progress_failed.call_args[0][1]['error_message']
+                      amqp_utils.progress_errored.call_args[0][1]['traceback'])
+
+        err = amqp_utils.progress_errored.call_args[0][1]['error_message']
         self.assertIn(str(worker.cmd), err)
         self.assertIn(str(worker.output), err)
         self.assertEqual(1, worker.store.put_file.call_count)
@@ -199,9 +218,10 @@
         worker = _worker(amqp_utils, 1)
         worker._on_message(msg)
         msg.channel.basic_ack.assert_called_once_with('foo')
-        self.assertEqual(1, amqp_utils.progress_failed.call_count)
+        self.assertEqual(0, amqp_utils.progress_failed.call_count)
+        self.assertEqual(1, amqp_utils.progress_errored.call_count)
         self.assertEqual(
-            'queue-name', amqp_utils.progress_failed.call_args[0][0])
+            'queue-name', amqp_utils.progress_errored.call_args[0][0])
 
     @mock.patch('ci_utils.amqp_worker.amqp_utils')
     def testNoQueue(self, amqp_utils):