Merge lp:~joetalbott/uci-engine/split_infra_failures into lp:uci-engine

Proposed by Joe Talbott on 2015-01-07
Status: Merged
Merged at revision: 918
Proposed branch: lp:~joetalbott/uci-engine/split_infra_failures
Merge into: lp:uci-engine
Diff against target: 152 lines (+57/-19)
3 files modified
ci-utils/ci_utils/amqp_utils.py (+9/-0)
ci-utils/ci_utils/amqp_worker.py (+10/-1)
ci-utils/ci_utils/tests/test_amqp_worker.py (+38/-18)
To merge this branch: bzr merge lp:~joetalbott/uci-engine/split_infra_failures
Reviewer Review Type Date Requested Status
Paul Larson Approve on 2015-01-08
Francis Ginther Approve on 2015-01-08
Vincent Ladeuil (community) Approve on 2015-01-08
Larry E Works (community) 2015-01-07 Approve on 2015-01-07
Review via email: mp+245789@code.launchpad.net

Commit message

amqp_worker - Treat DataStoreExceptions as infrastructure failures.

Description of the change

amqp_worker - Treat DataStoreExceptions as infrastructure failures.

To post a comment you must log in.
Larry E Works (larry-e-works) wrote :

LGTM; tested successfully.

review: Approve
Francis Ginther (fginther) wrote :

When this task was originally discussed, I understood that we wanted to focus on only the test_runner first and so we would need to subclass and override this code just for the test_runner. However, there is much here that I think makes sense and I think the approach should apply to all AMQPWorker components.

The test_runner component does define a couple of exception classes that represent infrastructure failures (for example, TestBedException), but I think the approach of generically treating all non-KeyboardException exceptions as infrastructure failures should work (see inline comment). The test_runner can still subclass this if it needs to refine the handling of infrastructure failures.

review: Needs Fixing
Vincent Ladeuil (vila) wrote :

Good job !

I think that's the right approach (especially introducing progress_errored !).

I agree with Francis remarks overall.

I'll add that we don't want to introduce a coupling between amqp_worker and data_store so I'd rather override in the test runner which knows about data_store.

https://code.launchpad.net/~vila/uci-engine/nova-failure-as-testbed-exception/+merge/245831 is attempting to land right now and we now also want to catch TestBedException.

As Francis said, we don't want to list all possible exceptions in amqp_worker and TestBedException is definitely specific to the test runner only.

Vincent Ladeuil (vila) wrote :

> I'll add that we don't want to introduce a coupling between amqp_worker and
> data_store so I'd rather override in the test runner which knows about
> data_store.

Wrong, amqp_worker already import data_store.

> TestBedException is definitely specific to the test runner only.

But that is still valid.

Vincent Ladeuil (vila) wrote :

I was vaguely considering allowing daughter classes to define their exceptions but that's not needed at that point.

review: Approve
Francis Ginther (fginther) wrote :

Looks good! I also like the introduction of progress_errored(), just forgot to mention it earlier.

review: Approve
Paul Larson (pwlars) wrote :

I like this, and have a dumb question about it because I haven't had a chance to try it for myself yet. If you run a ticket through with this and get an infrastructure failure, is this enough so that it actually shows up in the ticket completion status to the user as "error" rather than "failure"? If so, even better, but I wonder if as a later task we should actually explain the difference on the UI somehow. +1 from m e

review: Approve
Ubuntu CI Bot (uci-bot) wrote :
Download full text (111.9 KiB)

The attempt to merge lp:~joetalbott/uci-engine/split_infra_failures into lp:uci-engine failed. Below is the output from the failed tests.

Checking juju status
Private PPAs: disabled
Preparing local branch upload...
Uploading local branch, fingerprint 19728412d2e1dd2f0cbf9586e1070c1bde990b6b
Installing keys from bzr+ssh://bazaar.launchpad.net/~ci-engineering-private/+junk/ci-airline-dev-keys/
Running juju-deployer -v -c /tmp/tmpHRJjww/deployer/relations.yaml -c /tmp/tmpHRJjww/deployer/services.yaml ci-airline
Tests running...
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testConnectFailed ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testProcessQueue ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testRunForever ... OK (0.102 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testSent ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestProgressTrigger.testProgress ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testCancel ... OK (0.105 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoQueue ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoTicket ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageCalledProcessError ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageDataStoreException ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageFail ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageKilled ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageSimple ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageUnexpected ... OK (0.006 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testSaveLastRun ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCBRuns ... OK (0.021 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCanCancel ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_invalid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_valid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_abspath ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_basename ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_relpath ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_empty_single_config ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_multiple_config ... OK (0.006 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_single_config ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_features.TestJujuControlBucket.test_no_control_bucket ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_features.TestJujuGetEnvsFromYaml.test_broken_environments_file ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_features.TestJujuGetEnvsFromYaml.test_inconsistent_environments_file ... OK (0.003 secs)
ci-utils.ci_utils.tests.t...

Ubuntu CI Bot (uci-bot) wrote :
Download full text (59.8 KiB)

The attempt to merge lp:~joetalbott/uci-engine/split_infra_failures into lp:uci-engine failed. Below is the output from the failed tests.

2015-01-09 10:12:22 INFO juju.cmd supercommand.go:37 running jujud [1.20.14.1-precise-amd64 gc]
2015-01-09 10:12:22 DEBUG juju.agent agent.go:377 read agent config, format "1.18"
2015-01-09 10:12:22 INFO juju.jujud unit.go:78 unit agent unit-ci-airline-ts-django-0 start (1.20.14.1-precise-amd64 [gc])
2015-01-09 10:12:22 INFO juju.worker runner.go:260 start "api"
2015-01-09 10:12:22 INFO juju.state.api apiclient.go:242 dialing "wss://10.0.3.1:17070/"
2015-01-09 10:12:22 INFO juju.state.api apiclient.go:176 connection established to "wss://10.0.3.1:17070/"
2015-01-09 10:12:22 INFO juju.state.api apiclient.go:242 dialing "wss://10.0.3.1:17070/"
2015-01-09 10:12:22 INFO juju.state.api apiclient.go:176 connection established to "wss://10.0.3.1:17070/"
2015-01-09 10:12:23 INFO juju.state.api apiclient.go:242 dialing "wss://10.0.3.1:17070/"
2015-01-09 10:12:23 INFO juju.state.api apiclient.go:176 connection established to "wss://10.0.3.1:17070/"
2015-01-09 10:12:23 INFO juju.worker runner.go:260 start "upgrader"
2015-01-09 10:12:23 INFO juju.worker runner.go:260 start "logger"
2015-01-09 10:12:23 DEBUG juju.worker.logger logger.go:35 initial log config: "<root>=DEBUG"
2015-01-09 10:12:23 INFO juju.worker runner.go:260 start "uniter"
2015-01-09 10:12:23 DEBUG juju.worker.logger logger.go:60 logger setup
2015-01-09 10:12:23 INFO juju.worker runner.go:260 start "apiaddressupdater"
2015-01-09 10:12:23 INFO juju.worker runner.go:260 start "rsyslog"
2015-01-09 10:12:23 DEBUG juju.worker.rsyslog worker.go:75 starting rsyslog worker mode 1 for "unit-ci-airline-ts-django-0" "tarmac-local"
2015-01-09 10:12:23 DEBUG juju.worker.logger logger.go:45 reconfiguring logging from "<root>=DEBUG" to "<root>=WARNING;unit=DEBUG"
2015-01-09 10:12:38 INFO juju-log Making dir /srv/ci-airline-ts-django/code/ root:root 555
2015-01-09 10:12:39 INFO juju-log Adding dependencies.
2015-01-09 10:12:39 INFO juju-log installing apt packages...
2015-01-09 10:12:40 INFO config-changed gpg: keyring `/tmp/tmp1rpXVF/secring.gpg' created
2015-01-09 10:12:40 INFO config-changed gpg: keyring `/tmp/tmp1rpXVF/pubring.gpg' created
2015-01-09 10:12:40 INFO config-changed gpg: requesting key 6A8DFC40 from hkp server keyserver.ubuntu.com
2015-01-09 10:12:40 INFO config-changed gpg: /tmp/tmp1rpXVF/trustdb.gpg: trustdb created
2015-01-09 10:12:40 INFO config-changed gpg: key 6A8DFC40: public key "Launchpad PPA for Canonical CI Engineering" imported
2015-01-09 10:12:40 INFO config-changed gpg: Total number processed: 1
2015-01-09 10:12:40 INFO config-changed gpg: imported: 1 (RSA: 1)
2015-01-09 10:12:41 INFO config-changed OK
2015-01-09 10:12:44 INFO config-changed Get:1 http://ppa.launchpad.net precise Release.gpg [316 B]
2015-01-09 10:12:44 INFO config-changed Get:2 http://ubuntu-cloud.archive.canonical.com precise-updates/cloud-tools Release.gpg [543 B]
2015-01-09 10:12:46 INFO config-changed Hit http://archive.ubuntu.com precise Release.gpg
2015-01-09 10:12:46 INFO config-changed Get:3 http://ppa.launchpad.net ...

Vincent Ladeuil (vila) wrote :

Merging https://code.launchpad.net/~joetalbott/uci-engine/split_infra_failures into https://code.launchpad.net/~canonical-ci-engineering/uci-engine/trunk failed: Test command "/usr/bin/env python bin/called-by-tarmac.py" failed.
Firing tarmac_post_merge hook
:32:45 INFO pgsql-relation-joined TCP/IP connections on port 5432?
2015-01-09 10:32:45 INFO pgsql-relation-joined Syncing...
2015-01-09 10:32:45 INFO juju-log pgsql:12: migration command failed, retrying...
2015-01-09 10:32:51 INFO pgsql-relation-joined OperationalError: could not connect to server: Connection refused
2015-01-09 10:32:51 INFO pgsql-relation-joined Is the server running on host "10.0.3.134" and accepting
2015-01-09 10:32:51 INFO pgsql-relation-joined TCP/IP connections on port 5432?
2015-01-09 10:32:51 INFO pgsql-relation-joined Syncing...
2015-01-09 10:32:51 INFO juju-log pgsql:12: migration command failed, retrying...
2015-01-09 10:33:00 INFO juju-log pgsql:12: migration command failed
2015-01-09 10:33:00 INFO pgsql-relation-joined Traceback (most recent call last):
2015-01-09 10:33:00 INFO pgsql-relation-joined File "/var/lib/juju/agents/unit-ci-airline-ts-django-0/charm/hooks/pgsql-relation-joined", line 624, in <module>
2015-01-09 10:33:00 INFO pgsql-relation-joined hooks.execute(sys.argv)
2015-01-09 10:33:00 INFO pgsql-relation-joined File "/var/lib/juju/agents/unit-ci-airline-ts-django-0/charm/hooks/charmhelpers/core/hookenv.py", line 480, in execute
2015-01-09 10:33:00 INFO pgsql-relation-joined self._hooks[hook_name]()
2015-01-09 10:33:00 INFO pgsql-relation-joined File "/var/lib/juju/agents/unit-ci-airline-ts-django-0/charm/hooks/pgsql-relation-joined", line 561, in pgsql_relation_joined_changed
2015-01-09 10:33:00 INFO pgsql-relation-joined _db_migrate()
2015-01-09 10:33:00 INFO pgsql-relation-joined File "/var/lib/juju/agents/unit-ci-airline-ts-django-0/charm/hooks/pgsql-relation-joined", line 453, in _db_migrate
2015-01-09 10:33:00 INFO pgsql-relation-joined raise Exception("Migration failed")
2015-01-09 10:33:00 INFO pgsql-relation-joined Exception: Migration failed
2015-01-09 10:33:00 ERROR juju.worker.uniter uniter.go:486 hook failed: exit status 1
C

Ubuntu CI Bot (uci-bot) wrote :
Download full text (111.9 KiB)

The attempt to merge lp:~joetalbott/uci-engine/split_infra_failures into lp:uci-engine failed. Below is the output from the failed tests.

Checking juju status
Private PPAs: disabled
Preparing local branch upload...
Uploading local branch, fingerprint 7aa345427040a370c9ba2359863bb19c86ef0b17
Installing keys from bzr+ssh://bazaar.launchpad.net/~ci-engineering-private/+junk/ci-airline-dev-keys/
Running juju-deployer -v -c /tmp/tmpP7YcJC/deployer/relations.yaml -c /tmp/tmpP7YcJC/deployer/services.yaml ci-airline
Tests running...
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testConnectFailed ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testProcessQueue ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testRunForever ... OK (0.102 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testSent ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp.TestProgressTrigger.testProgress ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testCancel ... OK (0.107 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoQueue ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoTicket ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageCalledProcessError ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageDataStoreException ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageFail ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageKilled ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageSimple ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageUnexpected ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testSaveLastRun ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCBRuns ... OK (0.021 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCanCancel ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_invalid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_valid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_abspath ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_basename ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_relpath ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_empty_single_config ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_multiple_config ... OK (0.006 secs)
ci-utils.ci_utils.tests.test_deployer.TestDeployer.test_load_single_config ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_features.TestJujuControlBucket.test_no_control_bucket ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_features.TestJujuGetEnvsFromYaml.test_broken_environments_file ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_features.TestJujuGetEnvsFromYaml.test_inconsistent_environments_file ... OK (0.003 secs)
ci-utils.ci_utils.tests.t...

Vincent Ladeuil (vila) wrote :

======================================================================
FAIL: tests.test_webui.WebUITest.test_edit_ticket_with_invalid_mps
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tarmac/branch.rvSApj/bin/../tests/test_webui.py", line 490, in test_edit_ticket_with_invalid_mps
    wait_for(get_element_by_css, 'a.yui3-pagview-link-page-active')
  File "/dev/shm/venv-grSLE6/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 145, in inner
    return func(*args, **kwargs)
  File "/dev/shm/venv-grSLE6/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 852, in wait_for
    return _wait_for(condition, False, _TIMEOUT, _POLL, *args, **kwargs)
  File "/dev/shm/venv-grSLE6/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 816, in _wait_for
    _raise(error)
  File "/dev/shm/venv-grSLE6/local/lib/python2.7/site-packages/sst-0.2.4-py2.7.egg/sst/actions.py", line 118, in _raise
    raise AssertionError(msg)
AssertionError: Timed out waiting for: get_element_by_css
Error during wait: Could not identify element: 0 elements found

Ursula Junque (ursinha) wrote :

Test failures are valid, not just tarmac hiccups (as I supposed). Celso has a branch to fix that: (MP: https://code.launchpad.net/~cprov/uci-engine/webui-native-tickets/+merge/245953)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'ci-utils/ci_utils/amqp_utils.py'
2--- ci-utils/ci_utils/amqp_utils.py 2014-10-23 12:09:49 +0000
3+++ ci-utils/ci_utils/amqp_utils.py 2015-01-08 14:43:53 +0000
4@@ -211,6 +211,15 @@
5 return send(progress_trigger, json.dumps(data))
6
7
8+def progress_errored(progress_trigger, data=None):
9+ if data is None:
10+ data = {}
11+ data['state'] = 'COMPLETED'
12+ data['result'] = 'ERROR'
13+ data['exit'] = True
14+ return send(progress_trigger, json.dumps(data))
15+
16+
17 def progress_failed(progress_trigger, data=None):
18 if data is None:
19 data = {}
20
21=== modified file 'ci-utils/ci_utils/amqp_worker.py'
22--- ci-utils/ci_utils/amqp_worker.py 2014-10-29 16:43:39 +0000
23+++ ci-utils/ci_utils/amqp_worker.py 2015-01-08 14:43:53 +0000
24@@ -213,12 +213,18 @@
25 return t
26
27 def _handle_exception(self, e, store, trigger, logger, ret):
28+ error = False
29 try:
30 err = 'Unexpected exception occurred'
31 if isinstance(e, KeyboardInterrupt):
32 err = 'Worker terminated'
33 elif isinstance(e, subprocess.CalledProcessError) and logger:
34 err = 'error running {}: {}'.format(e.cmd, e.output)
35+ error = True
36+ # All exceptions except those that get whitelisted here are
37+ # considered infrastructure failures.
38+ else:
39+ error = True
40 ret['error_message'] = err
41 ret['traceback'] = traceback.format_exc()
42 if logger:
43@@ -226,7 +232,10 @@
44 self._store_worker_log(store, logger, ret)
45 finally:
46 if trigger:
47- amqp_utils.progress_failed(trigger, ret)
48+ if error:
49+ amqp_utils.progress_errored(trigger, ret)
50+ else:
51+ amqp_utils.progress_failed(trigger, ret)
52
53 def _on_message(self, msg):
54 log.info('on_message: %s', msg.body)
55
56=== modified file 'ci-utils/ci_utils/tests/test_amqp_worker.py'
57--- ci-utils/ci_utils/tests/test_amqp_worker.py 2014-10-07 10:04:01 +0000
58+++ ci-utils/ci_utils/tests/test_amqp_worker.py 2015-01-08 14:43:53 +0000
59@@ -21,7 +21,10 @@
60
61 import mock
62
63-from ci_utils import amqp_worker
64+from ci_utils import (
65+ amqp_worker,
66+ data_store,
67+)
68
69
70 class _worker(amqp_worker.AMQPWorker):
71@@ -111,6 +114,31 @@
72 self.assertEqual('queue-name.output.log',
73 retval['artifacts'][0]['name'])
74
75+ def _assert_infrastructure_failure(self, err, worker, msg, amqp_utils):
76+ #self.assertEqual(1, amqp_utils.progress_update.call_count)
77+ msg.channel.basic_ack.assert_called_once_with('foo')
78+ self.assertEqual(0, amqp_utils.progress_failed.call_count)
79+ self.assertEqual(1, amqp_utils.progress_errored.call_count)
80+ self.assertEqual(
81+ 'queue-name', amqp_utils.progress_errored.call_args[0][0])
82+ self.assertEqual(1, worker.store.put_file.call_count)
83+
84+ @mock.patch('ci_utils.amqp_worker.amqp_utils')
85+ def testOnMessageDataStoreException(self, amqp_utils):
86+ '''Test on message logic for an unexpected failure'''
87+ msg = mock.Mock()
88+ msg.delivery_tag = 'foo'
89+ msg.body = json.dumps({
90+ 'ticket_id': 1,
91+ 'progress_trigger': 'queue-name',
92+ })
93+ err = 'this is the worker error'
94+ worker = _worker(None, None, data_store.DataStoreException, err)
95+ worker._on_message(msg)
96+ self._assert_infrastructure_failure(err, worker, msg, amqp_utils)
97+ self.assertIn(
98+ err, amqp_utils.progress_errored.call_args[0][1]['traceback'])
99+
100 @mock.patch('ci_utils.amqp_worker.amqp_utils')
101 def testOnMessageUnexpected(self, amqp_utils):
102 '''Test on message logic for an unexpected failure'''
103@@ -123,14 +151,9 @@
104 err = 'this is the worker error'
105 worker = _worker(None, None, RuntimeError, err)
106 worker._on_message(msg)
107- self.assertEqual(1, amqp_utils.progress_update.call_count)
108- msg.channel.basic_ack.assert_called_once_with('foo')
109- self.assertEqual(1, amqp_utils.progress_failed.call_count)
110- self.assertEqual(
111- 'queue-name', amqp_utils.progress_failed.call_args[0][0])
112+ self._assert_infrastructure_failure(err, worker, msg, amqp_utils)
113 self.assertIn(
114- err, amqp_utils.progress_failed.call_args[0][1]['traceback'])
115- self.assertEqual(1, worker.store.put_file.call_count)
116+ err, amqp_utils.progress_errored.call_args[0][1]['traceback'])
117
118 @mock.patch('ci_utils.amqp_worker.amqp_utils')
119 def testOnMessageCalledProcessError(self, amqp_utils):
120@@ -154,15 +177,11 @@
121
122 worker = _calledprocess_worker(None, None, None, None)
123 worker._on_message(msg)
124-
125- self.assertEqual(1, amqp_utils.progress_update.call_count)
126- msg.channel.basic_ack.assert_called_once_with('foo')
127- self.assertEqual(1, amqp_utils.progress_failed.call_count)
128- self.assertEqual(
129- 'queue-name', amqp_utils.progress_failed.call_args[0][0])
130+ self._assert_infrastructure_failure(err, worker, msg, amqp_utils)
131 self.assertIn('CalledProcessError',
132- amqp_utils.progress_failed.call_args[0][1]['traceback'])
133- err = amqp_utils.progress_failed.call_args[0][1]['error_message']
134+ amqp_utils.progress_errored.call_args[0][1]['traceback'])
135+
136+ err = amqp_utils.progress_errored.call_args[0][1]['error_message']
137 self.assertIn(str(worker.cmd), err)
138 self.assertIn(str(worker.output), err)
139 self.assertEqual(1, worker.store.put_file.call_count)
140@@ -199,9 +218,10 @@
141 worker = _worker(amqp_utils, 1)
142 worker._on_message(msg)
143 msg.channel.basic_ack.assert_called_once_with('foo')
144- self.assertEqual(1, amqp_utils.progress_failed.call_count)
145+ self.assertEqual(0, amqp_utils.progress_failed.call_count)
146+ self.assertEqual(1, amqp_utils.progress_errored.call_count)
147 self.assertEqual(
148- 'queue-name', amqp_utils.progress_failed.call_args[0][0])
149+ 'queue-name', amqp_utils.progress_errored.call_args[0][0])
150
151 @mock.patch('ci_utils.amqp_worker.amqp_utils')
152 def testNoQueue(self, amqp_utils):

Subscribers

People subscribed via source and target branches