Ubuntu CI Engine

Merge lp:~vila/uci-engine/lxc-not-mounted into lp:uci-engine

  1. lxc-not-mounted
  2. Merge into trunk
Proposed by Vincent Ladeuil
Status: Merged
Approved by: Vincent Ladeuil
Approved revision: 897
Merged at revision: 897
Proposed branch: lp:~vila/uci-engine/lxc-not-mounted
Merge into: lp:uci-engine
Diff against target: 15 lines (+4/-1)
1 file modified
bin/called-by-tarmac.py (+4/-1)
To merge this branch: bzr merge lp:~vila/uci-engine/lxc-not-mounted
Reviewer Review Type Date Requested Status
Evan (community) Approve
PS Jenkins bot (community) continuous-integration Approve
Francis Ginther Needs Fixing
Review via email: mp+241822@code.launchpad.net

Commit message

Log mount output to get a better understanding about why we end up with /var/lib/lxc not mounted issues.

Description of the change

Let's look at why this /var/lib/lxc mount check is failing...

To post a comment you must log in.
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

PASSED: Continuous integration, rev:896
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1713/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1713/rebuild

review: Approve (continuous-integration)
Revision history for this message
Francis Ginther (fginther) wrote :

I can't come up with any other good commands for diagnosing a missing mount. This is as good of place to start as any. Hopefully the next time we see this we'll have the logs available to do better diagnostics.

review: Needs Fixing
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

PASSED: Continuous integration, rev:897
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1715/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1715/rebuild

review: Approve (continuous-integration)
Revision history for this message
Evan (ev) wrote :

Looks better. Thanks.

review: Approve
Revision history for this message
Ubuntu CI Bot (uci-bot) wrote :
Download full text (116.6 KiB)

The attempt to merge lp:~vila/uci-engine/lxc-not-mounted into lp:uci-engine failed. Below is the output from the failed tests.

Running cm...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
uploading webui-content.tgz to swift
Updating source dependencies...
Checking juju status
Private PPAs: disabled
Preparing local branch upload...
Uploading local branch, fingerprint bf456db0b8c6830f33b801acae6ea966addf360b
Building charm: lander
Building charm: wsgi-app
Building charm: rabbitmq-worker
Building charm: chroot-builder
Building charm: key-secret-subordinate
Building charm: webui
Building charm: system-image-server
Installing keys from bzr+ssh://bazaar.launchpad.net/~ci-engineering-private/+junk/ci-airline-dev-keys/
Running juju-deployer -v -c /tmp/tmpl9wgml/deployer/branch-source-builder.yaml -c /tmp/tmpl9wgml/deployer/britney-proxy.yaml -c /tmp/tmpl9wgml/deployer/coverage-retriever.yaml -c /tmp/tmpl9wgml/deployer/gatekeeper.yaml -c /tmp/tmpl9wgml/deployer/image-builder.yaml -c /tmp/tmpl9wgml/deployer/lander.yaml -c /tmp/tmpl9wgml/deployer/nf-stats-service.yaml -c /tmp/tmpl9wgml/deployer/ppa-creator.yaml -c /tmp/tmpl9wgml/deployer/publisher.yaml -c /tmp/tmpl9wgml/deployer/relations.yaml -c /tmp/tmpl9wgml/deployer/test-runner.yaml -c /tmp/tmpl9wgml/deployer/ticket-system.yaml -c /tmp/tmpl9wgml/deployer/validator.yaml -c /tmp/tmpl9wgml/deployer/webui.yaml ci-airline
Tests running...
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testConnectFailed ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testProcessQueue ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testRunForever ... OK (0.102 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testSent ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp.TestProgressTrigger.testProgress ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testCancel ... OK (0.106 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoQueue ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoTicket ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageCalledProcessError ... OK (0.005 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageFail ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageKilled ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageSimple ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageUnexpected ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testSaveLastRun ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCBRuns ... OK (0.021 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCanCancel ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_invalid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_valid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_d...

Revision history for this message
Vincent Ladeuil (vila) wrote :
Download full text (4.6 KiB)

======================================================================
FAIL: tests.test_test_runner.TestTestRunner.test_process_ticket
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tarmac/branch.qanMru/bin/../tests/test_test_runner.py", line 94, in test_process_ticket
    assertions.assertLength(self, 6, msgs)
  File "/dev/shm/venv-3DwiXg/local/lib/python2.7/site-packages/ucitests-0.1.5-py2.7.egg/ucitests/assertions.py", line 67, in assertLength
    ' for {!r}'.format(length, len(obj_with_len), obj_with_len))
  File "/usr/lib/python2.7/unittest/case.py", line 412, in fail
    raise self.failureException(msg)
AssertionError: Incorrect length: wanted 6, got 3 for [{u'progress_trigger': u'test-progress-process', u'series': u'precise', u'ppa_list': [], u'package_list': [u'libpng'], u'image_id': u'uci/cloudimg/precise-amd64.img', u'state': u'STATUS', u'ticket_id': u'process-ticket-precise-amd64', u'architecture': u'amd64'}, {u'message': u'Setting up the testbed for ticket process-ticket-precise-amd64', u'state': u'STATUS'}, {u'artifacts': [{u'type': u'LOGS', u'name': u'image_test.libpng-testbed-cloud-init.log', u'reference': u'http://172.20.161.145:8080/v1/AUTH_bcb5bfaa8655475988210d34dab13c34/ticket-process-ticket-precise-amd64/libpng-testbed-cloud-init.log'}, {u'type': u'LOGS', u'name': u'image_test.output.log', u'reference': u'http://172.20.161.145:8080/v1/AUTH_bcb5bfaa8655475988210d34dab13c34/ticket-process-ticket-precise-amd64/image_test.output.log'}], u'request': {u'progress_trigger': u'test-progress-process', u'series': u'precise', u'ppa_list': [], u'package_list': [u'libpng'], u'image_id': u'uci/cloudimg/precise-amd64.img', u'state': u'STATUS', u'ticket_id': u'process-ticket-precise-amd64', u'architecture': u'amd64'}, u'state': u'COMPLETED', u'exit': True, u'result': u'FAILED'}]

http://172.20.161.145:8080/v1/AUTH_bcb5bfaa8655475988210d34dab13c34/ticket-process-ticket-precise-amd64/image_test.output.log

[2014-11-18 09:07:38,431] image_test:INFO:Running in /home/ubuntu with HOME: /home/ubuntu
[2014-11-18 09:07:38,431] image_test:INFO:Setting up the testbed for ticket process-ticket-precise-amd64
[2014-11-18 09:08:00,145] image_test:INFO:Got IP 172.19.1.69 for 9a50846a-7932-4726-8270-b28f47fdd18e
[2014-11-18 09:09:01,222] image_test:ERROR:The testbed setup for process-ticket-precise-amd64 failed
Traceback (most recent call last):
  File "./test_runner/tstrun/run_worker.py", line 181, in setup_testbed
  File "/srv/ci-airline-tr-rabbit-worker/code/r1_bf456db0b8c6830f33b801acae6ea966addf360b/test_runner/tstrun/testbed.py", line 306, in setup
    self.wait_for_cloud_init()
  File "/srv/ci-airline-tr-rabbit-worker/code/r1_bf456db0b8c6830f33b801acae6ea966addf360b/test_runner/tstrun/testbed.py", line 391, in wait_for_cloud_init
    console = self.get_cloud_init_console(10)
  File "/srv/ci-airline-tr-rabbit-worker/code/r1_bf456db0b8c6830f33b801acae6ea966addf360b/test_runner/tstrun/testbed.py", line 382, in get_cloud_init_console
    return self.nova.get_server_console(self.instance, length)
  File "/srv/ci-airline-tr-rabbit-worker/code/r1_bf456db0b8c6830f33b801acae6ea966addf3...

Read more...

Revision history for this message
Ubuntu CI Bot (uci-bot) wrote :
Download full text (115.7 KiB)

The attempt to merge lp:~vila/uci-engine/lxc-not-mounted into lp:uci-engine failed. Below is the output from the failed tests.

Running cm...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
uploading webui-content.tgz to swift
Updating source dependencies...
Checking juju status
Private PPAs: disabled
Preparing local branch upload...
Uploading local branch, fingerprint 2f8d7faf0d1094bc5cde23a58cc5ff7c64d8d239
Building charm: lander
Building charm: wsgi-app
Building charm: rabbitmq-worker
Building charm: chroot-builder
Building charm: key-secret-subordinate
Building charm: webui
Building charm: system-image-server
Installing keys from bzr+ssh://bazaar.launchpad.net/~ci-engineering-private/+junk/ci-airline-dev-keys/
Running juju-deployer -v -c /tmp/tmpQrRKXU/deployer/branch-source-builder.yaml -c /tmp/tmpQrRKXU/deployer/britney-proxy.yaml -c /tmp/tmpQrRKXU/deployer/coverage-retriever.yaml -c /tmp/tmpQrRKXU/deployer/gatekeeper.yaml -c /tmp/tmpQrRKXU/deployer/image-builder.yaml -c /tmp/tmpQrRKXU/deployer/lander.yaml -c /tmp/tmpQrRKXU/deployer/nf-stats-service.yaml -c /tmp/tmpQrRKXU/deployer/ppa-creator.yaml -c /tmp/tmpQrRKXU/deployer/publisher.yaml -c /tmp/tmpQrRKXU/deployer/relations.yaml -c /tmp/tmpQrRKXU/deployer/test-runner.yaml -c /tmp/tmpQrRKXU/deployer/ticket-system.yaml -c /tmp/tmpQrRKXU/deployer/validator.yaml -c /tmp/tmpQrRKXU/deployer/webui.yaml ci-airline
Tests running...
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testConnectFailed ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testProcessQueue ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testRunForever ... OK (0.102 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testSent ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestProgressTrigger.testProgress ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testCancel ... OK (0.105 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoQueue ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoTicket ... OK (0.005 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageCalledProcessError ... OK (0.009 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageFail ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageKilled ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageSimple ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageUnexpected ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testSaveLastRun ... OK (0.003 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCBRuns ... OK (0.021 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCanCancel ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_invalid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_valid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_d...

Revision history for this message
Vincent Ladeuil (vila) wrote :

======================================================================
FAIL: tests.test_data_store.TestDataStore.test_add_file
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/tarmac/branch.kK00Qz/bin/../tests/test_data_store.py", line 64, in test_add_file
    self.assertEqual([self.filename], files)
  File "/usr/lib/python2.7/unittest/case.py", line 515, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python2.7/unittest/case.py", line 744, in assertListEqual
    self.assertSequenceEqual(list1, list2, msg, seq_type=list)
  File "/usr/lib/python2.7/unittest/case.py", line 726, in assertSequenceEqual
    self.fail(msg)
  File "/usr/lib/python2.7/unittest/case.py", line 412, in fail
    raise self.failureException(msg)
AssertionError: Lists differ: ['test_file'] != []

First list contains 1 additional elements.
First extra element 0:
test_file

- ['test_file']
+ []

Seen yesterday ina different MP, probably swift being to slow to really delete the artifact in the container.

I'm starting to wonder if this test becomes inherently flaky because of that...
The artifact deletion succeeded, it takes time for the operation to complete, we have no idea how long it will take, when can the test decide it's taking too long ?

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'bin/called-by-tarmac.py'
2--- bin/called-by-tarmac.py 2014-11-13 16:25:58 +0000
3+++ bin/called-by-tarmac.py 2014-11-17 08:47:33 +0000
4@@ -19,7 +19,10 @@
5
6 def check_lxc_mounted():
7 out = subprocess.check_output(['mount'])
8- return ' /var/lib/lxc ' in out
9+ if ' /var/lib/lxc ' not in out:
10+ logging.error('No /var/lib/lxc in mount output: {}'.format(out))
11+ return False
12+ return True
13
14
15 def _run_and_log_on_failure(cmd):

Subscribers

People subscribed via source and target branches