units newly added to juju 1.25 service are failing

Bug #1746119 reported by Paul Collins
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
juju-core
Won't Fix
Undecided
Unassigned

Bug Description

I'm working on adding capacity to a production OpenStack cloud, which is currently running Juju 1.25.13, having been upgraded from 1.24.7 a week or so ago. I believe these may be the first newly added units since the Juju upgrade.

Here's the current tabular status for the service (Canonical-only link: https://pastebin.canonical.com/208704/; the new units are 6 and 7).

The service had 56 units before I started and now has 58, both of which are behaving as follows.

Newly added units are repeatedly running hooks and logging the following message at the end:

ERROR juju.worker.uniter.filter filter.go:137 tomb: dying

And here's a snippet of juju status-history:

TIME TYPE STATUS MESSAGE
29 Jan 2018 22:44:31Z workload unknown Waiting for agent initialization to finish
29 Jan 2018 22:44:31Z agent allocating
29 Jan 2018 22:44:49Z workload maintenance installing charm software
29 Jan 2018 22:44:49Z agent executing running install hook
29 Jan 2018 22:45:47Z agent executing running leader-settings-changed hook
29 Jan 2018 22:45:48Z agent executing running config-changed hook
29 Jan 2018 22:45:55Z agent executing running start hook
29 Jan 2018 22:45:57Z workload unknown
29 Jan 2018 22:45:57Z workload unknown
29 Jan 2018 22:45:58Z agent executing running update-status hook
29 Jan 2018 22:46:01Z agent failed update relations [1 29 30 31 32 33 96 97 98 100 114 127 217 248 259]
29 Jan 2018 22:46:24Z agent executing running leader-settings-changed hook
29 Jan 2018 22:46:24Z agent executing running config-changed hook
29 Jan 2018 22:46:33Z agent executing running neutron-plugin-relation-joined hook
29 Jan 2018 22:46:36Z agent failed update relations [1 29 30 31 32 33 96 97 98 100 114 127 217 248 259]
29 Jan 2018 22:47:10Z agent executing running leader-settings-changed hook
29 Jan 2018 22:47:10Z agent executing running config-changed hook
29 Jan 2018 22:47:17Z agent executing running amqp-relation-joined hook
29 Jan 2018 22:47:20Z agent executing running image-service-relation-joined hook
29 Jan 2018 22:47:21Z agent failed update relations [1 29 30 31 32 33 96 97 98 100 114 127 217 248 259]
[ and on and on ]

Seemingly similar bugs:

https://bugs.launchpad.net/juju-core/1.25/+bug/1613992 - Fix Released
https://bugs.launchpad.net/juju-core/+bug/1661681 - Expired

This environment is deployed on Ubuntu 14.04 LTS.

Revision history for this message
Paul Collins (pjdc) wrote :

The two units that were evincing this problem somehow recovered and completed deploying successfully. I have two units left to deploy, which I will start now and report back.

Revision history for this message
Paul Collins (pjdc) wrote :

The other two units behaved similarly, logging "failed update relations [1 29 30 31 32 33 96 97 98 100 114 127 217 248 259]" but eventually finished successfully in "idle". However, we add units to this service on a regular basis and so it would be useful to prevent this from happening at all.

Revision history for this message
Anastasia (anastasia-macmood) wrote :

@Paul Collins,

Thank you for the report \o/

We use 'juju' project on launchpad to track Juju 2.x issues and suggestions.

This issue seems to be affecting only Juju 1.25.x installations. I'll re-target to 'juju-core' project. However, we are only supporting absolute production 'Critical' for juju 1.x. This means we are only fixing failures that neither juju nor the operator can recover from, say by restarting agents. Since in your report, Juju seems to be recovering after a period of time, this would not be considered critical. Hence, I am marking this as Won't Fix.

Also, this issue does not occur on Juju 2.x and we are working hard to provide an upgrade path.

Changed in juju-core:
status: New → Won't Fix
no longer affects: juju
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.