r196 upgraded charmhelpers and broke metadata sync for envs with leader election

Bug #1494443 reported by Robert C Jennings
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ubuntu-repository-cache (Juju Charms Collection)
Fix Released
Critical
Dan Watkins

Bug Description

The upgade of charmhelpers in r196[1] broke leadership election and metadata sync for juju environments where leader election is present.
This is a test escape as juju leader election could choose the lowest numbered unit and mask this problem.

charmhelpers.haproxy.cluster has the is_leader which was lowest number (prior to r196) and that is still assumed when determing the leader unit id in ubuntu_repository_cache/mirror specifically look for "leader_id". util.get_failover_host is also probably broken

Presently there could be timing issues for unit start that affect initial unit leadership so that the lowest numbered unit wouldn't be the leader but certainly when a unit is removed and added things will break.

With the introduction of leader election there are new hooks for leadership changes which we would use to determine who runs cron / metadata update. The trick will be running in environments with and without leadership election and testing both. <moderately difficult> OTOH, if leadership election is in the stable release of Juju now, we could just require that.

Also, this sort of transition seems like the sort of thing that charmhelpers was designed to smooth over but doesn't. core.hookenv returns NotImplementedError when leader election and settings aren't available, so it does little for us.

Suggested actions:
1) Triage by changing back to lowest numbered unit for our determination of is_leader
 - Replace our current call to charmhelpers.haproxy.cluser.is_leader with a function that does what we had been doing.
2) Fix root cause by updating the charm to user juju leadership throughout with hooks to handle leadership changes

[1] http://bazaar.launchpad.net/~charmers/charms/trusty/ubuntu-repository-cache/trunk/revision/196

Related branches

Robert C Jennings (rcj)
description: updated
description: updated
Revision history for this message
Robert C Jennings (rcj) wrote :

As a note for later when we enable juju leadership election, mirror._leader_update_metadata() and mirror._nonleader_update_metadata() will need updates to how they determine the leader unit id as it is currently using the lowest active host.

Changed in ubuntu-repository-cache (Juju Charms Collection):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.