[Precise] Potential for data corruption
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| pacemaker (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
| Precise |
Fix Released
|
Medium
|
Unassigned | ||
Bug Description
[Impact]
* Pacemaker designated controller can make wrong decisions based on uncleared node status on a rare specific situation. This situation can make the same resource starts on two nodes at the same time, resulting in data corruption.
[Test Case]
* The bug trigger is very hard hard to achieve:
1) If stonith was successful on fencing a node (any node was fenced).
2) If the target and origin are the same (node killed itself).
3) If we do not have a dc or the fenced node is our dc (our dc killed itself).
4) If the executor is not this node (at least 3 nodes).
5) If this node is elected new DC anytime in the future.
7) If a policy engine was not yet scheduled.
8) If takeover runs before policy engine.
* The bug couldn't be reproduced so far: the patch was made based on a community report (https://<email address hidden>
[Regression Potential]
* On logic before commit 82aa2d8d17 the node responsible for fencing (executioner) the dc was responsible also for updating cib. If this update failed (due to a executioner fail, for ex) the dc would be fenced a second time because the cluster would not know about fencing result. On upstream commit 82aa2d8d17, a logic trying to avoid this second dc fencing was introduced. This logic by itself is buggy.
* To minimize any kind of regression, instead of going forward on pacemaker versions, it was decided to go backwards removing only this piece of code.
* It is much more acceptable for SRU to restore old behavior, known to be safe even if it implies killing dc twice, than to backport several pieces of code to implement a logic that was not there on the stable version release.
[Other Info / Original Description]
Under certain conditions there is faulty logic in function tengine_
Conditions:
1. fenced node must have been the previous DC and been sufficiently functional to request its own fencing
2. fencing notification must arrive after the new DC has been elected but before it invokes the policy engine
Pacemaker versions affected:
1.1.6 - 1.1.9
Stable Ubuntu releases affected:
Ubuntu 12.04 LTS
Ubuntu 12.10 (EOL?)
Fix:
https:/
References:
https://<email address hidden>
http://
Related branches
| Changed in pacemaker (Ubuntu): | |
| status: | New → In Progress |
| Changed in pacemaker (Ubuntu): | |
| assignee: | nobody → Rafael David Tinoco (inaddy) |
| Rafael David Tinoco (rafaeldtinoco) wrote | #1 |
| Changed in pacemaker (Ubuntu Precise): | |
| assignee: | nobody → Rafael David Tinoco (inaddy) |
| Changed in pacemaker (Ubuntu): | |
| assignee: | Rafael David Tinoco (inaddy) → nobody |
| status: | In Progress → Fix Released |
| Changed in pacemaker (Ubuntu Precise): | |
| status: | New → In Progress |
| importance: | Undecided → Medium |
| description: | updated |
| Chris J Arges (arges) wrote | #2 |
| Brian Murray (brian-murray) wrote Please test proposed package | #3 |
| Changed in pacemaker (Ubuntu Precise): | |
| status: | In Progress → Fix Committed |
| tags: | added: verification-needed |
| Rafael David Tinoco (rafaeldtinoco) wrote | #4 |
| tags: |
added: verification-done removed: verification-needed |
| Launchpad Janitor (janitor) wrote | #5 |
| Changed in pacemaker (Ubuntu Precise): | |
| status: | Fix Committed → Fix Released |
| Colin Watson (cjwatson) wrote Update Released | #6 |
| Changed in pacemaker (Ubuntu Precise): | |
| assignee: | Rafael David Tinoco (inaddy) → nobody |
Here is the patch fixing corosync misbehavior described above.
Description: Remove buggy logic to prevent secondary dc fencing
On logic before commit 82aa2d8d17 the node responsible for fencing
(executioner) the dc was responsible also for updating cib. If this
update failed (due to a executioner fail, for ex) the dc would be
fenced a second time because the cluster would not know about fencing
result.
On upstream commit 82aa2d8d17, a logic trying to avoid this second
dc fencing was introduced. If this node was not the dc fence executioner
it would keep its name. With its name, in the case executioner node
died and this node became the new dc it would be able to update cib
telling the result of last dc fencing. Problem is that this list
is never cleaned and there might be cases wrong cib update is given
(when a dc takeover has to run) resulting in a bad, bad thing: same
resource running on different nodes.
It is much more acceptable for SRU to restore old behavior, known to
be safe even if it implies killing dc twice, than to backport several
pieces of code to implement a logic that was not there on the stable
version release.