mysql ocf script broken in 1:1.0.3-2ubuntu1 (lucid)

Bug #893352 reported by Devin Vance
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cluster-agents (Ubuntu)
Fix Released
Medium
Adam Gandelman
Hardy
Invalid
Undecided
Unassigned
Lucid
Fix Released
Medium
Unassigned
Natty
Fix Released
Undecided
Unassigned
Precise
Fix Released
Medium
Adam Gandelman

Bug Description

Due to ubuntu putting /var/run on a tempory fs the /var/run/mysqld directory is removed upon a reboot. Mysql will fail to start via the crm after next boot as it cannot access or find the the needed /var/run/mysqld directory.
This directory is created in the OCF script in the mysql_start() function , but for some reason not until after the script checks for the directory and write permissions, logs an error, and exits. (Line 345)

I have fixed it by merely moving the if statement which test -w the pid_dir to after the if statements which create the pid_dir (line 350)

SRU Justification:
[Impact]
/var/run/mysqld directory is typically created by pre-start in the mysql upstart job prior to starting the service every boot, as /var/run is tempfs and its contents ephemeral. Users of the Linux HA stack typically hand over control of specific services to the cluster managed by OCF scripts. The OCF script to manage mysql should take the same steps the required directories are present. A bug exists in the version of the mysql OCF script shipped in 1:1.0.3-2ubuntu1 which prevents the required directories from being created if they do not exist. If the clustered service is configured to use /var/run/mysqld (the ubuntu default for certain files), the cluster will fail to start mysql after a reboot. As most users of the cluster agent anticipate a *highly* available service, graceful system reboots should not result in an unavailable service.

[Development Fix]
As Florian Hass pointed out, this has been fixed upstream over a year ago and is not present in the current development release (precise, package renamed resource agents 1:3.9.2-5ubuntu1)

[Stable Fix]
A trivial fix at https://github.com/ClusterLabs/resource-agents/commit/a1801207e0602d9b975f336569b562c195a6eb8a solves the issue and is easily backported.

[Test Case]
See http://paste.ubuntu.com/746339/ for a minimal test case which involves configuration of a 1-node heartbeat/pacemaker cluster.

[Regression Potential]
Minimal chance of regression. Change simply ensures mysql directories exist before they are checked for write permissions. The change only affects the mysql OCF agent, all others are untouched.

Related branches

Revision history for this message
Florian Haas (fghaas) wrote :

I fixed this upstream more than a year ago:

https://github.com/ClusterLabs/resource-agents/commit/a1801207e0602d9b975f336569b562c195a6eb8a

Ubuntu maintainers are welcome to apply the patch and reroll the package.

Changed in cluster-agents (Ubuntu):
status: New → Fix Committed
Changed in cluster-agents (Ubuntu):
assignee: nobody → Adam Gandelman (gandelman-a)
status: Fix Committed → Confirmed
status: Confirmed → In Progress
Changed in cluster-agents (Ubuntu Hardy):
status: New → Incomplete
status: Incomplete → Invalid
Changed in cluster-agents (Ubuntu Lucid):
assignee: nobody → Adam Gandelman (gandelman-a)
status: New → In Progress
description: updated
Changed in cluster-agents (Ubuntu Lucid):
assignee: Adam Gandelman (gandelman-a) → nobody
Revision history for this message
James Page (james-page) wrote :

Uploaded to -proposed

Changed in cluster-agents (Ubuntu Lucid):
importance: Undecided → Medium
Changed in cluster-agents (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Martin Pitt (pitti) wrote :

Please fix this in precise first before SRUing. Presumably it is already, so please double-check and set the precise task to "fix released". Thank you!

Revision history for this message
Andres Rodriguez (andreserl) wrote :

Hi Martin,

I have checked and this is fixed since Natty.

Cheers.

Changed in cluster-agents (Ubuntu Natty):
status: New → Fix Released
Changed in cluster-agents (Ubuntu Precise):
status: In Progress → Fix Released
Revision history for this message
Clint Byrum (clint-fewbar) wrote : Please test proposed package

Hello Devin, or anyone else affected,

Accepted cluster-agents into lucid-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

Changed in cluster-agents (Ubuntu Lucid):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote : Verification still needed

The fix for the this bug has been awaiting testing feedback in the -proposed repository for lucid for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now the package will be removed from the -proposed repository.

Revision history for this message
Devin Vance (devin-p) wrote :

Proposed fix works. Thank you!

tags: added: verification-done
removed: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cluster-agents - 1:1.0.3-2ubuntu1.1

---------------
cluster-agents (1:1.0.3-2ubuntu1.1) lucid-proposed; urgency=low

  * debian/patches/mysql_move_writable_test.patch: Cherry-pick upstream
    commit (95a6eb8a). In mysql OCF, properly create $pid_dir before testing
    permissions on it. (LP: #893352)
 -- Adam Gandelman <email address hidden> Tue, 22 Nov 2011 11:20:06 -0800

Changed in cluster-agents (Ubuntu Lucid):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.