race condition updating statefile

Note: Exact same locking semantics are present in latest ifupdown (0.7.40) from upstream Debian.

Suggested fix: Lock a lockfile, not the statefile.

Bug reported upstream (to Debian) against latest version (0.7.40). Will update with Debain bug number when accepted.

Upsteam bug report:

#704003: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=704003

Status changed to 'Confirmed' because the bug affects multiple users.

This is trivial to reproduce if you have enough interfaces:
1. Setup a multicore (I used 4 CPU) VM with 12 interfaces. I experienced similar failures with both bonded interfaces and 12 static configs.
2. Add something to boot to detect if the interfaces are correct and then reboot. I added to /etc/rc.local:
sleep 60

if [ `ifconfig | grep eth | wc -l` = 12 ]; then
 echo "is equal to 12"
 reboot
else
 echo "Things are broken!!!"
fi
3. Set a terminal to ping the vm every 60 seconds. (ping 192.168.122.95 -i 60)

Eventually the icmp_req will be in sequential order for more than 3 pings. This likely means the script has been tripped and one of the interfaces didn't come up.

I've also never got it to reproduce with a single core VM. It can take 500+ reboots to reproduce with 4 interfaces. But with 12 interfaces it only takes around 5 reboots.

It can happen much more frequently than that.

I have three machines in particular that do weekly reboots. They all have two network interfaces, but only one of them is used. Every week, one or more of them will lose on an interface that matters (loopback or the used network interface), and sometimes it will continue to fail through several subsequent manual reboots before it comes back up again.

I've been able produce test fixes for p/q/r/s. I've tested P/S and have been able to continually reboot a machine for 12h and have the correct number of interfaces come up where before it took about 20m to reproduce the issue.

Here are the builds:
http://people.canonical.com/~arges/lp1160490/

This bug was fixed in the package ifupdown - 0.7.44ubuntu2

---------------
ifupdown (0.7.44ubuntu2) saucy; urgency=low

  * On upgrade from 0.7.5ubuntu4, handle the fact that our unmodified conffile
    may have a mismatch in the dpkg database and manually shuffle the file
    around on upgrade. LP: #1217263.

  [ Chris Arges ]
  * Backport a fix from upstream mercurial
    (http://anonscm.debian.org/hg/collab-maint/ifupdown/rev/a93db3ecb8f0)
    for a race condition when updating the state file. LP: #1160490
 -- Steve Langasek <email address hidden> Sun, 01 Sep 2013 06:43:53 +0000

Hello!

Just hit this bug again on 12.04.3 server.

Sep 10 13:03:39 inetgw2 kernel: [ 6.415157] init: network-interface (vlan7) pre-start process (1404) terminated with status 1

ifup: failed to overwrite statefile /run/network/ifstate: No such file or directory

Could you, please, release package with fix for 12.04?

Thank you!

Hello Ray, or anyone else affected,

Accepted ifupdown into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/ifupdown/0.7~beta2ubuntu9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Ray, or anyone else affected,

Accepted ifupdown into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/ifupdown/0.7.2ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Just confirmed the fix on 12.04, will check 12.10 now.

Confirmed fixed on 12.10 as well.

Bug 1226067 seems to possibly be fallout of this change.
I'm only suggesting that based on the 'statefile' and the fact that that bug exists in saucy but not (yet) in raring.

Ok. I just confirmed that this change caused bug 1226067.

 * boot fresh raring cloud image (20130914)
 * ifquery works as expected
 * enable saucy-proposed, apt-get update, apt-get install ifupdown
 * reboot
 * ifquery broken
    $ ifquery --list --allow auto
    ifquery: failed to lock lockfile /run/network/.ifstate.lock: Bad file descriptor
   $ dpkg-query --show ifupdown
   ifupdown 0.7.5ubuntu2.1

Patches that fix this regression are in bug 1226067.

Hello Ray, or anyone else affected,

Accepted ifupdown into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/ifupdown/0.7.5ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Ray, or anyone else affected,

Accepted ifupdown into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/ifupdown/0.7.2ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Ray, or anyone else affected,

Accepted ifupdown into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/ifupdown/0.7~beta2ubuntu10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Verified fixed on P, Q, and R; and also that the regression (1226067) was also fixed.

This bug was fixed in the package ifupdown - 0.7.5ubuntu2.2

---------------
ifupdown (0.7.5ubuntu2.2) raring-proposed; urgency=low

  * Backport a fix from upstream mercurial
    (http://anonscm.debian.org/hg/collab-maint/ifupdown/rev/fb3055c2c4f0)
    for a regression introduced by a93db3ecb8f0. LP: #1226067

ifupdown (0.7.5ubuntu2.1) raring; urgency=low

  * Backport a fix from upstream mercurial
    (http://anonscm.debian.org/hg/collab-maint/ifupdown/rev/a93db3ecb8f0)
    for a race condition when updating the state file. LP: #1160490
 -- Chris J Arges <email address hidden> Mon, 16 Sep 2013 11:37:31 -0500

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

This bug was fixed in the package ifupdown - 0.7.2ubuntu4

---------------
ifupdown (0.7.2ubuntu4) quantal-proposed; urgency=low

  * Backport a fix from upstream mercurial
    (http://anonscm.debian.org/hg/collab-maint/ifupdown/rev/fb3055c2c4f0)
    for a regression introduced by a93db3ecb8f0. LP: #1226067

ifupdown (0.7.2ubuntu3) quantal; urgency=low

  * Backport a fix from upstream mercurial
    (http://anonscm.debian.org/hg/collab-maint/ifupdown/rev/a93db3ecb8f0)
    for a race condition when updating the state file. LP: #1160490
 -- Chris J Arges <email address hidden> Mon, 16 Sep 2013 11:37:22 -0500

This bug was fixed in the package ifupdown - 0.7~beta2ubuntu10

---------------
ifupdown (0.7~beta2ubuntu10) precise-proposed; urgency=low

  * Backport a fix from upstream mercurial
    (http://anonscm.debian.org/hg/collab-maint/ifupdown/rev/fb3055c2c4f0)
    for a regression introduced by a93db3ecb8f0. LP: #1226067

ifupdown (0.7~beta2ubuntu9) precise; urgency=low

  * Backport a fix from upstream mercurial
    (http://anonscm.debian.org/hg/collab-maint/ifupdown/rev/a93db3ecb8f0)
    for a race condition when updating the state file. LP: #1160490
 -- Chris J Arges <email address hidden> Mon, 16 Sep 2013 11:37:14 -0500