Duplicate public addresses

Bug #408917 reported by danleyden
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Eucalyptus
Fix Released
Critical
chris grzegorczyk

Bug Description

We have found occasions where ec2din reports 2 instances with the same public IP address, but only one is accessible.

During investigations we were lucky enough to be able to reproduce this and capture the log files. This led us to believe that there is a race condition in AddressManager.java.

If, during the very early stages of ec2run, another request is received by the cloud controller for DescribeAddresses, an Address whose instanceId is set to "available" will be unassigned by the "housekeeping" code in AddressManager.DesecribeAddresses(). This means that the address will then be released the next time that AddressManager.updateAddressingMode() is called.

This issue leads to ec2din reporting incorrectly and an "orphaned" instance on the vm-container. The database and iptables appear to be correct though.

We suggest a couple of options for resolution:

1. Remove the "housekeeping" code from DescribeAddresses.

2. Amend the "housekeeping" code to ignore Addresses that are isPending().

Revision history for this message
chris grzegorczyk (chris-grze) wrote :

Indeed, this is a regression. Solution #2 which you suggest should be done.

Changed in eucalyptus:
assignee: nobody → chris grzegorczyk (chris-grze)
importance: Undecided → Critical
status: New → Confirmed
Revision history for this message
danleyden (dan-52degreesnorth) wrote :

Thanks for the rapid response on this - its much appreciated.

Will amend the code as per #2.

Out of interest, can you give a heads-up on the changes being made for 1.6? There are a number of bugs that I've seen lately with the 1.5.2 codebase that have been reported and I'm wondering if it is worth looking to patch them locally. Is 1.6 a major rewrite, or are you expecting it to be relatively easy for us to update the code? Any idea roughly when the 1.6 code is likely to start appearing up here so we can start to see what the improvements are going to look like?

Revision history for this message
chris grzegorczyk (chris-grze) wrote : Re: [Bug 408917] Re: Duplicate public addresses

On Wed, Aug 5, 2009 at 1:57 AM, danleyden<dan@52degreesnorth.com> wrote:
> Thanks for the rapid response on this - its much appreciated.

Happy to help. Once v1.6 rolls out I'll have the headspace to
backport these fixes for 1.5. They wont fall through the cracks...

> Out of interest, can you give a heads-up on the changes being made for
> 1.6? ... Is 1.6 a major rewrite, or are you
> expecting it to be relatively easy for us to update the code?

Its still too early to know how pervasive the changes will be. So, I
cant say anything actionable yet, and I'd rather not give you a false
impression (be it positive or negative).

> Any idea
> roughly when the 1.6 code is likely to start appearing up here so we can
> start to see what the improvements are going to look like?

Also hard to predict at the moment, but as soon as it is sane to do so.

> --
> Duplicate public addresses
> https://bugs.launchpad.net/bugs/408917
> You received this bug notification because you are a bug assignee.
>
> Status in Eucalyptus: Confirmed
>
> Bug description:
> We have found occasions where ec2din reports 2 instances with the same public IP address, but only one is accessible.
>
> During investigations we were lucky enough to be able to reproduce this and capture the log files. This led us to believe that there is a race condition in AddressManager.java.
>
> If, during the very early stages of ec2run, another request is received by the cloud controller for DescribeAddresses, an Address whose instanceId is set to "available" will be unassigned by the "housekeeping" code in AddressManager.DesecribeAddresses(). This means that the address will then be released the next time that AddressManager.updateAddressingMode() is called.
>
> This issue leads to ec2din reporting incorrectly and an "orphaned" instance on the vm-container. The database and iptables appear to be correct though.
>
> We suggest a couple of options for resolution:
>
> 1.       Remove the "housekeeping" code from DescribeAddresses.
>
> 2.       Amend the "housekeeping" code to ignore Addresses that are isPending().
>

--
Chris Grzegorczyk
Co-Founder and Engineer
Eucalyptus Systems, Inc.

130 Castilian St. | Goleta, CA | 93117
Office: 805-968-1400 x e^1 | Cell: 805-807-8237
Email: <email address hidden>
www.eucalyptus.com
________________________________________

Changed in eucalyptus:
status: Confirmed → Fix Committed
Revision history for this message
danleyden (dan-52degreesnorth) wrote :

Is this fix in 1.5?

What revision reflects the fix?

Revision history for this message
chris grzegorczyk (chris-grze) wrote :

Woops. Forgot the banner.

------------------------------------------------------------
revno: 433
committer: decker <decker@personal-army>
branch nick: 1.5
timestamp: Tue 2009-09-15 17:03:23 -0700
message:
  backport fixes and close dangling bugs for lp:399420, lp:403444,
lp:408917, lp:357440, lp:385435, lp:411543
------------------------------------------------------------

On Wed, Sep 16, 2009 at 2:11 AM, danleyden <dan@52degreesnorth.com> wrote:
> Is this fix in 1.5?
>
> What revision reflects the fix?
>
> --
> Duplicate public addresses
> https://bugs.launchpad.net/bugs/408917
> You received this bug notification because you are a bug assignee.
>
> Status in Eucalyptus: Fix Committed
>
> Bug description:
> We have found occasions where ec2din reports 2 instances with the same public IP address, but only one is accessible.
>
> During investigations we were lucky enough to be able to reproduce this and capture the log files. This led us to believe that there is a race condition in AddressManager.java.
>
> If, during the very early stages of ec2run, another request is received by the cloud controller for DescribeAddresses, an Address whose instanceId is set to "available" will be unassigned by the "housekeeping" code in AddressManager.DesecribeAddresses(). This means that the address will then be released the next time that AddressManager.updateAddressingMode() is called.
>
> This issue leads to ec2din reporting incorrectly and an "orphaned" instance on the vm-container. The database and iptables appear to be correct though.
>
> We suggest a couple of options for resolution:
>
> 1.       Remove the "housekeeping" code from DescribeAddresses.
>
> 2.       Amend the "housekeeping" code to ignore Addresses that are isPending().
>

--
Chris Grzegorczyk
Co-Founder and Engineer
Eucalyptus Systems, Inc.

130 Castilian St. | Goleta, CA | 93117
Office: 805-968-1400 x e^1 | Cell: 805-807-8237
Email: <email address hidden>
www.eucalyptus.com
________________________________________

Changed in eucalyptus:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.