OpenStack Compute (nova)

Quantum Network Manager - not clearing ips

Bug #988034 reported by Endre Karlson
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Somik Behera
OpenStack Compute (nova) 2012.2 "folsom"
Essex
Fix Released
Undecided
Unassigned
OpenStack Compute (nova) 2012.1.1

Bug Description

2012-04-24 20:09:19 ERROR nova.rpc.amqp [req-0c7d42c0-ce49-44c6-98fb-cad674b902c2 04458e4341404f609270b48ad40c1087 49e8978cba5b4e00917fd134eccd6e21] Exception during message handling
2012-04-24 20:09:19 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-04-24 20:09:19 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 252, in _process_data
2012-04-24 20:09:19 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-04-24 20:09:19 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line 556, in deallocate_for_instance
2012-04-24 20:09:19 TRACE nova.rpc.amqp vif, project_id)
2012-04-24 20:09:19 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line 628, in update_dhcp
2012-04-24 20:09:19 TRACE nova.rpc.amqp self.driver.kill_dhcp(dev)
2012-04-24 20:09:19 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 655, in kill_dhcp
2012-04-24 20:09:19 TRACE nova.rpc.amqp _execute('kill', '-9', pid, run_as_root=True)
2012-04-24 20:09:19 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 812, in _execute
2012-04-24 20:09:19 TRACE nova.rpc.amqp return utils.execute(*cmd, **kwargs)
2012-04-24 20:09:19 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/utils.py", line 243, in execute
2012-04-24 20:09:19 TRACE nova.rpc.amqp cmd=' '.join(cmd))
2012-04-24 20:09:19 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2012-04-24 20:09:19 TRACE nova.rpc.amqp Command: sudo nova-rootwrap kill -9 26037
2012-04-24 20:09:19 TRACE nova.rpc.amqp Exit code: 99
2012-04-24 20:09:19 TRACE nova.rpc.amqp Stdout: 'Unauthorized command: kill -9 26037\n'
2012-04-24 20:09:19 TRACE nova.rpc.amqp Stderr: ''

Related branches

lp:~openstack-ubuntu-testing/nova/precise-essex-proposed
Chuck Short: Pending requested
Diff: 56 lines (+14/-4)
3 files modified
debian/changelog (+8/-0)
debian/control (+6/-3)
debian/nova-console.install (+0/-1)
Revision history for this message
Maru Newby (maru) wrote :

Am I missing something, or is this a nova bug? If so, you'll want to file it against nova rather than quantum.

Revision history for this message
Maru Newby (maru) wrote :

There also doesn't seem to be much in the way of information to aid in reproducing the problem. The sequence of events that led up to the bug would be helpful for whoever ends up fixing it.

Revision history for this message
dan wendlandt (danwent) wrote : Re: [Bug 988034] Re: Quantum Network Manager - not clearing ips

I'm not sure how the traceback is related, but this may be similar to the
issue I ran into while giving the demo at the openstack conference :)

Brad, I think you were look at this, right?

Dan

On Tue, Apr 24, 2012 at 2:24 PM, Maru Newby <email address hidden> wrote:

> There also doesn't seem to be much in the way of information to aid in
> reproducing the problem. The sequence of events that led up to the bug
> would be helpful for whoever ends up fixing it.
>
> --
> You received this bug notification because you are a member of Netstack
> Core Developers, which is subscribed to quantum.
> https://bugs.launchpad.net/bugs/988034
>
> Title:
> Quantum Network Manager - not clearing ips
>
> Status in OpenStack Quantum (virtual network service):
> New
>
> Bug description:
> 2012-04-24 20:09:19 ERROR nova.rpc.amqp
> [req-0c7d42c0-ce49-44c6-98fb-cad674b902c2 04458e4341404f609270b48ad40c1087
> 49e8978cba5b4e00917fd134eccd6e21] Exception during message handling
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Traceback (most recent call last):
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 252, in
> _process_data
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp rval =
> node_func(context=ctxt, **node_args)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 556, in deallocate_for_instance
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp vif, project_id)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 628, in update_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp self.driver.kill_dhcp(dev)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 655, in
> kill_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp _execute('kill', '-9', pid,
> run_as_root=True)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 812, in
> _execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp return utils.execute(*cmd,
> **kwargs)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/utils.py", line 243, in execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp cmd=' '.join(cmd))
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected
> error while running command.
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Command: sudo nova-rootwrap kill
> -9 26037
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Exit code: 99
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stdout: 'Unauthorized command:
> kill -9 26037\n'
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stderr: ''
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/quantum/+bug/988034/+subscriptions
>

--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~

Revision history for this message
Brad Hall (bgh) wrote :

Review coming shortly..

Changed in quantum:
assignee: nobody → Brad Hall (bgh)
status: New → In Progress
importance: Undecided → High
Revision history for this message
Endre Karlson (endre-karlson) wrote :

It's taken from nova.network. It's a thing that occurs when using OVS.

Revision history for this message
Endre Karlson (endre-karlson) wrote :

Any news on this? It renders quantum kinda useless.

Revision history for this message
dan wendlandt (danwent) wrote :

Hi Endre, can you try out this patch and confirm it solves your issue: https://review.openstack.org/#/c/6949/1 ?

Revision history for this message
Endre Karlson (endre-karlson) wrote :
Download full text (13.3 KiB)

Doesn't seem to free up still when I destroy the instances.
So I go in and try to manually cleanup fixed ip's for the 2 networks:

+---------------------+---------------------+------------+---------+------+--------------+------------+-------------+-----------+--------+----------+----------------------+------+
| created_at | updated_at | deleted_at | deleted | id | address | network_id | instance_id | allocated | leased | reserved | virtual_interface_id | host |
 +---------------------+---------------------+------------+---------+------+--------------+------------+-------------+-----------+--------+----------+----------------------+------+
| 2012-04-30 12:40:50 | 2012-05-02 11:39:12 | NULL | 0 | 1297 | 172.16.65.16 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:39:12 | NULL | 0 | 1298 | 172.16.65.17 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:39:12 | NULL | 0 | 1299 | 172.16.65.18 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:08:38 | NULL | 0 | 1300 | 172.16.65.19 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:07:56 | NULL | 0 | 1301 | 172.16.65.20 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:11:05 | NULL | 0 | 1302 | 172.16.65.21 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:11:05 | NULL | 0 | 1303 | 172.16.65.22 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:24:49 | NULL | 0 | 1304 | 172.16.65.23 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:24:49 | NULL | 0 | 1305 | 172.16.65.24 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:24:48 | NULL | 0 | 1306 | 172.16.65.25 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:24:49 | NULL | 0 | 1307 | 172.16.65.26 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:24:49 | NULL | 0 | 1308 | 172.16.65.27 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:32:45 | NULL | 0 | 1309 | 172.16.65.28 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-04-30 12:40:50 | 2012-05-02 11:32:44 | NULL | 0 | 1310 | 172.16.65.29 | 7 | NULL | 0 | 0 | 0 | NULL | NULL |
| 2012-0...

Revision history for this message
Somik Behera (somikbehera) wrote :

Can you try this patch - https://review.openstack.org/#/c/6949/

and let us know if that resolves your issue? Also, in the above example, did you try the patch above and it still doesn't release your ips?

Revision history for this message
Endre Karlson (endre-karlson) wrote :

I tried the first patch yes, no positive feedback there.

2012/5/3 Somik Behera <email address hidden>

> Can you try this patch - https://review.openstack.org/#/c/6949/
>
> and let us know if that resolves your issue? Also, in the above example,
> did you try the patch above and it still doesn't release your ips?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/988034
>
> Title:
> Quantum Network Manager - not clearing ips
>
> Status in OpenStack Quantum (virtual network service):
> In Progress
>
> Bug description:
> 2012-04-24 20:09:19 ERROR nova.rpc.amqp
> [req-0c7d42c0-ce49-44c6-98fb-cad674b902c2 04458e4341404f609270b48ad40c1087
> 49e8978cba5b4e00917fd134eccd6e21] Exception during message handling
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Traceback (most recent call last):
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 252, in
> _process_data
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp rval =
> node_func(context=ctxt, **node_args)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 556, in deallocate_for_instance
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp vif, project_id)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 628, in update_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp self.driver.kill_dhcp(dev)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 655, in
> kill_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp _execute('kill', '-9', pid,
> run_as_root=True)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 812, in
> _execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp return utils.execute(*cmd,
> **kwargs)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/utils.py", line 243, in execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp cmd=' '.join(cmd))
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected
> error while running command.
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Command: sudo nova-rootwrap kill
> -9 26037
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Exit code: 99
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stdout: 'Unauthorized command:
> kill -9 26037\n'
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stderr: ''
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/quantum/+bug/988034/+subscriptions
>

Revision history for this message
Endre Karlson (endre-karlson) wrote :

Isn't that the same as before?
Den 1. mai 2012 03:50 skrev "dan wendlandt" <email address hidden>
følgende:

> Hi Endre, can you try out this patch and confirm it solves your issue:
> https://review.openstack.org/#/c/6949/1 ?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/988034
>
> Title:
> Quantum Network Manager - not clearing ips
>
> Status in OpenStack Quantum (virtual network service):
> In Progress
>
> Bug description:
> 2012-04-24 20:09:19 ERROR nova.rpc.amqp
> [req-0c7d42c0-ce49-44c6-98fb-cad674b902c2 04458e4341404f609270b48ad40c1087
> 49e8978cba5b4e00917fd134eccd6e21] Exception during message handling
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Traceback (most recent call last):
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 252, in
> _process_data
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp rval =
> node_func(context=ctxt, **node_args)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 556, in deallocate_for_instance
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp vif, project_id)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 628, in update_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp self.driver.kill_dhcp(dev)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 655, in
> kill_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp _execute('kill', '-9', pid,
> run_as_root=True)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 812, in
> _execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp return utils.execute(*cmd,
> **kwargs)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/utils.py", line 243, in execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp cmd=' '.join(cmd))
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected
> error while running command.
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Command: sudo nova-rootwrap kill
> -9 26037
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Exit code: 99
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stdout: 'Unauthorized command:
> kill -9 26037\n'
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stderr: ''
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/quantum/+bug/988034/+subscriptions
>

Revision history for this message
Somik Behera (somikbehera) wrote :

The new patch has an additional fix to allow the default manager to clean up the ips in the DB as well, so its got an additional fix that was in my environment.

Revision history for this message
Endre Karlson (endre-karlson) wrote :

Yeah, after replacing my manager.py file it now seems to be ok.

Revision history for this message
Somik Behera (somikbehera) wrote :

Great! Glad to hear that. Did you just replace the manager.py from the patch link above or did you pull a certain version from the Quantum trunk?

Revision history for this message
Endre Karlson (endre-karlson) wrote :

I just replaced it. By the way can you take a look at my other bug reported
where it doesn't setup .conf files for dnsmasq?

2012/5/4 Somik Behera <email address hidden>

> Great! Glad to hear that. Did you just replace the manager.py from the
> patch link above or did you pull a certain version from the Quantum
> trunk?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/988034
>
> Title:
> Quantum Network Manager - not clearing ips
>
> Status in OpenStack Quantum (virtual network service):
> In Progress
>
> Bug description:
> 2012-04-24 20:09:19 ERROR nova.rpc.amqp
> [req-0c7d42c0-ce49-44c6-98fb-cad674b902c2 04458e4341404f609270b48ad40c1087
> 49e8978cba5b4e00917fd134eccd6e21] Exception during message handling
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Traceback (most recent call last):
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 252, in
> _process_data
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp rval =
> node_func(context=ctxt, **node_args)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 556, in deallocate_for_instance
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp vif, project_id)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/quantum/manager.py", line
> 628, in update_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp self.driver.kill_dhcp(dev)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 655, in
> kill_dhcp
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp _execute('kill', '-9', pid,
> run_as_root=True)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 812, in
> _execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp return utils.execute(*cmd,
> **kwargs)
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp File
> "/usr/lib/python2.6/site-packages/nova/utils.py", line 243, in execute
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp cmd=' '.join(cmd))
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp ProcessExecutionError: Unexpected
> error while running command.
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Command: sudo nova-rootwrap kill
> -9 26037
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Exit code: 99
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stdout: 'Unauthorized command:
> kill -9 26037\n'
> 2012-04-24 20:09:19 TRACE nova.rpc.amqp Stderr: ''
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/quantum/+bug/988034/+subscriptions
>

Revision history for this message
Endre Karlson (endre-karlson) wrote :

Is there any news on this? Quantum is rendered useless at the moment.

Revision history for this message
Somik Behera (somikbehera) wrote :

Fix has been committed to Nova trunk - https://review.openstack.org/#/c/6949/

Changed in quantum:
assignee: Brad Hall (bgh) → Somik Behera (somikbehera)
status: In Progress → Fix Committed
Thierry Carrez (ttx)
affects: quantum → nova
Revision history for this message
Endre Karlson (endre-karlson) wrote :

I replaced my nova.network.quantum > manager and nova_ipam_lib files with the ones from upstream - restarted components and it works gloriously!

Devin Carlen (devcamcar)
Changed in nova:
milestone: none → folsom-1
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/essex)

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/7970

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/essex)

Reviewed: https://review.openstack.org/7970
Committed: http://github.com/openstack/nova/commit/3426030d3524e4e38c5843be0b9515805b67b04f
Submitter: Jenkins
Branch: stable/essex

commit 3426030d3524e4e38c5843be0b9515805b67b04f
Author: Somik Behera <email address hidden>
Date: Mon Apr 30 17:43:14 2012 -0700

    Fix bug 988034 - Quantum Network Manager - not clearing ips

    Added a line in nova_ipam_lib.QuantumNovaIPAMLib init
    method to ensure we clean up IPs and explanation of whats
    going on.

    Also, nuked release_fixed_ip as that is not
    needed in QuantumManager anymore.

    Change-Id: I3971b4d4c464ce68ba9370c9f5d8ca7840e0a72a

tags: added: in-stable-essex
Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-1 → 2012.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.