linux_net.ensure_metadata_ip() fails on precise

Bug #929127 reported by Adam Gandelman
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Adam Gandelman
nova (Ubuntu)
Fix Released
High
Unassigned

Bug Description

Now that nova-rootwrap is returning real return codes, ensure_metadata_ip() is failing on return code 2 when creating the metadata ip, because it already exists:

2012-02-08 15:30:13,309 DEBUG nova.utils [-] Running cmd (subprocess): sudo nova-rootwrap ip addr add 169.254.169.254/32 scope link dev lo from (pid=8942) execute /usr/lib/python2.7/dist-packages/nova/utils.py:208
2012-02-08 15:30:13,339 DEBUG nova.utils [-] Result was 2 from (pid=8942) execute /usr/lib/python2.7/dist-packages/nova/utils.py:224
2012-02-08 15:30:13,340 CRITICAL nova [-] Unexpected error while running command.
Command: sudo nova-rootwrap ip addr add 169.254.169.254/32 scope link dev lo
Exit code: 2
Stdout: ''
Stderr: 'RTNETLINK answers: File exists\n'
(nova): TRACE: Traceback (most recent call last):
(nova): TRACE: File "/usr/bin/nova-network", line 49, in <module>
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 414, in wait
(nova): TRACE: _launcher.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 134, in wait
(nova): TRACE: service.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 166, in wait
(nova): TRACE: return self._exit_event.wait()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
(nova): TRACE: return hubs.get_hub().switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 177, in switch
(nova): TRACE: return self.greenlet.switch()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
(nova): TRACE: result = function(*args, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 104, in run_server
(nova): TRACE: server.start()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/service.py", line 164, in start
(nova): TRACE: self.manager.init_host()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/manager.py", line 1558, in init_host
(nova): TRACE: self.l3driver.initialize()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/l3.py", line 81, in initialize
(nova): TRACE: linux_net.ensure_metadata_ip()
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 479, in ensure_metadata_ip
(nova): TRACE: run_as_root=True, check_exit_code=[0, 254])
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 839, in _execute
(nova): TRACE: return utils.execute(*cmd, **kwargs)
(nova): TRACE: File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 232, in execute
(nova): TRACE: cmd=' '.join(cmd))
(nova): TRACE: ProcessExecutionError: Unexpected error while running command.
(nova): TRACE: Command: sudo nova-rootwrap ip addr add 169.254.169.254/32 scope link dev lo
(nova): TRACE: Exit code: 2
(nova): TRACE: Stdout: ''
(nova): TRACE: Stderr: 'RTNETLINK answers: File exists\n'
(nova): TRACE:

There seems to have been a change upstream wrt return codes from iproute:

root@ip-10-252-15-139:~# dpkg -l | grep iproute
ii iproute 20110629-1 networking and traffic control tools
root@ip-10-252-15-139:~# sudo ip addr add 169.254.169.254/32 scope link dev lo ; echo $?
RTNETLINK answers: File exists
254

Processing triggers for man-db ...
Setting up iproute (20111117-1ubuntu1) ...
root@ip-10-252-15-139:~# sudo ip addr add 169.254.169.254/32 scope link dev lo ; echo $?
RTNETLINK answers: File exists
2

Update, the correct upstream commit:
http://git.kernel.org/?p=linux/kernel/git/shemminger/iproute2.git;a=commit;h=7397944de6c11519a5951fc1bcff20225e71c4bd

Seems like a crazy change, but in any case, precise is now on this version and nova-network should check for both 2 and 254.

description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/3934

Changed in nova:
assignee: nobody → Adam Gandelman (gandelman-a)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/3934
Committed: http://github.com/openstack/nova/commit/4804690ba401a23d62df489875da6c546d3224bb
Submitter: Jenkins
Branch: master

commit 4804690ba401a23d62df489875da6c546d3224bb
Author: Adam Gandelman <email address hidden>
Date: Wed Feb 8 14:16:08 2012 -0800

    linux_net: Also ignore shell error 2 from ip addr

    Ignores error code 2 from 'ip addr {add, del}' which, in recent
    versions of iproute2, means the address has already been removed
    or added.

    Fixes bug 929127

    Change-Id: I09274454847ff834ac3da0e38022b30a7a7f1676

Changed in nova:
status: In Progress → Fix Committed
Dave Walker (davewalker)
Changed in nova (Ubuntu):
status: New → Fix Committed
importance: Undecided → High
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nova - 2012.1~e4~20120217.12709-0ubuntu1

---------------
nova (2012.1~e4~20120217.12709-0ubuntu1) precise; urgency=low

  [ Dave Walker (Daviey) ]
  * New upstream snapshot
  * debian/patches/temp_fix_linux_net.patch:
    - Dropped, applied upstream. LP: #929127
  * debian/patches/libvirt-use-console-pipe.patch:
    - Rebased against latest trunk

  [ Chuck Short ]
  * debian/nova.conf: Re-enable default iscsi_helper.
  * debian/nova.conf: More fixups.
  * debian/control: Dont depend and conflicts on nova-compute-
    hypervisor. (LP: #923681)
  * debian/patches/libvirt-us-console-pipe.patch: Refreshed.
  * Temporarily disable console patch. (LP: #932787)
  * New usptream version.
 -- Chuck Short <email address hidden> Fri, 17 Feb 2012 10:59:59 -0500

Changed in nova (Ubuntu):
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: none → essex-4
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: essex-4 → 2012.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.