linux bridge agent disables ipv6 before adding an ipv6 address

IPv6 is disabled on bridge devices because it was exposing the hypervisor to possible attack from local VMs running on a compute node - i.e. you could ping and try to ssh to that IP, especially in the OVS hybrid case. Could be we're being overly aggressive in disabling it.

I'm not super knowledgable on the linux bridge agent, but if you had a stack track on how this is getting triggered it could probably be fixed.

Neutron now disables IPv6 on each bridge to prevent inadvertently exposing the hypervisor on tenant networks: In most cases each hypervisor is just an L2 hop on the tenant network, so there isn't a need for it to participate at layers 3 and higher on those networks. Any L3 features (such as routing to DHCP) should be provided within a isolated network namespace. If a user specifically wants to connect a hypervisor to a tenant network, they can either use two physical interfaces connected to the same VALN on the switch, or create a veth interface pair and attach on end to tenant bridge and assign an IP to the other.

I'm not sure why Neutron is now trying to assign an IPv6 address to the bridge interface. This behavior seems incorrect.

Does disabling IPv6 on the bridge just mean that no IPv6 address can be plumbed to the bridge but it will forward IPv6? In other words, I can still create an IPv6 subnet on the network? Don't know enough about IPv6 to know if the Routing Advertisement will forward through the bridge to the VM for the VM to be able to use SLAAC to auto-configure.

I think the problem may be that since I have IPv6 auto-configuring on the physical NIC, the process of creating a bridge is attempting to move all addresses from the physical NIC to the bridge. But it's not smart enough to know not to move IPv6 addresses since it has been disabled for that. It seems to get into a loop of continuously attempting to add the address:
2017-02-06 13:43:31.249 4358 INFO neutron.plugins.ml2.drivers.agent._common_agent [req-1abef784-7bf3-4fda-b379-67084ad37f74 - - - - -] Linux bridge agent Agent out of sync with plugin!
2017-02-06 13:43:31.250 4358 DEBUG neutron.plugins.ml2.drivers.agent._common_agent [req-1abef784-7bf3-4fda-b379-67084ad37f74 - - - - -] Agent loop found changes! {'current': set(['tapd0444968-ca']), 'timestamps': {'tapd0444968-ca': 11}, 'removed': set([]), 'added': set(['tapd0444968-ca']), 'updated': set([])} daemon_loop /usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py:450

Right, the bridge will still pass IPv6 between ports even if it doesn't have an IPv6 address.

This case seems to be the initial bridge creation, passing a device (say eth0) along with the IPs on the device. I do see that ensure_bridge() will call update_interface_ip_details() to move any IPs it was given from the interface being added to the bridge, to the bridge itself. I think that's a valid operation since without it I don't think the IPs will be usable.

That said, this will probably still expose the hypervisor, even for the IPv4 address, but clearly we aren't disallowing that. The intent of disabling IPv6 was to not accept an RA from a tenant network and configure an address on their subnet. We don't want to undo that change, which leaves you in a broken state.

A quick hack you can do to just move the IPv4 addresses is this change to get_interface_details():

- ips = device.addr.list(scope='global')
--
+ ips = device.addr.list(scope='global', ip_version=constants.IP_VERSION_4)

Like I said, that will leave the IPv6 addresses behind, but will they still be usable? Someone from the ML2 team that better understands the linux bridge driver would have to comment on this as well.

Fix proposed to branch: master
Review: https://review.openstack.org/431619

Reviewed: https://review.openstack.org/431619
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b6d4d382a9fc1088752ae9b010fb25d0fcdd5845
Submitter: Jenkins
Branch: master

commit b6d4d382a9fc1088752ae9b010fb25d0fcdd5845
Author: Brian Haley <email address hidden>
Date: Thu Feb 9 10:48:07 2017 -0500

    Fix linuxbridge agent startup issue with IPv6

    The linuxbridge agent tries to move all the IP addresses
    from the first interface attached to a bridge at startup.
    This can fail if the interface has an IPv6 address since
    IPv6 is always disabled on bridge devices.

    Change ensure_bridge() to not disable IPv6, and instead
    move all IPs and default route by family - IPv4 and IPv6.

    Change-Id: Ic236de04c0203633df49967a9a4528fda13c51df
    Closes-bug: #1662324

This issue was fixed in the openstack/neutron 11.0.0.0b1 development milestone.

The problem, or a very similar one, seems to exist for Ubuntu 19.04 (Disco Dingo) with its OpenStack Stein packages:
* Compute node is configured with Neutron
* Physical interface eth1 for bridging has no IPv4 or IPv6 configuration. net.ipv6.conf.eth1.autoconf=0.
* Two subnets are defined, one for IPv4, another one for IPv6.
* When creating an instance, Neutron creates a bridge device brq21b1bce8-6e. It has IPv6 disabled.

Error log:
2019-05-28 12:52:13.124 1921 DEBUG oslo_concurrency.lockutils [-] Lock "privileged-ip-lib" acquired by "neutron.privileged.agent.linux.ip_lib.interface_exists" :: waited 0.000s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:327
2019-05-28 12:52:13.127 1921 DEBUG oslo_concurrency.lockutils [-] Lock "privileged-ip-lib" released by "neutron.privileged.agent.linux.ip_lib.interface_exists" :: held 0.003s inner /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:339
2019-05-28 12:52:13.127 1921 DEBUG oslo.privsep.daemon [-] privsep: reply[140589694763752]: (4, True) _call_back /usr/lib/python3/dist-packages/oslo_privsep/daemon.py:472
2019-05-28 12:52:13.128 1462 ERROR neutron.plugins.ml2.drivers.agent._common_agent [req-00ac9618-e4fb-4001-8690-243a99ac1036 - - - - -] Error in agent loop. Devices info: {'current': {'tap29ab982f-05'}, 'timestamps': {'tap29ab982f-05': 8}, 'added': {'tap29ab982f-05'}, 'removed': set(), 'updated': set()}: neutron_lib.exceptions.ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: Error: IPv6 is disabled on nexthop device.
2019-05-28 12:52:13.128 1462 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2019-05-28 12:52:13.128 1462 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 466, in daemon_loop
2019-05-28 12:52:13.128 1462 ERROR neutron.plugins.ml2.drivers.agent._common_agent sync = self.process_network_devices(device_info)
2019-05-28 12:52:13.128 1462 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 159, in wrapper
2019-05-28 12:52:13.128 1462 ERROR neutron.plugins.ml2.drivers.agent._common_agent result = f(*args, **kwargs)
2019-05-28 12:52:13.128 1462 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 215, in process_network_devices
2019-05-28 12:52:13.128 1462 ERROR neutron.plugins.ml2.drivers.agent._common_agent resync_a = self.treat_devices_added_updated(devices_added_updated)
2019-05-28 12:52:13.128 1462 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/osprofiler/profiler.py", line 159, in wrapper
2019-05-28 12:52:13.128 1462 ERROR neutron.plugins.ml2.drivers.agent._common_agent result = f(*args, **kwargs)
2019-05-28 12:52:13.128 1462 ERROR neutron.plugins.ml2.drivers.agent._common_agent File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 232, in treat_devices_added_updated
2019-05-28 12:52:13.128 1462 ERROR neutron.plugins.ml2.driv...

This is a very old closed bug, if you are seeing an issue with a newer release, e.g. Stein, you should open a new bug.

Thanks Seyeong. A new version of neutron including your patch has been uploaded to the xenial unapproved queue.
https://launchpad.net/ubuntu/xenial/+queue?queue_state=1&queue_text=neutron

Since the patch is not entirely trivial, since some of the methods have had their argument lists changed, I would appreciate if we had a bit more regression potential analysis. What do you think could, in the worst unpredictable scenario, break after this is released? In which parts should we expect issues? Could it somehow affect non IPv6 scenarios?

I updated regression

Please let me know if I need to put more.

Thanks.

I'm sorry, I don't see answers to any of Lukasz's questions in the text you've written. Please look again, or perhaps ask your sponsor for help. The documentation at https://wiki.ubuntu.com/StableReleaseUpdates#Procedure also has some guidance.

Thanks for your guide

Hello David, or anyone else affected,

Accepted neutron into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:8.4.0-0ubuntu7.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello

I verified this patch is working fine on testing.

After this patch, disable_ipv6 contains 0 which is proper value for ipv6.

Thanks a lot.

ii neutron-common 2:8.4.0-0ubuntu7.5 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:8.4.0-0ubuntu7.5 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 2:8.4.0-0ubuntu7.5 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-linuxbridge-agent 2:8.4.0-0ubuntu7.5 all Neutron is a virtual network service for Openstack - linuxbridge agent
ii neutron-metadata-agent 2:8.4.0-0ubuntu7.5 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-plugin-ml2 2:8.4.0-0ubuntu7.5 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-server 2:8.4.0-0ubuntu7.5 all Neutron is a virtual network service for Openstack - server
ii python-neutron 2:8.4.0-0ubuntu7.5 all Neutron is a virtual network service for Openstack - Python library

Hello David, or anyone else affected,

Accepted neutron into mitaka-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:mitaka-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-mitaka-needed to verification-mitaka-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-mitaka-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I'm going to verify mitaka this week. building trusty-mitaka env now

@corey.bryant

Hello. I just checked that even I did "sudo add-apt-repository cloud-archive:mitaka-proposed"

neutron version is the same as mitaka-staging

neutron-linuxbridge-agent/trusty-proposed,trusty,now 2:8.4.0-0ubuntu7.5~cloud0 all [installed]

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package neutron - 2:8.4.0-0ubuntu7.5

---------------
neutron (2:8.4.0-0ubuntu7.5) xenial; urgency=medium

  * d/p/0001-Fix-linuxbridge-agent-startup-issue-with-IPv6.patch
    - Ensure network enable_ipv6 when using linuxbridge (LP: #1662324)

 -- Seyeong Kim <email address hidden> Fri, 22 May 2020 01:29:23 -0700

@seyeong, yes that's correct, 2:8.4.0-0ubuntu7.5~cloud0 has the fix that needs testing. Would you be able to test mitaka-proposed and report back?

@corey Thanks.

it works fine as belows. disable_ipv6 is 0

6: ens4f1.1103@ens4f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master brqc1308c59-e3 state UP group default
    link/ether 52:54:01:97:26:35 brd ff:ff:ff:ff:ff:ff
7: brqc1308c59-e3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 52:54:01:97:26:35 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7067:acff:fe3f:2c7c/64 scope link
       valid_lft forever preferred_lft forever
root@maas-node-18:~# cat /proc/sys/net/ipv6/conf/brqc1308c59-e3/disable_ipv6
0

ii neutron-common 2:8.4.0-0ubuntu7.5~cloud0 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:8.4.0-0ubuntu7.5~cloud0 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 2:8.4.0-0ubuntu7.5~cloud0 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-linuxbridge-agent 2:8.4.0-0ubuntu7.5~cloud0 all Neutron is a virtual network service for Openstack - linuxbridge agent
ii neutron-metadata-agent 2:8.4.0-0ubuntu7.5~cloud0 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-plugin-ml2 2:8.4.0-0ubuntu7.5~cloud0 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-server 2:8.4.0-0ubuntu7.5~cloud0 all Neutron is a virtual network service for Openstack - server
ii python-neutron 2:8.4.0-0ubuntu7.5~cloud0 all Neutron is a virtual network service for Openstack - Python library

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package neutron - 2:8.4.0-0ubuntu7.5~cloud0
---------------

 neutron (2:8.4.0-0ubuntu7.5~cloud0) trusty-mitaka; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:8.4.0-0ubuntu7.5) xenial; urgency=medium
 .
   * d/p/0001-Fix-linuxbridge-agent-startup-issue-with-IPv6.patch
     - Ensure network enable_ipv6 when using linuxbridge (LP: #1662324)