Ubuntu
neutron package

Neutron remote security group does not work in UCA Rocky and Stein - fixed upstream

Bug #1877797 reported by James Troup
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Fix Released
Critical
Unassigned
Rocky
Fix Released
Critical
Unassigned
Stein
Fix Released
Critical
Unassigned
Train
Fix Released
Critical
Unassigned
Ussuri
Fix Released
Critical
Unassigned
neutron (Ubuntu)
Fix Released
Critical
James Page
Eoan
Fix Released
Critical
Unassigned
Focal
Fix Released
Critical
Unassigned
Groovy
Fix Released
Critical
James Page

Bug Description

[Impact]
OpenStack deployments using the OVS firewall driver are broken when remote security groups are used due to a regression caused by bug 1854131.

[Test Case]
Deploy OpenStack (using charms)
Follow reproduction steps as detailed in bug 1862703
# create bastion-sec-grp to allow ssh from anywhere
openstack security group create bastion-sec-grp
openstack security group rule create --ethertype=IPv4 --protocol tcp --remote-ip 0.0.0.0/0 --ingress --dst-port=22 bastion-sec-grp

# create application-sec-grp
openstack security group create application-sec-grp

# Allow ssh to egress from the bastion group to the application group
openstack security group rule create --ethertype=IPv4 --protocol tcp --remote-group application-sec-grp --egress --dst-port=22 bastion-sec-grp

# Allow ssh to ingress to the application group from the bastion group
openstack security group rule create --ethertype=IPv4 --protocol tcp --remote-group bastion-sec-grp --ingress --dst-port=22 application-sec-grp

# create servers and associate with security groups
openstack server create --wait --image rhel7 --flavor small --security-group bastion-sec-grp bastion-server
openstack server create --wait --image rhel7 --flavor small --security-group application-sec-grp application-server

After boot, bastion-server and application-server are landed on different HVs and we can ssh to bastion-server but cannot ssh to application-server from there. Neutron debug log from application-server's HV shows:

2020-02-05 22:57:05,825 DEBUG [neutron.agent.linux.openvswitch_firewall.firewall] /opt/openstack/venv/neutron/lib/python2.7/site-packages/neutron/agent/linux/openvswitch_firewall/firewall.py:_build_addr_conj_id_map:297 No member for SG <BASTION_SEC_GRP_ID>

[Regression Potential]
Low - the fix is upstream across multiple releases and resolves a previous regression in functionality.

[Original Bug Report]
Remote security groups are broken in the UCA Rocky and Stein versions of Neutron.

The broken patch was introduced in LP #1854131 and fixed in LP #1862703.

The relevant fixed has landed in Neutron 13.0.7 for Rocky¹.

The relevant fixed landed in Neutron 14.1.0-37 for Stein², alternatively the specific fix is available here:

  https://github.com/openstack/neutron/commit/4193c6ca0e0165a2bcc7a11eee775df15019e755

The Queens version of Neutron currently in UCA (12.1.0) doesn't appear to have the bad patch from #1854131 in it.

We ran into this while upgrading a customer cloud and it caused several hours of VM connectivity downtime while we diagnosed it. Please upgrade Neutron in the Ubuntu Cloud Archive to have this fix available for at least Rocky and Stein.

I realise Rocky is no longer supported, but given that the supported upgrade path from Queens is via Rocky, I think it needs fixed there too.

¹ https://docs.openstack.org/releasenotes/neutron/rocky.html
² https://docs.openstack.org/releasenotes/neutron/stein.html

See original description
Revision history for this message
James Troup (elmo) wrote :

Subscribed ~field-high due to network outages on upgrade.

summary: - rocky neutron-openvswitch-agent has a bug which causes VM connectivity
- problems during Rocky upgrade (to get to Stein)
+ UCA rocky neutron-openvswitch-agent has a bug which causes VM
+ connectivity problems during Rocky upgrade (to get to Stein)
Revision history for this message
James Troup (elmo) wrote : Re: UCA rocky neutron-openvswitch-agent has a bug which causes VM connectivity problems during Rocky upgrade (to get to Stein)

OK, so it turns out this isn't even fixed in the Stein version of neutron in UCA. Upgrading to field-critical.

The Stein patch is here:

 https://github.com/openstack/neutron/commit/4193c6ca0e0165a2bcc7a11eee775df15019e755

James Troup (elmo)
summary: - UCA rocky neutron-openvswitch-agent has a bug which causes VM
- connectivity problems during Rocky upgrade (to get to Stein)
+ Neutron remote security group does not work in UCA Rocky and Stein -
+ fixed upstream
James Troup (elmo)
description: updated
James Troup (elmo)
description: updated
Revision history for this message
James Troup (elmo) wrote :

For reference this is the debdiff I used to build fixed packages for Bionic/Stein:

 https://paste.ubuntu.com/p/bHVZFz29dN/

Those packages are available in a PPA here:

 https://launchpad.net/~elmo/+archive/ubuntu/neutron-lp-1862703-public

Revision history for this message
James Page (james-page) wrote :

Rocky/13.0.7 is in rocky-proposed under bug 1875462 - looking at stein now.

James Page (james-page)
Changed in neutron (Ubuntu):
assignee: nobody → James Page (james-page)
importance: Undecided → Critical
Revision history for this message
James Page (james-page) wrote :

Ussuri has the required fix - rocky,stein,train are all impacted as they don't container the followup fix under bug 1862703. Queens does not have the regression (bug 1854131) or the fix (bug 1862703) but needs to be addressed as part of the next set of SRU's

James Page (james-page)
no longer affects: cloud-archive/queens
Changed in neutron (Ubuntu Groovy):
status: New → Fix Released
Changed in neutron (Ubuntu Focal):
status: New → Fix Released
Changed in neutron (Ubuntu Eoan):
status: New → Triaged
importance: Undecided → Critical
Changed in neutron (Ubuntu Focal):
importance: Undecided → Critical
description: updated
Revision history for this message
James Page (james-page) wrote :

Uploaded to eoan for SRU team review.

description: updated
description: updated
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello James, or anyone else affected,

Accepted neutron into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:15.0.2-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in neutron (Ubuntu Eoan):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-eoan
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Could we maybe get the reproduction steps from bug LP: #1862703 copied over to the bug description for readability? Anyway, accepted, I like that this fix comes with an unit test as well.

Revision history for this message
James Page (james-page) wrote :

Hello James, or anyone else affected,

Accepted neutron into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-train-needed
Revision history for this message
James Page (james-page) wrote :

Hello James, or anyone else affected,

Accepted neutron into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed
description: updated
Revision history for this message
James Page (james-page) wrote :

eoan/proposed

# apt-cache policy python3-neutron
python3-neutron:
  Installed: 2:15.0.2-0ubuntu1.1
  Candidate: 2:15.0.2-0ubuntu1.1
  Version table:
 *** 2:15.0.2-0ubuntu1.1 500
        500 http://archive.ubuntu.com/ubuntu eoan-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2:15.0.2-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu eoan-updates/main amd64 Packages
     2:15.0.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu eoan/main amd64 Packages

No log errors seem, able to SSH from bastion to application server.

Fix verified. Also ran tempest smoke to ensure no other regressions.

tags: added: verification-done verification-done-eoan
removed: verification-needed verification-needed-eoan
Revision history for this message
James Page (james-page) wrote :

UCA bionic-train/proposed

$ apt-cache policy python3-neutron
python3-neutron:
  Installed: 2:15.0.2-0ubuntu1.1~cloud0
  Candidate: 2:15.0.2-0ubuntu1.1~cloud0
  Version table:
 *** 2:15.0.2-0ubuntu1.1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/train/main amd64 Packages
        100 /var/lib/dpkg/status

No log errors seem, able to SSH from bastion to application server.

Fix verified. Also ran tempest smoke to ensure no other regressions.

tags: added: verification-train-done
removed: verification-train-needed
Revision history for this message
James Page (james-page) wrote :

UCA bionic-stein/proposed

$ apt-cache policy python3-neutron
python3-neutron:
  Installed: 2:14.1.0-0ubuntu1~cloud1
  Candidate: 2:14.1.0-0ubuntu1~cloud1
  Version table:
 *** 2:14.1.0-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/stein/main amd64 Packages
        100 /var/lib/dpkg/status

No log errors seem, able to SSH from bastion to application server.

Fix verified. Also ran tempest smoke to ensure no other regressions.

tags: added: verification-stein-done
removed: verification-stein-needed
Revision history for this message
James Page (james-page) wrote :

UCA bionic-rocky/proposed

$ apt-cache policy python3-neutron
python3-neutron:
  Installed: 2:13.0.7-0ubuntu1~cloud1
  Candidate: 2:13.0.7-0ubuntu1~cloud1
  Version table:
 *** 2:13.0.7-0ubuntu1~cloud1 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/rocky/main amd64 Packages
        100 /var/lib/dpkg/status

No log errors seem, able to SSH from bastion to application server.

Fix verified. Also ran tempest smoke to ensure no other regressions.

tags: added: verification-rocky-done
Revision history for this message
James Page (james-page) wrote :

Verification completed across all UCA and Ubuntu series.

It would be good to get this released today - waiting on SRU team for eoan release before releasing to the UCA pockets.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:15.0.2-0ubuntu1.1

---------------
neutron (2:15.0.2-0ubuntu1.1) eoan; urgency=medium

  * d/p/lp1877797.patch: Cherry pick fix to resolve issues with
    remote security groups when used with the OVS firewall driver
    (LP: #1877797).

 -- James Page <email address hidden> Mon, 11 May 2020 08:24:20 +0100

Changed in neutron (Ubuntu Eoan):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for neutron has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
James Page (james-page) wrote :

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
James Page (james-page) wrote :

This bug was fixed in the package neutron - 2:15.0.2-0ubuntu1.1~cloud0
---------------

 neutron (2:15.0.2-0ubuntu1.1~cloud0) bionic-train; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:15.0.2-0ubuntu1.1) eoan; urgency=medium
 .
   * d/p/lp1877797.patch: Cherry pick fix to resolve issues with
     remote security groups when used with the OVS firewall driver
     (LP: #1877797).

Revision history for this message
James Page (james-page) wrote :

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
James Page (james-page) wrote :

This bug was fixed in the package neutron - 2:14.1.0-0ubuntu1~cloud1
---------------

 neutron (2:14.1.0-0ubuntu1~cloud1) bionic-stein; urgency=medium
 .
   * d/p/lp1877797.patch: Cherry pick fix to resolve issues with
     remote security groups when used with the OVS firewall driver
     (LP: #1877797).

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.