[SRU] neutron doesn't correctly handle unknown protocols and should whitelist known and handled protocols

Bug #1749667 reported by Ian Kumlien
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Cloud Archive
Fix Released
Undecided
Unassigned
Ocata
Fix Released
Critical
Unassigned
neutron
Fix Released
Critical
Brian Haley

Bug Description

[Impact]
Neutron allowed users to create security group rules that would translate to invalid iptables rules thus causing neutron to fail when it attempted to apply them. This is now fixed for >= Pike and we are backporting for Ocata.

[Test Case]
  * deploy openstack ocata
  * create an invalid security group rule e.g.

openstack security group rule create --protocol gre --dst-port 0:255 jmclane

  * check that request is rejected with e.g.

Error while executing command: BadRequestException: Unknown error, {"NeutronError": {"message": "Invalid protocol 47 for port range, only supported for TCP, UDP, UDPLITE│·········································································································
, SCTP and DCCP.", "type": "SecurityGroupInvalidProtocolForPortRange", "detail": ""}}

[Regression Potential]
Upgrading to this patch will reject new api requests that try to create invalid rules but will not cleanup invalid rules already extant.

Note also that the backported ocata patch is unchanged from pike.

-----------------------------------------------------------------------------

We have had problems with openvswitch agent continuously restarting and never actually completing setup because of this:

# Completed by iptables_manager
; Stdout: ; Stderr: iptables-restore v1.4.21: multiport only works with TCP, UDP, UDPLITE, SCTP and DCCP
Error occurred at line: 83
Try `iptables-restore -h' or 'iptables-restore --help' for more information.

    83. -I neutron-openvswi-<id> 69 -s <ip> -p 112 -m multiport --dports 1:65535 -j RETURN
---

Someone has managed to inject a rule that is, effectively, a DoS.

Revision history for this message
Ian Kumlien (pomac) wrote :

The web ui seems to handle this, but the API doesn't

Revision history for this message
Brian Haley (brian-haley) wrote :

I can reproduce this, for example:

$ openstack security group rule create --ingress --protocol 70 --dst-port 70:71 default

Although the security DB code does check the range, it only does it for protocols it knows support it like TCP. There should be a check to reject a range if it's not a known good protocol.

I'm working on a change now.

Changed in neutron:
status: New → Confirmed
importance: Undecided → Critical
assignee: nobody → Brian Haley (brian-haley)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/545091

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
Thomas Morin (tmmorin-orange) wrote : Re: neutron doesn't correctly handle unknown protocols and should whitelist known and handled protocols

I think it would also be needed to have the iptables code to fail as cleanly as possible in case iptables rejets what we ask.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/545091
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b564871bb759a38cf96527f94e7c7d4cc760b1c9
Submitter: Zuul
Branch: master

commit b564871bb759a38cf96527f94e7c7d4cc760b1c9
Author: Brian Haley <email address hidden>
Date: Thu Feb 15 13:57:32 2018 -0500

    Only allow SG port ranges for whitelisted protocols

    Iptables only supports port-ranges for certain protocols,
    others will generate failures, possibly leaving the agent
    looping trying to apply rules. Change to not allow port
    ranges outside of the list of known good protocols.

    Change-Id: I5867f77fc5aedc169b42f50def0424ff209c164c
    Closes-bug: #1749667

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 13.0.0.0b1

This issue was fixed in the openstack/neutron 13.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/566921

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/566922

Revision history for this message
Florian Haas (fghaas) wrote : Re: neutron doesn't correctly handle unknown protocols and should whitelist known and handled protocols

Considering this bug is described as a potential DOS attack, I think it merits backports to pike and queens. Brian, your patch is a clean cherry-pick to queens and has only an apparently trivial test case conflict in pike, so I've taken a stab at just chucking those up. This is not meant to indicate that I know what I'm doing, so please take this with a grain of salt. :)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.openstack.org/566921
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7559512135d4dde16e019c94990a33b2603587b1
Submitter: Zuul
Branch: stable/queens

commit 7559512135d4dde16e019c94990a33b2603587b1
Author: Brian Haley <email address hidden>
Date: Thu Feb 15 13:57:32 2018 -0500

    Only allow SG port ranges for whitelisted protocols

    Iptables only supports port-ranges for certain protocols,
    others will generate failures, possibly leaving the agent
    looping trying to apply rules. Change to not allow port
    ranges outside of the list of known good protocols.

    Change-Id: I5867f77fc5aedc169b42f50def0424ff209c164c
    Closes-bug: #1749667
    (cherry picked from commit b564871bb759a38cf96527f94e7c7d4cc760b1c9)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/pike)

Reviewed: https://review.openstack.org/566922
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d8f9c9447020f1708ebac2b210e3bfe1dbefe3fa
Submitter: Zuul
Branch: stable/pike

commit d8f9c9447020f1708ebac2b210e3bfe1dbefe3fa
Author: Brian Haley <email address hidden>
Date: Thu Feb 15 13:57:32 2018 -0500

    Only allow SG port ranges for whitelisted protocols

    Iptables only supports port-ranges for certain protocols,
    others will generate failures, possibly leaving the agent
    looping trying to apply rules. Change to not allow port
    ranges outside of the list of known good protocols.

    This backport is based on commit
    b564871bb759a38cf96527f94e7c7d4cc760b1c9, excluding validation
    and tests for protocols where support for port ranges was
    added later (in Pike, only TCP and UDP are supported).

    Conflicts:
        neutron/tests/unit/db/test_securitygroups_db.py

    Change-Id: I5867f77fc5aedc169b42f50def0424ff209c164c
    Closes-bug: #1749667
    (cherry picked from commit b564871bb759a38cf96527f94e7c7d4cc760b1c9)

tags: added: in-stable-pike
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 12.0.3

This issue was fixed in the openstack/neutron 12.0.3 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 11.0.5

This issue was fixed in the openstack/neutron 11.0.5 release.

Changed in cloud-archive:
status: New → Fix Released
Revision history for this message
Edward Hope-Morley (hopem) wrote :
description: updated
summary: - neutron doesn't correctly handle unknown protocols and should whitelist
- known and handled protocols
+ [SRU] neutron doesn't correctly handle unknown protocols and should
+ whitelist known and handled protocols
tags: added: sts sts-sru-needed
description: updated
Revision history for this message
Corey Bryant (corey.bryant) wrote : Please test proposed package

Hello Ian, or anyone else affected,

Accepted neutron into ocata-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ocata-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ocata-needed to verification-ocata-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ocata-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-ocata-needed
Revision history for this message
Edward Hope-Morley (hopem) wrote :

Verified:

⟫ openstack security group rule create --protocol gre --dst-port 0:255 default
Error while executing command: BadRequestException: Unknown error, {"NeutronError": {"message": "Invalid protocol 47 for port range, only supported for TCP and UDP.", "type": "SecurityGroupInvalidProtocolForPortRange", "detail": ""}}

tags: added: verification-ocata-done
removed: verification-ocata-needed
Revision history for this message
Corey Bryant (corey.bryant) wrote : Update Released

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package neutron - 2:10.0.7-0ubuntu1~cloud1
---------------

 neutron (2:10.0.7-0ubuntu1~cloud1) xenial-ocata; urgency=medium
 .
   * Backport security-group multiport iptables rule fix (LP: #1749667)
     - d/p/Only-allow-SG-port-ranges-for-whitelisted-protocols.patch

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.