metadata service calls to nova-api-metadata with IP based SAN's fails

Bug #1790598 reported by James Page
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Undecided
Unassigned
neutron (Ubuntu)
Fix Released
High
James Page
Xenial
Triaged
Low
Unassigned
Bionic
Fix Released
High
Unassigned
Cosmic
Fix Released
High
James Page

Bug Description

[Impact]
If the nova-api-metadata service is secured with a certificate that makes use of IP based SAN's, under Python 2 certificate validation will fail as the ssl module does not support use of IP addresses in cert SAN fields (and httplib2 which is used to make the request uses ssl directly).

Master branch of neutron has switched (see [0]) to using requests to make these calls, supporting use of certs with IP address based SAN's (via urllib3 which does support IP address based SAN's under Python 2).

[0] https://github.com/openstack/neutron/commit/7e0dd2f18d4919964655cfce7a282d1c5c131fc4

[Test Case]
Deploy OpenStack, securing metadata service using certs with IPAddress based SAN's (openstack charms + vault can do this).
Boot instance - instance will fail to get metadata due to neutron->nova cert verification failure.

[Regression Potential]
Patch switches communication between neutron and nova for metadata queries to use requests over httplib2; so its a fairly like-for-like switch - both are used across openstack for various purposes.

James Page (james-page)
Changed in neutron (Ubuntu Cosmic):
status: New → Triaged
Changed in neutron (Ubuntu Bionic):
status: New → Triaged
Changed in neutron (Ubuntu Xenial):
status: New → Triaged
importance: Undecided → High
Changed in neutron (Ubuntu Bionic):
importance: Undecided → High
Changed in neutron (Ubuntu Cosmic):
importance: Undecided → High
description: updated
description: updated
Changed in neutron:
status: New → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/599537

James Page (james-page)
Changed in neutron (Ubuntu Cosmic):
status: Triaged → In Progress
assignee: nobody → James Page (james-page)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/599541

James Page (james-page)
Changed in neutron (Ubuntu Xenial):
importance: High → Low
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:13.0.0-0ubuntu2

---------------
neutron (2:13.0.0-0ubuntu2) cosmic; urgency=medium

  * d/p/metadata-use-requests-for-comms-with-nova-api.patch: Cherry
    pick of fix to support use of certs with IP based SAN's on Nova
    API endpoints when making metadata service calls (LP: #1790598).
  * d/control: Bump minimum requests version inline with above patch.

 -- James Page <email address hidden> Tue, 04 Sep 2018 14:59:36 +0100

Changed in neutron (Ubuntu Cosmic):
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/rocky)

Reviewed: https://review.openstack.org/599537
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c28e4963b75414f093e432c9934f8658a4e56b98
Submitter: Zuul
Branch: stable/rocky

commit c28e4963b75414f093e432c9934f8658a4e56b98
Author: James Page <email address hidden>
Date: Mon Aug 20 15:22:10 2018 +0100

    metadata: use requests for comms with nova api

    httplib2 makes use of the ssl module provided by Python; under Python 2,
    the ssl module does not support IP addresses as subject alternate names
    (SAN's) which although an optional part of the associated RFC, is awkward
    to work with in environments where certificate management approaches
    rely on use of IP addresses in SAN's.

    The requests module is more than happy to deal with this scenario; switch
    to requests in preference of httplib2 for metadata proxy calls.

    httplib2 is retained as its used elsewhere in the codebase.

    Closes-Bug: 1790598
    Change-Id: Ife4adf09ddbf7116da2f8596c80aed53fb6790df
    (cherry picked from commit 7e0dd2f18d4919964655cfce7a282d1c5c131fc4)

tags: added: in-stable-rocky
Revision history for this message
Corey Bryant (corey.bryant) wrote :

The stable/queens fix has been included in neutron 2:12.0.4-0ubuntu1, currently in the bionic unapproved queue awaiting SRU team review.

description: updated
James Page (james-page)
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote :

As an FYI the "Regression Potential" part of the SRU description is supposed to be about how things can go wrong not a statement regarding the chances of their being a regression.

Revision history for this message
Brian Murray (brian-murray) wrote : Proposed package upload rejected

An upload of neutron to bionic-proposed has been rejected from the upload queue for the following reason: "coreycb said he was going to add some more patches in #ubuntu-devel.".

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Neutron 2:12.0.4-0ubuntu1 is now ready for review in the unapproved queue.

James Page (james-page)
description: updated
Revision history for this message
Timo Aaltonen (tjaalton) wrote : Please test proposed package

Hello James, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:12.0.4-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in neutron (Ubuntu Bionic):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-bionic
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Regression testing successful for bionic-proposed (tempest results):

======
Totals
======
Ran: 92 tests in 1318.6413 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 494.8999 sec.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Regression testing successful for queens-proposed (tempest results):

======
Totals
======
Ran: 92 tests in 1000.6584 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 465.0920 sec.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Looks like this bug is verified but not marked as verification-done-bionic. Is there any more testing you want to perform on this bug before release?

Revision history for this message
Corey Bryant (corey.bryant) wrote :

I checked with jamespage and he said regression is enough for verifying this. Tagged as verified.

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 13.0.2

This issue was fixed in the openstack/neutron 13.0.2 release.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello James, or anyone else affected,

Accepted neutron into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:12.0.5-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed verification-needed-bionic
removed: verification-done verification-done-bionic
Revision history for this message
Corey Bryant (corey.bryant) wrote :

Regression testing was successful against xenial-proposed:

======
Totals
======
Ran: 92 tests in 1034.1765 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 465.6833 sec.

Regression testing was successful against queens-proposed:

======
Totals
======
Ran: 92 tests in 1106.9986 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 548.8946 sec.

Revision history for this message
Corey Bryant (corey.bryant) wrote :

Sorry the first set of testing above was against bionic-proposed not xenial-proposed.

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package neutron - 2:12.0.5-0ubuntu1

---------------
neutron (2:12.0.5-0ubuntu1) bionic; urgency=medium

  * New stable point release for OpenStack Queens (LP: #1795424).
  * d/p/metadata-use-requests-for-comms-with-nova-api.patch: Cherry-picked
    from https://review.openstack.org/#/c/599541/ to enable cert management
    where IP addresses are used in subject alternate names (LP: #1790598).

 -- Corey Bryant <email address hidden> Tue, 06 Nov 2018 11:43:51 -0500

Changed in neutron (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/631434

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/631434
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1685982a97f1bc82698724eb9a47e92416dc0aac
Submitter: Zuul
Branch: master

commit 1685982a97f1bc82698724eb9a47e92416dc0aac
Author: Pawel Suder <email address hidden>
Date: Thu Jan 17 07:30:45 2019 +0100

    Use status_code instead of status in requests

    It fixes raising exception for response with not recognized
    status code.

    Co-Authored-By: Brian Haley <email address hidden>
    Change-Id: I174ff62cb6599e4c7bdc86cb2d0786f9f2499b00
    Related-Bug: 1790598

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/rocky)

Related fix proposed to branch: stable/rocky
Review: https://review.openstack.org/632072

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/queens)

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/632076

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/rocky)

Reviewed: https://review.openstack.org/632072
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9f003cf497fd4f4ffcdd5c633824531b465aa69c
Submitter: Zuul
Branch: stable/rocky

commit 9f003cf497fd4f4ffcdd5c633824531b465aa69c
Author: Pawel Suder <email address hidden>
Date: Thu Jan 17 07:30:45 2019 +0100

    Use status_code instead of status in requests

    It fixes raising exception for response with not recognized
    status code.

    Co-Authored-By: Brian Haley <email address hidden>
    Change-Id: I174ff62cb6599e4c7bdc86cb2d0786f9f2499b00
    Related-Bug: 1790598
    (cherry picked from commit 1685982a97f1bc82698724eb9a47e92416dc0aac)

tags: added: neutron-proactive-backport-potential
tags: removed: neutron-proactive-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/599541
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ae542f685466dc65967c6d74d38d8935685256f5
Submitter: Zuul
Branch: stable/queens

commit ae542f685466dc65967c6d74d38d8935685256f5
Author: James Page <email address hidden>
Date: Mon Aug 20 15:22:10 2018 +0100

    metadata: use requests for comms with nova api

    httplib2 makes use of the ssl module provided by Python; under Python 2,
    the ssl module does not support IP addresses as subject alternate names
    (SAN's) which although an optional part of the associated RFC, is awkward
    to work with in environments where certificate management approaches
    rely on use of IP addresses in SAN's.

    The requests module is more than happy to deal with this scenario; switch
    to requests in preference of httplib2 for metadata proxy calls.

    httplib2 is retained as its used elsewhere in the codebase.

    Closes-Bug: 1790598
    Change-Id: Ife4adf09ddbf7116da2f8596c80aed53fb6790df
    (cherry picked from commit 7e0dd2f18d4919964655cfce7a282d1c5c131fc4)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (stable/queens)

Reviewed: https://review.opendev.org/632076
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4091dc33d7172a7ae6333ae29c1f0620aa50c400
Submitter: Zuul
Branch: stable/queens

commit 4091dc33d7172a7ae6333ae29c1f0620aa50c400
Author: Pawel Suder <email address hidden>
Date: Thu Jan 17 07:30:45 2019 +0100

    Use status_code instead of status in requests

    It fixes raising exception for response with not recognized
    status code.

    Co-Authored-By: Brian Haley <email address hidden>
    Change-Id: I174ff62cb6599e4c7bdc86cb2d0786f9f2499b00
    Related-Bug: 1790598
    (cherry picked from commit 1685982a97f1bc82698724eb9a47e92416dc0aac)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 12.1.0

This issue was fixed in the openstack/neutron 12.1.0 release.

Changed in neutron:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.