keystonemiddleware

auth_token does not updated when endpoint is updated

Bug #1813739 reported by Yang Youseok
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
keystonemiddleware
Fix Released
Undecided
Yang Youseok

Bug Description

nova-api using auth_token middleware does not find identity server when identity endpoint is not created yet emitting EndpointNotfound Exception. It's working normally since there is no proper endpoint in auth_token.

The problem is even after user create proper identity endpoint used by nova-api, auth_token does not re-authenticate til expiration time is reached. Operator can restart nova-api and re-initialize token auth though, I think it should be working without restart.

It does not related to caching in auth_token, since every different token also failed after identity endpoint created. It results from adapter in _identity_server does not be updated using original auth (original catalog which does not have identity endpoint yet).

So I think it's better to have any ways to be notified when endpoint updated in auth_token.

Thanks.

OpenStack Infra (hudson-openstack)
Changed in keystonemiddleware:
assignee: nobody → Yang Youseok (ileixe)
status: New → In Progress
Revision history for this message
Colleen Murphy (krinkle) wrote :

Could you include the traceback that you are seeing when this happens?

Revision history for this message
Yang Youseok (ileixe) wrote :
Download full text (14.1 KiB)

@Collen Sure. This is nova-api.log

2019-02-04 15:57:30.257 3569 DEBUG keystonemiddleware.auth_token [-] Identity endpoint not found. fetch_token /opt/openstack/src/keystonemiddleware/keystonemiddleware/auth_token/__init__.py:780
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token Traceback (most recent call last):
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/src/keystonemiddleware/keystonemiddleware/auth_token/__init__.py", line 762, in fetch_token
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token allow_expired=allow_expired)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/src/keystonemiddleware/keystonemiddleware/auth_token/_identity.py", line 219, in verify_token
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token allow_expired=allow_expired)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/src/keystonemiddleware/keystonemiddleware/auth_token/_identity.py", line 108, in verify_token
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token allow_expired=allow_expired)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token return wrapped(*args, **kwargs)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/keystoneclient/v3/tokens.py", line 110, in validate
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token allow_expired=allow_expired)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/positional/__init__.py", line 101, in inner
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token return wrapped(*args, **kwargs)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/keystoneclient/v3/tokens.py", line 89, in get_token_data
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token resp, body = self._client.get(url, headers=headers)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 217, in get
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token return self.request(url, 'GET', **kwargs)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 374, in request
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 142, in request
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token return self.session.request(url, method, **kwargs)
2019-02-04 15:57:30.257 3569 ERROR keystonemiddleware.auth_token File "/opt/openstack/lib/python2...

Revision history for this message
Yang Youseok (ileixe) wrote :

Traceback pastebin above: https://pastebin.com/75a7zjfg

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to keystonemiddleware (master)

Reviewed: https://review.openstack.org/633695
Committed: https://git.openstack.org/cgit/openstack/keystonemiddleware/commit/?id=4e51cb8e6b4968fcb68903dce7e773b218f85bb7
Submitter: Zuul
Branch: master

commit 4e51cb8e6b4968fcb68903dce7e773b218f85bb7
Author: Yang Youseok <email address hidden>
Date: Tue Jan 29 18:59:12 2019 +0900

    Add auth invalidation in auth_token for identity endpoint update

    Currently auth_token middleware does not concern identity endpoint
    update since service catalog is not updated after service having
    auth_token middleware started.

    Add invalidation logic when EndpointNotfound exception occurs so
    that auth_token middleware can be notified of sevice catalog update
    without restart.

    Change-Id: I631ee1538883d732fe3987b172d987f703dad5c0
    Closes-Bug: #1813739

Changed in keystonemiddleware:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/keystonemiddleware 6.0.0

This issue was fixed in the openstack/keystonemiddleware 6.0.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.