This happens with oslo.config >= 6.3.0([1]) + python3 + "keystone" paste_deploy + current glance(before https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 it works)
Testing in devstack: https://review.openstack.org/#/c/596380/
The glance api service fails to start with below Error, reproducing here: https://review.openstack.org/#/c/596380/:-
ERROR: dictionary changed size during iteration , see logs below
Failure logs from job:- http://logs.openstack.org/80/596380/2/check/tempest-full-py3/514fa29/controller/logs/screen-g-api.txt.gz#_Aug_27_07_26_10_698243
The Runtime Error is returned at keystonemiddleware:- https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py#L551
Adding code snippet here:-
if self._conf.oslo_conf_obj != cfg.CONF: <-- Fails here
oslo_cache.configure(self._conf.oslo_conf_obj)
So with pdb found that an additional key(fatal_deprecations) was added to cfg.CONF at ^^, so Error is returned in python3. With python2 same key is added but no Error.
There are multiple ways to avoid it, like use the paste_deploy configuration that works(ex: keystone+cachemanagement), use oslo.config <= 6.2.0, Use python2 or update glance(https://review.openstack.org/#/c/532503/10/glance/common/store_utils.py@30 as use_user_token is deprecated since long)
with keystone+cachemanagement, all the config items were added before reaching the Failure point in keystonemiddleware and self._conf.oslo_conf_obj != cfg.CONF didn't raised an error and returned Boolean. Don't know why.
But it seems a real issue to me as it may happen in python3 at different places. So it would be good if Teams from affected projects(oslo.config, keystonemiddleware, glance) can look at it and fix(not avoid) at the best place.
To me it looks like keystonemiddleware is not handling(comparing the dict) it properly for python3, as the conf is dynamically updated(how ? and when ?).
- so can oslo.config Team check if glance and keystonmiddleware are handling/using oslo.config properly.
- i checked keystone+cachemanagement is default in devstack from last 6 years, is "keystone" flavor supported? if yes it should be fixed. Also it would be good to cleanup the deprecated options those are deprecated since Mitaka.
- If it's wrongly used in keystonemiddleware/glance, it would be good to fix there.
Initially detected while testing with Fedora[2], but later digged on why it's working in CI with Ubuntu and started [3].
[1] https://review.openstack.org/#/c/560094/
[2] https://review.rdoproject.org/r/#/c/14921/
[3] https://review.openstack.org/#/c/596380/
I suspect that != line should be changed to "is not" for an object identity comparison instead of equality comparison.