rate_limit error when executing openstack network commands

Bug #1819020 reported by Ghada Khalil
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Critical
Scott Little

Bug Description

Title
-----
rate_limit error when executing openstack network commands
This issue was reported by Shuicheng Lin

Brief Description
-----------------
When attempting to run openstack networking commands as part of system provisioning, an error is returned as follows:
controller-0:/home/wrsroot# openstack network segment range create ${PHYSNET0}-a --network-type vlan --physical-network ${PHYSNET0} --minimum 400 --maximum 499 --private --project ${ADMINID}
__init__() got an unexpected keyword argument 'rate_limit'
controller-0:/home/wrsroot# openstack network list
__init__() got an unexpected keyword argument 'rate_limit'

Severity
--------
Major: This impacts configuring networking on recent StarlingX loads; started on Mar 5.

Steps to Reproduce
------------------
run any openstack network command

Expected Behavior
------------------
Command should pass

Actual Behavior
----------------
Commands fail as per above

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
Any

Branch/Pull Time/Commit
-----------------------
master as of March 5 afternoon

From Joseph Richard:
In https://github.com/openstack/openstacksdk/blame/da4ec3e70c20db3ea37cd54dec6dc1e10b309274/lower-constraints.txt, a dependency was added for keystoneauth1==3.13.0, however it looks like StarlingX is only at 3.10.0. Because we are building openstacksdk from master, we are hitting this. We will likely need to upversion keystoneauth1 to resolve this.

Timestamp/Logs
--------------
Not required; issue is reproducible

Revision history for this message
Ghada Khalil (gkhalil) wrote :

From Don Penney:
This should just be a matter of updating the RPM version being downloaded, as we aren’t rebuilding it:
stx-tools/centos-mirror-tools/rpms_3rdparties.lst:
python2-keystoneauth1-3.10.0-1.el7.noarch.rpm#https://cbs.centos.org/kojifiles/packages/python-keystoneauth1/3.10.0/1.el7/noarch/python2-keystoneauth1-3.10.0-1.el7.noarch.rpm

There should be a check of other RPM versions, as well, though, in case anything else needs to be updated.

Changed in starlingx:
importance: Undecided → Critical
description: updated
description: updated
Ghada Khalil (gkhalil)
description: updated
Changed in starlingx:
assignee: nobody → Scott Little (slittle1)
Ghada Khalil (gkhalil)
tags: added: stx.2019.05 stx.build stx.networking
tags: added: stx.distro.openstack
removed: stx.build
Revision history for this message
Scott Little (slittle1) wrote :

No pre-built rpms available for 3.13 or newer at this time. We will have to compile our own rpm from git or tarball.

Revision history for this message
Juan Carlos Alonso (juancarlosa) wrote :

This issue is reproducible using our test framework, same behaviour with neutron commands and openstack commands:

$: openstack network list
__init__() got an unexpected keyword argument 'rate_limit'

$: neutron net-list
neutron CLI is deprecated and will be removed in the future. Use openstack CLI instead.
Connection to neutron failed: Failed to connect Neutron server

Revision history for this message
Joseph Richard (josephrichard) wrote :

The issue does not affect neutronclient.

That error you are seeing looks like you have the wrong endpoint.
Set OS_CLOUD=openstack_helm and do not source /etc/nova/openrc when accessing neutron.

https://wiki.openstack.org/w/index.php?title=StarlingX/Containers/Installation#Verify_the_cluster_endpoints

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-upstream (master)

Fix proposed to branch: master
Review: https://review.openstack.org/642036

Changed in starlingx:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-tools (master)

Fix proposed to branch: master
Review: https://review.openstack.org/642037

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-manifest (master)

Fix proposed to branch: master
Review: https://review.openstack.org/642040

Revision history for this message
Scott Little (slittle1) wrote :

    Investigation showed that there is currently no centos rpm
providing python2-keystoneauth1-3.13.0 or better. Solution
is to build python2-keystoneauth1-3.13.1 from openstack's git.
We can revert to a downloaded rpm when one becomes available.

Code reviews:

stx-manifest: https://review.openstack.org/642040
stx-upstream: https://review.openstack.org/642036
stx-tools: https://review.openstack.org/642037

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-manifest (master)

Reviewed: https://review.openstack.org/642040
Committed: https://git.openstack.org/cgit/openstack/stx-manifest/commit/?id=03eb066b2c5d934709d6440002777db94887804c
Submitter: Zuul
Branch: master

commit 03eb066b2c5d934709d6440002777db94887804c
Author: Scott Little <email address hidden>
Date: Thu Mar 7 12:38:45 2019 -0500

    Upgrade python-keystoneauth1 to 3.13.0

    Issue:
       'rate_limit' error when executing openstack network commands.
    Failure was tracked to a failure to satisfy the lower-constraints.txt
    of openstacksdk, keystoneauth1==3.13.0. We currently only provide
    python2-keystoneauth1-3.10.0-1.el7.noarch.rpm.

    Solution:
        Investigation showed that there is currently no centos rpm
    providing python2-keystoneauth1-3.13.0 or better. Solution
    is to build python2-keystoneauth1-3.13.1 from openstack's git.
    We can revert to a downloaded rpm when one becomes available.

    Closes-Bug: 1819020
    Change-Id: I715ee3f05dacaa320083dd40c2b23f0f599b910e
    Signed-off-by: Scott Little <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-upstream (master)

Reviewed: https://review.openstack.org/642036
Committed: https://git.openstack.org/cgit/openstack/stx-upstream/commit/?id=eb6883ff55b9fa7a10dcccac685cfcebd06cfd62
Submitter: Zuul
Branch: master

commit eb6883ff55b9fa7a10dcccac685cfcebd06cfd62
Author: Scott Little <email address hidden>
Date: Thu Mar 7 13:10:06 2019 -0500

    Upgrade python-keystoneauth1 to 3.13.0

    Issue:
       'rate_limit' error when executing openstack network commands.
    Failure was tracked to a failure to satisfy the lower-constraints.txt
    of openstacksdk, keystoneauth1==3.13.0. We currently only provide
    python2-keystoneauth1-3.10.0-1.el7.noarch.rpm.

    Solution:
        Investigation showed that there is currently no centos rpm
    providing python2-keystoneauth1-3.13.0 or better. Solution
    is to build python2-keystoneauth1-3.13.1 from openstack's git.
    We can revert to a downloaded rpm when one becomes available.

    Note:
        I'm taking this opportunity to clean up stale patches
    that pre-date python2-keystoneauth1-3.10.0. None of these
    where in use.

    Closes-Bug: 1819020
    Change-Id: Ie2300f352dffc2fc87fbc84a9fc98d2a4e9484d4
    Depends-On: I715ee3f05dacaa320083dd40c2b23f0f599b910e
    Signed-off-by: Scott Little <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-tools (master)

Reviewed: https://review.openstack.org/642037
Committed: https://git.openstack.org/cgit/openstack/stx-tools/commit/?id=67e57499198a700a36a5464e17b7fab30d67685b
Submitter: Zuul
Branch: master

commit 67e57499198a700a36a5464e17b7fab30d67685b
Author: Scott Little <email address hidden>
Date: Thu Mar 7 12:50:35 2019 -0500

    Upgrade python-keystoneauth1 to 3.13.0

    Issue:
       'rate_limit' error when executing openstack network commands.
    Failure was tracked to a failure to satisfy the lower-constraints.txt
    of openstacksdk, keystoneauth1==3.13.0. We currently only provide
    python2-keystoneauth1-3.10.0-1.el7.noarch.rpm.

    Solution:
        Investigation showed that there is currently no centos rpm
    providing python2-keystoneauth1-3.13.0 or better. Solution
    is to build python2-keystoneauth1-3.13.1 from openstack's git.
    We can revert to a downloaded rpm when one becomes available.

    Closes-Bug: 1819020
    Change-Id: I30332820a8f4ae877b0719148022bade1943eadb
    Depends-On: Ie2300f352dffc2fc87fbc84a9fc98d2a4e9484d4
    Signed-off-by: Scott Little <email address hidden>

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-tools (master)

Fix proposed to branch: master
Review: https://review.openstack.org/642364

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-manifest (master)

Fix proposed to branch: master
Review: https://review.openstack.org/642365

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-upstream (master)

Fix proposed to branch: master
Review: https://review.openstack.org/642368

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on stx-upstream (master)

Change abandoned by Saul Wold (<email address hidden>) on branch: master
Review: https://review.openstack.org/642368
Reason: Did something wrong will try again tomorrow.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on stx-manifest (master)

Change abandoned by Saul Wold (<email address hidden>) on branch: master
Review: https://review.openstack.org/642365
Reason: Did something wrong will try again tomorrow.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on stx-tools (master)

Change abandoned by Saul Wold (<email address hidden>) on branch: master
Review: https://review.openstack.org/642364
Reason: Did something wrong will try again tomorrow.

Ken Young (kenyis)
tags: added: stx.2.0
removed: stx.2019.05
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.