Merge lp:~tribaal/charms/trusty/nova-compute/enable-api-rate-limiting into lp:~openstack-charmers-archive/charms/trusty/nova-compute/next

[Chris Glass]

This branch introduces a new api-rate-limit-rules charm config option controlling the rate-limiting
of the nova API by writing the passed value to the /etc/nova/api-paste.ini file.

[lp:~tribaal/charms/trusty/nova-compute/enable-api-rate-limiting]

165. By Chris Glass on 2015-09-30

Forgot to check the output.

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #11089 nova-compute-next for tribaal mp272884
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/11089/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #10295 nova-compute-next for tribaal mp272884
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/10295/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #11090 nova-compute-next for tribaal mp272884
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/11090/

[uosci-testing-bot (uosci-testing-bot)]

charm_unit_test #10297 nova-compute-next for tribaal mp272884
    UNIT OK: passed

Build: http://10.245.162.77:8080/job/charm_unit_test/10297/

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #6896 nova-compute-next for tribaal mp272884
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12625830/
Build: http://10.245.162.77:8080/job/charm_amulet_test/6896/

[uosci-testing-bot (uosci-testing-bot)]

charm_lint_check #11151 nova-compute-next for tribaal mp272884
    LINT OK: passed

Build: http://10.245.162.77:8080/job/charm_lint_check/11151/

[Liam Young (gnuoy)]

nova-compute install hook fails on ./tests/014-basic-precise-icehouse amulet test.

0 upgraded, 0 newly installed, 0 to remove and 19 not upgraded.
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/install.real", line 434, in <module>
    main()
  File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/install.real", line 428, in main
    hooks.execute(sys.argv)
  File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/core/hookenv.py", line 672, in execute
    self._hooks[hook_name]()
  File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 777, in wrapped_f
    set_os_workload_status(configs, required_interfaces, charm_func)
  File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 790, in set_os_workload_status
    incomplete_rel_data = incomplete_relation_data(configs, required_interfaces)
  File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/utils.py", line 920, in incomplete_relation_data
    complete_ctxts = configs.complete_contexts()
  File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 295, in complete_contexts
    for i in six.itervalues(self.templates)]
  File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 120, in complete_contexts
    self.context()
  File "/var/lib/juju/agents/unit-nova-compute-0/charm/hooks/charmhelpers/contrib/openstack/templating.py", line 107, in context
    ctxt.update(_ctxt)
TypeError: 'APIRateLimitingContext' object is not iterable

review: Needs Fixing

[lp:~tribaal/charms/trusty/nova-compute/enable-api-rate-limiting]

166. By Chris Glass on 2015-10-01

Fixing iterable idiocy.

[Liam Young (gnuoy)]

Manually ran amulet for trusty and precise. All good! Approved

review: Approve

[uosci-testing-bot (uosci-testing-bot)]

charm_amulet_test #6926 nova-compute-next for tribaal mp272884
    AMULET FAIL: amulet-test failed

AMULET Results (max last 2 lines):
make: *** [test] Error 1
ERROR:root:Make target returned non-zero.

Full amulet test output: http://paste.ubuntu.com/12633828/
Build: http://10.245.162.77:8080/job/charm_amulet_test/6926/

[Ryan Beisner (1chb1n)]

fwiw ^ The re-run (with test enviro looking sound) says precise-icehouse didn't have nova-api service started on the nova-compute/0 unit.

=== modified file 'config.yaml'
--- config.yaml	2015-09-28 07:56:16 +0000
+++ config.yaml	2015-10-01 15:08:20 +0000
@@ -265,3 +265,13 @@
     description: |
       The pecentage of system memory to use for hugepages eg '10%' or the total
       number of 2M hugepages - eg "1024".
+  api-rate-limit-rules:
+    type: string
+    default:
+    description: |
+      The API rate-limit rules to use for the deployed nova API, if any.
+      Contents of this config options will be inserted in the api-paste.ini file
+      under the "filter:ratelimit" section as "limits". The syntax for these
+      rules is documented at
+      http://docs.openstack.org/kilo/config-reference/content/configuring-compute-API.html
+    

=== modified file 'hooks/nova_compute_context.py'
--- hooks/nova_compute_context.py	2015-09-28 16:28:13 +0000
+++ hooks/nova_compute_context.py	2015-10-01 15:08:20 +0000
@@ -496,3 +496,12 @@
             ctxt['host_ip'] = host_ip
 
         return ctxt
+
+
+class APIRateLimitingContext(context.OSContextGenerator):
+    def __call__(self):
+        ctxt = {}
+        rate_rules = config('api-rate-limit-rules')
+        if rate_rules:
+            ctxt['api_rate_limit_rules'] = rate_rules
+        return ctxt

=== modified file 'hooks/nova_compute_utils.py'
--- hooks/nova_compute_utils.py	2015-09-29 15:14:35 +0000
+++ hooks/nova_compute_utils.py	2015-10-01 15:08:20 +0000
@@ -74,6 +74,7 @@
     CEPH_CONF,
     ceph_config_file,
     HostIPContext,
+    APIRateLimitingContext
 )
 
 CA_CERT_PATH = '/usr/local/share/ca-certificates/keystone_juju_ca_cert.crt'
@@ -150,6 +151,7 @@
 LIBVIRT_BIN = '/etc/default/libvirt-bin'
 LIBVIRT_BIN_OVERRIDES = '/etc/init/libvirt-bin.override'
 NOVA_CONF = '%s/nova.conf' % NOVA_CONF_DIR
+API_PASTE_INI = '%s/api-paste.ini' % NOVA_CONF_DIR
 
 BASE_RESOURCE_MAP = {
     NOVA_CONF: {
@@ -175,6 +177,9 @@
                      HostIPContext(),
                      context.LogLevelContext()],
     },
+    API_PASTE_INI: {
+        'services': ['nova-compute'],
+        'contexts': [APIRateLimitingContext()]}
 }
 
 LIBVIRT_RESOURCE_MAP = {

=== added file 'templates/api-paste.ini'
--- templates/api-paste.ini	1970-01-01 00:00:00 +0000
+++ templates/api-paste.ini	2015-10-01 15:08:20 +0000
@@ -0,0 +1,141 @@
+###############################################################################
+# [ WARNING ]
+# Configuration file maintained by Juju. Local changes may be overwritten.
+###############################################################################
+############
+# Metadata #
+############
+[composite:metadata]
+use = egg:Paste#urlmap
+/: meta
+
+[pipeline:meta]
+pipeline = ec2faultwrap logrequest metaapp
+
+[app:metaapp]
+paste.app_factory = nova.api.metadata.handler:MetadataRequestHandler.factory
+
+#######
+# EC2 #
+#######
+
+[composite:ec2]
+use = egg:Paste#urlmap
+/: ec2cloud
+
+[composite:ec2cloud]
+use = call:nova.api.auth:pipeline_factory
+noauth = ec2faultwrap logrequest ec2noauth cloudrequest validator ec2executor
+noauth2 = ec2faultwrap logrequest ec2noauth cloudrequest validator ec2executor
+keystone = ec2faultwrap logrequest ec2keystoneauth cloudrequest validator ec2executor
+
+[filter:ec2faultwrap]
+paste.filter_factory = nova.api.ec2:FaultWrapper.factory
+
+[filter:logrequest]
+paste.filter_factory = nova.api.ec2:RequestLogging.factory
+
+[filter:ec2lockout]
+paste.filter_factory = nova.api.ec2:Lockout.factory
+
+[filter:ec2keystoneauth]
+paste.filter_factory = nova.api.ec2:EC2KeystoneAuth.factory
+
+[filter:ec2noauth]
+paste.filter_factory = nova.api.ec2:NoAuth.factory
+
+[filter:cloudrequest]
+controller = nova.api.ec2.cloud.CloudController
+paste.filter_factory = nova.api.ec2:Requestify.factory
+
+[filter:authorizer]
+paste.filter_factory = nova.api.ec2:Authorizer.factory
+
+[filter:validator]
+paste.filter_factory = nova.api.ec2:Validator.factory
+
+[app:ec2executor]
+paste.app_factory = nova.api.ec2:Executor.factory
+
+#############
+# OpenStack #
+#############
+
+[composite:osapi_compute]
+use = call:nova.api.openstack.urlmap:urlmap_factory
+/: oscomputeversions
+/v1.1: openstack_compute_api_v2
+/v2: openstack_compute_api_v2
+/v2.1: openstack_compute_api_v21
+/v3: openstack_compute_api_v3
+
+[composite:openstack_compute_api_v2]
+use = call:nova.api.auth:pipeline_factory
+noauth = compute_req_id faultwrap sizelimit noauth ratelimit osapi_compute_app_v2
+noauth2 = compute_req_id faultwrap sizelimit noauth2 ratelimit osapi_compute_app_v2
+keystone = compute_req_id faultwrap sizelimit authtoken keystonecontext ratelimit osapi_compute_app_v2
+keystone_nolimit = compute_req_id faultwrap sizelimit authtoken keystonecontext osapi_compute_app_v2
+
+[composite:openstack_compute_api_v21]
+use = call:nova.api.auth:pipeline_factory_v21
+noauth = compute_req_id faultwrap sizelimit noauth osapi_compute_app_v21
+noauth2 = compute_req_id faultwrap sizelimit noauth2 osapi_compute_app_v21
+keystone = compute_req_id faultwrap sizelimit authtoken keystonecontext osapi_compute_app_v21
+
+[composite:openstack_compute_api_v3]
+use = call:nova.api.auth:pipeline_factory_v21
+noauth = request_id faultwrap sizelimit noauth_v3 osapi_compute_app_v3
+noauth2 = request_id faultwrap sizelimit noauth_v3 osapi_compute_app_v3
+keystone = request_id faultwrap sizelimit authtoken keystonecontext osapi_compute_app_v3
+
+[filter:request_id]
+paste.filter_factory = oslo.middleware:RequestId.factory
+
+[filter:compute_req_id]
+paste.filter_factory = nova.api.compute_req_id:ComputeReqIdMiddleware.factory
+
+[filter:faultwrap]
+paste.filter_factory = nova.api.openstack:FaultWrapper.factory
+
+[filter:noauth]
+paste.filter_factory = nova.api.openstack.auth:NoAuthMiddlewareOld.factory
+
+[filter:noauth2]
+paste.filter_factory = nova.api.openstack.auth:NoAuthMiddleware.factory
+
+[filter:noauth_v3]
+paste.filter_factory = nova.api.openstack.auth:NoAuthMiddlewareV3.factory
+
+[filter:ratelimit]
+paste.filter_factory = nova.api.openstack.compute.limits:RateLimitingMiddleware.factory
+{% if api_rate_limit_rules -%}
+limits = {{ api_rate_limit_rules}}
+{% endif -%}
+
+[filter:sizelimit]
+paste.filter_factory = oslo.middleware:RequestBodySizeLimiter.factory
+
+[app:osapi_compute_app_v2]
+paste.app_factory = nova.api.openstack.compute:APIRouter.factory
+
+[app:osapi_compute_app_v21]
+paste.app_factory = nova.api.openstack.compute:APIRouterV21.factory
+
+[app:osapi_compute_app_v3]
+paste.app_factory = nova.api.openstack.compute:APIRouterV3.factory
+
+[pipeline:oscomputeversions]
+pipeline = faultwrap oscomputeversionapp
+
+[app:oscomputeversionapp]
+paste.app_factory = nova.api.openstack.compute.versions:Versions.factory
+
+##########
+# Shared #
+##########
+
+[filter:keystonecontext]
+paste.filter_factory = nova.api.auth:NovaKeystoneContext.factory
+
+[filter:authtoken]
+paste.filter_factory = keystonemiddleware.auth_token:filter_factory

=== modified file 'tests/basic_deployment.py'
--- tests/basic_deployment.py	2015-07-21 18:02:54 +0000
+++ tests/basic_deployment.py	2015-10-01 15:08:20 +0000
@@ -68,7 +68,9 @@
     def _configure_services(self):
         """Configure all of the services."""
         nova_config = {'config-flags': 'auto_assign_floating_ip=False',
-                       'enable-live-migration': 'False'}
+                       'enable-live-migration': 'False',
+                       'api-rate-limit-rules':
+                            '( POST, *, .*, 9999, MINUTE );'}
         nova_cc_config = {}
         if self.git:
             amulet_http_proxy = os.environ.get('AMULET_HTTP_PROXY')
@@ -466,6 +468,20 @@
                 message = "nova config error: {}".format(ret)
                 amulet.raise_status(amulet.FAIL, msg=message)
 
+    def test_api_paste_config(self):
+        """Check that the rate limiting is set on the nova api (for POSTs)."""
+        unit = self.nova_compute_sentry
+        conf = '/etc/nova/api-paste.ini'
+        section = "filter:ratelimit"
+        factory = ("nova.api.openstack.compute.limits:RateLimitingMiddleware"
+                   ".factory")
+        expected = {"paste.filter_factory": factory,
+                    "limits": "( POST, *, .*, 9999, MINUTE );"}
+        ret = u.validate_config_data(unit, conf, section, expected)
+        if ret:
+            message = "api paste config error: {}".format(ret)
+            amulet.raise_status(amulet.FAIL, msg=message)
+
     def test_image_instance_create(self):
         """Create an image/instance, verify they exist, and delete them."""
         # NOTE(coreycb): Skipping failing test on essex until resolved. essex