Im having similar problem. Deployed Mitaka on Trusty with MaaS 1.94 and Juju 1.25.6. In the log for nova-api-metadata service when VM boots and requests instance metadata: --- LOG --- 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler [req-28a7fa71-aa04-4e9e-99c1-6b251a396174 - - - - -] Failed to get metadata for instance id: 7bd7798b-9491-47b4-a972-7d979a26dd20 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler Traceback (most recent call last): 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/nova/api/metadata/handler.py", line 311, in _get_meta_by_instance_id 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler remote_address) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/nova/api/metadata/handler.py", line 106, in get_metadata_by_instance_id 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler data = base.get_metadata_by_instance_id(instance_id, address) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/nova/api/metadata/base.py", line 551, in get_metadata_by_instance_id 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler return InstanceMetadata(instance, address) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/nova/api/metadata/base.py", line 134, in __init__ 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler ctxt, instance) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 417, in get_instance_security_groups 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler context, servers, detailed) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 381, in get_instances_security_groups_bindings 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler ports = self._get_ports_from_server_list(servers, neutron) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/nova/network/security_group/neutron_driver.py", line 340, in _get_ports_from_server_list 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler ports.extend(neutron.list_ports(**search_opts).get('ports')) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 97, in with_params 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler ret = self.function(instance, *args, **kwargs) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 644, in list_ports 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler **_params) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 373, in list 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler for r in self._pagination(collection, path, **params): 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 388, in _pagination 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler res = self.get(path, params=params) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 358, in get 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler headers=headers, params=params) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 335, in retry_request 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler headers=headers, params=params) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 286, in do_request 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler resp, replybody = self.httpclient.do_request(action, method, body=body) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 305, in do_request 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler self._check_uri_length(url) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 298, in _check_uri_length 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler uri_len = len(self.endpoint_url) + len(url) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 312, in endpoint_url 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler return self.get_endpoint() 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 132, in get_endpoint 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler return self.session.get_endpoint(auth or self.auth, **kwargs) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 661, in get_endpoint 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler return auth.get_endpoint(self, **kwargs) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 210, in get_endpoint 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler service_catalog = self.get_access(session).service_catalog 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 136, in get_access 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler self.auth_ref = self.get_auth_ref(session) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 179, in get_auth_ref 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler self._plugin = self._do_create_plugin(session) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/generic/base.py", line 118, in _do_create_plugin 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler authenticated=False) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler return func(*args, **kwargs) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneauth1/identity/base.py", line 308, in get_discovery 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler authenticated=authenticated) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler return func(*args, **kwargs) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneauth1/discover.py", line 141, in __init__ 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler authenticated=authenticated) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler return func(*args, **kwargs) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneauth1/discover.py", line 40, in get_version_data 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler resp = session.get(url, headers=headers, authenticated=authenticated) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 564, in get 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler return self.request(url, 'GET', **kwargs) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler return func(*args, **kwargs) 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 394, in request 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler raise exceptions.EndpointNotFound() 2016-08-31 22:59:42.347 24078 ERROR nova.api.metadata.handler EndpointNotFound: Could not find requested endpoint in Service Catalog. --- LOG ---
nova-api-metadata service nova.conf has: --- EXAMPLE --- [neutron] url= auth_url=://: auth_type=password project_domain_name=default user_domain_name=default region= project_name= username= password= service_metadata_proxy=True metadata_proxy_shared_secret=xxxxxxx... --- EXAMPLE ---
While nova-cloud-controller nova.conf has: --- EXAMPLE --- [neutron] url = http://<ip>:9696 auth_strategy = keystone auth_section = keystone_authtoken auth_plugin = password [keystone_authtoken] auth_uri = http://<ip>:5000 auth_url = http://<ip>:35357 auth_type = password project_domain_name = default user_domain_name = default project_name = services username = nova password = <pass> signing_dir = /var/cache/nova --- EXAMPLE ---
Merging this back to nova-api-metadata nova.conf seems to cure the problem.
Im having similar problem. Deployed Mitaka on Trusty with MaaS 1.94 and Juju 1.25.6. metadata. handler [req-28a7fa71- aa04-4e9e- 99c1-6b251a3961 74 - - - - -] Failed to get metadata for instance id: 7bd7798b- 9491-47b4- a972-7d979a26dd 20 metadata. handler Traceback (most recent call last): metadata. handler File "/usr/lib/ python2. 7/dist- packages/ nova/api/ metadata/ handler. py", line 311, in _get_meta_ by_instance_ id metadata. handler remote_address) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ nova/api/ metadata/ handler. py", line 106, in get_metadata_ by_instance_ id metadata. handler data = base.get_ metadata_ by_instance_ id(instance_ id, address) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ nova/api/ metadata/ base.py" , line 551, in get_metadata_ by_instance_ id metadata. handler return InstanceMetadat a(instance, address) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ nova/api/ metadata/ base.py" , line 134, in __init__ metadata. handler ctxt, instance) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ nova/network/ security_ group/neutron_ driver. py", line 417, in get_instance_ security_ groups metadata. handler context, servers, detailed) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ nova/network/ security_ group/neutron_ driver. py", line 381, in get_instances_ security_ groups_ bindings metadata. handler ports = self._get_ ports_from_ server_ list(servers, neutron) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ nova/network/ security_ group/neutron_ driver. py", line 340, in _get_ports_ from_server_ list metadata. handler ports.extend( neutron. list_ports( **search_ opts).get( 'ports' )) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 97, in with_params metadata. handler ret = self.function( instance, *args, **kwargs) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 644, in list_ports metadata. handler **_params) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 373, in list metadata. handler for r in self._paginatio n(collection, path, **params): metadata. handler File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 388, in _pagination metadata. handler res = self.get(path, params=params) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 358, in get metadata. handler headers=headers, params=params) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 335, in retry_request metadata. handler headers=headers, params=params) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ v2_0/client. py", line 286, in do_request metadata. handler resp, replybody = self.httpclient .do_request( action, method, body=body) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ client. py", line 305, in do_request metadata. handler self._check_ uri_length( url) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ client. py", line 298, in _check_uri_length metadata. handler uri_len = len(self. endpoint_ url) + len(url) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ neutronclient/ client. py", line 312, in endpoint_url metadata. handler return self.get_endpoint() metadata. handler File "/usr/lib/ python2. 7/dist- packages/ keystoneauth1/ adapter. py", line 132, in get_endpoint metadata. handler return self.session. get_endpoint( auth or self.auth, **kwargs) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ keystoneauth1/ session. py", line 661, in get_endpoint metadata. handler return auth.get_ endpoint( self, **kwargs) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ keystoneauth1/ identity/ base.py" , line 210, in get_endpoint metadata. handler service_catalog = self.get_ access( session) .service_ catalog metadata. handler File "/usr/lib/ python2. 7/dist- packages/ keystoneauth1/ identity/ base.py" , line 136, in get_access metadata. handler self.auth_ref = self.get_ auth_ref( session) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ keystoneauth1/ identity/ generic/ base.py" , line 179, in get_auth_ref metadata. handler self._plugin = self._do_ create_ plugin( session) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ keystoneauth1/ identity/ generic/ base.py" , line 118, in _do_create_plugin metadata. handler authenticated= False) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ positional/ __init_ _.py", line 94, in inner metadata. handler return func(*args, **kwargs) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ keystoneauth1/ identity/ base.py" , line 308, in get_discovery metadata. handler authenticated= authenticated) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ positional/ __init_ _.py", line 94, in inner metadata. handler return func(*args, **kwargs) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ keystoneauth1/ discover. py", line 141, in __init__ metadata. handler authenticated= authenticated) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ positional/ __init_ _.py", line 94, in inner metadata. handler return func(*args, **kwargs) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ keystoneauth1/ discover. py", line 40, in get_version_data metadata. handler resp = session.get(url, headers=headers, authenticated= authenticated) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ keystoneauth1/ session. py", line 564, in get metadata. handler return self.request(url, 'GET', **kwargs) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ positional/ __init_ _.py", line 94, in inner metadata. handler return func(*args, **kwargs) metadata. handler File "/usr/lib/ python2. 7/dist- packages/ keystoneauth1/ session. py", line 394, in request metadata. handler raise exceptions. EndpointNotFoun d() metadata. handler EndpointNotFound: Could not find requested endpoint in Service Catalog.
In the log for nova-api-metadata service when VM boots and requests instance metadata:
--- LOG ---
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
2016-08-31 22:59:42.347 24078 ERROR nova.api.
--- LOG ---
nova-api-metadata service nova.conf has: domain_ name=default name=default metadata_ proxy=True proxy_shared_ secret= xxxxxxx. ..
--- EXAMPLE ---
[neutron]
url=
auth_url=://:
auth_type=password
project_
user_domain_
region=
project_name=
username=
password=
service_
metadata_
--- EXAMPLE ---
While nova-cloud- controller nova.conf has: authtoken]
--- EXAMPLE ---
[neutron]
url = http://<ip>:9696
auth_strategy = keystone
auth_section = keystone_authtoken
auth_plugin = password
[keystone_
auth_uri = http://<ip>:5000
auth_url = http://<ip>:35357
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = services
username = nova
password = <pass>
signing_dir = /var/cache/nova
--- EXAMPLE ---
Merging this back to nova-api-metadata nova.conf seems to cure the problem.