Tag evaluation over RPC on rack fails when TLS is enabled

Bug #2054808 reported by Adam Collard
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Committed
High
Anton Troyanov
3.5
Fix Committed
High
Anton Troyanov

Bug Description

https://paste.ubuntu.com/p/JhfPdXHZGw/

Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: provisioningserver.rpc.common: [critical] Unhandled failure dispatching AMP command. This is probably a bug. Please ensure that this error is handled within application code or declared in the signature of the b'EvaluateTag' command. [maas-system-maas:pid=9499:cmd=E
valuateTag:ask=4]
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: Traceback (most recent call last):
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/usr/lib/python3/dist-packages/twisted/internet/asyncioreactor.py", line 271, in _onTimer
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: self.runUntilCurrent()
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/usr/lib/python3/dist-packages/twisted/internet/base.py", line 991, in runUntilCurrent
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: call.func(*call.args, **call.kw)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 700, in errback
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: self._startRunCallbacks(fail)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 763, in _startRunCallbacks
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: self._runCallbacks()
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: --- <exception caught here> ---
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 857, in _runCallbacks
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: current.result = callback( # type: ignore[misc]
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/usr/lib/python3/dist-packages/twisted/protocols/amp.py", line 1138, in checkKnownErrors
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: key = error.trap(*command.allErrors)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/usr/lib/python3/dist-packages/twisted/python/failure.py", line 451, in trap
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: self.raiseException()
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/usr/lib/python3/dist-packages/twisted/python/failure.py", line 475, in raiseException
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: raise self.value.with_traceback(self.tb)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 244, in inContext
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: result = inContext.theWork() # type: ignore[attr-defined]
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 260, in <lambda>
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: inContext.theWork = lambda: context.call( # type: ignore[attr-defined]
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/usr/lib/python3/dist-packages/twisted/python/context.py", line 117, in callWithContext
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: return self.currentContext().callWithContext(ctx, func, *args, **kw)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/usr/lib/python3/dist-packages/twisted/python/context.py", line 82, in callWithContext
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: return func(*args, **kw)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/lib/python3.10/site-packages/provisioningserver/utils/twisted.py", line 203, in wrapper
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: result = func(*args, **kwargs)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/lib/python3.10/site-packages/provisioningserver/rpc/tags.py", line 39, in evaluate_tag
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: process_node_tags(
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/lib/python3.10/site-packages/provisioningserver/tags.py", line 339, in process_node_tags
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: process_all(
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/lib/python3.10/site-packages/provisioningserver/tags.py", line 303, in process_all
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: nodes_matched, nodes_unmatched = classify(
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/lib/python3.10/site-packages/provisioningserver/utils/__init__.py", line 75, in classify
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: for ident, subject in subjects:
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/lib/python3.10/site-packages/provisioningserver/tags.py", line 279, in gen_node_details
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: for system_id, details in get_details(batch).items():
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/lib/python3.10/site-packages/provisioningserver/tags.py", line 78, in get_details_for_nodes
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: data = process_response(client.get(path, op="details"))
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/lib/python3.10/site-packages/apiclient/maas_client.py", line 270, in get
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: return self.dispatcher.dispatch_query(
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/snap/maas/33772/lib/python3.10/site-packages/apiclient/maas_client.py", line 122, in dispatch_query
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: res = opener.open(req)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/usr/lib/python3.10/urllib/request.py", line 525, in open
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: response = meth(req, response)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: response = self.parent.error(
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/usr/lib/python3.10/urllib/request.py", line 557, in error
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: result = self._call_chain(*args)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: result = func(*args)
Feb 23 12:05:35 maas-system-maas maas-log[9784]: maas.tags: [error] Tag all (true()) could not be evaluated on rack controller maas-system-maas (nbmwp6): (b'UNHANDLED', 'Unknown Error [maas-system-maas:pid=9499:cmd=EvaluateTag:ask=4]')
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/usr/lib/python3.10/urllib/request.py", line 749, in http_error_302
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: return self.parent.open(new, timeout=req.timeout)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/usr/lib/python3.10/urllib/request.py", line 519, in open
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: response = self._open(req, data)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/usr/lib/python3.10/urllib/request.py", line 536, in _open
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: result = self._call_chain(self.handle_open, protocol, protocol +
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: result = func(*args)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/usr/lib/python3.10/urllib/request.py", line 1391, in https_open
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: return self.do_open(http.client.HTTPSConnection, req,
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: File "/usr/lib/python3.10/urllib/request.py", line 1351, in do_open
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: raise URLError(err)
Feb 23 12:05:35 maas-system-maas maas-rackd[9499]: urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)>

Related branches

description: updated
description: updated
Changed in maas:
status: New → Triaged
milestone: none → 3.5.0
importance: Undecided → High
Changed in maas:
assignee: nobody → Anton Troyanov (troyanov)
no longer affects: maas/3.6
Changed in maas:
milestone: 3.5.0 → 3.6.0
Changed in maas:
status: Triaged → In Progress
Changed in maas:
status: In Progress → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.