when TLS is enabled racks try to connect to region through https

Bug #1980269 reported by Diego Mascialino
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Invalid
Medium
Unassigned
MAAS CI
Fix Released
Undecided
Adam Collard

Bug Description

We are executing system-tests wih TLS enabled.

Test are passing OK, but I found this reading config and logs:

/etc/maas/rackd.conf
```
cluster_uuid: 62f103ef-bbb5-49e4-9149-b193da51dd55
debug: true
maas_url:
- https://maas.under.test:5443/MAAS/
```

/var/log/maas/rackd.log
```
2022-06-27 22:41:36 provisioningserver.rpc.clusterservice: [critical] Failed to contact region. (While requesting RPC info at https://maas.under.test:5443/MAAS/).
        Traceback (most recent call last):
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 460, in callback
            self._startRunCallbacks(result)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
            self._runCallbacks()
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 654, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1475, in gotResult
            _inlineCallbacks(r, g, status)
        --- <exception caught here> ---
          File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1292, in _doUpdate
            eventloops, maas_url = yield self._get_rpc_info(urls)
          File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1549, in _get_rpc_info
            raise config_exc
          File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1520, in _get_rpc_info
            eventloops, maas_url = yield self._parallel_fetch_rpc_info(urls)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 654, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1494, in handle_responses
            errors[0].raiseException()
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 467, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1455, in _serial_fetch_rpc_info
            raise last_exc
          File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1447, in _serial_fetch_rpc_info
            response = yield self._fetch_rpc_info(url, orig_url)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
            result = g.send(result)
          File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1549, in _get_rpc_info
            raise config_exc
          File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1520, in _get_rpc_info
            eventloops, maas_url = yield self._parallel_fetch_rpc_info(urls)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 654, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1494, in handle_responses
            errors[0].raiseException()
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 467, in raiseException
            raise self.value.with_traceback(self.tb)
          File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1455, in _serial_fetch_rpc_info
            raise last_exc
          File "/usr/lib/python3/dist-packages/provisioningserver/rpc/clusterservice.py", line 1447, in _serial_fetch_rpc_info
            response = yield self._fetch_rpc_info(url, orig_url)
        twisted.web._newclient.ResponseNeverReceived: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]>]
```

and also:

```
2022-06-27 22:41:42 provisioningserver.rpc.clusterservice: [info] Making connections to event-loops: maas-system-maas:pid=19046, maas-system-maas:pid=19048, maas-system-maas:pid=19049
2022-06-27 22:41:42 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.157.204.177', port=44806, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.157.204.177', port=5252, flowInfo=0, scopeID=0))
2022-06-27 22:41:42 provisioningserver.rpc.common: [debug] [RPC -> sent] AmpBox({b'message': b'X\xc0\x05\x9eG\x14|0\xb5"\xf4\xf7mO\xd3\xaf', b'_command': b'Authenticate'})
2022-06-27 22:41:42 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.157.204.177', port=51706, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.157.204.177', port=5251, flowInfo=0, scopeID=0))
2022-06-27 22:41:42 provisioningserver.rpc.common: [debug] [RPC -> sent] AmpBox({b'message': b'\xe2\x00v\xb8\xd87\xc8K\x17\xe0\x03\\M(\xce\xb7', b'_command': b'Authenticate'})
2022-06-27 22:41:42 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(type='TCP', host='::ffff:10.245.136.20', port=39022, flowInfo=0, scopeID=0) PEER:IPv6Address(type='TCP', host='::ffff:10.245.136.20', port=5253, flowInfo=0, scopeID=0))
2022-06-27 22:41:42 provisioningserver.rpc.common: [debug] [RPC -> sent] AmpBox({b'message': b'\xec\x13\xc0\xd0R8p\ri\x01\xac\x89\xd0\xcb$.', b'_command': b'Authenticate'})
2022-06-27 22:41:42 provisioningserver.rpc.common: [debug] [RPC <- received] AmpBox({b'_ask': b'1', b'_command': b'Authenticate', b'message': b"B\xe5)i\x8b\xc4\xda\xcc'\x8d\xa7k\xe2\x17\xc9\xd7"})
2022-06-27 22:41:42 provisioningserver.rpc.common: [debug] [RPC -> responding] AmpBox({b'digest': b'<I0T\x01/p\x85\x9c\x82M\xcd\x10\x1f_)\xcf\xaa\x91\x167\xd4\xe0\x17\x8cj-j\xc5\xd6\xb3H', b'salt': b'\x95\x04p\x0e!\xfd4\x80\xf1c|\xd7\x1e\x84\xc2\xf6', b'_answer': b'1'})
2022-06-27 22:41:42 provisioningserver.rpc.common: [debug] [RPC <- received] AmpBox({b'_ask': b'1', b'_command': b'Authenticate', b'message': b'\tq\xb7]\xba\x17-,\xc7S\x8e\xf7<-\xa8~'})
2022-06-27 22:41:42 provisioningserver.rpc.common: [debug] [RPC -> responding] AmpBox({b'digest': b'\x8eu\xee\xdf\xe0\x04\x7f\xa8\x93\xb5\xeb\xfeL\x86\xe2>\x10\xb2/1\x94;O\x83\xfb\xd0B\x1cI\x06\xff\xc8', b'salt': b'\xd6\xfc\x01d\xd6\x15\xcd\xdeY\xa2R}\x15\xf0\xe2-', b'_answer': b'1'})
2022-06-27 22:41:42 provisioningserver.rpc.common: [debug] [RPC <- recieved] AmpBox({b'_answer': b'1', b'digest': b'\xcaQ\xc7vA\xfc\xa0tX\xcc$\x06\x9f\xfa\xeb\xce\xce\xe2N\xaeco\xd58\xf1Ekq\xf1\xd7\x15\x16', b'salt': b'xkW\xc8\xf26JT\xf8g3^\x19({{'})
2022-06-27 22:41:42 provisioningserver.rpc.clusterservice: [info] Event-loop 'maas-system-maas:pid=19049' authenticated.
```

Here are the artifacts of this execution:
http://maas-integration-ci.internal:8080/job/maas-system-tests/1350/artifact/

Changed in maas:
status: New → Triaged
Revision history for this message
Anton Troyanov (troyanov) wrote :

It happens because `rackd.conf` contains wrong URL and we didn't try to cover the case when rackd can reach regiond over HTTPS.

I think the simple workaround for now is to set a non-HTTPS URL in the rackd.config

One problem that I see is that for rack+region setup, that config is populated with URL from the database, which is HTTPS (because it was enabled)

Changed in maas:
importance: Undecided → Medium
milestone: none → 3.2.1
milestone: 3.2.1 → 3.3.0
no longer affects: maas/3.2
Changed in maas:
status: Triaged → Invalid
Changed in maas-ci:
status: New → Incomplete
status: Incomplete → Confirmed
assignee: nobody → Adam Collard (adam-collard)
Revision history for this message
Adam Collard (adam-collard) wrote :

This is a bug in the system-tests, where they are (wrongly) calling `maas config --maas-url=https://...`

Changed in maas-ci:
status: Confirmed → Fix Released
Changed in maas:
milestone: 3.3.0 → none
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.