Bug #1841807 “maas cli: rack-controller list-boot-images Connect...” : Bugs : MAAS

Adam Collard (adam-collard) on 2019-09-09

Changed in maas:
importance:	Undecided → Medium

Adam Collard (adam-collard) on 2019-09-09

Changed in maas:
status:	New → Triaged

Revision history for this message

Blake Rouse (blake-rouse) wrote on 2019-09-09:

#1

Download full text (4.0 KiB)

This seems to be more of a network issue in your lab, digging through the logs I was able to determine this:

2019-08-27 21:33:02 provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: leafeon:pid=9545
2019-08-27 21:33:02 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(TCP, '::ffff:192.168.33.23', 53010) PEER:IPv6Address(TCP, '::ffff:192.168.33.22', 5251))
2019-08-27 21:33:02 provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: swoobat:pid=9969
2019-08-27 21:33:02 provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: swoobat:pid=9961
2019-08-27 21:33:02 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(TCP, '::ffff:192.168.33.23', 54470) PEER:IPv6Address(TCP, '::ffff:192.168.33.24', 5252))
2019-08-27 21:33:02 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(TCP, '::ffff:192.168.33.23', 52882) PEER:IPv6Address(TCP, '::ffff:192.168.33.24', 5251))
2019-08-27 21:33:03 provisioningserver.rpc.clusterservice: [info] Making connections to event-loops: leafeon:pid=9545, swoobat:pid=9961, swoobat:pid=9969
2019-08-27 21:33:05 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04-edge:xenial: to_add=['20190822'] to_remove=[]
2019-08-27 21:33:09 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04-lowlatency-edge:xenial: to_add=['20190822'] to_remove=[]
2019-08-27 21:33:13 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04-lowlatency:xenial: to_add=['20190822'] to_remove=[]
2019-08-27 21:33:18 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04:xenial: to_add=['20190822'] to_remove=[]
2019-08-27 21:33:18 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(TCP, '::ffff:192.168.33.23', 53780) PEER:IPv6Address(TCP, '::ffff:192.168.33.24', 5251))
2019-08-27 21:33:18 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(TCP, '::ffff:192.168.33.23', 53900) PEER:IPv6Address(TCP, '::ffff:192.168.33.22', 5251))
2019-08-27 21:33:18 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(TCP, '::ffff:192.168.33.23', 55368) PEER:IPv6Address(TCP, '::ffff:192.168.33.24', 5252))
2019-08-27 21:33:18 provisioningserver.rpc.clusterservice: [info] Event-loop 'leafeon:pid=9545' authenticated.
2019-08-27 21:33:18 provisioningserver.rpc.clusterservice: [info] Event-loop 'swoobat:pid=9961' authenticated.
2019-08-27 21:33:18 provisioningserver.rpc.clusterservice: [info] Event-loop 'swoobat:pid=9969' authenticated.
2019-08-27 21:33:18 provisioningserver.rpc.clusterservice: [info] Rack controller 'ba6gn7' registered (via leafeon:pid=9545) with MAAS version 2.6.0-7802-g59416a869-0ubuntu1~18.04.1.
2019-08-27 21:33:19 provisioningserver.rpc.clusterservice: [info] Rack controller 'ba6gn7' registered (via swoobat:pid=9969) with MAAS version 2.6.0-7802-g59416a869-0ubuntu1~18.04.1.
2019-08-27 21:33:19 provisioningserver.rpc.clusterservice: [info] Rack controller 'ba6gn7' registered (via swoobat:pid=9961) with MAAS version 2.6.0-7802-g59416a869-0ubuntu1~1...

This seems to be more of a network issue in your lab, digging through the logs I was able to determine this:

2019-08-27 21:33:02 provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: leafeon:pid=9545
2019-08-27 21:33:02 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(TCP, '::ffff:192.168.33.23', 53010) PEER:IPv6Address(TCP, '::ffff:192.168.33.22', 5251))
2019-08-27 21:33:02 provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: swoobat:pid=9969
2019-08-27 21:33:02 provisioningserver.rpc.clusterservice: [info] Failure on ping dropping connection to event-loop: swoobat:pid=9961
2019-08-27 21:33:02 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(TCP, '::ffff:192.168.33.23', 54470) PEER:IPv6Address(TCP, '::ffff:192.168.33.24', 5252))
2019-08-27 21:33:02 ClusterClient,client: [info] ClusterClient connection lost (HOST:IPv6Address(TCP, '::ffff:192.168.33.23', 52882) PEER:IPv6Address(TCP, '::ffff:192.168.33.24', 5251))
2019-08-27 21:33:03 provisioningserver.rpc.clusterservice: [info] Making connections to event-loops: leafeon:pid=9545, swoobat:pid=9961, swoobat:pid=9969
2019-08-27 21:33:05 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04-edge:xenial: to_add=['20190822'] to_remove=[]
2019-08-27 21:33:09 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04-lowlatency-edge:xenial: to_add=['20190822'] to_remove=[]
2019-08-27 21:33:13 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04-lowlatency:xenial: to_add=['20190822'] to_remove=[]
2019-08-27 21:33:18 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-16.04:xenial: to_add=['20190822'] to_remove=[]
2019-08-27 21:33:18 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(TCP, '::ffff:192.168.33.23', 53780) PEER:IPv6Address(TCP, '::ffff:192.168.33.24', 5251))
2019-08-27 21:33:18 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(TCP, '::ffff:192.168.33.23', 53900) PEER:IPv6Address(TCP, '::ffff:192.168.33.22', 5251))
2019-08-27 21:33:18 Uninitialized: [info] ClusterClient connection established (HOST:IPv6Address(TCP, '::ffff:192.168.33.23', 55368) PEER:IPv6Address(TCP, '::ffff:192.168.33.24', 5252))
2019-08-27 21:33:18 provisioningserver.rpc.clusterservice: [info] Event-loop 'leafeon:pid=9545' authenticated.
2019-08-27 21:33:18 provisioningserver.rpc.clusterservice: [info] Event-loop 'swoobat:pid=9961' authenticated.
2019-08-27 21:33:18 provisioningserver.rpc.clusterservice: [info] Event-loop 'swoobat:pid=9969' authenticated.
2019-08-27 21:33:18 provisioningserver.rpc.clusterservice: [info] Rack controller 'ba6gn7' registered (via leafeon:pid=9545) with MAAS version 2.6.0-7802-g59416a869-0ubuntu1~18.04.1.
2019-08-27 21:33:19 provisioningserver.rpc.clusterservice: [info] Rack controller 'ba6gn7' registered (via swoobat:pid=9969) with MAAS version 2.6.0-7802-g59416a869-0ubuntu1~18.04.1.
2019-08-27 21:33:19 provisioningserver.rpc.clusterservice: [info] Rack controller 'ba6gn7' registered (via swoobat:pid=9961) with MAAS version 2.6.0-7802-g59416a869-0ubuntu1~18.04.1.
2019-08-27 21:33:21 sstreams: [info] maas:v2:download/maas:boot:ubuntu:amd64:hwe-18.04-edge:bionic: to_add=['20190822'] to_remove=[]
2019-08-27 21:33:22 provisioningserver.rpc.clusterservice: [info] Fully connected to all 12 event-loops on all 3 region controllers (leafeon, meinfoo, swoobat).

The rackd does a ping check often to the region controller and noticed that it was not able to get a ping response from that region controller. So it dropped the connection, that connection drop is the reason you got the error from the CLI.

The CLI is using the RPC connection to the rack controller to list the boot images, but that TCP connection is dropped by the rack controller because it was not able to ping. I included the remaining of the log in the snippet above, where you see that the rackd then reconnected cleanly and was fully connected.

Seems to me something in the network connection between the regiond and rackd bounced.

Blake Rouse (blake-rouse) on 2019-09-10

Changed in maas:
assignee:	nobody → Blake Rouse (blake-rouse)
milestone:	none → 2.7.0alpha1

MAAS Lander (maas-lander) on 2019-09-11

Changed in maas:
status:	Triaged → Fix Committed

Revision history for this message

Jason Hobbs (jason-hobbs) wrote on 2019-10-22:

#2

Can you please clarify your comment? Is this a network issue on our side or a bug in maas? I see there is a commit landed for it.

We're seeing this quite a bit on 2.6. This should be considered for backport to 2.6.

Adam Collard (adam-collard) on 2019-12-04

Changed in maas:
status:	Fix Committed → Fix Released

MAAS

maas cli: rack-controller list-boot-images Connection to the other side was lost in a non-clean fashion

Bug Description

Related branches

Other bug subscribers

Remote bug watches