rack controller fails to register because of failure updating interfaces

Bug #1863597 reported by Alberto Donato
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Medium
Alberto Donato
2.7
Fix Released
Medium
Alberto Donato

Bug Description

Running a local development maas from master, I got into a situation where the rackd fails to register to regiond because of a duplicated interface name.

The error is as follows:

2020-02-17 08:54:35 maasserver.rpc.regionservice: [info] Rack controller 'None' disconnected.
2020-02-17 08:54:35 RegionServer,51,::ffff:172.16.99.1: [info] RegionServer connection lost (HOST:IPv6Address(TCP, '::ffff:172.16.99.1', 5251) PEER:IPv6Address(TCP, '::ffff:172.16.99.1', 37082))
2020-02-17 08:54:35 maasserver.rpc.regionservice: [critical] Failed to register rack controller 'None' with the master. Connection will be dropped.
        Traceback (most recent call last):
          File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1442, in gotResult
            _inlineCallbacks(r, g, deferred)
          File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1384, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/failure.py", line 408, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
        --- <exception caught here> ---
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/rpc/regionservice.py", line 743, in _register
            version=version,
          File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
            result = inContext.theWork()
          File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
            inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
          File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
            return self.currentContext().callWithContext(ctx, func, *args, **kw)
          File "/snap/maas/x1/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
            return func(*args,**kw)
          File "/snap/maas/x1/lib/python3.6/site-packages/provisioningserver/utils/twisted.py", line 885, in callInContext
            return func(*args, **kwargs)
          File "/snap/maas/x1/lib/python3.6/site-packages/provisioningserver/utils/twisted.py", line 217, in wrapper
            result = func(*args, **kwargs)
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/utils/orm.py", line 692, in call_with_connection
            return func(*args, **kwargs)
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/utils/__init__.py", line 193, in call_with_lock
            return func(*args, **kwargs)
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/utils/orm.py", line 737, in call_within_transaction
            return func_outside_txn(*args, **kwargs)
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/utils/orm.py", line 540, in retrier
            return func(*args, **kwargs)
          File "/snap/maas/x1/usr/lib/python3.6/contextlib.py", line 52, in inner
            return func(*args, **kwds)
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/rpc/rackcontrollers.py", line 168, in register
            rackcontroller.update_interfaces(interfaces, create_fabrics=create_fabrics)
          File "/snap/maas/x1/lib/python3.6/site-packages/provisioningserver/utils/twisted.py", line 217, in wrapper
            result = func(*args, **kwargs)
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/utils/orm.py", line 692, in call_with_connection
            return func(*args, **kwargs)
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/utils/__init__.py", line 193, in call_with_lock
            return func(*args, **kwargs)
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/utils/orm.py", line 721, in call_within_transaction
            return func_within_txn(*args, **kwargs)
          File "/snap/maas/x1/usr/lib/python3.6/contextlib.py", line 52, in inner
            return func(*args, **kwds)
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/models/node.py", line 6827, in update_interfaces
            hints=topology_hints,
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/models/node.py", line 6130, in _update_interface
            name, config, create_fabrics=create_fabrics, hints=hints
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/models/node.py", line 6162, in _update_physical_interface
            defaults={"node": self, "name": name, "enabled": is_enabled},
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/models/interface.py", line 432, in get_or_create
            *args, **kwargs
          File "/snap/maas/x1/usr/lib/python3/dist-packages/django/db/models/manager.py", line 85, in manager_method
            return getattr(self.get_queryset(), name)(*args, **kwargs)
          File "/snap/maas/x1/usr/lib/python3/dist-packages/django/db/models/query.py", line 466, in get_or_create
            return self._create_object_from_params(lookup, params)
          File "/snap/maas/x1/usr/lib/python3/dist-packages/django/db/models/query.py", line 498, in _create_object_from_params
            obj = self.create(**params)
          File "/snap/maas/x1/usr/lib/python3/dist-packages/django/db/models/query.py", line 394, in create
            obj.save(force_insert=True, using=self.db)
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/models/interface.py", line 1643, in save
            return super().save(*args, **kwargs)
          File "/snap/maas/x1/lib/python3.6/site-packages/maasserver/models/cleansave.py", line 187, in save
            self.validate_unique(exclude=[self._meta.pk.name])
          File "/snap/maas/x1/usr/lib/python3/dist-packages/django/db/models/base.py", line 1041, in validate_unique
            raise ValidationError(errors)
        django.core.exceptions.ValidationError: {'__all__': ['Interface with this Node and Name already exists.']}

The issue is the mpqemubr0-dummy interface (created by multipass), which somehow now got a different MAC address (but still has the previous name).

In MAAS' database an interface already existed for the node with that name, but a different MAC. The current logic looks up the MAC, doesn't find it, so tries to create a new interface, but the name is duplicated, so the constraint fails.

We should rework the Node.update_interfaces() logic so that it first figures out which interfaces need to be delete and which to be updated, delete first, then update and create new ones.

Related branches

Alberto Donato (ack)
Changed in maas:
milestone: none → 2.8.0b1
Alberto Donato (ack)
Changed in maas:
assignee: nobody → Alberto Donato (ack)
status: Triaged → In Progress
Changed in maas:
status: In Progress → Fix Committed
Alberto Donato (ack)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.