[2.2rc1] rackd — 75% connected to region controllers.

Bug #1680277 reported by Andres Rodriguez
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Blake Rouse

Bug Description

The rack controller constantly showing the " rackd — 75% connected to region controllers." message. However, this may just be a UI message provided that there are no tracebacks on the logs.

2017-04-05 22:51:47 regiond: [info] 127.0.0.1 GET /MAAS/api/2.0/machines/ HTTP/1.1 --> 200 OK (referrer: -; agent: Python-urllib/2.7)
2017-04-05 22:52:00 maasserver.regiondservices.service_monitor_service: [critical] Failed to monitor services and update database.

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 434, in errback
    self._startRunCallbacks(fail)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
    self._runCallbacks()
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/usr/lib/python3/dist-packages/maasserver/regiondservices/service_monitor_service.py", line 60, in _updateDatabase
    self._saveIntoDatabase, advertising.process_id, services)
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246, in inContext
    result = inContext.theWork()
  File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 863, in callInContext
    return func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 231, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 686, in call_within_transaction
    return func_outside_txn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 504, in retrier
    return func(*args, **kwargs)
  File "/usr/lib/python3.5/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3/dist-packages/maasserver/regiondservices/service_monitor_service.py", line 66, in _saveIntoDatabase
    process = RegionControllerProcess.objects.get(id=processId)
  File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 334, in get
    self.model._meta.object_name
maasserver.models.regioncontrollerprocess.DoesNotExist: RegionControllerProcess matching query does not exist.

Related branches

no longer affects: maas (Ubuntu)
Changed in maas:
milestone: none → 2.2.0rc2
importance: Undecided → Critical
Revision history for this message
Brendan Donegan (brendan-donegan) wrote :

Does this happen all the time or only some of the time? I don't think CI would pass if it was all the time. I have also seen similar in CI, but saying '25% connected'

Changed in maas:
status: New → Incomplete
Revision history for this message
Andres Rodriguez (andreserl) wrote :

This is on a system that has tons of machines, and there's a valid traceback there. This should not be marked as incomplete.

Changed in maas:
status: Incomplete → Triaged
Revision history for this message
Andres Rodriguez (andreserl) wrote :

fwiw, this happens constantly.

Changed in maas:
assignee: nobody → Blake Rouse (blake-rouse)
Changed in maas:
status: Triaged → In Progress
Changed in maas:
status: In Progress → Fix Committed
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.