MAAS

RackControllerService flooding log with: 'RegionAdvertisingService' object has no attribute 'processId'

Bug #1578800 reported by Lee Trager
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Gavin Panella
MAAS 2.0.0

Bug Description

I pulled trunk today and I'm noticing that regiond.log is being flooded with exceptions from the RackControllerService. This happens in both the dev(make test) and a live environment.

==> logs/regiond/current <==
2016-05-05 19:51:54 [-] Log opened.
2016-05-05 19:51:54 [-] twistd 16.0.0 (/usr/bin/python3.5 3.5.1) starting up.
2016-05-05 19:51:54 [-] reactor class: twisted.internet.epollreactor.EPollReactor.
2016-05-05 19:51:54 [-] Site starting on 5240
2016-05-05 19:51:54 [-] Factory starting on 5250
2016-05-05 19:51:54 [-] Traceback (most recent call last):
2016-05-05 19:51:54 [-] File "/home/lee/maas/trunk/bin/twistd.region", line 40, in <module>
2016-05-05 19:51:54 [-] sys.exit(twisted.scripts.twistd.run())
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/scripts/twistd.py", line 29, in run
2016-05-05 19:51:54 [-] app.run(runApp, ServerOptions)
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 617, in run
2016-05-05 19:51:54 [-] runApp(config)
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/scripts/twistd.py", line 25, in runApp
2016-05-05 19:51:54 [-] _SomeApplicationRunner(config).run()
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 352, in run
2016-05-05 19:51:54 [-] self.postApplication()
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/scripts/_twistd_unix.py", line 207, in postApplication
2016-05-05 19:51:54 [-] self.startApplication(self.application)
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/scripts/_twistd_unix.py", line 403, in startApplication
2016-05-05 19:51:54 [-] app.startApplication(application, not self.config['no_save'])
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/application/app.py", line 633, in startApplication
2016-05-05 19:51:54 [-] service.IService(application).startService()
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/application/service.py", line 283, in startService
2016-05-05 19:51:54 [-] service.startService()
2016-05-05 19:51:54 [-] File "/usr/lib/python3/dist-packages/twisted/application/service.py", line 283, in startService
2016-05-05 19:51:54 [-] service.startService()
2016-05-05 19:51:54 [-] File "/home/lee/maas/trunk/src/provisioningserver/utils/twisted.py", line 140, in wrapper
2016-05-05 19:51:54 [-] return func(*args, **kwargs)
2016-05-05 19:51:54 [-] File "/home/lee/maas/trunk/src/maasserver/rack_controller.py", line 125, in startService
2016-05-05 19:51:54 [-] self.starting = self.advertisingService.processId.get()
2016-05-05 19:51:54 [-] AttributeError: 'RegionAdvertisingService' object has no attribute 'processId'
/home/lee/maas/trunk/src/maasserver/websockets/protocol.py:27: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9.
  from django.utils.importlib import import_module

Related branches

lp:~allenap/maas/rack-controller-service--bug-1578800
Mike Pontillo (community): Approve
Diff: 233 lines (+51/-37)
2 files modified
src/maasserver/rack_controller.py (+5/-5)
src/maasserver/tests/test_rack_controller.py (+46/-32)
lp:~allenap/maas/rack-controller-service--bug-1578800--service-monitor
LaMont Jones (community): Approve
Diff: 157 lines (+26/-27)
3 files modified
src/maasserver/service_monitor.py (+3/-2)
src/maasserver/tests/test_rack_controller.py (+14/-15)
src/maasserver/tests/test_service_monitor.py (+9/-10)
Lee Trager (ltrager)
summary: - advertisingService flooding log with exceptions
+ RackControllerService flooding log with exceptions
Revision history for this message
LaMont Jones (lamont) wrote : Re: RackControllerService flooding log with exceptions

introduced in 4986: [r=mpontillo][bug=][author=allenap] Address situations where RegionAdvertisingService was not thread-safe.

Revision history for this message
Lee Trager (ltrager) wrote :

I think there may be a hidden conflict between 4985 and 4986. If I run 4985 I don't see the problem. However if I take trunk and revert 4986 the problem still exists, if I revert 4985 it goes away.

Gavin Panella (allenap)
Changed in maas:
status: New → Triaged
status: Triaged → In Progress
LaMont Jones (lamont)
summary: - RackControllerService flooding log with exceptions
+ RackControllerService flooding log with: 'RegionAdvertisingService'
+ object has no attribute 'processId'
Changed in maas:
status: In Progress → Fix Committed
Revision history for this message
Gordon Bennett (gor-blimey) wrote :

I tried this patch on 2.0.0~beta5+bzr4998-0ubuntu1~xenial1
and although the GUI does now open the regiond.log still reports errors related to RegionAdvertisingService.

Traceback (most recent call last):
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 393, in callback
     self._startRunCallbacks(result)
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
     self._runCallbacks()
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
     current.result = callback(current.result, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1274, in unwindGenerator
     return _inlineCallbacks(None, gen, Deferred())
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1128, in _inlineCallbacks
     result = g.send(result)
   File "/usr/lib/python3/dist-packages/maasserver/service_monitor.py", line 101, in _updateDatabase
     processId = yield self.advertisingService.processId.get()
 builtins.AttributeError: 'RegionAdvertisingService' object has no attribute 'processId'

Revision history for this message
Gavin Panella (allenap) wrote :

Gordon, I'm going to look into that tomorrow morning. Thanks for letting us know.

Andres Rodriguez (andreserl)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.