MAAS

Editing/creating a cluster interface can cause an internal server error if cluster disconnected

Bug #1439476 reported by Mike Pontillo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Gavin Panella
MAAS 1.9.0

Bug Description

The following exceptions were seen in the log after doing the following:

 - Installing MAAS with a fresh database
 - Importing boot images
 - Adding a chassis with 2 nodes
 - Attempting commissioning (which fails since the cluster is unmanaged)
 *** At this point, I think a network has automatically been created,
 *** and the two nodes have been added to it
 - Editing the cluster interface to provide dynamic and static ranges,
   and change it to DNS+DHCP managed

2015-04-01 19:04:17 [-] 127.0.0.1 - - [02/Apr/2015:02:04:16 +0000] "GET
  /MAAS/clusters/992334f5-f5f0-4e61-88e9-e372ae174b1e/interfaces/eth0/edit/
  HTTP/1.1" 200 2804
  "http://maas-trusty/MAAS/clusters/992334f5-f5f0-4e61-88e9-e372ae174b1e/edit/"
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36
  (KHTML, like Gecko) Chrome/41.0.2272.104 Safari/537.36"
2015-04-01 19:04:29 [maas] WARNING: Failed to create Network when
  adding/editing cluster interface maas-eth0 with error [{'ip':
  [u'Network with this Ip already exists.'], 'name': [u'Network with
  this Name already exists.']}]. This is OK if it already exists.
2015-04-01 19:04:29 [HTTPChannel,3,127.0.0.1] 500 Error -
  /MAAS/clusters/992334f5-f5f0-4e61-88e9-e372ae174b1e/interfaces/eth0/edit/

Traceback (most recent call last):
  File ".../twisted/web/wsgi.py", line 315, in run
    appIterator = self.application(self.environ, self.startResponse)
  File ".../django/core/handlers/wsgi.py", line 206, in __call__
    response = self.get_response(request)
  File ".../maasserver/utils/views.py", line 166, in get_response
    response = get_response(request)
  File ".../maasserver/utils/views.py", line 155, in get_response
    request, get_resolver(None), sys.exc_info())
--- <exception caught here> ---
  File ".../maasserver/utils/views.py", line 140, in get_response
    return django_get_response(request)
  File ".../maasserver/utils/orm.py", line 331, in __exit__
    self.fire()
  File ".../provisioningserver/utils/twisted.py", line 151, in wrapper
    return func(*args, **kwargs)
  File ".../maasserver/utils/async.py", line 180, in fire
    self._fire_in_reactor(hook).wait()
  File ".../crochet/_eventloop.py", line 219, in wait
    result.raiseException()
  File "<string>", line 2, in raiseException

This causes a generic "Internal server error" to be seen when saving the
cluster interface configuration.

See original description
Tags: oil

Related branches

lp:~allenap/maas/edit-disconnected-cluster--bug-1439476
Blake Rouse (community): Approve
Diff: 475 lines (+204/-39)
4 files modified
src/maasserver/dhcp.py (+31/-12)
src/maasserver/rpc/regionservice.py (+35/-3)
src/maasserver/rpc/tests/test_regionservice.py (+76/-0)
src/maasserver/tests/test_dhcp.py (+62/-24)
Mike Pontillo (mpontillo)
Changed in maas:
importance: Undecided → High
description: updated
Revision history for this message
Raphaël Badin (rvb) wrote :

This is a crash, it should be critical.

Changed in maas:
importance: High → Critical
Revision history for this message
Larry Michel (lmic) wrote :

I am seeing this with 1.8b4. I noticed that the change goes through since I can get to it when go back.

tags: added: oil
Revision history for this message
Larry Michel (lmic) wrote :

Correction from my previous comment.. the change appears to go through following "internal server error" but functionality is broken.

Earlier, I was changing an interface to unmanaged to DHCP and DNS, and maas-dhcp would not start.. I just retried a second a go by changing the interface from DHCP and DNS to just DHCP and the save took without the "internal server error".. and maas-dhcp started!

Andres Rodriguez (andreserl)
Changed in maas:
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for MAAS because there has been no activity for 60 days.]

Changed in maas:
status: Incomplete → Expired
Revision history for this message
Andres Rodriguez (andreserl) wrote :

==> /var/log/maas/regiond.log <==
2015-10-21 21:52:48 [HTTPChannel,8,127.0.0.1] 500 Error - /MAAS/clusters/bd8ff259-e344-40a7-a0ad-a786712b6d83/interfaces/add/
 Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/twisted/web/wsgi.py", line 315, in run
     appIterator = self.application(self.environ, self.startResponse)
   File "/usr/lib/django16/django/core/handlers/wsgi.py", line 206, in __call__
     response = self.get_response(request)
   File "/usr/lib/python2.7/dist-packages/maasserver/utils/views.py", line 236, in get_response
     response = get_response(request)
   File "/usr/lib/python2.7/dist-packages/maasserver/utils/views.py", line 225, in get_response
     request, get_resolver(None), sys.exc_info())
 --- <exception caught here> ---
   File "/usr/lib/python2.7/dist-packages/maasserver/utils/views.py", line 210, in get_response
     return django_get_response(request)
   File "/usr/lib/python2.7/dist-packages/maasserver/utils/orm.py", line 357, in __exit__
     self.fire()
   File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 200, in wrapper
     return func(*args, **kwargs)
   File "/usr/lib/python2.7/dist-packages/maasserver/utils/async.py", line 203, in fire
     self._fire_in_reactor(hook).wait()
   File "/usr/lib/python2.7/dist-packages/crochet/_eventloop.py", line 219, in wait
     result.raiseException()
   File "<string>", line 2, in raiseException

 provisioningserver.rpc.exceptions.NoConnectionsAvailable: Unable to connect to cluster bd8ff259-e344-40a7-a0ad-a786712b6d83; no connections available.

summary: - Editing a cluster interface can cause an internal server error
+ Editing/creating a cluster interface can cause an internal server error
+ if cluster disconnected
Changed in maas:
status: Expired → New
milestone: none → 1.9.0
Gavin Panella (allenap)
Changed in maas:
status: New → Triaged
assignee: nobody → Gavin Panella (allenap)
status: Triaged → In Progress
description: updated
Revision history for this message
Mike Pontillo (mpontillo) wrote :

With Gavin's help, today we looked into possibilities regarding how this might occur.

I think it's likely that dhcp_post_change_Subnet() (maasserver/models/signals/dhcp.py) is the cause of this bug.

I'm still confused about the stack traces in this bug, though. I think we need to prioritize cleaning up the stack traces we get from Twisted so that we can track these down easier...

Gavin Panella (allenap)
Changed in maas:
status: In Progress → Fix Committed
Andres Rodriguez (andreserl)
Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.