Failed to upload leases

Bug #1387515 reported by Andres Rodriguez
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Gavin Panella
1.7
Fix Released
Critical
Gavin Panella

Bug Description

==> maas.log <== [45/530]
Oct 30 05:18:16 maas-trusty-back-may22 maas.lease_upload_service: [INFO] Uploading 1779 DHCP leases to region controller.

==> pserv.log <==
2014-10-30 05:18:16+0000 [-] Unhandled Error
        Traceback (most recent call last):
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 423, in errback
            self._startRunCallbacks(fail)
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 490, in _startRunCallbacks
            self._runCallbacks()
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 577, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1155, in gotResult
            _inlineCallbacks(r, g, deferred)
        --- <exception caught here> ---
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/pserv_services/lease_upload_service.py", line 113, in _get_client_and_start_upload
            yield self._start_upload(client) [24/530]
          File "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
            result = g.send(result)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/pserv_services/lease_upload_service.py", line 129, in _start_upload
            UpdateLeases, uuid=self.uuid, mappings=mappings)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/utils/twisted.py", line 102, in wrapper
            return func(*args, **kwargs)
          File "/usr/lib/python2.7/dist-packages/provisioningserver/rpc/common.py", line 125, in __call__
            return self._conn.callRemote(cmd, **kwargs)
          File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 821, in callRemote
            return co._doCommand(self)
          File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 1778, in _doCommand
            self.requiresAnswer)
          File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 752, in _sendBoxCommand
            box._sendTo(self.boxSender)
          File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 577, in _sendTo
            proto.sendBox(self)
          File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 2153, in sendBox
            self.transport.write(box.serialize())
          File "/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 555, in serialize
            raise TooLong(False, True, v, k)
        twisted.protocols.amp.TooLong:

==> maas.log <==
Oct 30 05:18:16 maas-trusty-back-may22 maas.lease_upload_service: [ERROR] Failed to upload leases:

Related branches

Revision history for this message
Julian Edwards (julian-edwards) wrote :

AMP's maximum message size is now not big enough to cope with all the leases, which is slowing down the pserv and causing this stacktrace.

We have a couple of options:
 1. Get rid of lease parsing. Nothing needs it.
 2. Switch to the things previously discussed, namely a daemon that uses inotify to pick up changes as they happen

The problem is that the current parser tries to upload the current status every time, which is probably unnecessary, it only needs to be done once on startup.

Changed in maas:
status: New → Triaged
importance: Undecided → High
milestone: none → next
tags: added: oil
tags: added: dhcp provisioning
Revision history for this message
Julian Edwards (julian-edwards) wrote :

If there's nothing in the future that we need lease parsing for, I really suggest we get rid of it. Andres was talking about auto-discovery of things on the network, but I don't think using DHCP leases for that is a great idea, I'd rather we had some way of analysing the network traffic and/or actively probing.

Revision history for this message
Raphaël Badin (rvb) wrote :

It's a crash, marking this critical.

Changed in maas:
importance: High → Critical
Gavin Panella (allenap)
Changed in maas:
assignee: nobody → Gavin Panella (allenap)
status: Triaged → In Progress
Revision history for this message
Christian Reis (kiko) wrote :

Can someone file a real bug for getting rid of lease parsing?

Revision history for this message
Adam Conrad (adconrad) wrote : Please test proposed package

Hello Andres, or anyone else affected,

Accepted maas into utopic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/maas/1.7.5+bzr3369-0ubuntu1~14.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed
Revision history for this message
Andres Rodriguez (andreserl) wrote :

This issue has been verified to work both on upgrade and fresh install, and has been QA'd. Marking verification-done.

tags: added: verification-done
removed: verification-needed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.