Unable to deploy machine with static IP address

Bug #1904810 reported by Lee Trager
38
This bug affects 7 people
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Alberto Donato
2.8
Fix Released
Critical
Alberto Donato
2.9
Fix Released
Critical
Alberto Donato

Bug Description

I have a node which I manually assigned a static IP address. I went to deploy the node but I get a "Error:Failed to allocate the required AUTO IP addresses". My MAAS has no other machines deployed so the address space is fully available. Switching to auto assign works around this.

2020-11-19 03:07:22 maasserver.websockets.protocol: [critical] Error on request (90) machine.action: Failed to allocate the required AUTO IP addresses
 Traceback (most recent call last):
   File "/usr/lib/python3.8/threading.py", line 870, in run
     self._target(*self._args, **self._kwargs)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 825, in worker
     return target()
   File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line 46, in work
     task()
   File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in doWork
     task()
 --- <exception caught here> ---
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 250, in inContext
     result = inContext.theWork()
   File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 266, in <lambda>
     inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 122, in callWithContext
     return self.currentContext().callWithContext(ctx, func, *args, **kw)
   File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 85, in callWithContext
     return func(*args,**kw)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 860, in callInContext
     return func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 192, in wrapper
     result = func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 737, in call_within_transaction
     return func_outside_txn(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 602, in __exit__
     self.fire()
   File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py", line 192, in wrapper
     result = func(*args, **kwargs)
   File "/usr/lib/python3/dist-packages/maasserver/utils/asynchronous.py", line 207, in fire
     self._fire_in_reactor(hook).wait(LONGTIME)
   File "/usr/lib/python3/dist-packages/crochet/_eventloop.py", line 231, in wait
     result.raiseException()
   File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 467, in raiseException
     raise self.value.with_traceback(self.tb)
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
     result = result.throwExceptionIntoGenerator(g)
   File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
     return g.throw(self.type, self.value, self.tb)
   File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 5546, in claim_auto_ips
     yield self._claim_auto_ips()
   File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
     result = g.send(result)
   File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 4506, in _claim_auto_ips
     raise StaticIPAddressExhaustion(
 maasserver.exceptions.StaticIPAddressExhaustion: Failed to allocate the required AUTO IP addresses

Related branches

Alberto Donato (ack)
Changed in maas:
assignee: nobody → Alberto Donato (ack)
status: Triaged → In Progress
Alberto Donato (ack)
Changed in maas:
milestone: 2.9.0rc2 → none
no longer affects: maas/2.9
Changed in maas:
milestone: none → next
no longer affects: maas/2.9
Changed in maas:
milestone: next → 2.9.0rc2
Changed in maas:
status: In Progress → Fix Committed
Changed in maas:
milestone: 2.9.0rc2 → 2.10-next
Revision history for this message
Matt Rae (mattrae) wrote :

I seem to be having this problem after the maas snap auto-updated to 2.8.3/4 from 2.8.2

Deploying with a static ip, in a reserved range, to any subnet gives an error in the regiond log:

maasserver.exceptions.StaticIPAddressExhaustion: Failed to allocate the required AUTO IP addresses

There doesn't appear to be any unexpected ips in the subnet when I go to the subnet page, or when i ping the subnet.

Revision history for this message
Tatu Ylonen (ylo) wrote :

This may be same as #1902425, but in any case I am very much suffering from similar issues (same AUTO IP crash). 2.8/stable (2.8.4-8597-g.05313b458). Completely unable to deploy new machines. I'm using static addresses for the hosts I'm trying to deploy. I had unexplained unused ip addresses on two subnets inside dynamic ranges for managed networks, and one actual misconfigured switch management port inside a dynamic range. The misconfigured host has been reconfigured to be outside dynamic ranges. I have deleted the unknown-usage ip addresses and related interfaces using psql from maasserver_staticipaddress, maasserver_interface_ip_addresses, and maasserver_interfaces. I have tried unmanaging and re-managing all subnets. I've tried disabling and re-enabling DHCP. I've rebooted. I've removed and reinstalled the maas snap (I use a separate postgresql database as per production deployment instructions). I tried upgrading to 2.9/stable and then 2.9/edge but doing so breaks deployments via API, possibly due to an apparmor configuration bug (see #1917640). I have restored the postgresql database from a backup taken before 2.9 deployment and reinstalled maas 2.8/stable but the AUTO IP failure persists. So far nothing has worked.

Changed in maas:
milestone: 3.0.0 → none
status: Fix Committed → Fix Released
Revision history for this message
Jeff Lane  (bladernr) wrote :

Is this actually fixed? I am seeing this currently on our MAAS using the 2.9 snap (2.9.2 (9165-g.c3e7848d1))

maas.log shows this for each failed deployment:
2021-04-20T18:29:27.482734+00:00 lxc-maas maas.node: [info] lalande: Status transition from ALLOCATED to DEPLOYING
2021-04-20T18:29:27.796705+00:00 lxc-maas maas.subnet: [warn] Next IP address to allocate from 'Internal-IPs (10.1.10.0/23)' has been observed previously: 10.1.11.231 was last claimed by 00:21:9b:9b:53:1e via eth1 (physical) on lxc-maas at 2021-02-09 16:23:16.398982.
2021-04-20T18:29:28.357973+00:00 lxc-maas maas.subnet: [warn] Next IP address to allocate from 'Internal-IPs (10.1.10.0/23)' has been observed previously: 10.1.11.231 was last claimed by 00:21:9b:9b:53:1e via eth1 (physical) on lxc-maas at 2021-02-09 16:23:16.398982.
2021-04-20T18:29:28.504844+00:00 lxc-maas maas.node: [info] lalande: Aborting DEPLOYING and reverted to ALLOCATED. Unable to power control the node. Please check power credentials.
2021-04-20T18:29:28.515866+00:00 lxc-maas maas.node: [info] lalande: Status transition from DEPLOYING to ALLOCATED

There is nothing in regiond.log or rackd.log around those timestamps. Nor is there any clue in any other log.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.