celeryd 100% cpu when large dhcpd lease file

Number of unique entries in the dhcpd.lease file.

ubuntu@maas-trusty:~$ grep hardware /var/lib/maas/dhcp/dhcpd.leases | awk '{print $3}' | sort | uniq | wc -l
3931

This is related to bug 1153077, and is probably a dupe of it.

I think it is some what related, but in this case the queue was actually not filling up, it was doing it just fast enough to not cause that issue. But each celeryd process was consuming larges amounts of cpu time.

I think Gavin is right: they are related. It's just that since bug 1153077, we set an expiration date of the tasks so they don't really pile up if they can't be processed in time. But the task in question is fired every minute; if it takes ~147.9s to complete, it will take hold of all celery's workers pretty quickly.

I'm testing something completely unrelated and, in /var/log/syslog, I have this:

Apr 25 11:49:15 3d8eb172-cc4f-11e3-a506-9c4e363b1c94 dhcpd: Can't create new lease file: Permission denied

The message seems to indicate that the dhcp server is failing to "rotate" its lease file. That might explain why we end of with huge lease files.

Can you check the containing directory's permissions? Remember that in the precise → raring transition we went from root-owned to dhcpd-owned directories as the dhcpd now drops permissions as needed.

rvb@maas:~$ ls -la /var/lib/maas/dhcp/
total 16
drwxr-xr-x 2 root root 4096 Apr 30 10:01 .
drwxr-xr-x 5 maas maas 4096 Apr 29 18:38 ..
-rw-r--r-- 1 root root 1228 Apr 30 14:50 dhcpd.leases
-rw-r--r-- 1 root root 817 Apr 30 09:58 dhcpd.leases~

rvb@maas:~$ ls -la /var/lib/maas/
total 52
drwxr-xr-x 5 maas maas 4096 Apr 29 18:38 .
drwxr-xr-x 46 root root 4096 Apr 29 18:32 ..
drwxr-xr-x 5 maas maas 4096 Apr 30 11:31 boot-resources
-rw-r--r-- 1 maas maas 16384 Apr 30 16:08 celerybeat-cluster-schedule
-rw-r--r-- 1 maas maas 12288 Apr 30 10:01 celerybeat-region-schedule
drwxr-xr-x 2 root root 4096 Apr 30 10:01 dhcp
-rw-r--r-- 1 root root 4 Apr 29 18:38 dhcpd-interfaces
drwxr-xr-x 2 maas maas 4096 Apr 29 18:34 media

Btw, if you have a look at the dhcpd.leases file posted on a related bug (https://bugs.launchpad.net/maas/+bug/1153077/+attachment/3689992/+files/dhcpd.leases.gz), you'll see that, although the file weights 28M, there are only 48 different IP<->MAC mappings in it. i.e. a *lot* of repetitions.

On 01/05/14 02:13, Raphaël Badin wrote:
> Btw, if you have a look at the dhcpd.leases file posted on a related bug
> (https://bugs.launchpad.net/maas/+bug/1153077/+attachment/3689992/+files/dhcpd.leases.gz),
> you'll see that, although the file weights 28M, there are only 48
> different IP<->MAC mappings in it. i.e. a *lot* of repetitions.
>

Apparently DHCPD re-writes the leases file every hour and that's a
compiled-in default.

I guess they didn't expect external code to be scanning it!

My DHCPD on trusty seems to be rotating lease files fine, there is no permission error anywhere.

My directory permissions are the same as rvb's above.

ubuntu@maas:/var/lib/maas$ ls -la
total 60
drwxr-xr-x 7 maas maas 4096 Mar 28 14:30 .
drwxr-xr-x 61 root root 4096 Jan 31 12:22 ..
drwxr-xr-x 14 root root 4096 May 7 16:03 boot-resources
-rw-r--r-- 1 maas maas 16384 May 8 11:29 celerybeat-cluster-schedule
-rw-r--r-- 1 maas maas 12288 May 8 10:20 celerybeat-region-schedule
drwxr-xr-x 2 root root 4096 May 8 10:19 dhcp
-rw-r--r-- 1 root root 4 Feb 13 13:17 dhcpd-interfaces
drwxr-xr-x 7 maas maas 4096 Feb 13 18:16 ephemeral
drwxr-xr-x 3 maas maas 4096 Jan 31 12:31 media
drwxr-xr-x 4 maas maas 4096 Mar 28 14:31 tftp
ubuntu@maas:/var/lib/maas$ ls -la dhcp
total 124
drwxr-xr-x 2 root root 4096 May 8 10:19 .
drwxr-xr-x 7 maas maas 4096 Mar 28 14:30 ..
-rw-r--r-- 1 root root 3316 May 8 10:19 dhcpd.leases

branch which fixes this and other 1.5.2 release bugs for SRU is linked to bug #1325038

Hello Blake, or anyone else affected,

Accepted maas into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/maas/1.5.2+bzr2282-0ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I tested an upgrade from 1.5.1 - upgraded, started an existing node, recommissioned an existing node, deleted the node, then enlisted/commissioned/installed to it without issue.

Hello Blake, or anyone else affected,

Accepted maas into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/maas/1.5.2+bzr2282-0ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package maas - 1.5.2+bzr2282-0ubuntu0.2

---------------
maas (1.5.2+bzr2282-0ubuntu0.2) trusty-proposed; urgency=medium

  * debian/control:
   - Add missing dependency in maas-cluster-controller for grub-common
     (LP: #1328231)
   - Move dependency from maas-cluster-controller to maas-provisioningserver
     for python-seamicroclient (LP: #1332532)

maas (1.5.2+bzr2282-0ubuntu0.1) trusty-proposed; urgency=medium

  * New upstream release:
    - Remove workaround for fixed Django bug 1311433 (LP: #1311433)
    - Ensure that validation errors are returned when adding a node over
      the API and its cluster controller is not contactable. (LP: #1305061)
    - Hardware enablement support for PowerKVM (LP: #1325038)
    - Shorten the time taken for a cluster to initially connect to the region
      via RPC to around 2 seconds (LP: #1317682)
    - Faster DHCP leases parser (LP: #1305102)
    - Documentation fixed explaining how to enable an ephemeral backdoor
      (LP: #1321696)
    - Use probe-and-enlist-hardware to enlist all virtual machine inside
      a libvirt machine, allow password qemu+ssh connections.
      (LP: #1315155, LP: #1315157)
    - Rename ppc64el boot loader to PowerKVM (LP: #1315154)
    - Fix NodeForm's is_valid() method so that it uses Django's way of setting
      errors on forms instead of putting text in self.errors['architecture']
      (LP: #1301465)
    - Change BootMethods to return their own IReader per-request, update method
      names to reflect new usage. (LP: #1315154)
    - Return early and stop the DHCP server when the list of managed interfaces
      of the nodegroup is empty. (LP: #1324944)
    - Fix invalid attribute references in the VirshSSH class. Added more test
      for the VirshSSH class. (LP: #1324966)
  * debian/control:
    - Add missing dependency in maas-cluster-controller for python-pexpect
      (LP: #1322151)
 -- Greg Lutostanski <email address hidden> Fri, 20 Jun 2014 10:10:47 -0500

The verification of the Stable Release Update for maas has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.