Need /etc/sysctl.d/10-juju.conf

Bug #1631038 reported by Michael Foord
34
This bug affects 5 people
Affects Status Importance Assigned to Milestone
juju-release-tools
Fix Released
High
Nicholas Skaggs
juju-core (Ubuntu)
Fix Released
Undecided
Nicholas Skaggs
Xenial
Fix Released
Undecided
Nicholas Skaggs
Yakkety
Fix Released
Undecided
Nicholas Skaggs
Zesty
Fix Released
Undecided
Nicholas Skaggs

Bug Description

In order to increase the number of containers that can be used with the lxd provider we need an /etc/sysctl.d/10-juju.conf file setting:

    fs.inotify.max_user_watches = 524288
    fs.inotify.max_user_instances = 256

This is a partial fix for bug #1602192.

[SRU Information]

[Impact]
This represent a partial workaround to the problem of being unable to launch double digit containers via LXD. While upstream work within the kernel and LXD should provide a better solution, this seeks to mitigate the specific case of running a charm bundle locally via LXD.

[Verification]
First confirm the values are now set by checking with

sysctl -a | grep fs.inotify.max_user_

Then, try running a bunch of lxc containers. You should be able to start around 20 before they fail.

To test juju, start 13 containers using lxc. Then perform a juju bootstrap and deploy the ubuntu charm. Using the old package, the deploy should never complete. With the changes, you should be able to deploy. NOTE: You will still hit an upper limit depending on your machine of around 20 or so containers. This test is intended to demonstrate you can deploy even with more than 13 running lxc containers.

As an additional verification, you can attempt to deploy a large bundle like canonical-kubernetes, hadoop, or another bigdata charm. Be sure to remove any pre-existing containers before deploying as these bundles will start more than a dozen on there own.

[Regression Potential]
As these values are not currently set, there is no potential for regressions. Rather, the risk is on setting the values.

[Other]
This change tweaks kernel settings that will change the users machine for all running binaries -- not just LXD and juju. These values have been chosen as reasonable defaults and as safe to implement. See the discussion on bug 1602192.

Curtis Hovey (sinzui)
Changed in juju-release-tools:
status: New → Triaged
importance: Undecided → High
Changed in juju-release-tools:
assignee: nobody → Torsten Baumann (torbaumann)
Revision history for this message
Curtis Hovey (sinzui) wrote :

I think we need to create d/juju-2.0.sysctl with
   fs.inotify.max_user_watches = 524288
   fs.inotify.max_user_instances = 256

then add this to to d/rules:
    install -m 644 -D debian/juju-2.0.sysctl debian/juju-2.0/etc/sysctl.d/10-juju.conf

tags: added: conjure
Revision history for this message
Martin Pitt (pitti) wrote :

Please ship it in /usr/lib/sysctl.d/ instead of /etc/.

Revision history for this message
Martin Pitt (pitti) wrote :

Also, shouldn't this rather be shipped by lxd itself? Trying to guess every *user* of lxd sounds wrong, and if multiple consumers start shipping such snippets they might even collide on files.

Changed in juju-release-tools:
status: Triaged → In Progress
assignee: Torsten Baumann (torbaumann) → Nicholas Skaggs (nskaggs)
status: In Progress → Fix Committed
Revision history for this message
Nicholas Skaggs (nskaggs) wrote :

This is pending next upload into the archive, but will be available in the GA version of juju that will appear in the stable ppa.

Changed in juju-core (Ubuntu Zesty):
status: New → Fix Released
Changed in juju-core (Ubuntu Xenial):
status: New → In Progress
Changed in juju-core (Ubuntu Yakkety):
status: New → In Progress
Changed in juju-core (Ubuntu Zesty):
assignee: nobody → Nicholas Skaggs (nskaggs)
Changed in juju-core (Ubuntu Yakkety):
assignee: nobody → Nicholas Skaggs (nskaggs)
Changed in juju-core (Ubuntu Xenial):
assignee: nobody → Nicholas Skaggs (nskaggs)
description: updated
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Michael, or anyone else affected,

Accepted juju-core into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/juju-core/2.0.0-0ubuntu0.16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in juju-core (Ubuntu Yakkety):
status: In Progress → Fix Committed
tags: added: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Michael, or anyone else affected,

Accepted juju-core into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/juju-core/2.0.0-0ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in juju-core (Ubuntu Xenial):
status: In Progress → Fix Committed
Revision history for this message
Nicholas Skaggs (nskaggs) wrote :

Updating to say this works properly in xenial, but not in yakkety. Yakkety is missing the sysctl file; it's not installed. This needs to be corrected in the debian/rules.

tags: added: verification-done verification-failed
removed: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

The tags should be v-done-xenial and v-failed-yakkety.

tags: removed: verification-done verification-failed
tags: added: verification-needed
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Michael, or anyone else affected,

Accepted juju-core into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/juju-core/2.0.0-0ubuntu0.16.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: v-done-xenial v-failed-yakkety
removed: verification-needed
tags: removed: v-failed-yakkety
tags: added: verification-needed
Revision history for this message
Steve Langasek (vorlon) wrote :

Hello Michael, or anyone else affected,

Accepted juju-core into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/juju-core/2.0.0-0ubuntu0.16.10.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Nicolas Thomas (thomnico) wrote :

Deploying those ridiculously low parameters in /usr/lib/sysctl.d/juju-2.0.conf prevent to override in /etc/sysctl.conf as one should expect ..

Please revert this setup it prevents to use juju on lxd at all and is not coherent with :
https://github.com/lxc/lxd/blob/master/doc/production-setup.md

Revision history for this message
Nicholas Skaggs (nskaggs) wrote :

@Nicolas, /usr/lib/sysctl.d/ does override /etc/sysctl.conf. Note, these settings are already an increase from the default. These are lower values than in the wiki, but are deemed safe enough to deploy to all users. You should absolutely bump these values should you wish to run a more production environment. That wiki page suggest modifying /etc/sysctl.conf. I would suggest not modifying that file directly, and instead adding something to /usr/lib/sysctl.d/.

We can also inform folks of the existence of this file so folks don't fall over. If you make changes to the values, indeed you no longer want the juju default settings used. Let's improve that page to help make these clear. Any other suggestions? I would expect someone wanting to set there own settings to modify or remove the juju.conf.

tags: added: v-done-yakkety
removed: verification-needed
Revision history for this message
Nicholas Skaggs (nskaggs) wrote :

Works properly for yakkety. juju.conf installed and values set properly.

tags: added: verification-done-xenial verification-done-yakkety
removed: v-done-xenial v-done-yakkety
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package juju-core - 2.0.0-0ubuntu0.16.10.3

---------------
juju-core (2.0.0-0ubuntu0.16.10.3) yakkety; urgency=medium

  * debian/control: Fix the build failure due to dependency issues.
    (LP: #1617440)
    - Nope, we really don't want a Build-Depends on golang-github-coreos-go-
      systemd-dev.
    - Drop golang-go.crypto from Build-Depends.

juju-core (2.0.0-0ubuntu0.16.10.2) yakkety; urgency=medium

  * Restore golang-github-coreos-go-systemd-dev
  * Fix missing dh_install for sysctl file (LP: #1631038)

juju-core (2.0.0-0ubuntu0.16.10.1) yakkety; urgency=medium

  * New upstream release 2.0.0 (LP: #1617440)
  * d/copyright updated for 2.0.0 vendored packages.
  * Add sysctl files for lxd provider (LP: #1631038)
  * Update d/watch file
  * Add upstream signing key
  * Restore golang-golang-x-[crypto,net]-dev dependencies
  * Update bootstrap order for autpopkgtests for juju cli changes

 -- Mathieu Trudel-Lapierre <email address hidden> Thu, 03 Nov 2016 11:51:54 -0600

Changed in juju-core (Ubuntu Yakkety):
status: Fix Committed → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Update Released

The verification of the Stable Release Update for juju-core has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package juju-core - 2.0.0-0ubuntu0.16.04.2

---------------
juju-core (2.0.0-0ubuntu0.16.04.2) xenial; urgency=medium

  * DPKG_MAINTSCRIPT_ARCH doesn't work in .config, use dpkg check (LP: #1614969)

 -- Nicholas Skaggs <email address hidden> Mon, 31 Oct 2016 15:55:49 -0400

Changed in juju-core (Ubuntu Xenial):
status: Fix Committed → Fix Released
Changed in juju-release-tools:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.