Merge lp:~vila/otto/update-setup-instructions into lp:otto

[Vincent Ladeuil]

I updated the setup instructions while setting up qa-radeon-7750.

There are still a few places where I followed the existing instructions
blindly and I'd be glad if some explanations could be given/added for the
unaware ;)

1) swapaccount: I wasn't able to verify that the CONFIG_MEMCG_SWAP_ENABLED
   was set in the kernel config:

ubuntu@qa-radeon-7750:/usr/src/linux-headers-3.11.0-12-generic$ grep CONFIG_MEMCG_SWAP_ENABLED .config
# CONFIG_MEMCG_SWAP_ENABLED is not set

Of course the above was too naive, so:

ubuntu@qa-radeon-7750:/usr/src/linux-headers-3.11.0-12-generic$ grep MEMCG .config
CONFIG_MEMCG=y
CONFIG_MEMCG_SWAP=y
# CONFIG_MEMCG_SWAP_ENABLED is not set
# CONFIG_MEMCG_KMEM is not set

But the question is really: why do we want to have swap in the container ?
Can't we just get rid of that constraint (and the associated host grub
configuration) ?

2) Raising the memory limit seems to have an impact on test passing or
not. What do we know about that memory consumption ? 4G seems a huge
requirement for autopilot tests...

[Martin Pitt (pitti)]

Thanks Vincent for your efforts to document that better. I recently re-tried otto myself on saucy/trusty and it was certainly far harder than necessary.

> 1) swapaccount: I wasn't able to verify that the CONFIG_MEMCG_SWAP_ENABLED was set in the kernel config:

That's indeed the biggest pain point right now. At least saucy/trusty indeed don't set this in the default kernel config, so otto does not work at all out of the box. I think that should be addressed in the code, not in the documentation. My current workaround is this:

--- lxc.defaults/config 2013-07-05 21:00:06 +0000
+++ lxc.defaults/config 2013-10-17 14:22:47 +0000
@@ -67,5 +67,5 @@
 lxc.loglevel = 1

 # Restrict memory usage of the container
-lxc.cgroup.memory.limit_in_bytes = 2G
-lxc.cgroup.memory.memsw.limit_in_bytes = 2560M
+#lxc.cgroup.memory.limit_in_bytes = 2G
+#lxc.cgroup.memory.memsw.limit_in_bytes = 2560M

But I think otto should dynamically enable that feature if the kernel supports it.

+ * branch otto from launchpad and setup a ~/bin directory

I think this should become a separate documentation item, as it is optional. Like:

  * If you don't want to call otto with full path of the checkout, you can put it into $PATH with:

    mkdir -p ~/bin
    for f in `pwd`/otto/bin/*; do ln -s $f ~/bin; done

[Martin Pitt (pitti)]

BTW, this is a better command to check for it, as it doesn't depend on the headers to be installed:

 $ grep CONFIG_MEMCG_SWAP_ENABLED /boot/config-`uname -r`
 # CONFIG_MEMCG_SWAP_ENABLED is not set

[Vincent Ladeuil (vila)]

> +#lxc.cgroup.memory.limit_in_bytes = 2G
> +#lxc.cgroup.memory.memsw.limit_in_bytes = 2560M

Right so you just don't care about memory limits. I had a gut feeling it's a better approach to start with and only if/when needed 1) limit the memory, 2) limit memory and swap, the later requiring a properly configured kernel.

> BTW, this is a better command to check for it

Excellent, thanks. Included in the next revision.

[lp:~vila/otto/update-setup-instructions]

259. By Vincent Ladeuil on 2013-10-29

Feedback from pitti.

[Jean-Baptiste Lallement (jibel)]

I agree with Martin that it should handled in the code and removes this configuration if it is not supported by the kernel with a Warning in that case.

This configuration is required in production to keep control of the machines if a test goes crazy. It occurred lot of times during Saucy cycle due to memory fragmentation caused by recordmydesktop on long running testsuite like Unity. Without this directive OOM killer simply kills anything on the machine instead of just processes in the container.

[Jean-Baptiste Lallement (jibel)]

But swapaccount=1 on the kernel command line is the only thing you should need to enable this feature. On production machines the kernel has not been changed in any way.

[Jean-Baptiste Lallement (jibel)]

Why do you do
$ sudo ln -s /etc/apparmor.d/usr.bin.rsyslogd /etc/apparmor.d/disable/
?

There is a typo line 63 (missing semicolon or new line).

[Vincent Ladeuil (vila)]

> This configuration is required in production to keep control of the machines if a test goes crazy.

Why the swap part though ? Just setting a memory limit should be enough no ?

> Why do you do
> $ sudo ln -s /etc/apparmor.d/usr.bin.rsyslogd /etc/apparmor.d/disable/
> ?

I found it on dx-autopilot-intel, dx-autopilot-nvidia, dx-autopilot-ati, ps-nvidia-gt630, ps-intel-4600, ps-radeon-hd8350, qa-nvidia-gtx660 and qa-intel-4000. So I thought it was required.

If it's not, I'd rather remove it both in that doc and in the ci lab.

> There is a typo line 63 (missing semicolon or new line).

Good catch, fixed.

[lp:~vila/otto/update-setup-instructions]

260. By Vincent Ladeuil on 2013-10-30

Fix typo.

[Jean-Baptiste Lallement (jibel)]

On 10/30/2013 09:37 AM, Vincent Ladeuil wrote:
>> This configuration is required in production to keep control of the machines if a test goes crazy.
>
> Why the swap part though ? Just setting a memory limit should be enough no ?
Because it happened that the machine used swap because there was not
enough RAM (run unity7 testsuite with recordmydesktop enabled with
default settings twice and the machine will crash)

FTR, recordmydesktop (+the way autopilot uses it?) causes memory
fragmentation which leads to memory exhaustion and sometimes requiring
swap to finish the test.

>
>> Why do you do
>> $ sudo ln -s /etc/apparmor.d/usr.bin.rsyslogd /etc/apparmor.d/disable/
>> ?
>
> I found it on dx-autopilot-intel, dx-autopilot-nvidia, dx-autopilot-ati, ps-nvidia-gt630, ps-intel-4600, ps-radeon-hd8350, qa-nvidia-gtx660 and qa-intel-4000. So I thought it was required.
>
> If it's not, I'd rather remove it both in that doc and in the ci lab.
>
>> There is a typo line 63 (missing semicolon or new line).
>
> Good catch, fixed.
>

--
Jean-Baptiste
IRC: jibel

[lp:~vila/otto/update-setup-instructions]

261. By Vincent Ladeuil on 2013-10-30

Remove /etc/apparmor.d/usr.bin.rsyslogd symlink creation, otto has nothing to do with that.

[Jean-Baptiste Lallement (jibel)]

LGTM, thanks Vincent!

review: Approve

=== modified file 'doc/README'
--- doc/README	2013-08-07 10:05:54 +0000
+++ doc/README	2013-10-30 09:00:44 +0000
@@ -23,34 +23,43 @@
     $ sudo apt-get install bzr --no-install-recommends
     $ sudo apt-get install python3-lxc aufs-tools
 
-  * branch otto from launchpad
-
-    $ bzr branch lp:otto
-    $ cd otto
+  * If you don't want to call otto with the full path of the checkout, you
+    can put it into $PATH with (running in your otto checkout/branch):
+    $ mkdir ~/bin
+    $ for f in `pwd`/otto/bin/*; do ln -s $f ~/bin; done
 
   * Try to run otto:
     $ sudo bin/otto --help
 
-  * You'll need to run lxc-start unconfined to allow mounting /dev/loop0 
-    during the pre-start hook
+  * You'll need to run lxc-start unconfined to allow mounting /dev/loop0
+    during the pre-start hook (which is required to boot from a desktop
+    image):
 
     $ sudo apparmor_parser -R /etc/apparmor.d/usr.bin.lxc-start
     $ sudo ln -s /etc/apparmor.d/usr.bin.lxc-start /etc/apparmor.d/disable/
 
   * On Ubuntu swap resource control is enable in the kernel but disabled by
     default. To enable it make sure CONFIG_MEMCG_SWAP_ENABLED=Y is set in kernel
-    configuration file and swapaccount=1 on the kernel boot parameters (set it
-    permanently in /etc/default/grub then run update-grub)
+    configuration file:
+    $ grep MEMCG /boot/config-`uname -r`
+    Set swapaccount=1 on the kernel boot parameters (set it
+    permanently in /etc/default/grub then run update-grub) to get:
+    $ grep GRUB_CMDLINE_LINUX_DEFAULT /etc/default/grub
+    GRUB_CMDLINE_LINUX_DEFAULT="quiet swapaccount=1"
 
   * You may also want to install apport to enable crash reporting from
     applications inside the container.
 
 == Integration with Jenkins ==
-  * Declare the node jenkins server (use the same name than hostname by
+  * Declare the node on the jenkins server (use the same name than hostname by
     default)
   * install jenkins-slave
     $ sudo apt-get install --no-install-recommends jenkins-slave
   * Add user 'jenkins' to /etc/sudoers
+    # cat <<EOF >/etc/sudoers.d/jenkins
+    jenkins ALL=(ALL) NOPASSWD: ALL
+    EOF
+    # chmod 440 /etc/sudoers.d/jenkins
   * Edit jenkins-slave configuration in /etc/default/jenkins-slave and set the
     URL of the server. You can also set the name of the slave if you do not
     want to use the default (hostname)
@@ -58,7 +67,9 @@
     $ sudo start jenkins-slave
   * Verify that the node becomes online on the server.
   * Follow the steps described in the previous section (Installation) as user
-    'jenkins' instead of 'ubuntu'.
+    'jenkins' instead of 'ubuntu'. And add symlinks for the jenkins scripts.
+    $ for f in `pwd`/otto/jenkins/*; do ln -s $f ~/bin; done
+ 
 
 = Running Otto =
 

=== modified file 'jenkins/update_host'
--- jenkins/update_host	2013-08-05 07:52:18 +0000
+++ jenkins/update_host	2013-10-30 09:00:44 +0000
@@ -46,7 +46,7 @@
 
 This script downloads an Ubuntu Desktop ISO from the given URL for the release
 and architecture matching the host system.  Then the system is upgraded and
-rebooted.
+rebooted (if needed).
 
 EOF
     exit 1

=== modified file 'ottolib/container.py'
--- ottolib/container.py	2013-08-09 12:36:03 +0000
+++ ottolib/container.py	2013-10-30 09:00:44 +0000
@@ -73,7 +73,7 @@
         required and replaced by using an Ubuntu image directly.
         It creates the minimum files required by LXC to be a valid container:
         rootfs/, config and fstab, copies a pre-mount script that is used
-        during when the container starts to prepare the container to run from
+        when the container starts to prepare the container to run from
         a disk image and share the devices from the host. The directory guest
         will be rsynced to the guest FS by the pre-mount script to install
         additional packages into the container.