Merge lp:~serge-hallyn/serverguide/serverguide-lxc into lp:~ubuntu-core-doc/serverguide/precise

[Serge Hallyn]

This merge introduces a new LXC section. Some subsections are yet to be written, because they are contingent on work still going into precise.

[Peter Matulis (petermatulis)]

This is a very significant contribution to the guide. Thank you!

Technical:

All tests performed using default settings and with a simple ubuntu-based container. I did not check all commands.

1. I tried to start a container (cn1) on a KVM guest. I was able to log in but shutting down threw warnings/errors. Normal?

------------------------
$ sudo poweroff
[sudo] password for ubuntu:
$
Broadcast message from ubuntu@cn1
        (/dev/lxc/console) at 18:17 ...

The system is going down for power off NOW!
 * Asking all remaining processes to terminate...
   ...done.
 * All processes ended within 1 seconds....
   ...done.
 * Deconfiguring network interfaces...
   ...done.
 * Deactivating swap...
   ...fail!
umount: /run/lock: not mounted
umount: /dev/shm: not mounted
mount: / is busy
 * Will now halt
------------------------

2. This command output doesn't look right. Normal?:

------------------------
$ sudo lxc-start -n cn1 -d
$ lxc-ls
cn1
cn1

$ lxc-list
RUNNING

STOPPED
------------------------

3. Should include how to escape from a container console (Ctrl-a q).

Style:

Under "Host Setup", for /etc/default/lxc, since you say "true by default" for value of USE_LXC_BRIDGE, it makes sense to also say "true by default" for value of LXC_AUTO.

In general, consider using italics when introducing new terms/commands/package_names or when trying to emphasize a word. Example: "a package was introduced called ``lxcguest''..." or "...have various ``leaks'' which allow...". This quoting style is awkward.

We should encourage proper practice and prepend all commands requiring privileged access with 'sudo'.

I question the section title of "Container Introspection". The term introspection pertains to the fields of philosophy and psychology. Maybe "Inspection" is better.

Under "Advanced namespace usage", there is a block of code that is not formatted properly. It shows '<pre>' tags and I don't think the red colour is called for. I also think you should provide an external resource/link to 'private namespaces' as well as giving a one-line description of a basic use-case.

Standardize on using "LXC" and not "lxc" or "Lxc" except when referring to a package name? Dunno, 'lxc.sourceforge.net' shows kind of the reverse: "LXC is the userspace control package for Linux Containers" and "Linux Containers (lxc) implement:". Can be confusing to readers. Proceed as you see fit.

"IP address" instead of "ip address".

Awkward: "The type can be one of several types."

Missing a 'The'? "Following is an example configuration file..."

Extra character? "...which require cap_sys_admin}."

You've made bold man page references before. "See capabilities(7) for a list..."

Rework: "For instance, if a package's postinst fails if it cannot open a block device..."

The standard is to capitalize release codenames: "In the natty and oneiric releases of Ubuntu..."

Add resources section (external links, man pages) at end of page. See end of https://help.ubuntu.com/11.10/serverguide/C/openldap-server.html for an example.

review: Needs Fixing

[Serge Hallyn (serge-hallyn)]

On 03/16/2012 02:52 PM, Peter Matulis wrote:
> Review: Needs Fixing
>
> This is a very significant contribution to the guide. Thank you!
>
> Technical:
>
> All tests performed using default settings and with a simple ubuntu-based container. I did not check all commands.
>
> 1. I tried to start a container (cn1) on a KVM guest. I was able to log in but shutting down threw warnings/errors. Normal?

Yes, the errors are normal. Should that be explained somewhere?

> ------------------------
> $ sudo poweroff
> [sudo] password for ubuntu:
> $
> Broadcast message from ubuntu@cn1
> (/dev/lxc/console) at 18:17 ...
>
> The system is going down for power off NOW!
> * Asking all remaining processes to terminate...
> ...done.
> * All processes ended within 1 seconds....
> ...done.
> * Deconfiguring network interfaces...
> ...done.
> * Deactivating swap...
> ...fail!
> umount: /run/lock: not mounted
> umount: /dev/shm: not mounted
> mount: / is busy
> * Will now halt
> ------------------------
>
>
> 2. This command output doesn't look right. Normal?:
>
> ------------------------
> $ sudo lxc-start -n cn1 -d
> $ lxc-ls
> cn1
> cn1
>
> $ lxc-list
> RUNNING
>
> STOPPED
> ------------------------

The lxc-list output doesn't look right - cn1 should show up in both
lists. The lxc-ls output shouldn't have the third (empty) line but
otherwise looks fine. I can't reproduce this.

I will address the rest in a merge proposal update. Thanks for the
comments!

[lp:~serge-hallyn/serverguide/serverguide-lxc]

50. By Serge Hallyn on 2012-03-18

Address a number of pmatulis' comments.

51. By Serge Hallyn on 2012-03-18

sudo

52. By Serge Hallyn on 2012-03-18

standardize use of LXC

53. By Serge Hallyn on 2012-03-18

remove <pre>

54. By Serge Hallyn on 2012-03-18

fix parse errors

55. By Serge Hallyn on 2012-03-18

namespac

56. By Serge Hallyn on 2012-03-18

write security section

57. By Serge Hallyn on 2012-03-18

remove udev comment

58. By Serge Hallyn on 2012-03-18

comment out virt-install libvirt-lxc section (it never worked for me)

[Serge Hallyn (serge-hallyn)]

I believe the comments are now addressed, thanks.

review: Needs Resubmitting

[Peter Matulis (petermatulis)]

All good. Very nice!

review: Approve

=== modified file 'serverguide/C/virtualization.xml'
--- serverguide/C/virtualization.xml	2012-03-11 16:42:45 +0000
+++ serverguide/C/virtualization.xml	2012-03-18 23:46:47 +0000
@@ -2215,4 +2215,1768 @@
 
     </sect2>
   </sect1>  
+  <sect1 id='lxc' status='review'>
+    <title>LXC</title>
+    <para>
+    Containers are a lightweight virtualization technology.  They are
+    more akin to an enhanced chroot than to full virtualization like
+    Qemu or VMware, both because they do not emulate hardware and
+    because containers share the same operating system as the host.
+    Therefore containers are better compared to Solaris zones or BSD
+    jails.  Linux-vserver and OpenVZ are two pre-existing, independently
+    developed implementations of containers-like functionality for
+    Linux.  In fact, containers came about as a result of the work to
+    upstream the vserver and OpenVZ functionality.  Some vserver and
+    OpenVZ functionality is still missing in containers, however
+    containers can <emphasis>boot</emphasis> many Linux distributions and have the
+    advantage that they can be used with an un-modified upstream kernel.
+    </para>
+
+    <para>
+    There are two user-space implementations of containers, each
+    exploiting the same kernel features.  Libvirt allows the use of
+    containers through the LXC driver by connecting to 'lxc:///'.  This
+    can be very convenient as it supports the same usage as its other
+    drivers.  The other implementation, called simply 'LXC', is not
+    compatible with libvirt, but is more flexible with more userspace
+    tools.  It is possible to switch between the two, though there are
+    peculiarities which can cause confusion.
+    </para>
+
+    <para>
+    In this document we will mainly describe the <application>lxc</application> package.  Toward
+    the end, we will describe how to use the libvirt LXC driver.
+    </para>
+    
+    <para>
+    In this document, a container name will be shown as CN, C1, or C2.
+    </para>
+
+    <sect2 id="lxc-installation" status="review">
+      <title>Installation</title>
+    <para>
+    The <application>lxc</application> package can be installed using
+    </para>
+
+<screen>
+<command>
+sudo apt-get install lxc
+</command>
+</screen>
+
+        <para>
+        This will pull in the required and recommended dependencies, including
+        cgroup-lite, lvm2, and debootstrap.  To use libvirt-lxc, install libvirt-bin.
+        LXC and libvirt-lxc can be installed and used at the same time.
+        </para>
+   </sect2>
+
+    <sect2 id="lxc-hostsetup" status="review">
+      <title>Host Setup</title>
+      <sect3 id="lxc-layout" status="review">
+        <title>Basic layout of LXC files</title>
+    <para>
+    Following is a description of the files and directories which
+    are installed and used by LXC.
+    </para>
+
+    <itemizedlist>
+    <listitem>
+    <para>There are two upstart jobs:</para>
+
+      <itemizedlist> <!-- nested list -->
+          <listitem>
+          <para>
+          <filename>/etc/init/lxc-net.conf:</filename> is an optional job which
+          only runs if <filename> /etc/default/lxc</filename> specifies
+          USE_LXC_BRIDGE (true by default).  It sets up a NATed bridge for
+          containers to use.
+          </para>
+          </listitem>
+
+          <listitem>
+          <para>
+          <filename>/etc/init/lxc.conf:</filename> runs if LXC_AUTO (true by
+	  default) is set to
+          true in <filename>/etc/default/lxc</filename>.  It looks for entries
+          under <filename>/etc/lxc/auto/</filename> which are symbolic links to
+          configuration files for the containers which should be started at boot.
+          </para>
+  	      </listitem>
+      </itemizedlist>
+
+      </listitem>
+      <listitem>
+      <para>
+      <filename>/etc/lxc/lxc.conf:</filename>
+      There is a default container creation configuration file,
+      <filename>/etc/lxc/lxc.conf</filename>, which directs containers to use
+      the LXC bridge created by the lxc-net upstart job.  If no configuration
+      file is specified when creating a container, then this one will be used.
+      </para>
+      </listitem>
+
+      <listitem>
+      <para>
+    Examples of other container creation configuration files are
+    found under <filename>/usr/share/doc/lxc/examples</filename>.  These show how to
+    create containers without a private network, or using macvlan,
+    vlan, or other network layouts.
+      </para>
+      </listitem>
+
+      <listitem>
+      <para>
+    The various container administration tools are found under
+    <filename>/usr/bin</filename>.
+      </para>
+      </listitem>
+
+      <listitem>
+      <para>
+    <filename>/usr/lib/lxc/lxc-init</filename> is a very minimal and lightweight init
+    binary which is used by lxc-execute.  Rather than `booting' a
+    full container, it manually mounts a few filesystems, especially
+    <filename>/proc</filename>, and executes its arguments.  You are not likely to need to
+    manually refer to this file.
+      </para>
+      </listitem>
+
+      <listitem>
+      <para>
+    <filename>/usr/lib/lxc/templates/</filename> contains the `templates' which can be
+    used to create new containers of various distributions and
+    flavors.  Not all templates are currently supported.
+      </para>
+      </listitem>
+
+      <listitem>
+      <para>
+    <filename>/etc/apparmor.d/usr.bin.lxc-start</filename> contains the (active by default)
+    apparmor MAC policy which works to protect the host from containers.
+    Please see the <xref linkend="lxc-security">Security</xref> section for more information.
+      </para>
+      </listitem>
+
+      <listitem>
+      <para>
+    There are various man pages for the LXC administration tools as well
+    as the <filename>lxc.conf</filename> container configuration file.
+      </para>
+      </listitem>
+
+      <listitem>
+      <para>
+    <filename>/var/lib/lxc</filename> is where containers and their configuration information
+    are stored.
+      </para>
+      </listitem>
+
+      <listitem>
+      <para>
+    <filename>/var/cache/lxc</filename> is where caches of distribution data are stored to
+    speed up multiple container creations.
+      </para>
+      </listitem>
+      </itemizedlist>
+      </sect3>
+
+      <sect3 id="lxcbr0" status="review">
+        <title>lxcbr0</title>
+    <para>
+    When USE_LXC_BRIDGE is set to true in /etc/default/lxc (as it is by
+    default), a bridge called lxcbr0 is created at startup.  This bridge is
+    given the private address 10.0.3.1, and containers using this bridge will
+    have a 10.0.3.0/24 address.  A dnsmasq instance is run listening on that
+    bridge, so if another dnsmasq has bound all interfaces before the lxc-net
+    upstart job runs, lxc-net will fail to start and lxcbr0 will not exist.
+    </para>
+
+    <para>
+    If you have another bridge - libvirt's default virbr0, or a br0
+    bridge for your default NIC - you can use that bridge in place of
+    lxcbr0 for your containers.
+    </para>
+       </sect3>
+
+      <sect3 id="lxc-partitions" status="review">
+        <title>Using a separate filesystem for the container store</title>
+    <para>
+    LXC stores container information and (with the default backing store) root
+    filesystems under <filename>/var/lib/lxc</filename>.  Container creation
+    templates also tend to store cached distribution information under
+    <filename>/var/cache/lxc</filename>.
+    </para>
+
+    <para>
+    If you wish to use another filesystem than
+    <filename>/var</filename>, you can mount a filesystem which has more space into those
+    locations.  If you have a disk dedicated for this, you can simply
+    mount it at <filename>/var/lib/lxc</filename>.  If you'd like to use another location, like
+    <filename>/srv</filename>, you can bind mount it or use a symbolic link.  For instance, if
+    <filename>/srv</filename> is a large mounted filesystem, create and symlink two directories:
+    </para>
+
+<screen>
+<command>
+sudo mkdir /srv/lxclib /srv/lxccache
+sudo rm -rf /var/lib/lxc /var/cache/lxc
+sudo ln -s /srv/lxclib /var/lib/lxc
+sudo ln -s /srv/lxccache /var/cache/lxc
+</command>
+</screen>
+
+    <para>
+    or, using bind mounts:
+    </para>
+
+<screen>
+<command>
+sudo mkdir /srv/lxclib /srv/lxccache
+sudo sed -i '$a \
+/srv/lxclib /var/lib/lxc    none defaults,bind 0 0 \
+/srv/lxccache /var/cache/lxc none defaults,bind 0 0' /etc/fstab
+sudo mount -a
+</command>
+</screen>
+
+      </sect3>
+
+      <sect3 id="lxc-lvm" status="review">
+        <title>Containers backed by lvm</title>
+
+    <para>
+    It is possible to use LVM partitions as the backing stores for
+    containers.  Advantages of this include flexibility in storage
+    management and fast container cloning.  The tools
+    default to using a VG (volume group) named <emphasis>lxc</emphasis>, but another
+    VG can be used through command line options.  When a LV is used
+    as a container backing store, the container's configuration file
+    is still <filename>/var/lib/lxc/CN/config</filename>, but the root fs
+    entry in that file (<emphasis>lxc.rootfs</emphasis>) will point to the lV block
+    device name, i.e. <filename>/dev/lxc/CN</filename>.
+    </para>
+
+    <para>
+    Containers with directory tree and LVM backing stores can
+    co-exist.
+    </para>
+      </sect3>
+
+      <sect3 id="lxc-btrfs" status="review">
+        <title>Btrfs</title>
+        <para>
+        If your host has a btrfs <filename>/var</filename>, the LXC administration
+        tools will detect this and automatically exploit it by
+        cloning containers using btrfs snapshots.
+        </para>
+      </sect3>
+
+      <sect3 id="lxc-apparmor" status="review">
+        <title>Apparmor</title>
+        <para>
+        LXC ships with an apparmor profile intended to protect the host
+        from accidental misuses of privilege inside the container.  For
+        instance, the container will not be able to write to
+        <filename>/proc/sysrq-trigger</filename> or to most <filename>/sys</filename> files.
+        </para>
+      </sect3>
+
+      <sect3 id="lxc-cgroups" status="review">
+        <title>Control Groups</title>
+        <para>
+        Control groups (cgroups) are a kernel feature providing hierarchical
+        task grouping and per-cgroup resource accounting and limits.  They are
+        used in containers to limit block and character device access and to
+        freeze (suspend) containers.  They can be further used to limit memory
+        use and block i/o, guarantee minimum cpu shares, and to lock containers
+        to specific cpus.  By default, LXC depends on the cgroup-lite package to be installed, which
+        provides the proper cgroup initialization at boot.  The cgroup-lite
+        package mounts each cgroup subsystem separately under
+        <filename>/sys/fs/cgroup/SS</filename>, where SS is the subsystem name.  For instance
+        the freezer subsystem is mounted under <filename>/sys/fs/cgroup/freezer</filename>.
+        LXC cgroup are kept under <filename>/sys/fs/cgroup/SS/INIT/lxc</filename>, where
+        INIT is the init task's cgroup.  This is <filename>/</filename> by default, so
+        in the end the freezer cgroup for container CN would be
+        <filename>/sys/fs/cgroup/freezer/lxc/CN</filename>.
+        </para>
+      </sect3>
+
+      <sect3 id="lxc-privs" status="review">
+        <title>Privilege</title>
+        <para>
+        The container administration tools must be run with root user
+        privilege.  A utility called <filename>lxc-setup</filename> was written with the
+        intention of providing the tools with the needed file capabilities to
+        allow non-root users to run the tools with sufficient privilege.
+        However, as root in a container cannot yet be reliably contained, this
+        is not worthwhile.  It is therefore recommended to not use
+        <filename>lxc-setup</filename>, and to provide the LXC administrators the needed
+        sudo privilege.
+        </para>
+
+        <para>
+        The user namespace, which is expected to be available in the next Long Term
+        Support (LTS) release, will allow containment of the container root user, as
+        well as reduce the amount of privilege required for creating and administering
+        containers.
+        </para>
+      </sect3>
+
+      <sect3 id="lxc-upstart" status="review">
+        <title>LXC Upstart Jobs</title>
+        <para>
+        As listed above, the <application>lxc</application> package includes two upstart jobs.  The
+        first, <filename>lxc-net</filename>, is always started when the other,
+        <filename>lxc</filename>, is about to begin, and stops when it stops.  If the
+        USE_LXC_BRIDGE variable is set to false in <filename>/etc/defaults/lxc</filename>,
+        then it will immediately exit.  If it is true, and an error occurs
+        bringing up the LXC bridge, then the <filename>lxc</filename> job will not start.
+        <filename>lxc-net</filename> will bring down the LXC bridge when stopped, unless
+        a container is running which is using that bridge.
+        </para>
+
+        <para>
+        The <filename>lxc</filename> job starts on runlevel 2-5.  If the LXC_AUTO variable
+        is set to true, then it will look under <filename>/etc/lxc</filename> for containers
+        which should be started automatically.  When the <filename>lxc</filename> job is
+        stopped, either manually or by entering runlevel 0, 1, or 6, it will
+        stop those containers.
+        </para>
+
+        <para>
+        To register a container to start automatically, create a symbolic
+        link <filename>/etc/default/lxc/name.conf</filename> pointing to the container's
+        config file.  For instance, the configuration file for a container
+        <filename>CN</filename> is <filename>/var/lib/lxc/CN/config</filename>.  To make that container
+        auto-start, use the command:
+        </para>
+
+<screen>
+<command>
+sudo ln -s /var/lib/lxc/CN/config /etc/lxc/auto/CN.conf
+</command>
+</screen>
+      </sect3>
+
+   </sect2>
+
+    <sect2 id="lxc-admin" status="review">
+      <title>Container Administration</title>
+      <sect3 id="lxc-creation" status="review">
+        <title>Creating Containers</title>
+
+    <para>
+    The easiest way to create containers is using <command>lxc-create</command>.  This
+    script uses distribution-specific templates under
+    <filename>/usr/lib/lxc/templates/</filename> to set up container-friendly chroots under
+    <filename>/var/lib/lxc/CN/rootfs</filename>, and initialize the configuration in
+    <filename>/var/lib/lxc/CN/fstab</filename> and
+    <filename>/var/lib/lxc/CN/config</filename>, where CN is the container name
+    </para>
+
+    <para>
+    The simplest container creation command would look like:
+    </para>
+
+<screen>
+<command>
+sudo lxc-create -t ubuntu -n CN
+</command>
+</screen>
+
+    <para>
+    This tells lxc-create to use the ubuntu template (-t ubuntu) and to call
+    the container CN (-n CN).  Since no configuration file was specified
+    (which would have been done with `-f file'), it will use the default
+    configuration file under <filename>/etc/lxc/lxc.conf</filename>.  This gives the container
+    a single veth network interface attached to the lxcbr0 bridge.
+    </para>
+
+    <para>
+    The container creation templates can also accept arguments.  These can
+    be listed after --.  For instance
+    </para>
+
+<screen>
+<command>
+sudo lxc-create -t ubuntu -n oneiric1 -- -r oneiric
+</command>
+</screen>
+
+    <para>
+    passes the arguments '-r oneiric1' to the ubuntu template.
+    </para>
+
+      <sect4 id="lxc-help" status="review">
+        <title>Help</title>
+    <para>
+    Help on the lxc-create command can be seen by using<command> lxc-create -h</command>.
+    However, the templates also take their own options.  If you do
+    </para>
+
+<screen>
+<command>
+sudo lxc-create -t ubuntu -h
+</command>
+</screen>
+
+    <para>
+    then the general <command>lxc-create</command> help will be followed by help output
+    specific to the ubuntu template.  If no template is specified, then only
+    help for <command>lxc-create</command> itself will be shown.
+    </para>
+       </sect4>
+
+       <sect4 id="lxc-ubuntu" status="review">
+        <title>Ubuntu template</title>
+
+    <para>
+    The ubuntu template can be used to create Ubuntu system containers with any
+    release at least as new as 10.04 LTS.  It uses debootstrap to create
+    a cached container filesystem which gets copied into place each time a
+    container is created.  The cached image is saved and only re-generated
+    when you create a container
+    using the <emphasis>-F</emphasis> (flush) option to the template, i.e.:
+    </para>
+
+<screen>
+<command>
+sudo lxc-create -t ubuntu -n CN -- -F
+</command>
+</screen>
+
+    <para>
+    The Ubuntu release installed by the template will be the same as that on
+    the host, unless otherwise specified with the <emphasis>-r</emphasis> option, i.e.
+    </para>
+
+<screen>
+<command>
+sudo lxc-create -t ubuntu -n CN -- -r lucid
+</command>
+</screen>
+
+    <para>
+    If you want to create a 32-bit container on a 64-bit host, pass <emphasis>-a i386</emphasis>
+    to the container.  If you have the qemu-user-static package installed, then you can
+    create a container using any architecture supported by qemu-user-static.
+    </para>
+
+    <para>
+    The container will have a user named <emphasis>ubuntu</emphasis> whose password is <emphasis>ubuntu</emphasis>
+    and who is a member of the <emphasis>sudo</emphasis> group.  If you wish to inject a public ssh
+    key for the <emphasis>ubuntu</emphasis> user, you can do so with <emphasis>-S sshkey.pub</emphasis>.
+    </para>
+
+    <para>
+    You can also <emphasis>bind</emphasis> user jdoe from the host into the container using
+    the <emphasis>-b jdoe</emphasis> option.  This will copy jdoe's password and shadow
+    entries into the container, make sure his default group and shell are
+    available, add him to the sudo group, and bind-mount his home directory
+    into the container when the container is started.
+    </para>
+
+    <para>
+    When a container is created, the <filename>release-updates</filename> archive is added
+    to the container's <filename>sources.list</filename>, and its package archive will be
+    updated.  If the container release is older than 12.04 LTS, then the
+    lxcguest package will be automatically installed.  Alternatively, if the <emphasis>--trim</emphasis>
+    option is specified, then the lxcguest package will not be installed,
+    and many services will be removed from the container.  This will result
+    in a faster-booting, but less upgrade-able container.
+    </para>
+       </sect4>
+
+       <sect4 id="lxc-ubuntu-cloud" status="review">
+        <title>Ubuntu-cloud template</title>
+
+    <para>
+    The ubuntu-cloud template  creates Ubuntu containers by downloading and
+    extracting the published Ubuntu cloud images.  It accepts some of the same
+    options as the ubuntu template, namely <emphasis>-r release</emphasis>, <emphasis>-S sshkey.pub</emphasis>,
+    <emphasis>-a arch</emphasis>, and <emphasis>-F</emphasis> to flush the cached image.  It also accepts a few
+    extra options.  The <emphasis>-C</emphasis> option will create a <emphasis>cloud</emphasis> container,
+    configured for use with a metadata service.  The <emphasis>-u</emphasis> option accepts a
+    cloud-init user-data file to configure the container on start.  If <emphasis>-L</emphasis>
+    is passed, then no locales will be installed.  The <emphasis>-T</emphasis> option can be
+    used to choose a tarball location to extract in place of the published
+    cloud image tarball.  Finally the <emphasis>-i</emphasis> option sets a host id for
+    cloud-init, which by default is set to a random string.
+    </para>
+       </sect4>
+
+       <sect4 id="lxc-other-templates" status="review">
+        <title> Other templates</title>
+
+    <para>
+    The ubuntu and ubuntu-cloud templates are well supported.  Other
+    templates are available however.  The debian template creates a
+    Debian based container, using debootstrap much as the ubuntu
+    template does.  By default it installs a <emphasis>debian squeeze</emphasis>
+    image.  An alternate release can be chosen by setting the SUITE
+    environment variable, i.e.:
+    </para>
+
+<screen>
+<command>
+sudo SUITE=sid lxc-create -t debian -n d1
+</command>
+</screen>
+
+    <para>
+    Since debian cannot be safely booted inside a container, debian
+    containers will be trimmed as with the <emphasis>--trim</emphasis> option to
+    the ubuntu template.
+    </para>
+
+    <para>
+    To purge the container image cache, call the template directly
+    and pass it the <emphasis>--clean</emphasis> option.
+    </para>
+
+<screen>
+<command>
+sudo SUITE=sid /usr/lib/lxc/templates/lxc-debian --clean
+</command>
+</screen>
+
+    <para>
+    A fedora template exists, which creates containers based on
+    fedora releases &lt;= 14.  Fedora release 15 and higher are
+    based on systemd, which the template is not yet able to convert
+    into a container-bootable setup.  Before the fedora template is
+    able to run, you'll need to make sure that <command>yum</command> and <command>curl</command>
+    are installed.  A fedora 12 container can be created with
+    </para>
+
+<screen>
+<command>
+sudo lxc-create -t fedora -n fedora12 -- -R 12
+</command>
+</screen>
+
+    <para>
+    A OpenSuSE template exists, but it requires the <command>zypper</command> program,
+    which is not yet packaged.  The OpenSuSE template is therefore
+    not supported.
+    </para>
+
+    <para>
+    Two more templates exist mainly for experimental purposes.  The
+    busybox template creates a very small system container based
+    entirely on busybox.  The sshd template creates an application
+    container running sshd in a private network namespace.  The
+    host's library and binary directories are bind-mounted into the
+    container, though not its <filename>/home</filename> or
+    <filename>/root</filename>.  To create, start, and ssh into an ssh
+    container, you might:
+    </para>
+
+<screen>
+<command>
+sudo lxc-create -t sshd -n ssh1
+ssh-keygen -f id
+sudo mkdir /var/lib/lxc/ssh1/rootfs/root/.ssh
+sudo cp id.pub /var/lib/lxc/ssh1/rootfs/root/.ssh/authorized_keys
+sudo lxc-start -n ssh1 -d
+ssh -i id root@ssh1.
+</command>
+</screen>
+
+       </sect4>
+
+       <sect4 id="lxc-backing-stores" status="review">
+        <title> Backing Stores</title>
+
+    <para>
+By default, <command>lxc-create</command> places the container's root
+filesystem as a directory tree at <filename>/var/lib/lxc/CN/rootfs.</filename>
+Another option is to use LVM logical volumes.  If a volume group named <emphasis>lxc</emphasis>
+exists, you can create an lvm-backed container called CN using:
+    </para>
+
+<screen>
+<command>
+sudo lxc-create -t ubuntu -n CN -B lvm
+</command>
+</screen>
+
+    <para>
+    If you want to use a volume group named schroots, with a 5G xfs
+    filesystem, then you would use
+    </para>
+
+<screen>
+<command>
+sudo lxc-create -t ubuntu -n CN -B lvm --vgname schroots --fssize 5G --fstype xfs
+</command>
+</screen>
+      </sect4>
+
+     </sect3>
+
+     <sect3 id="lxc-cloning" status="review">
+      <title>Cloning</title>
+
+    <para>
+    For rapid provisioning, you may wish to customize a canonical
+    container according to your needs and then make multiple copies of it.
+    This can be done with the <command>lxc-clone</command> program.  Given an existing
+    container called C1, a new container called C2 can be created
+    using
+    </para>
+
+
+<screen>
+<command>
+sudo lxc-clone -o C1 -n C2
+</command>
+</screen>
+
+    <para>
+    If <filename>/var/lib/lxc</filename> is a btrfs filesystem, then
+    <command>lxc-clone</command> will create C2's filesystem as a snapshot of
+    C1's.  If the container's root filesystem is lvm backed, then you can
+    specify the <emphasis>-s</emphasis> option to create the new rootfs as a lvm snapshot of the
+    original as follows:
+    </para>
+
+<screen>
+<command>
+sudo lxc-clone -s -o C1 -n C2
+</command>
+</screen>
+
+    <para>
+    Both lvm and btrfs snapshots will provide fast cloning with very
+    small initial disk usage.
+    </para>
+    </sect3>
+
+     <sect3 id="lxc-start-stop" status="review">
+      <title>Starting and stopping</title>
+
+    <para>
+    To start a container, use <command>lxc-start -n CN</command>.  By default
+    <command>lxc-start</command> will execute <filename>/sbin/init</filename>
+    in the container.  You can provide a different program to execute, plus
+    arguments, as further arguments to <command>lxc-start</command>:
+    </para>
+
+<screen>
+<command>
+sudo lxc-start -n container /sbin/init loglevel=debug
+</command>
+</screen>
+
+    <para>
+    If you do not specify the <emphasis>-d</emphasis> (daemon) option, then you will see a
+    console (on the container's <filename>/dev/console</filename>, see 
+    <xref linkend="lxc-consoles"/> for more information) on the terminal.  If
+    you specify the <emphasis>-d</emphasis> option, you will not see that console, and lxc-start
+    will immediately exit success - even if a later part of container startup
+    has failed.  You can use <command>lxc-wait</command> or
+    <command>lxc-monitor</command> (see <xref
+    linkend="lxc-monitoring"/>) to check on the success or failure of the
+    container startup.
+    </para>
+
+    <para>
+    To obtain LXC debugging information, use <emphasis>-o filename -l debuglevel</emphasis>,
+    for instance:
+    </para>
+
+<screen>
+<command>
+sudo lxc-start -o lxc.debug -l DEBUG -n container
+</command>
+</screen>
+
+    <para>
+    Finally, you can specify configuration parameters inline using <emphasis>-s</emphasis>.
+    However, it is generally recommended to place them in the container's
+    configuration file instead.  Likewise, an entirely alternate config
+    file can be specified with the <emphasis>-f</emphasis> option, but this is not
+    generally recommended.
+    </para>
+
+    <para>
+    While <command>lxc-start</command> runs the container's
+    <filename>/sbin/init</filename>, <command>lxc-execute</command> uses a
+    minimal init program called <command>lxc-init</command>, which attempts to
+    mount <filename>/proc</filename>, <filename>/dev/mqueue</filename>, and
+    <filename>/dev/shm</filename>, executes the programs specified on the
+    command line, and waits for those to finish executing.
+    <command>lxc-start</command> is intended to be used for <emphasis>system containers</emphasis>,
+    while <command>lxc-execute</command> is intended for <emphasis>application
+        containers</emphasis> (see <ulink url="https://www.ibm.com/developerworks/linux/library/l-lxc-containers/">
+    this article</ulink> for more).
+    </para>
+
+    <para>
+    You can stop a container several ways.  You can use <command>shutdown</command>,
+    <command>poweroff</command> and <command>reboot</command> while logged into
+    the container.  To cleanly shut down a container externally (i.e. from the host), you can issue
+    the <command>sudo lxc-shutdown -n CN</command> command.  This takes an optional
+    timeout value.  If not specified, the command issues a SIGPWR signal to the
+    container and immediately returns.  If the option is used, as in
+    <command>sudo lxc-shutdown -n CN -t 10</command>, then the command will wait the
+    specified number of seconds for the container to cleanly shut down.  Then,
+    if the container is still running, it will kill it (and any running
+    applications).  You can also immediately kill the container (without any
+    chance for applications to cleanly shut down) using
+    <command>sudo lxc-stop -n CN</command>.  Finally,
+    <command>lxc-kill</command> can be used more generally to send any signal
+    number to the container's init.
+    </para>
+
+    <para>
+    While the container is shutting down, you can expect to see some (harmless)
+    error messages, as follows:
+    </para>
+
+<screen>
+$ sudo poweroff
+[sudo] password for ubuntu: =
+
+$ =
+
+Broadcast message from ubuntu@cn1
+        (/dev/lxc/console) at 18:17 ...
+
+The system is going down for power off NOW!
+ * Asking all remaining processes to terminate...
+   ...done.
+ * All processes ended within 1 seconds....
+   ...done.
+ * Deconfiguring network interfaces...
+   ...done.
+ * Deactivating swap...
+   ...fail!
+umount: /run/lock: not mounted
+umount: /dev/shm: not mounted
+mount: / is busy
+ * Will now halt
+</screen>
+
+    <para>
+    A container can be frozen with <command>sudo lxc-freeze -n CN</command>.  This
+    will block all its processes until the container is later unfrozen using
+    <command>sudo lxc-unfreeze -n CN</command>.
+    </para>
+
+    </sect3>
+
+     <sect3 id="lxc-monitoring" status="review">
+      <title>Monitoring container status </title>
+
+    <para>
+    Two commands are available to monitor container state changes.
+    <command>lxc-monitor</command> monitors one or more containers for any
+    state changes.  It takes a container name as usual with the <emphasis>-n</emphasis> option,
+    but in this case the container name can be a posix regular expression to
+    allow monitoring desirable sets of containers.
+    <command>lxc-monitor</command> continues running as it prints container
+    changes.  <command>lxc-wait</command> waits for a specific state change and
+    then exits.  For instance,
+    </para>
+
+
+<screen>
+<command>
+sudo lxc-monitor -n cont[0-5]*
+</command>
+</screen>
+
+    <para>
+    would print all state changes to any containers matching the
+    listed regular expression, whereas
+    </para>
+
+<screen>
+<command>
+sudo lxc-wait -n cont1 -s 'STOPPED|FROZEN'
+</command>
+</screen>
+
+    <para>
+    will wait until container cont1 enters state STOPPED or state FROZEN
+    and then exit.
+    </para>
+    </sect3>
+
+     <sect3 id="lxc-consoles" status="review">
+      <title>Consoles</title>
+
+    <para>
+    Containers have a configurable number of consoles.  One always exists on
+    the container's <filename>/dev/console.</filename>  This is shown on the
+    terminal from which you ran <command>lxc-start</command>, unless the <emphasis>-d</emphasis>
+    option is specified.  The output on <filename>/dev/console</filename> can
+    be redirected to a file using the <emphasis>-c console-file</emphasis> option to
+    <command>lxc-start</command>.  The number of extra consoles is specified by
+    the <command>lxc.tty</command> variable, and is usually set to 4.  Those
+    consoles are shown on <filename>/dev/ttyN</filename> (for 1 &lt;= N &lt;=
+    4).  To log into console 3 from the host, use
+    </para>
+
+<screen>
+<command>
+sudo lxc-console -n container -t 3
+</command>
+</screen>
+
+    <para>
+    or if the <emphasis>-t N</emphasis> option is not specified, an unused console will be
+    automatically chosen.  To exit the console, use the escape sequence
+    Ctrl-a q.  Note that the escape sequence does not work in the console
+    resulting from <command>lxc-start</command> without the <emphasis>-d</emphasis>
+    option.
+    </para>
+
+    <para>
+    Each container console is actually a Unix98 pty in the host's (not the
+    guest's) pty mount, bind-mounted over the guest's
+    <filename>/dev/ttyN</filename> and <filename>/dev/console</filename>.
+    Therefore, if the guest unmounts those or otherwise tries to access the
+    actual character device <command>4:N</command>, it will not be serving
+    getty to the LXC consoles.  (With the default settings, the container will
+    not be able to access that character device and getty will therefore fail.)
+    This can easily happen when a boot script blindly mounts a new
+    <filename>/dev</filename>.
+    </para>
+    </sect3>
+
+     <sect3 id="lxc-introspection" status="review">
+      <title>Container Inspection</title>
+
+    <para>
+    Several commands are available to gather information on existing
+    containers.  <command>lxc-ls</command> will report all existing containers
+    in its first line of output, and all running containers in the second line.
+    <command>lxc-list</command> provides the same information in a more verbose
+    format, listing running containers first and stopped containers next.
+    <command>lxc-ps</command> will provide lists of processes in containers.
+    To provide <command>ps</command> arguments to <command>lxc-ps</command>,
+    prepend them with <command>--</command>.  For instance, for listing of all
+    processes in container plain,
+    </para>
+
+<screen>
+<command>
+sudo lxc-ps -n plain -- -ef
+</command>
+</screen>
+
+    <para>
+    <command>lxc-info</command> provides the state of a container and the pid of its init
+    process.  <command>lxc-cgroup</command> can be used to query or set the values of a
+    container's control group limits and information.  This can be more convenient
+    than interacting with the <command>cgroup</command> filesystem.  For instance, to query
+    the list of devices which a running container is allowed to access,
+    you could use
+    </para>
+
+<screen>
+<command>
+sudo lxc-cgroup -n CN devices.list
+</command>
+</screen>
+
+    <para>
+    or to add mknod, read, and write access to <filename>/dev/sda</filename>,
+    </para>
+
+<screen>
+<command>
+sudo lxc-cgroup -n CN devices.allow "b 8:* rwm"
+</command>
+</screen>
+
+    <para>
+    and, to limit it to 300M of RAM,
+    </para>
+
+<screen>
+<command>
+lxc-cgroup -n CN memory.limit_in_bytes 300000000
+</command>
+</screen>
+
+    <para>
+    <command>lxc-netstat</command> executes <command>netstat</command> in the
+    running container, giving you a glimpse of its network state.
+    </para>
+
+    <para>
+    <command>lxc-backup</command> will create backups of the root filesystems
+    of all existing containers (except lvm-based ones), using
+    <command>rsync</command> to back the contents up under
+    <filename>/var/lib/lxc/CN/rootfs.backup.1</filename>.  These backups can be
+    restored using <command>lxc-restore.</command>  However,
+    <command>lxc-backup</command> and <command>lxc-restore</command> are
+    fragile with respect to customizations and therefore their use is not
+    recommended.
+    </para>
+
+    </sect3>
+
+     <sect3 id="lxc-destroying" status="review">
+      <title>Destroying containers</title>
+
+    <para>
+    Use <command>lxc-destroy</command> to destroy an existing container.
+    </para>
+
+<screen>
+<command>
+sudo lxc-destroy -n CN
+</command>
+</screen>
+
+    <para>
+    If the container is running, <command>lxc-destroy</command> will exit with a message
+    informing you that you can force stopping and destroying the container
+    with
+    </para>
+
+<screen>
+<command>
+sudo lxc-destroy -n CN -f
+</command>
+</screen>
+
+    </sect3>
+
+     <sect3 id="lxc-namespaces" status="review">
+      <title>Advanced namespace usage</title>
+
+    <para>
+    One of the Linux kernel features used by LXC to create containers is
+    private namespaces.  Namespaces allow a set of tasks to have private
+    mappings of names to resources for things like pathnames and process
+    IDs.  (See <xref linkend="lxc-resources">Resources</xref> for a link
+    to more information).  Unlike control groups and other mount features which
+    are also used to create containers, namespaces cannot be manipulated using
+    a filesystem interface.  Therefore, LXC ships with the <command>lxc-unshare</command>
+    program, which is mainly for testing.  It provides the ability to create
+    new tasks in private namespaces.  For instance,
+    </para>
+
+<screen>
+<command>
+sudo lxc-unshare -s 'MOUNT|PID' /bin/bash
+</command>
+</screen>
+
+    <para>
+    creates a bash shell with private pid and mount namespaces.
+    In this shell, you can do
+    </para>
+
+<screen>
+root@ubuntu:~# mount -t proc proc /proc
+root@ubuntu:~# ps -ef
+UID        PID  PPID  C STIME TTY          TIME CMD
+root         1     0  6 10:20 pts/9    00:00:00 /bin/bash
+root       110     1  0 10:20 pts/9    00:00:00 ps -ef
+</screen>
+
+    <para>
+    so that <command>ps</command> shows only the tasks in your new namespace.
+    </para>
+    </sect3>
+
+     <sect3 id="lxc-ephemeral" status="review">
+      <title>Ephemeral containers</title>
+
+    <para>
+    Ephemeral containers are one-time containers.  Given an existing
+    container CN, you can run a command in an ephemeral container
+    created based on CN, with the host's jdoe user bound into the
+    container, using:
+    </para>
+
+<screen>
+<command>
+lxc-start-ephemeral -b jdoe -o CN -- /home/jdoe/run_my_job
+</command>
+</screen>
+
+    <para>
+    When the job is finished, the container will be discarded.
+    </para>
+
+     </sect3>
+     <sect3 id="lxc-commands" status="review">
+      <title>Container Commands</title>
+
+Following is a table of all container commands:
+
+<table>
+<title> Container commands</title>
+<tgroup cols="2" rowsep="1">
+<thead>
+ <row>
+  <entry valign="left"><para>Command</para></entry>
+  <entry valign="left"><para>Synopsis</para></entry>
+ </row>
+</thead>
+<tbody>
+ <row>
+  <entry><para>lxc-attach </para></entry>
+  <entry><para>(NOT SUPPORTED) Run a command in a running container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-backup </para></entry>
+  <entry><para>Back up the root filesystems for all lvm-backed containers</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-cgroup </para></entry>
+  <entry><para>View and set container control group settings</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-checkconfig </para></entry>
+  <entry><para>Verify host support for containers</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-checkpoint </para></entry>
+  <entry><para>(NOT SUPPORTED) Checkpoint a running container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-clone </para></entry>
+  <entry><para>Clone a new container from an existing one</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-console </para></entry>
+  <entry><para>Open a console in a running container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-create </para></entry>
+  <entry><para>Create a new container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-destroy </para></entry>
+  <entry><para>Destroy an existing container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-execute </para></entry>
+  <entry><para>Run a command in a (not running) application container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-freeze </para></entry>
+  <entry><para>Freeze a running container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-info </para></entry>
+  <entry><para>Print information on the state of a container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-kill </para></entry>
+  <entry><para>Send a signal to a container's init</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-list </para></entry>
+  <entry><para>List all containers</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-ls </para></entry>
+  <entry><para>List all containers with shorter output than lxc-list</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-monitor </para></entry>
+  <entry><para>Monitor state changes of one or more containers</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-netstat </para></entry>
+  <entry><para>Execute netstat in a running container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-ps </para></entry>
+  <entry><para>View process info in a running container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-restart </para></entry>
+  <entry><para>(NOT SUPPORTED) Restart a checkpointed container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-restore </para></entry>
+  <entry><para>Restore containers from backups made by lxc-backup</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-setcap </para></entry>
+  <entry><para>(NOT RECOMMENDED) Set file capabilities on LXC tools</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-setuid </para></entry>
+  <entry><para>(NOT RECOMMENDED) Set or remove setuid bits on LXC tools</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-shutdown </para></entry>
+  <entry><para>Safely shut down a container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-start </para></entry>
+  <entry><para>Start a stopped container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-start-ephemeral </para></entry>
+  <entry><para>Start an ephemeral (one-time) container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-stop </para></entry>
+  <entry><para>Immediately stop a running container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-unfreeze </para></entry>
+  <entry><para>Unfreeze a frozen container</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-unshare </para></entry>
+  <entry><para>Testing tool to manually unshare namespaces</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-version </para></entry>
+  <entry><para>Print the version of the LXC tools</para></entry>
+ </row>
+ <row>
+  <entry><para>lxc-wait </para></entry>
+  <entry><para>Wait for a container to reach a particular state</para></entry>
+ </row>
+ </tbody>
+ </tgroup>
+</table>
+
+   </sect3>
+   </sect2>
+
+    <sect2 id="lxc-conf" status="review">
+      <title>Configuration File</title>
+
+    <para>
+    LXC containers are very flexible.  The Ubuntu <application>lxc</application> package sets defaults
+    to make creation of Ubuntu system containers as simple as possible.
+    If you need more flexibility, this chapter will show how to fine-tune
+    your containers as you need.
+    </para>
+
+    <para>
+    Detailed information is available in the <command>lxc.conf(5)</command> man page.
+    Note that the default configurations created by the ubuntu templates
+    are reasonable for a system container and usually do not need
+    customization.
+    </para>
+
+      <sect3 id="lxc-conf-options" status="review">
+        <title>Choosing configuration files and options</title>
+
+    <para>
+    The container setup is controlled by the LXC configuration options.
+    Options can be specified at several points:
+    </para>
+
+    <itemizedlist>
+    <listitem><para>
+    During container creation, a configuration file can be specified.
+    However, creation templates often insert their own configuration
+    options, so we usually specify only network configuration options at
+    this point.  For other configuration, it is usually better to edit the
+    configuration file after container creation.
+    </para></listitem>
+
+    <listitem><para>
+    The file <filename>/var/lib/lxc/CN/config</filename> is used at
+    container startup by default.
+    </para></listitem>
+
+    <listitem><para>
+    <command>lxc-start</command> accepts an alternate configuration file with
+    the <emphasis>-f filename</emphasis> option.
+    </para></listitem>
+
+    <listitem><para>
+    Specific configuration variables can be overridden at <command>lxc-start</command>
+    using <emphasis>-s key=value</emphasis>.  It is generally better to edit the container
+    configuration file.
+    </para></listitem>
+
+    </itemizedlist>
+
+     </sect3>
+
+      <sect3 id="lxc-conf-net" status="review">
+        <title>Network Configuration</title>
+
+    <para>
+    Container networking in LXC is very flexible.  It is triggered by
+    the <command>lxc.network.type</command> configuration file entries.
+    If no such entries exist, then the container will share the host's
+    networking stack.  Services and connections started in the container
+    will be using the host's IP address.
+    If at least one <command>lxc.network.type</command> entry is present, then the container
+    will have a private (layer 2) network stack.  It will have its own
+    network interfaces and firewall rules.  There are several options
+    for <command>lxc.network.type</command>:
+    </para>
+
+    <itemizedlist>
+    <listitem><para>
+    <command>lxc.network.type=empty</command>:
+    The container will have no network interfaces other than loopback.
+    </para></listitem>
+
+    <listitem><para>
+    <command>lxc.network.type=veth</command>:
+    This is the default when using the ubuntu or ubuntu-cloud templates,
+    and creates a veth network tunnel.  One end of this tunnel
+    becomes the network interface inside the container.  The other end
+    is attached to a bridged on the host.  Any number of such tunnels
+    can be created by adding more <command>lxc.network.type=veth</command>
+    entries in the container configuration file.  The bridge to which the
+    host end of the tunnel will be attached is specified with
+    <command>lxc.network.link = lxcbr0</command>.
+    </para></listitem>
+
+    <listitem><para>
+    <command>lxc.network.type=phys</command>
+    A physical network interface (i.e. eth2) is passed into the container.
+    </para></listitem>
+    </itemizedlist>
+
+    <para>
+    Two other options are to
+    use vlan or macvlan, however their use is more complicated and is
+    not described here.  A few other networking options exist:
+    </para>
+
+    <itemizedlist>
+    <listitem><para>
+    <command>lxc.network.flags</command> can only be set to <emphasis>up</emphasis> and ensures that the network interface is up.
+    </para></listitem>
+
+    <listitem><para>
+    <command>lxc.network.hwaddr</command> specifies a mac address to assign the the
+    nic inside the container.
+    </para></listitem>
+
+    <listitem><para>
+    <command>lxc.network.ipv4</command> and <command>lxc.network.ipv6</command>
+    set the respective IP addresses, if those should be static.
+    </para></listitem>
+
+    <listitem><para>
+    <command>lxc.network.name</command> specifies a name to assign inside the
+    container.  If this is not specified, a good default (i.e. eth0 for the
+    first nic) is chosen.
+    </para></listitem>
+
+    <listitem><para>
+    <command>lxc.network.lxcscript.up</command> specifies a script to be called
+    after the host side of the networking has been set up.  See the
+    <command>lxc.conf(5)</command> manual page for details.
+    </para></listitem>
+    </itemizedlist>
+
+     </sect3>
+
+      <sect3 id="lxc-conf-cgroup" status="review">
+        <title>Control group configuration</title>
+
+    <para>
+    Cgroup options can be specified using <command>lxc.cgroup</command>
+    entries.  <command>lxc.cgroup.subsystem.item = value</command> instructs
+    LXC to set cgroup <command>subsystem</command>'s <command>item</command> to
+    <command>value</command>.  It is perhaps simpler to realize that this will
+    simply write <command>value</command> to the file <command>item</command>
+    for the container's control group for subsystem
+    <command>subsystem</command>.  For instance, to set the memory limit to
+    320M, you could add
+    </para>
+
+<screen>
+<command>
+lxc.cgroup.memory.limit_in_bytes = 320000000
+</command>
+</screen>
+
+    <para>
+    which will cause 320000000 to be written to the file
+    <filename>/sys/fs/cgroup/memory/lxc/CN/limit_in_bytes</filename>.
+    </para>
+     </sect3>
+
+      <sect3 id="lxc-conf-mounts" status="review">
+        <title>Rootfs, mounts and fstab</title>
+
+    <para>
+    An important part of container setup is the mounting of various
+    filesystems into place.  The following is an example configuration file
+    excerpt demonstrating the commonly used configuration options:
+    </para>
+
+<screen>
+<command>
+lxc.rootfs = /var/lib/lxc/CN/rootfs
+lxc.mount.entry=proc /var/lib/lxc/CN/rootfs/proc proc nodev,noexec,nosuid 0 0
+lxc.mount = /var/lib/lxc/CN/fstab
+</command>
+</screen>
+
+    <para>
+    The first line says that the container's root filesystem is already mounted
+    at <filename>/var/lib/lxc/CN/rootfs</filename>.  If the filesystem is a
+    block device (such as an LVM logical volume), then the path to the block
+    device must be given instead.
+    </para>
+
+    <para>
+    Each <command>lxc.mount.entry</command> line should contain an item to
+    mount in valid fstab format.  The target directory should be prefixed by
+    <filename>/var/lib/lxc/CN/rootfs</filename>, even if
+    <command>lxc.rootfs</command> points to a block device.
+    </para>
+
+    <para>
+    Finally, <command>lxc.mount</command> points to a file, in fstab format,
+    containing further items to mount.  Note that all of these entries will be
+    mounted by the host before the container init is started.  In this way it
+    is possible to bind mount various directories from the host into the
+    container.
+    </para>
+     </sect3>
+
+      <sect3 id="lxc-conf-other" status="review">
+        <title>Other configuration options</title>
+
+    <itemizedlist>
+
+    <listitem>
+    <para>
+    <command>lxc.cap.drop</command> can be used to prevent the container from having
+    or ever obtaining the listed capabilities.  For instance, including
+    </para>
+
+<screen>
+<command>
+lxc.cap.drop = sys_admin
+</command>
+</screen>
+
+    <para>
+    will prevent the container from mounting filesystems, as well as all other
+    actions which require cap_sys_admin.  See the <command>capabilities(7)</command>
+    manual page for a list of capabilities and their meanings.
+    </para>
+    </listitem>
+
+    <listitem><para>
+    <command>lxc.console=/path/to/consolefile</command> will cause console
+    messages to be written to the specified file.
+    </para></listitem>
+
+    <listitem><para>
+    <command>lxc.arch</command> specifies the architecture for the container, for instance
+    x86, or x86_64.
+    </para></listitem>
+
+    <listitem><para>
+    <command>lxc.tty=5</command> specifies that 5 consoles (in addition to
+    <filename>/dev/console</filename>) should be created.  That is, consoles
+    will be available on <filename>/dev/tty1</filename> through
+    <filename>/dev/tty5</filename>.  The ubuntu templates set this value to 4.
+    </para></listitem>
+
+    <listitem>
+    <para>
+    <command>lxc.pts=1024</command> specifies that the container should have a
+    private (Unix98) devpts filesystem mount.  If this is not specified, then
+    the container will share <filename>/dev/pts</filename> with the host, which
+    is rarely desired.  The number 1024 means that 1024 ptys should be allowed
+    in the container, however this number is currently ignored.  Before
+    starting the container init, LXC will do (essentially) a
+    </para>
+
+<screen>
+<command>
+sudo mount -t devpts -o newinstance devpts /dev/pts
+</command>
+</screen>
+
+    <para>
+    inside the container.  It is important to realize that the container should
+    not mount devpts filesystems of its own.  It may safely do bind or move
+    mounts of its mounted <filename>/dev/pts</filename>.  But if it does
+    </para>
+
+<screen>
+<command>
+sudo mount -t devpts devpts /dev/pts
+</command>
+</screen>
+
+    <para>
+    it will remount the host's devpts
+    instance.  If it adds the newinstance mount option, then it will mount a new
+    private (empty) instance.  In neither case will it remount the instance
+    which was set up by LXC.  For this reason, and to prevent the container
+    from using the host's ptys, the default apparmor policy will not allow
+    containers to mount devpts filesystems after the container's init has been
+    started.
+    </para>
+    </listitem>
+
+    <listitem><para>
+    <command>lxc.devttydir</command> specifies a directory under
+    <filename>/dev</filename> in which LXC will create its console devices.  If
+    this option is not specified, then the ptys will be bind-mounted over
+    <filename>/dev/console</filename> and <filename>/dev/ttyN.</filename>
+    However, rare package updates may try to blindly <emphasis>rm -f</emphasis> and then
+    <emphasis>mknod</emphasis> those devices.  They will fail (because the file has been
+    bind-mounted), causing the package update to fail.  When
+    <command>lxc.devttydir</command> is set to LXC, for instance, then LXC will
+    bind-mount the console ptys onto <filename>/dev/lxc/console</filename> and
+    <filename>/dev/lxc/ttyN,</filename> and subsequently symbolically link them
+    to <filename>/dev/console</filename> and <filename>/dev/ttyN.</filename>
+    This allows the package updates to succeed, at the risk of making future
+    gettys on those consoles fail until the next reboot.  This problem will be
+    ideally solved with device namespaces.
+    </para></listitem>
+
+    </itemizedlist>
+
+     </sect3>
+
+   </sect2>
+
+    <sect2 id="lxc-container-updates" status="review">
+      <title>Updates in Ubuntu containers</title>
+
+    <para>
+    Because of some limitations which are placed on containers, package upgrades at
+    times can fail.  For instance, a package install or upgrade might fail if it
+    is not allowed to create or open a block device.  This often blocks all future
+    upgrades until the issue is resolved.  In some cases,
+    you can work around this by chrooting into the container, to avoid the
+    container restrictions, and completing the upgrade in the chroot.
+    </para>
+
+    <para>
+    Some of the specific things known to occasionally impede package
+    upgrades include:
+    </para>
+
+    <itemizedlist>
+    <listitem><para>
+    The container modifications performed when creating containers with the
+    --trim option.
+    </para></listitem>
+    <listitem><para>
+    Actions performed by lxcguest.  For instance, because
+    <filename>/lib/init/fstab</filename> is bind-mounted from another file,
+    mountall upgrades which insist on replacing that file can fail.
+    </para></listitem>
+    <listitem><para>
+    The over-mounting of console devices with ptys from the host can
+    cause trouble with udev upgrades.
+    </para></listitem>
+    <listitem><para>
+    Apparmor policy and devices cgroup restrictions can prevent
+    package upgrades from performing certain actions.
+    </para></listitem>
+    <listitem><para>
+    Capabilities dropped by use of <command>lxc.cap.drop</command> can likewise stop package
+    upgrades from performing certain actions.
+    </para></listitem>
+    </itemizedlist>
+    </sect2>
+
+    <sect2 id="lxc-libvirt" status="review">
+      <title>Libvirt LXC</title>
+
+    <para>
+Libvirt is a powerful hypervisor management solution with which you can
+administer Qemu, Xen and LXC virtual machines, both locally and remote.
+The libvirt LXC driver is a separate implementation from what we normally
+call <emphasis>LXC</emphasis>.  A few differences include:
+    </para>
+
+    <itemizedlist>
+    <listitem><para>
+	Configuration is stored in xml format
+    </para></listitem>
+    <listitem><para>
+	There no tools to facilitate container creation
+    </para></listitem>
+    <listitem><para>
+	By default there is no console on <filename>/dev/console</filename>
+    </para></listitem>
+    <listitem><para>
+	There is no support (yet) for container reboot or full shutdown
+    </para></listitem>
+    </itemizedlist>
+
+<!--
+      <sect3 id="lxc-libvirt-virtinst" status="review">
+        <title>virt-install</title>
+
+    <para>
+    virt-install can be used to create an LXC container.  (test and
+    verify).  Serge hasn't gotten this to work.
+    </para>
+
+      </sect3>
+      -->
+
+      <sect3 id="lxc-libvirt-convert" status="review">
+        <title>Converting a LXC container to libvirt-lxc</title>
+
+    <para>
+
+    <xref linkend="lxc-creation"/> showed how to create LXC containers.
+    If you've created a valid LXC container in this way, you can
+    manage it with libvirt.  Fetch a sample xml file from
+    </para>
+
+<screen>
+<command>
+wget http://people.canonical.com/~serge/o1.xml
+</command>
+</screen>
+
+    <para>
+    Edit this file to replace the container name and root
+    filesystem locations.  Then you can define the container with:
+    </para>
+
+<screen>
+<command>
+virsh -c lxc:/// define o1.xml
+</command>
+</screen>
+      </sect3>
+
+      <sect3 id="lxc-libvirt-fromcloud" status="review">
+        <title>Creating a container from cloud image</title>
+
+    <para>
+If you prefer to create a pristine new container just for LXC, you
+can download an ubuntu cloud image, extract it, and point a libvirt
+LXC xml file to it.  For instance, find the url for a root tarball
+for the latest daily Ubuntu 12.04 LTS cloud image using
+    </para>
+
+<screen>
+<command>
+url1=`ubuntu-cloudimg-query precise daily $arch --format "%{url}\n"`
+url=`echo $url1 | sed -e 's/.tar.gz/-root\0/'`
+wget $url
+filename=`basename $url`
+</command>
+</screen>
+
+    <para>
+    Extract the downloaded tarball, for instance
+    </para>
+
+<screen>
+<command>
+mkdir $HOME/c1
+cd $HOME/c1
+sudo tar zxf $filename
+</command>
+</screen>
+
+    <para>
+    Download the xml template 
+    </para>
+
+<screen>
+<command>
+wget http://people.canonical.com/~serge/o1.xml
+</command>
+</screen>
+
+    <para>
+    In the xml template, replace the name o1 with c1 and the source directory
+    <filename>/var/lib/lxc/o1/rootfs</filename> with
+    <filename>$HOME/c1</filename>.  Then define the container using
+    </para>
+
+<screen>
+<command>
+virsh define o1.xml
+</command>
+</screen>
+
+      </sect3>
+
+      <sect3 id="lxc-libvirt-interacting" status="review">
+        <title>Interacting with libvirt containers</title>
+
+    <para>
+    As we've seen, you can create a libvirt-lxc container using
+    </para>
+
+<screen>
+<command>
+virsh -c lxc:/// define container.xml
+</command>
+</screen>
+
+    <para>
+    To start a container called <emphasis>container</emphasis>, use
+    </para>
+
+<screen>
+<command>
+virsh -c lxc:/// start container
+</command>
+</screen>
+
+    <para>
+    To stop a running container, use
+    </para>
+
+<screen>
+<command>
+virsh -c lxc:/// destroy container
+</command>
+</screen>
+
+    <para>
+    Note that whereas the <command>lxc-destroy</command> command deletes the
+    container, the <command>virsh destroy</command> command stops a running
+    container.  To delete the container definition, use
+    </para>
+
+<screen>
+<command>
+virsh -c lxc:/// undefine container
+</command>
+</screen>
+
+    <para>
+    To get a console to a running container, use
+    </para>
+
+<screen>
+<command>
+virsh -c lxc:/// console container
+</command>
+</screen>
+
+    <para>
+    Exit the console by simultaneously pressing control and ].
+    </para>
+
+      </sect3>
+
+    </sect2>
+
+    <sect2 id="lxc-guest" status="review">
+      <title>The lxcguest package</title>
+
+    <para>
+    In the 11.04 (Natty) and 11.10 (Oneiric) releases of Ubuntu, a package was introduced called
+    <emphasis role="italic">lxcguest</emphasis>.  An unmodified root image could not be safely booted inside a
+    container, but an image with the lxcguest package installed could be 
+    booted as a container, on bare hardware, or in a Xen, kvm, or VMware virtual
+    machine.
+    </para>
+
+    <para>
+    As of the 12.04 LTS release, the work previously done by the lxcguest package
+    was pushed into the core packages, and the lxcguest package was removed.
+    As a result, an unmodified 12.04 LTS image can be booted as a
+    container, on bare hardware, or in a Xen, kvm, or VMware virtual machine.
+    To use an older release, the lxcguest package should still be used.
+    </para>
+
+    </sect2>
+
+    <sect2 id="lxc-security" status="review">
+      <title>Security</title>
+
+    <para>
+    A namespace maps ids to resources.  By not providing a container any id
+    with which to reference a resource, the resource can be protected.  This
+    is the basis of some of the security afforded to container users.  For
+    instance, IPC namespaces are completely isolated.  Other namespaces,
+    however, have various <emphasis role="italic">leaks</emphasis> which allow privilege to be
+    inappropriately exerted from a container into another container or to
+    the host.
+    </para>
+
+    <para>
+    By default, LXC containers are started under a apparmor policy to
+    restrict some actions.  However, while stronger security is a goal
+    for future releases, in 1204 LTS the goal of the apparmor policy is not
+    to stop malicious actions but rather to stop accidental harm of the
+    host by the guest.
+    </para>
+
+    <para>
+    See the <ulink url="http://wiki.ubuntu.com/LxcSecurity">LXC security</ulink>
+    wiki page for more, uptodate information.
+    </para>
+
+      <sect3 id="lxc-seccomp" status="review">
+        <title>Exploitable system calls</title>
+
+    <para>
+    It is a core container feature that containers share a kernel with the
+    host.  Therefore, if the kernel contains any exploitable system calls,
+    the container can exploit these as well.  Once the container controls the
+    kernel it can fully control any resource known to the host.
+    </para>
+
+      </sect3>
+    </sect2>
+
+		<sect2 id="lxc-resources" status="review">
+		<title>Resources</title>
+	<itemizedlist>
+
+	<listitem>
+	<para>
+        The DeveloperWorks article <ulink url="https://www.ibm.com/developerworks/linux/library/l-lxc-containers/">LXC: Linux container tools</ulink> was an early introduction to the use of containers.
+	</para>
+	</listitem>
+
+	<listitem>
+	<para>
+        The <ulink url="http://www.ibm.com/developerworks/linux/library/l-lxc-security/index.html"> Secure Containers Cookbook</ulink> demonstrated the use of security modules to make containers more secure.
+	</para>
+	</listitem>
+
+	<listitem>
+	<para>
+	Manual pages referenced above can be found at:
+<programlisting>
+<ulink url="http://manpages.ubuntu.com/manpages/en/man7/capabilities.7.html">capabilities</ulink>
+<ulink url="http://manpages.ubuntu.com/manpages/en/man5/lxc.conf.5.html">lxc.conf</ulink>
+</programlisting>
+	</para>
+	</listitem>
+
+	<listitem>
+	<para>
+	The upstream LXC project is hosted at <ulink url="http://lxc.sf.net">Sourceforge</ulink>.
+	</para>
+	</listitem>
+
+	<listitem>
+	<para>
+	LXC security issues are listed and discussed at <ulink url="http://wiki.ubuntu.com/LxcSecurity">the LXC Security wiki page</ulink>
+	</para>
+	</listitem>
+
+	<listitem>
+	<para> For more on namespaces in Linux, see: S. Bhattiprolu, E. W. Biederman, S. E. Hallyn, and D. Lezcano. Virtual Servers and Check- point/Restart in Mainstream Linux. SIGOPS Op- erating Systems Review, 42(5), 2008.</para>
+	</listitem>
+
+	</itemizedlist>
+		</sect2>
+   </sect1>
 </chapter>