Merge lp:~raharper/curtin/trunk.documentation-update into lp:~curtin-dev/curtin/trunk

[Scott Moser]

[lp:~raharper/curtin/trunk.documentation-update]

389. By Ryan Harper on 2016-05-19

Flesh out some more networking; Pull in an previous version of storage documentation.

390. By Ryan Harper on 2016-05-19

fix indentation warnings.

391. By Ryan Harper on 2016-05-20

Move curtin VERSION into module; use in setup.py, and doc building.

392. By Ryan Harper on 2016-05-20

doc: Content update to storage.rst.

393. By Ryan Harper on 2016-05-20

doc: use rtd theme

394. By Ryan Harper on 2016-05-23

Apply some common formatting to list elements that are keys in dictionaries. Remove redundent lvm entry.

395. By Ryan Harper on 2016-05-25

Clear up network configuration types and subnet keys.

396. By Ryan Harper on 2016-06-01

Change net/storage table of contents to include the syntax formatting

397. By Ryan Harper on 2016-06-01

aspell -c overview.rst

398. By Ryan Harper on 2016-06-01

aspell -c networking.rst

399. By Ryan Harper on 2016-06-01

aspell -c development.rst

400. By Ryan Harper on 2016-06-01

aspell -c integration-testing.rst

401. By Ryan Harper on 2016-06-01

aspell -c reporting.rst

402. By Ryan Harper on 2016-06-01

aspell -c storage.rst

403. By Ryan Harper on 2016-07-14

Add curtin top-level config documentation.

404. By Ryan Harper on 2016-07-14

Fixes suggested by smoser.

405. By Ryan Harper on 2016-07-14

Fix swap multiplier values

406. By Ryan Harper on 2016-08-18

drop mention of device in route config

407. By Ryan Harper on 2016-08-18

merge with trunk

408. By Ryan Harper on 2016-08-18

cleanup make docs after merge

409. By Ryan Harper on 2016-08-29

merge from trunk

410. By Ryan Harper on 2016-08-29

merge from trunk

=== modified file 'Makefile'
--- Makefile	2016-07-26 13:30:35 +0000
+++ Makefile	2016-08-29 23:08:04 +0000
@@ -49,5 +49,7 @@
 sync-images:
 	@$(CWD)/tools/vmtest-sync-images
 
+clean:
+	rm -rf doc/_build
 
-.PHONY: all test pyflakes pyflakes3 pep8 build
+.PHONY: all clean test pyflakes pyflakes3 pep8 build
 
=== modified file 'curtin/__init__.py'
--- curtin/__init__.py	2016-07-15 17:20:36 +0000
+++ curtin/__init__.py	2016-08-29 23:08:04 +0000
@@ -37,4 +37,6 @@
     'APT_CONFIG_V1',
 ]
 
+__version__ = "0.1.0"
+
 # vi: ts=4 expandtab syntax=python
 
=== modified file 'doc/conf.py'
--- doc/conf.py	2015-08-27 18:24:31 +0000
+++ doc/conf.py	2016-08-29 23:08:04 +0000
@@ -13,6 +13,11 @@
 
 import sys, os
 
+# Fix path so we can import curtin.__version__
+sys.path.insert(1, os.path.realpath(os.path.join(
+                                    os.path.dirname(__file__), '..')))
+import curtin
+
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
@@ -41,16 +46,16 @@
 
 # General information about the project.
 project = u'curtin'
-copyright = u'2013, Scott Moser'
+copyright = u'2016, Scott Moser, Ryan Harper'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
 # built documents.
 #
 # The short X.Y version.
-version = '0.3'
+version = curtin.__version__
 # The full version, including alpha/beta/rc tags.
-release = '0.3'
+release = version
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -93,6 +98,18 @@
 # a list of builtin themes.
 html_theme = 'classic'
 
+# on_rtd is whether we are on readthedocs.org, this line of code grabbed from
+# docs.readthedocs.org
+on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
+
+if not on_rtd:  # only import and set the theme if we're building docs locally
+    import sphinx_rtd_theme
+    html_theme = 'sphinx_rtd_theme'
+    html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
+
+# otherwise, readthedocs.org uses their theme by default, so no need to specify
+# it
+
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
@@ -120,7 +137,7 @@
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['static']
+#html_static_path = ['static']
 
 # If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
 # using the given strftime format.
 
=== removed file 'doc/devel/README-vmtest.txt'
--- doc/devel/README-vmtest.txt	2016-08-02 08:18:23 +0000
+++ doc/devel/README-vmtest.txt	1970-01-01 00:00:00 +0000
@@ -1,221 +0,0 @@
-== Background ==
-Curtin includes a mechanism called 'vmtest' that allows it to actually
-do installs and validate a number of configurations.
-
-The general flow of the vmtests is:
- 1. each test has an associated yaml config file for curtin in examples/tests
- 2. uses curtin-pack to create the user-data for cloud-init to trigger install
- 3. create and install a system using 'tools/launch'.
-    3.1 The install environment is booted from a maas ephemeral image.
-    3.2 kernel & initrd used are from maas images (not part of the image)
-    3.3 network by default is handled via user networking
-    3.4 It creates all empty disks required
-    3.5 cloud-init datasource is provided by launch
-      a) like: ds=nocloud-net;seedfrom=http://10.7.0.41:41518/
-         provided by python webserver start_http
-      b) via -drive file=/tmp/launch.8VOiOn/seed.img,if=virtio,media=cdrom
-         as a seed disk (if booted without external kernel)
-    3.6 dependencies and other preparations are installed at the beginning by
-        curtin inside the ephemeral image prior to configuring the target
- 4. power off the system.
- 5. configure a 'NoCloud' datasource seed image that provides scripts that
-    will run on first boot.
-    5.1 this will contain all our code to gather health data on the install
-    5.2 by cloud-init design this runs only once per instance, if you start
-        the system again this won't be called again
- 6. boot the installed system with 'tools/xkvm'.
-    6.1 reuses the disks that were installed/configured in the former steps
-    6.2 also adds an output disk
-    6.3 additionally the seed image for the data gathering is added
-    6.4 On this boot it will run the provided scripts, write their output to a
-        "data" disk and then shut itself down.
- 7. extract the data from the output disk
- 8. vmtest python code now verifies if the output is as expected.
-
-== Debugging ==
-At 3.1
-  - one can pull data out of the maas image with
-    sudo mount-image-callback your.img -- sh -c 'COMMAND'
-    e.g. sudo mount-image-callback your.img -- sh -c 'cp $MOUNTPOINT/boot/* .'
-At step 3.6 -> 4.
-  - tools/launch can be called in a way to give you console access
-    to do so just call tools/launch but drop the -serial=x parameter.
-    One might want to change "'power_state': {'mode': 'poweroff'}" to avoid
-    the auto reboot before getting control
-    Replace the directory usually seen in the launch calls with a clean fresh
-    directory
-  - In /curtin curtin and its config can be found
-  - if the system gets that far cloud-init will create a user ubuntu/passw0rd
-  - otherwise one can use a cloud-image from  https://cloud-images.ubuntu.com/
-    and add a backdoor user via
-    bzr branch lp:~maas-maintainers/maas/backdoor-image backdoor-image
-    sudo ./backdoor-image -v --user=<USER> --password-auth --password=<PW> IMG
-At step 6 -> 7
-  - You might want to keep all the temporary images around.
-    To do so you can set CURTIN_VMTEST_KEEP_DATA_PASS=all:
-    export CURTIN_VMTEST_KEEP_DATA_PASS=all CURTIN_VMTEST_KEEP_DATA_FAIL=all
-    That will keep the /tmp/tmpXXXXX directories and all files in there for
-    further execution.
-At step 7
-  - You might want to take a look at the output disk yourself.
-    It is a normal qcow image, so one can use mount-image-callback as described
-    above
-  - to invoke xkvm on your own take the command you see in the output and
-    remove the "-serial ..." but add -nographic instead
-    For graphical console one can add --vnc 127.0.0.1:1
-
-== Setup ==
-In order to run vmtest you'll need some dependencies.  To get them, you 
-can run:
-  make vmtest-deps
-
-That will install all necessary dependencies.
-
-== Running ==
-Running tests is done most simply by:
-
-  make vmtest
-
-If you wish to all tests in test_network.py, do so with:
-  sudo PATH=$PWD/tools:$PATH nosetests3 tests/vmtests/test_network.py
-
-Or run a single test with:
-  sudo PATH=$PWD/tools:$PATH nosetests3 tests/vmtests/test_network.py:WilyTestBasic
-
-Note:
-  * currently, the tests have to run as root.  The reason for this is that
-    the kernel and initramfs to boot are extracted from the maas ephemeral
-    image.  This should be fixed at some point, and then 'make vmtest'
-
-    The tests themselves don't actually have to run as root, but the
-    test setup does.
-  * the 'tools' directory must be in your path.
-  * test will set apt: { proxy } in the guests to the value of
-    'apt_proxy' environment variable.  If that is not set it will
-    look at the host's apt config and read 'Acquire::HTTP::Proxy'
-
-== Environment Variables ==
-Some environment variables affect the running of vmtest
-  * apt_proxy:
-    test will set apt_proxy in the guests to the value of 'apt_proxy'.
-    If that is not set it will look at the host's apt config and read
-    'Acquire::HTTP::Proxy'
-
-  * CURTIN_VMTEST_KEEP_DATA_PASS CURTIN_VMTEST_KEEP_DATA_FAIL:
-    default:
-      CURTIN_VMTEST_KEEP_DATA_PASS=none
-      CURTIN_VMTEST_KEEP_DATA_FAIL=all
-    These 2 variables determine what portions of the temporary
-    test data are kept.
-
-    The variables contain a comma ',' delimited list of directories
-    that should be kept in the case of pass or fail.  Additionally,
-    the values 'all' and 'none' are accepted.
-
-    Each vmtest that runs has its own sub-directory under the top level
-    CURTIN_VMTEST_TOPDIR.  In that directory are directories:
-      boot: inputs to the system boot (after install)
-      install: install phase related files
-      disks: the disks used for installation and boot
-      logs: install and boot logs
-      collect: data collected by the boot phase
-
-  * CURTIN_VMTEST_TOPDIR: default $TMPDIR/vmtest-<timestamp>
-    vmtest puts all test data under this value.  By default, it creates
-    a directory in TMPDIR (/tmp) named with as "vmtest-<timestamp>"
-
-    If you set this value, you must ensure that the directory is either
-    non-existant or clean.
-
-  * CURTIN_VMTEST_LOG: default $TMPDIR/vmtest-<timestamp>.log
-    vmtest writes extended log information to this file.
-    The default puts the log along side the TOPDIR.
-
-  * CURTIN_VMTEST_IMAGE_SYNC: default false (boolean)
-    if set to true, each run will attempt a sync of images.
-    If you want to make sure images are always up to date, then set to true.
-
-  * CURTIN_VMTEST_BRIDGE: default 'user'
-    the network devices will be attached to this bridge.  The default is
-    'user', which means to use qemu user mode networking.  Set it to
-    'virbr0' or 'lxcbr0' to use those bridges and then be able to ssh
-    in directly.
-
-  * CURTIN_VMTEST_BOOT_TIMEOUT: default 300
-    timeout before giving up on the boot of the installed system.
-
-  * CURTIN_VMTEST_INSTALL_TIMEOUT: default 3000
-    timeout before giving up on installation.
-
-  * CURTIN_VMTEST_PARALLEL: default ''
-    only supported through ./tools/jenkins-runner .
-       -1 : then run one per core.
-        0 or '': then run with no parallel
-       >0 : run with N processes
-    this modifies the  invocation of nosetets to add '--processes' and other
-    necessary nose arguments (--process-timeout)
-
-  * IMAGE_DIR: default /srv/images
-    vmtest keeps a mirror of maas ephemeral images in this directory.
-
-  * IMAGES_TO_KEEP: default 1
-    keep this number of images of each release in the IMAGE_DIR.
-
-Environment 'boolean' values:
-   For boolean environment variables the value is considered True
-   if it is any value other than case insensitive 'false', '' or "0"
-
-
-== Test Class Variables ==
-The base VMBaseClass defines several variables that help creating a new test
-easily. Among those the common ones are:
-
-Generic:
-- arch_skip
-  If a test is not supported on an architecture it can list the arch in this
-  variable to auto-skip the test if executed on that arch.
-- conf_file
-  The configuration that will be processed by this vmtest.
-- extra_kern_args
-  Extra arguments to the guest kernel on boot.
-
-Data Collection:
-- collect_scripts
-  The commands run when booting into the installed environment to collect the
-  data for the test to verify a proper execution.
-- boot_cloudconf
-  Extra cloud-init config content for the install phase.
-  This allows to gather content of the install phase if needed for test
-  verification.
-
-
-Disk Setup:
-- disk_block_size = 512
-- disk_driver = 'virtio-blk'
-  Used to set up the disks for the virtual environment used by the vmtest.
-  Will set the values used in extra_disks if not specified there.
-- extra_disks
-  A list of extra disks to be created for the testcase. The definition is
-  dpath:size:driver:block_size
-- multipath = False
-- multipath_num_paths = 2
-  Details for the (virtual) multipath setup
-- nvme_disks
-  a shortcut to provide extra disks with the nvme driver
-
-Timeouts:
-- boot_timeout
-- install_timeout
-  Usually set via CURTIN_VMTEST_BOOT_TIMEOUT/CURTIN_VMTEST_INSTALL_TIMEOUT
-  (see above) environment var, but can be overwritten by a testcase if needed.
-
-Checks:
-- disk_to_check
-  A disk name that is verified to be existing after the installation.
-- fstab_expected
-
-Release:
-- release = None
-- krel = None
-  Those two define the distribution and kernel release to be tested and are
-  usually set by importing and inheriting from tests/vmtests/releases.py
 
=== removed file 'doc/devel/README.txt'
--- doc/devel/README.txt	2015-03-04 13:30:53 +0000
+++ doc/devel/README.txt	1970-01-01 00:00:00 +0000
@@ -1,55 +0,0 @@
-## curtin development ##
-
-This document describes how to use kvm and ubuntu cloud images
-to develop curtin or test install configurations inside kvm.
-
-## get some dependencies ##
-sudo apt-get -qy install kvm libvirt-bin cloud-utils bzr
-
-## get cloud image to boot (-disk1.img) and one to install (-root.tar.gz)
-mkdir -p ~/download
-DLDIR=$( cd ~/download && pwd )
-rel="trusty"
-arch=amd64
-burl="http://cloud-images.ubuntu.com/$rel/current/"
-for f in $rel-server-cloudimg-${arch}-root.tar.gz $rel-server-cloudimg-${arch}-disk1.img; do
-  wget "$burl/$f" -O $DLDIR/$f; done
-( cd $DLDIR && qemu-img convert -O qcow $rel-server-cloudimg-${arch}-disk1.img $rel-server-cloudimg-${arch}-disk1.qcow2)
-
-BOOTIMG="$DLDIR/$rel-server-cloudimg-${arch}-disk1.qcow2"
-ROOTTGZ="$DLDIR/$rel-server-cloudimg-${arch}-root.tar.gz"
-
-## get curtin
-mkdir -p ~/src
-bzr init-repo ~/src/curtin
-( cd ~/src/curtin && bzr  branch lp:curtin trunk.dist )
-( cd ~/src/curtin && bzr branch trunk.dist trunk )
-
-## work with curtin
-cd ~/src/curtin/trunk
-# use 'launch' to launch a kvm instance with user data to pack
-# up local curtin and run it inside instance.
-./tools/launch $BOOTIMG --publish $ROOTTGZ -- curtin install "PUBURL/${ROOTTGZ##*/}"
-
-## notes about 'launch' ##
- * launch has --help so you can see that for some info.
- * '--publish' adds a web server at ${HTTP_PORT:-9923}
-   and puts the files you want available there.  You can reference
-   this url in config or cmdline with 'PUBURL'.  For example
-   '--publish foo.img' will put 'foo.img' at PUBURL/foo.img.
- * launch sets 'ubuntu' user password to 'passw0rd'
- * launch runs 'kvm -curses'
-   kvm -curses keyboard info:
-     'alt-2' to go to qemu console
- * launch puts serial console to 'serial.log' (look there for stuff)
- * when logged in
-   * you can look at /var/log/cloud-init-output.log
-   * archive should be extracted in /curtin
-   * shell archive should be in /var/lib/cloud/instance/scripts/part-002
- * when logged in, and archive available at
-
-
-## other notes ##
- * need to add '--install-deps' or something for curtin
-   cloud-image in 12.04 has no 'python3'
-   ideally 'curtin --install-deps install' would get the things it needs
 
=== modified file 'doc/index.rst'
--- doc/index.rst	2015-08-27 19:54:41 +0000
+++ doc/index.rst	2016-08-29 23:08:04 +0000
@@ -13,7 +13,13 @@
    :maxdepth: 2
 
    topics/overview
+   topics/config
+   topics/apt_source
+   topics/networking
+   topics/storage
    topics/reporting
+   topics/development
+   topics/integration-testing
 
 
 
 
=== modified file 'doc/topics/apt_source.rst'
--- doc/topics/apt_source.rst	2016-08-02 07:53:52 +0000
+++ doc/topics/apt_source.rst	2016-08-29 23:08:04 +0000
@@ -9,7 +9,7 @@
 The feature has an optional target argument which - by default - is used to modify the environment that curtin currently installs (@TARGET_MOUNT_POINT).
 
 Features
---------
+~~~~~~~~
 
 * Add PGP keys to the APT trusted keyring
 
@@ -39,7 +39,7 @@
 
 
 Configuration
--------------
+~~~~~~~~~~~~~
 
 The general configuration of the apt feature is under an element called ``apt``.
 
@@ -53,7 +53,8 @@
 The key is the filename and will be prepended by /etc/apt/sources.list.d/ if it doesn't start with a ``/``.
 There are certain cases - where no content is written into a source.list file where the filename will be ignored - yet it can still be used as index for merging.
 
-The values inside the entries consist of the following optional entries::
+The values inside the entries consist of the following optional entries
+
 * ``source``: a sources.list entry (some variable replacements apply)
 
 * ``keyid``: providing a key to import via shortid or fingerprint
@@ -62,7 +63,8 @@
 
 * ``keyserver``: specify an alternate keyserver to pull keys from that were specified by keyid
 
-The section "sources" is is a dictionary (unlike most block/net configs which are lists). This format allows merging between multiple input files than a list like::
+The section "sources" is is a dictionary (unlike most block/net configs which are lists). This format allows merging between multiple input files than a list like ::
+
   sources:
      s1: {'key': 'key1', 'source': 'source1'}
 
@@ -101,8 +103,7 @@
 
   - make an example with a partial mirror that doesn't mirror the backports suite, so backports have to be disabled
 
-That would be specified as
-::
+That would be specified as ::
 
   apt:
     primary:
@@ -125,26 +126,31 @@
 
 
 Common snippets
----------------
+~~~~~~~~~~~~~~~
 This is a collection of additional ideas people can use the feature for customizing their to-be-installed system.
 
 * enable proposed on installing
-  apt:
-    sources:
-      proposed.list: deb $MIRROR $RELEASE-proposed main restricted universe multiverse
+
+::
+
+ apt:
+   sources:
+     proposed.list: deb $MIRROR $RELEASE-proposed main restricted universe multiverse
 
 * Make debug symbols available
-  apt:
-    sources:
-      ddebs.list: |
-        deb http://ddebs.ubuntu.com $RELEASE main restricted universe multiverse
-        deb http://ddebs.ubuntu.com $RELEASE-updates main restricted universe multiverse
-        deb http://ddebs.ubuntu.com $RELEASE-security main restricted universe multiverse
-        deb http://ddebs.ubuntu.com $RELEASE-proposed main restricted universe multiverse
-
+
+::
+
+ apt:
+   sources:
+     ddebs.list: |
+       deb http://ddebs.ubuntu.com $RELEASE main restricted universe multiverse
+       deb http://ddebs.ubuntu.com $RELEASE-updates main restricted universe multiverse
+       deb http://ddebs.ubuntu.com $RELEASE-security main restricted universe multiverse
+       deb http://ddebs.ubuntu.com $RELEASE-proposed main restricted universe multiverse
 
 Timing
-------
+~~~~~~
 The feature is implemented at the stage of curthooks_commands, which runs just after curtin has extracted the image to the target.
 Additionally it can be ran as standalong command "curtin -v --config <yourconfigfile> apt-config".
 
@@ -153,6 +159,6 @@
 
 
 Dependencies
-------------
+~~~~~~~~~~~~
 Cloud-init might need to resolve dependencies and install packages in the ephemeral environment to run curtin.
 Therefore it is recommended to not only provide an apt configuration to curtin for the target, but also one to the install environment via cloud-init.
 
=== added file 'doc/topics/config.rst'
--- doc/topics/config.rst	1970-01-01 00:00:00 +0000
+++ doc/topics/config.rst	2016-08-29 23:08:04 +0000
@@ -0,0 +1,551 @@
+====================
+Curtin Configuration
+====================
+
+Curtin exposes a number of configuration options for controlling Curtin
+behavior during installation.
+
+
+Configuration options
+---------------------
+Curtin's top level config keys are as follows:
+
+
+- apt_mirrors (``apt_mirrors``)
+- apt_proxy (``apt_proxy``)
+- block-meta (``block``)
+- debconf_selections (``debconf_selections``)
+- disable_overlayroot (``disable_overlayroot``)
+- grub (``grub``)
+- http_proxy (``http_proxy``)
+- install (``install``)
+- kernel (``kernel``)
+- kexec (``kexec``)
+- multipath (``multipath``)
+- network (``network``)
+- power_state (``power_state``)
+- reporting (``reporting``)
+- restore_dist_interfaces: (``restore_dist_interfaces``)
+- sources (``sources``)
+- stages (``stages``)
+- storage (``storage``)
+- swap (``swap``)
+- system_upgrade (``system_upgrade``)
+- write_files (``write_files``)
+
+
+apt_mirrors
+~~~~~~~~~~~
+Configure APT mirrors for ``ubuntu_archive`` and ``ubuntu_security``
+
+**ubuntu_archive**: *<http://local.archive/ubuntu>*
+
+**ubuntu_security**: *<http://local.archive/ubuntu>*
+
+If the target OS includes /etc/apt/sources.list, Curtin will replace
+the default values for each key set with the supplied mirror URL.
+
+**Example**::
+
+  apt_mirrors:
+    ubuntu_archive: http://local.archive/ubuntu
+    ubuntu_security: http://local.archive/ubuntu
+
+
+apt_proxy
+~~~~~~~~~
+Curtin will configure an APT HTTP proxy in the target OS
+
+**apt_proxy**: *<URL to APT proxy>*
+
+**Example**::
+
+  apt_proxy: http://squid.mirror:3267/
+
+
+block-meta
+~~~~~~~~~~
+Configure how Curtin selects and configures disks on the target
+system without providing a custom configuration (mode=simple).
+
+**devices**: *<List of block devices for use>*
+
+The ``devices`` parameter is a list of block device paths that Curtin may
+select from with choosing where to install the OS.
+
+**boot-partition**: *<dictionary of configuration>*
+
+The ``boot-partition`` parameter controls how to configure the boot partition
+with the following parameters:
+
+**enabled**: *<boolean>*
+
+Enabled will forcibly setup a partition on the target device for booting.
+
+**format**: *<['uefi', 'gpt', 'prep', 'mbr']>*
+
+Specify the partition format.  Some formats, like ``uefi`` and ``prep``
+are restricted by platform characteristics.
+
+**fstype**: *<filesystem type: one of ['ext3', 'ext4'], defaults to 'ext4'>*
+
+Specify the filesystem format on the boot partition.
+
+**label**: *<filesystem label: defaults to 'boot'>*
+
+Specify the filesystem label on the boot partition.
+
+**Example**::
+
+  block-meta:
+      devices:
+        - /dev/sda
+        - /dev/sdb
+      boot-partition:
+        - enabled: True
+          format: gpt
+          fstype: ext4
+          label: my-boot-partition
+
+
+debconf_selections
+~~~~~~~~~~~~~~~~~~
+Curtin will update the target with debconf set-selection values.  Users will
+need to be familiar with the package debconf options.  Users can probe a
+packages' debconf settings by using ``debconf-get-selections``.
+
+**selection_name**: *<debconf-set-selections input>*
+
+``debconf-set-selections`` is in the form::
+
+  <packagename> <packagename/option-name> <type> <value>
+
+**Example**::
+
+  debconf_selections:
+    set1: |
+      cloud-init cloud-init/datasources multiselect MAAS
+      lxd lxd/bridge-name string lxdbr0
+    set2: lxd lxd/setup-bridge boolean true
+
+
+
+disable_overlayroot
+~~~~~~~~~~~~~~~~~~~
+Curtin disables overlayroot in the target by default.
+
+**disable_overlayroot**: *<boolean: default True>*
+
+**Example**::
+
+  disable_overlayroot: False
+
+
+grub
+~~~~
+Curtin configures grub as the target machine's boot loader.  Users
+can control a few options to tailor how the system will boot after
+installation.
+
+**install_devices**: *<list of block device names to install grub>*
+
+Specify a list of devices onto which grub will attempt to install.
+
+**replace_linux_default**: *<boolean: default True>*
+
+Controls whether grub-install will update the Linux Default target
+value during installation.
+
+**update_nvram**: *<boolean: default False>*
+
+Certain platforms, like ``uefi`` and ``prep`` systems utilize
+NVRAM to hold boot configuration settings which control the order in
+which devices are booted.  Curtin by default will not attempt to
+update the NVRAM settings to preserve the system configuration.
+Users may want to force NVRAM to be updated such that the next boot
+of the system will boot from the installed device.
+
+**Example**::
+
+  grub:
+     install_devices:
+       - /dev/sda1
+     replace_linux_default: False
+     update_nvram: True
+
+
+http_proxy
+~~~~~~~~~~
+Curtin will export ``http_proxy`` value into the installer environment.
+
+**http_proxy**: *<HTTP Proxy URL>*
+
+**Example**::
+
+  http_proxy: http://squid.proxy:3728/
+
+
+
+install
+~~~~~~~
+Configure Curtin's install options.
+
+**log_file**: *<path to write Curtin's install.log data>*
+
+Curtin logs install progress by default to /var/log/curtin/install.log
+
+**post_files**: *<List of files to read from host to include in reporting data>*
+
+Curtin by default will post the ``log_file`` value to any configured reporter.
+
+**save_install_config**: *<Path to save merged curtin configuration file>*
+
+Curtin will save the merged configuration data into the target OS at 
+the path of ``save_install_config``.  This defaults to /root/curtin-install-cfg.yaml
+
+**Example**::
+
+  install:
+     log_file: /tmp/install.log
+     post_files:
+       - /tmp/install.log
+       - /var/log/syslog
+     save_install_config: /root/myconf.yaml
+
+
+kernel
+~~~~~~
+Configure how Curtin selects which kernel to install into the target image.
+If ``kernel`` is not configured, Curtin will use the default mapping below
+and determine which ``package`` value by looking up the current release
+and current kernel version running.
+
+
+**fallback-package**: *<kernel package-name to be used as fallback>*
+
+Specify a kernel package name to be used if the default package is not
+available.
+
+**mapping**: *<Dictionary mapping Ubuntu release to HWE kernel names>*
+
+Default mapping for Releases to package names is as follows::
+
+ precise:
+    3.2.0: 
+    3.5.0: -lts-quantal
+    3.8.0: -lts-raring
+    3.11.0: -lts-saucy
+    3.13.0: -lts-trusty
+  trusty:
+    3.13.0: 
+    3.16.0: -lts-utopic
+    3.19.0: -lts-vivid
+    4.2.0: -lts-wily
+    4.4.0: -lts-xenial
+  xenial:
+    4.3.0:
+    4.4.0:
+ 
+
+**package**: *<Linux kernel package name>*
+
+Specify the exact package to install in the target OS.
+
+**Example**::
+
+  kernel:
+    fallback-package: linux-image-generic
+    package: linux-image-generic-lts-xenial
+    mapping:
+      - xenial:
+        - 4.4.0: -my-custom-kernel    
+
+
+kexec
+~~~~~
+Curtin can use kexec to "reboot" into the target OS.
+
+**mode**: *<on>*
+
+Enable rebooting with kexec.
+
+**Example**::
+
+  kexec: on
+
+
+multipath
+~~~~~~~~~
+Curtin will detect and autoconfigure multipath by default to enable
+boot for systems with multipath.  Curtin does not apply any advanced
+configuration or tuning, rather it uses distro defaults and provides
+enough configuration to enable booting.
+
+**mode**: *<['auto', ['disabled']>*
+
+Defaults to auto which will configure enough to enable booting on multipath
+devices.  Disabled will prevent curtin from installing or configuring
+multipath.
+
+**overwrite_bindings**: *<boolean>*
+
+If ``overwrite_bindings`` is True then Curtin will generate new bindings
+file for multipath, overriding any existing binding in the target image.
+
+**Example**::
+
+  multipath:
+      mode: auto
+      overwrite_bindings: True
+
+
+network
+~~~~~~~
+Configure networking (see Networking section for details).
+
+**network_option_1**: *<option value>*
+
+**Example**::
+
+  network:
+     version: 1
+     config:
+       - type: physical
+         name: eth0
+         mac_address: "c0:d6:9f:2c:e8:80"
+         subnets:
+           - type: dhcp4
+
+
+power_state
+~~~~~~~~~~~
+Curtin can configure the target machine into a specific power state after
+completing an installation.  Default is to do nothing.
+
+**delay**: *<Integer seconds to delay change in state>*
+
+Curtin will wait ``delay`` seconds before changing the power state.
+
+**mode**: *<New power state is one of: [halt, poweroff, reboot]>*
+
+Curtin will transition the node into one of the new states listed.
+
+``halt`` will stop a machine, but may not cut the power to the system.
+``poweroff`` will stop a machine and request it shut off the power.
+``reboot`` will perform a platform reset.
+
+**message**:  *<message string>*
+
+The ``message`` string will be broadcast to system consoles prior to
+power state change.
+
+
+**Example**::
+
+  power_state:
+    mode: poweroff
+    delay: 5
+    message: Bye Bye
+
+
+reporting
+~~~~~~~~~
+Configure installation reporting (see Reporting section for details).
+
+**Example**::
+
+  reporting:
+    maas:
+      level: DEBUG
+      type: webhook
+      endpoint: http://localhost:8000/
+
+
+restore_dist_interfaces
+~~~~~~~~~~~~~~~~~~~~~~~
+Curtin can restore a copy of /etc/network/interfaces built in to cloud images.
+
+**restore_dist_interfaces**: *<boolean>*
+
+If True, then Curtin will restore the interfaces file into the target.
+
+
+**Example**::
+
+  restore_dist_interfaces: True
+
+
+sources
+~~~~~~~
+Specify the root image to install on to the target system.  The URI also
+configures the method used to copy the image to the target system.
+
+**sources**: *<List of source URIs>*
+
+``source URI`` may be one of:
+
+- **dd-**:  Use ``dd`` command to write image to target.
+- **cp://**: Use ``rsync`` command to copy source directory to target.
+- **file://**: Use ``tar`` command to extract source to target.
+- **http[s]://**: Use ``wget | tar`` commands to extract source to target.
+
+**Example Cloud-image**::
+
+  sources: 
+    - https://cloud-images.ubuntu.com/xenial/current/xenial-server-cloudimg-amd64-root.tar.gz
+
+**Example Custom DD image**::
+
+  sources: 
+    - dd-img: https://localhost/raw_images/centos-6-3.img
+
+**Example Copy from booted environment**::
+
+  sources: 
+    - cp:///
+
+
+**Example Copy from local tarball**::
+
+  sources: 
+    - file:///tmp/root.tar.gz
+
+
+stages
+~~~~~~
+Curtin installation executes in stages.  At each stage, Curtin will look for
+a list of commands to run at each stage by reading in from the Curtin config
+*<stage_name>_commands* which is a dictionary and each key contains a list
+of commands to run.  Users may override the stages value to control
+what curtin stages execute.  During each stage, the commands are executed
+in C Locale sort order.  Users should name keys in a NN-XXX format where NN
+is a two-digit number to exercise control over execution order.
+
+The following stages are defined in Curtin and 
+run by default.
+
+- **early**: *Preparing for Installation*
+
+This stage runs before any actions are taken for installation.  By default
+this stage does nothing.
+
+- **partitioning**: *Select and partition disks for installation*
+
+This stage runs ``curtin block-meta simple`` by default.
+
+- **network**: *Probe and configure networking*
+
+This stage runs ``curtin net-meta auto`` by default.
+
+- **extract**: *Writing install sources to disk*
+
+This stage runs ``curtin extract`` by default.
+
+- **extract**: *Writing install sources to disk*
+
+This stage runs ``curtin extract`` by default.
+
+- **curthooks**: *Configuring installed system*
+
+This stage runs ``curtin curthooks`` by default.
+
+- **hooks**: *Finalizing installation*
+
+This stage runs ``curtin hook`` by default.
+
+- **late**: *Executing late commands*
+
+This stage runs after Curtin has completed the installation.  By default
+this stage does nothing.
+
+**Example Custom Stages**::
+
+  # Skip the whole install and just run `mystage`
+  stages: ['early', 'late', 'mystage']
+  mystage_commands:
+     00-cmd: ['/usr/bin/foo']
+
+**Example Early and Late commands**::
+
+  early_commands:
+      99-cmd:  ['echo', 'I ran last']
+      00-cmd:  ['echo', 'I ran first']
+  late_commands:
+      50-cmd: ['curtin', 'in-target' '--', 'touch', '/etc/disable_overlayroot']
+    
+
+swap
+~~~~
+Curtin can configure a swapfile on the filesystem in the target system.
+Size settings can be integer or string values with suffix.  Curtin
+supports the following suffixes which multiply the value.
+
+- **B**: *1*
+- **K[B]**: *1 << 10*
+- **M[B]**: *1 << 20*
+- **G[B]**: *1 << 30*
+- **T[B]**: *1 << 40*
+
+Curtin will use a heuristic to configure the swapfile size if the ``size``
+parameter is not set to a specific value.  The ``maxsize`` sets the upper
+bound of the heuristic calculation.
+
+**filename**: *<path to swap file>* 
+
+Configure the filename of the swap file. Defaults to /swap.img
+
+**maxsize**: *<Size string>*
+
+Configure the max size of the swapfile, defaults to 8GB
+
+**size**: *<Size string>*
+
+Configure the exact size of the swapfile.  Setting ``size`` to 0 will
+disable swap.
+
+**Example**::
+
+  swap:
+    filename: swap.img
+    size: None
+    maxsize: 4GB
+
+
+system_upgrade
+~~~~~~~~~~~~~~
+Control if Curtin runs `dist-upgrade` in target after install.  Defaults to
+False.
+
+**enabled**: *<boolean>*
+
+**Example**::
+
+  system_upgrade:
+    enabled: False
+
+
+write_files
+~~~~~~~~~~~
+Curtin supports writing out arbitrary data to a file.
+``write_files`` accepts a dictionary of entries formatted as follows:
+
+**path**: *<path and filename to save content>*
+
+Specify the name and location of where to write the content.
+
+**permissions**: *<Unix permission string>*
+
+Specify the permissions mode as an integer or string of numbers.
+
+**content**: *<data>*
+
+Specify the content.
+
+**Example**::
+
+  write_files:
+    f1:
+      path: /file1
+      content: !!binary |
+        f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAwARAAAAAAABAAAAAAAAAAJAVAAAAAAA
+    f2: {path: /file2, content: "foobar", permissions: '0666'}
 
=== added file 'doc/topics/development.rst'
--- doc/topics/development.rst	1970-01-01 00:00:00 +0000
+++ doc/topics/development.rst	2016-08-29 23:08:04 +0000
@@ -0,0 +1,68 @@
+=================
+Developing Curtin 
+=================
+
+Curtin developers make use of cloud-images and kvm to help develop and test new
+curtin features.
+
+Install dependencies
+====================
+
+Install some virtualization and cloud packages to get started.::
+
+  sudo apt-get -qy install kvm libvirt-bin cloud-utils bzr
+
+
+Download cloud images
+=====================
+Curtin will use two cloud images (-disk1.img) is for booting, 
+(-root.tar.gz) is used for installing.::
+
+  mkdir -p ~/download
+  DLDIR=$( cd ~/download && pwd )
+  rel="trusty"
+  arch=amd64
+  burl="http://cloud-images.ubuntu.com/$rel/current/"
+  for f in $rel-server-cloudimg-${arch}-root.tar.gz $rel-server-cloudimg-${arch}-disk1.img; do
+    wget "$burl/$f" -O $DLDIR/$f; done
+  ( cd $DLDIR && qemu-img convert -O qcow $rel-server-cloudimg-${arch}-disk1.img $rel-server-cloudimg-${arch}-disk1.qcow2)
+
+  export BOOTIMG="$DLDIR/$rel-server-cloudimg-${arch}-disk1.qcow2"
+  export ROOTTGZ="$DLDIR/$rel-server-cloudimg-${arch}-root.tar.gz"
+
+
+Getting the source
+==================
+Download curtin source from launchpad via `bzr` command.::
+
+  mkdir -p ~/src
+  bzr init-repo ~/src/curtin
+  ( cd ~/src/curtin && bzr  branch lp:curtin trunk.dist )
+  ( cd ~/src/curtin && bzr branch trunk.dist trunk )
+
+Using curtin
+============
+Use `launch` to launch a kvm instance with user data to pack up
+local curtin and run it inside an instance.::
+
+  cd ~/src/curtin/trunk
+  ./tools/launch $BOOTIMG --publish $ROOTTGZ -- curtin install "PUBURL/${ROOTTGZ##*/}"
+
+
+Notes about 'launch'
+====================
+
+- launch has --help so you can see that for some info.
+- `--publish` adds a web server at ${HTTP_PORT:-$RANDOM_PORT}
+  and puts the files you want available there.  You can reference
+  this url in config or cmdline with 'PUBURL'.  For example
+  `--publish foo.img` will put `foo.img` at PUBURL/foo.img.
+- launch sets 'ubuntu' user password to 'passw0rd'
+- launch runs 'kvm -curses'
+  kvm -curses keyboard info:
+  'alt-2' to go to qemu console
+- launch puts serial console to 'serial.log' (look there for stuff)
+- when logged in
+  - you can look at /var/log/cloud-init-output.log
+  - archive should be extracted in /curtin
+  - shell archive should be in /var/lib/cloud/instance/scripts/part-002
 
=== added file 'doc/topics/integration-testing.rst'
--- doc/topics/integration-testing.rst	1970-01-01 00:00:00 +0000
+++ doc/topics/integration-testing.rst	2016-08-29 23:08:04 +0000
@@ -0,0 +1,245 @@
+===================
+Integration Testing
+===================
+
+Curtin includes an in-tree integration suite that runs hundreds of tests
+validating various forms of custom storage and network configurations across
+all of the supported Ubuntu LTS releases as well as some of the currently 
+supported interim releases.
+
+Background
+==========
+
+Curtin includes a mechanism called 'vmtest' that allows it to actually
+do installs and validate a number of configurations.
+
+The general flow of the vmtests is:
+
+#. each test has an associated yaml config file for curtin in examples/tests
+#. uses curtin-pack to create the user-data for cloud-init to trigger install
+#. create and install a system using 'tools/launch'.
+
+   #. The install environment is booted from a maas ephemeral image.
+   #. kernel & initrd used are from maas images (not part of the image)
+   #. network by default is handled via user networking
+   #. It creates all empty disks required
+   #. cloud-init datasource is provided by launch
+
+      #. like: ds=nocloud-net;seedfrom=http://10.7.0.41:41518/
+         provided by python webserver start_http
+      #. via -drive file=/tmp/launch.8VOiOn/seed.img,if=virtio,media=cdrom
+         as a seed disk (if booted without external kernel)
+
+   #. dependencies and other preparations are installed at the beginning by
+      curtin inside the ephemeral image prior to configuring the target
+
+#. power off the system.
+#. configure a 'NoCloud' datasource seed image that provides scripts that
+   will run on first boot.
+
+   #. this will contain all our code to gather health data on the install
+   #. by cloud-init design this runs only once per instance, if you start
+      the system again this won't be called again
+
+#. boot the installed system with 'tools/xkvm'.
+
+   #. reuses the disks that were installed/configured in the former steps
+   #. also adds an output disk
+   #. additionally the seed image for the data gathering is added
+   #. On this boot it will run the provided scripts, write their output to a
+      "data" disk and then shut itself down.
+
+#. extract the data from the output disk
+#. vmtest python code now verifies if the output is as expected.
+
+Debugging
+=========
+
+At 3.1 one can pull data out of the maas image the command 
+``mount-image-callback``.  For example::
+
+  sudo mount-image-callback your.img -- sh -c 'cp $MOUNTPOINT/boot/* .'
+
+At step 3.6 through 4 ``tools/launch`` can be called in a way to give you
+console access.  To do so just call tools/launch but drop the -serial=x
+parameter.  One might want to change "'power_state': {'mode': 'poweroff'}" to
+avoid the auto reboot before getting control.  Replace the directory usually
+seen in the launch calls with a clean fresh directory.
+
+In /curtin curtin and its config can be found. If the system gets that far
+cloud-init will create a user creds: ubuntu/passw0rd , otherwise one can use a
+cloud-image from  https://cloud-images.ubuntu.com/ and add a backdoor user
+via::
+
+  bzr branch lp:~maas-maintainers/maas/backdoor-image backdoor-image
+  sudo ./backdoor-image -v --user=<USER> --password-auth --password=<PW> IMG
+
+At step 6 -> 7 you might want to keep all the temporary images around.  To do
+so you can set ``CURTIN_VMTEST_KEEP_DATA_PASS=all`` in your environment. ::
+
+  export CURTIN_VMTEST_KEEP_DATA_PASS=all CURTIN_VMTEST_KEEP_DATA_FAIL=all
+
+That will keep the /tmp/tmpXXXXX directories and all files in there for further
+execution.
+
+At step 7 you might want to take a look at the output disk yourself.  It is a
+normal qcow image, so one can use ``mount-image-callback`` as described above.
+
+To invoke xkvm on your own take the command you see in the output and remove
+the "-serial ..." but add ``-nographic`` instead For graphical console one can
+add ``--vnc 127.0.0.1:1``
+
+Setup
+=====
+
+In order to run vmtest you'll need some dependencies.  To get them, you 
+can run::
+
+  make vmtest-deps
+
+Running
+=======
+
+Running tests is done most simply by::
+
+  make vmtest
+
+If you wish to all tests in test_network.py, do so with::
+
+  nosetests3 tests/vmtests/test_network.py
+
+Or run a single test with::
+
+  nosetests3 tests/vmtests/test_network.py:WilyTestBasic
+
+
+Environment Variables
+=====================
+
+Some environment variables affect the running of vmtest
+
+- ``apt_proxy``:
+
+    test will set apt: { proxy } in the guests to the value of ``apt_proxy``
+    environment variable.  If that is not set it will look at the host's apt
+    config and read ``Acquire::HTTP::Proxy``
+
+- ``CURTIN_VMTEST_KEEP_DATA_PASS``: Defaults to none.
+- ``CURTIN_VMTEST_KEEP_DATA_FAIL``: Defaults to all.
+
+  These 2 variables determine what portions of the temporary
+  test data are kept.
+
+  The variables contain a comma ',' delimited list of directories
+  that should be kept in the case of pass or fail.  Additionally,
+  the values 'all' and 'none' are accepted.
+
+  Each vmtest that runs has its own sub-directory under the top level
+  ``CURTIN_VMTEST_TOPDIR``.  In that directory are directories:
+
+    - ``boot``: inputs to the system boot (after install)
+    - ``install``: install phase related files
+    - ``disks``: the disks used for installation and boot
+    - ``logs``: install and boot logs
+    - ``collect``: data collected by the boot phase
+
+- ``CURTIN_VMTEST_TOPDIR``: default $TMPDIR/vmtest-<timestamp>
+
+  Vmtest puts all test data under this value.  By default, it creates
+  a directory in TMPDIR (/tmp) named with as ``vmtest-<timestamp>``
+
+  If you set this value, you must ensure that the directory is either
+  non-existent or clean.
+
+- ``CURTIN_VMTEST_LOG``: default $TMPDIR/vmtest-<timestamp>.log
+
+  Vmtest writes extended log information to this file.
+  The default puts the log along side the TOPDIR.
+
+- ``CURTIN_VMTEST_IMAGE_SYNC``: default false (boolean)
+
+  If set to true, each run will attempt a sync of images.
+  If you want to make sure images are always up to date, then set to true.
+
+- ``CURTIN_VMTEST_BRIDGE``: ``user``
+
+  The network devices will be attached to this bridge.  The default is
+  ``user``, which means to use qemu user mode networking.  Set it to
+  ``virbr0`` or ``lxdbr0`` to use those bridges and then be able to ssh
+  in directly.
+
+- ``CURTIN_VMTEST_BOOT_TIMEOUT``: default 300
+
+    timeout before giving up on the boot of the installed system.
+
+- ``CURTIN_VMTEST_INSTALL_TIMEOUT``: default 3000
+
+    timeout before giving up on installation.
+
+- ``CURTIN_VMTEST_PARALLEL``: default ''
+
+    only supported through ./tools/jenkins-runner .
+
+    - ``-1``: then run one per core.
+    - ``0`` or ``''``: run with no parallel
+    - ``>0``: run with N processes
+
+    This modifies the  invocation of nosetets to add '--processes' and other
+    necessary nose arguments (--process-timeout)
+
+- ``IMAGE_DIR``: default /srv/images
+
+  Vmtest keeps a mirror of maas ephemeral images in this directory.
+
+- ``IMAGES_TO_KEEP``: default 1
+
+  Controls the number of images of each release retained in the IMAGE_DIR.
+
+Environment 'boolean' values
+============================
+
+For boolean environment variables the value is considered True
+if it is any value other than case insensitive 'false', '' or "0".
+
+Test Class Variables
+====================
+
+The base VMBaseClass defines several variables that help creating a new test
+easily. Among those the common ones are:
+
+Generic:
+
+- ``arch_skip``: 
+
+  If a test is not supported on an architecture it can list the arch in this
+  variable to auto-skip the test if executed on that arch.
+
+- ``conf_file``:
+
+  The configuration that will be processed by this vmtest.
+
+- ``extra_kern_args``:
+
+  Extra arguments to the guest kernel on boot.
+
+Data Collection:
+
+- ``collect_scripts``:
+
+  The commands run when booting into the installed environment to collect the
+  data for the test to verify a proper execution.
+
+- ``boot_cloudconf``:
+
+  Extra cloud-init config content for the install phase.  This allows to gather
+  content of the install phase if needed for test verification.
+
+Disk Setup:
+
+- ``disk_block_size``:
+
+  Default block size ``512`` bytes.
+
+- ``disk_driver``:
+
+  Default block device driver is ``virtio-blk``.
 
=== added file 'doc/topics/networking.rst'
--- doc/topics/networking.rst	1970-01-01 00:00:00 +0000
+++ doc/topics/networking.rst	2016-08-29 23:08:04 +0000
@@ -0,0 +1,522 @@
+==========
+Networking
+==========
+
+Curtin supports a user-configurable networking configuration format.
+This format lets users (including via MAAS) to customize their machines'
+networking interfaces by assigning subnet configuration, virtual device
+creation (bonds, bridges, vlans) routes and DNS configuration.
+
+Curtin accepts a YAML input under the top-level ``network`` key
+to indicate that a user would like to specify a custom networking
+configuration.  Required elements of a network configuration are
+``config`` and ``version``.  Currently curtin only supports 
+network config version=1. ::
+
+  network:
+    version: 1
+    config: []
+       
+Configuration Types
+-------------------
+Within the network ``config`` portion, users include a list of configuration
+types.  The current list of support ``type`` values are as follows:
+  
+- Physical (``physical``)
+- Bond (``bond``)
+- Bridge (``bridge``)
+- VLAN (``vlan``)
+- Nameserver (``nameserver``)
+- Route (``route``)
+
+Physical, Bond, Bridge and VLAN types may also include IP configuration under
+the key ``subnets``.
+
+- Subnet/IP (``subnets``)
+
+
+Physical
+~~~~~~~~
+The ``physical`` type configuration represents a "physical" network device,
+typically Ethernet-based.  At least one of of these entries is required for
+external network connectivity.  Type ``physical`` requires only one key:
+``name``.  A ``physical`` device may contain some or all of the following keys:
+
+**name**: *<desired device name>*
+
+A devices name must be less than 15 characters.  Names exceeding the maximum
+will be truncated. This is a limitation of the Linux kernel network-device
+structure.
+
+**mac_address**: *<MAC Address>*
+
+The MAC Address is a device unique identifier that most Ethernet-based network
+devices possess.  Specifying a MAC Address is optional.
+
+
+.. note::
+
+  Curtin will emit a udev rule to provide a persistent mapping between a
+  device's ``name`` and the ``mac_address``.
+
+**mtu**: *<MTU SizeBytes>* 
+
+The MTU key represents a device's Maximum Transmission Unit, the largest size
+packet or frame, specified in octets (eight-bit bytes), that can be sent in a
+packet- or frame-based network.  Specifying ``mtu`` is optional.
+
+.. note::
+
+  The possible supported values of a device's MTU is not available at
+  configuration time.  It's possible to specify a value too large or to
+  small for a device and may be ignored by the device.
+
+
+**Physical Example**::
+  
+  network:
+    version: 1
+    config:
+      # Simple network adapter
+      - type: physical
+        name: interface0
+        mac_address: 00:11:22:33:44:55
+      # Second nic with Jumbo frames
+      - type: physical
+        name: jumbo0
+        mac_address: aa:11:22:33:44:55
+        mtu: 9000
+      # 10G pair
+      - type: physical
+        name: gbe0
+        mac_address: cd:11:22:33:44:00
+      - type: physical
+        name: gbe1
+        mac_address: cd:11:22:33:44:02
+
+Bond
+~~~~
+A ``bond`` type will configure a Linux software Bond with one or more network
+devices.  A ``bond`` type requires the following keys:
+
+**name**: *<desired device name>*
+
+A devices name must be less than 15 characters.  Names exceeding the maximum
+will be truncated. This is a limitation of the Linux kernel network-device
+structure.
+
+**mac_address**: *<MAC Address>*
+
+When specifying MAC Address on a bond this value will be assigned to the bond
+device and may be different than the MAC address of any of the underlying 
+bond interfaces.  Specifying a MAC Address is optional.  If ``mac_address`` is
+not present, then the bond will use one of the MAC Address values from one of
+the bond interfaces.
+
+
+**bond_interfaces**: *<List of network device names>*
+
+The ``bond_interfaces`` key accepts a list of network device ``name`` values
+from the configuration.  This list may be empty.
+
+**params**:  *<Dictionary of key: value bonding parameter pairs>* 
+
+The ``params`` key in a bond holds a dictionary of bonding parameters.
+This dictionary may be empty. For more details on what the various bonding
+parameters mean please read the Linux Kernel Bonding.txt.
+
+Valid ``params`` keys are:
+
+  - ``active_slave``: Set bond attribute
+  - ``ad_actor_key``: Set bond attribute
+  - ``ad_actor_sys_prio``: Set bond attribute
+  - ``ad_actor_system``: Set bond attribute
+  - ``ad_aggregator``: Set bond attribute
+  - ``ad_num_ports``: Set bond attribute
+  - ``ad_partner_key``: Set bond attribute
+  - ``ad_partner_mac``: Set bond attribute
+  - ``ad_select``: Set bond attribute
+  - ``ad_user_port_key``: Set bond attribute
+  - ``all_slaves_active``: Set bond attribute
+  - ``arp_all_targets``: Set bond attribute
+  - ``arp_interval``: Set bond attribute
+  - ``arp_ip_target``: Set bond attribute
+  - ``arp_validate``: Set bond attribute
+  - ``downdelay``: Set bond attribute
+  - ``fail_over_mac``: Set bond attribute
+  - ``lacp_rate``: Set bond attribute
+  - ``lp_interval``: Set bond attribute
+  - ``miimon``: Set bond attribute
+  - ``mii_status``: Set bond attribute
+  - ``min_links``: Set bond attribute
+  - ``mode``: Set bond attribute
+  - ``num_grat_arp``: Set bond attribute
+  - ``num_unsol_na``: Set bond attribute
+  - ``packets_per_slave``: Set bond attribute
+  - ``primary``: Set bond attribute
+  - ``primary_reselect``: Set bond attribute
+  - ``queue_id``: Set bond attribute
+  - ``resend_igmp``: Set bond attribute
+  - ``slaves``: Set bond attribute
+  - ``tlb_dynamic_lb``: Set bond attribute
+  - ``updelay``: Set bond attribute
+  - ``use_carrier``: Set bond attribute
+  - ``xmit_hash_policy``: Set bond attribute
+ 
+**Bond Example**::
+
+   network:
+    version: 1
+    config:
+      # Simple network adapter
+      - type: physical
+        name: interface0
+        mac_address: 00:11:22:33:44:55
+      # 10G pair
+      - type: physical
+        name: gbe0
+        mac_address: cd:11:22:33:44:00
+      - type: physical
+        name: gbe1
+        mac_address: cd:11:22:33:44:02
+      - type: bond
+        name: bond0
+        bond_interfaces:
+          - gbe0
+          - gbe1
+        params:
+          bond-mode: active-backup
+ 
+Bridge
+~~~~~~
+Type ``bridge`` requires the following keys:
+
+- ``name``: Set the name of the bridge.
+- ``bridge_interfaces``: Specify the ports of a bridge via their ``name``.  This list may be empty.
+- ``params``:  A list of bridge params.  For more details, please read the bridge-utils-interfaces manpage.
+
+Valid keys are:
+
+  - ``bridge_ageing``: Set the bridge's ageing value.
+  - ``bridge_bridgeprio``: Set the bridge device network priority.
+  - ``bridge_fd``: Set the bridge's forward delay.
+  - ``bridge_hello``: Set the bridge's hello value.
+  - ``bridge_hw``: Set the bridge's MAC address.
+  - ``bridge_maxage``: Set the bridge's maxage value.
+  - ``bridge_maxwait``:  Set how long network scripts should wait for the bridge to be up.
+  - ``bridge_pathcost``:  Set the cost of a specific port on the bridge.
+  - ``bridge_portprio``:  Set the priority of a specific port on the bridge.
+  - ``bridge_ports``:  List of devices that are part of the bridge.
+  - ``bridge_stp``:  Set spanning tree protocol on or off.
+  - ``bridge_waitport``: Set amount of time in seconds to wait on specific ports to become available.
+
+
+**Bridge Example**::
+
+   network:
+    version: 1
+    config:
+      # Simple network adapter
+      - type: physical
+        name: interface0
+        mac_address: 00:11:22:33:44:55
+      # Second nic with Jumbo frames
+      - type: physical
+        name: jumbo0
+        mac_address: aa:11:22:33:44:55
+        mtu: 9000
+      - type: bridge
+        name: br0
+        bridge_interfaces:
+          - jumbo0
+        params:
+          bridge_ageing: 250
+		  bridge_bridgeprio: 22
+		  bridge_fd: 1
+          bridge_hello: 1
+          bridge_maxage: 10
+          bridge_maxwait: 0
+          bridge_pathcost:
+            - jumbo0 75
+          bridge_pathprio:
+            - jumbo0 28
+          bridge_stp: 'off'
+          bridge_maxwait:
+            - jumbo0 0
+
+  
+VLAN
+~~~~
+Type ``vlan`` requires the following keys:
+
+- ``name``: Set the name of the VLAN
+- ``vlan_link``: Specify the underlying link via its ``name``.
+- ``vlan_id``: Specify the VLAN numeric id.
+
+**VLAN Example**::
+
+   network:
+     version: 1
+     config:
+       # Physical interfaces.
+       - type: physical
+         name: eth0
+         mac_address: "c0:d6:9f:2c:e8:80"
+       # VLAN interface.
+       - type: vlan
+         name: eth0.101
+         vlan_link: eth0
+         vlan_id: 101
+         mtu: 1500
+
+Nameserver
+~~~~~~~~~~
+
+Users can specify a ``nameserver`` type.  Nameserver dictionaries include
+the following keys:
+
+- ``address``: List of IPv4 or IPv6 address of nameservers.
+- ``search``: List of of hostnames to include in the resolv.conf search path.
+
+**Nameserver Example**::
+
+  network:
+    version: 1
+    config:
+      - type: physical
+        name: interface0
+        mac_address: 00:11:22:33:44:55
+        subnets:
+           - type: static
+             address: 192.168.23.14/27
+             gateway: 192.168.23.1
+      - type: nameserver:
+        address: 
+          - 192.168.23.2
+          - 8.8.8.8
+        search:
+          - exemplary
+
+     
+
+Route
+~~~~~
+
+Users can include static routing information as well.  A ``route`` dictionary
+has the following keys:
+
+- ``destination``: IPv4 network address with CIDR netmask notation.
+- ``gateway``: IPv4 gateway address with CIDR netmask notation.
+- ``metric``: Integer which sets the network metric value for this route.
+
+**Route Example**::
+
+  network:
+    version: 1
+    config:
+      - type: physical
+        name: interface0
+        mac_address: 00:11:22:33:44:55
+        subnets:
+           - type: static
+             address: 192.168.23.14/24
+             gateway: 192.168.23.1
+      - type: route
+        destination: 192.168.24.0/24
+        gateway: 192.168.24.1
+        metric: 3
+
+Subnet/IP
+~~~~~~~~~
+
+For any network device (one of the Config Types) users can define a list of
+``subnets`` which contain ip configuration dictionaries.  Multiple subnet
+entries will create interface alias allowing a single interface to use different
+ip configurations.  
+
+Valid keys for for ``subnets`` include the following:
+
+- ``type``: Specify the subnet type.
+- ``control``: Specify manual, auto or hotplug.  Indicates how the interface will be handled during boot.
+- ``address``: IPv4 or IPv6 address.  It may include CIDR netmask notation.
+- ``netmask``: IPv4 subnet mask in dotted format or CIDR notation.
+- ``gateway``: IPv4 address of the default gateway for this subnet.
+- ``dns_nameserver``: Specify a list of IPv4 dns server IPs to end up in resolv.conf.
+- ``dns_search``: Specify a list of search paths to be included in resolv.conf.
+
+
+Subnet types are one of the following:
+
+- ``dhcp4``: Configure this interface with IPv4 dhcp.
+- ``dhcp``: Alias for ``dhcp4``
+- ``dhcp6``: Configure this interface with IPv6 dhcp.
+- ``static``: Configure this interface with a static IPv4.
+- ``static6``: Configure this interface with a static IPv6 .
+
+When making use of ``dhcp`` types, no additional configuration is needed in the
+subnet dictionary.
+
+
+**Subnet DHCP Example**::
+
+   network:
+     version: 1
+     config:
+       - type: physical
+         name: interface0
+         mac_address: 00:11:22:33:44:55
+         subnets:
+           - type: dhcp
+
+
+**Subnet Static Example**::
+
+   network:
+     version: 1
+     config:
+       - type: physical
+         name: interface0
+         mac_address: 00:11:22:33:44:55
+         subnets:
+           - type: static
+             address: 192.168.23.14/27
+             gateway: 192.168.23.1
+             dns_nameservers:
+               - 192.168.23.2
+               - 8.8.8.8
+             dns_search:
+               - exemplary.maas
+
+The following will result in an ``interface0`` using DHCP and ``interface0:1``
+using the static subnet configuration.
+
+**Multiple subnet Example**::
+
+   network:
+     version: 1
+     config:
+       - type: physical
+         name: interface0
+         mac_address: 00:11:22:33:44:55
+         subnets:
+           - type: dhcp
+           - type: static
+             address: 192.168.23.14/27
+             gateway: 192.168.23.1
+             dns_nameservers:
+               - 192.168.23.2
+               - 8.8.8.8
+             dns_search:
+               - exemplary
+
+
+Multi-layered configurations
+----------------------------
+
+Complex networking sometimes uses layers of configuration.  The syntax allows
+users to build those layers one at a time.  All of the virtual network devices
+supported allow specifying an underlying device by their ``name`` value.
+
+**Bonded VLAN Example**::
+
+  network:
+    version: 1
+    config:
+      # 10G pair
+      - type: physical
+        name: gbe0
+        mac_address: cd:11:22:33:44:00
+      - type: physical
+        name: gbe1
+        mac_address: cd:11:22:33:44:02
+      # Bond.
+      - type: bond
+        name: bond0
+        bond_interfaces:
+          - gbe0
+          - gbe1
+        params:
+          bond-mode: 802.3ad
+          bond-lacp-rate: fast
+      # A Bond VLAN.
+      - type: vlan
+          name: bond0.200
+          vlan_link: bond0
+          vlan_id: 200
+          subnets:
+              - type: dhcp4
+
+More Examples
+-------------
+Some more examples to explore the various options available.
+
+**Multiple VLAN example**::
+
+  network:
+    version: 1
+    config:
+    - id: eth0
+      mac_address: d4:be:d9:a8:49:13
+      mtu: 1500
+      name: eth0
+      subnets:
+      - address: 10.245.168.16/21
+        dns_nameservers:
+        - 10.245.168.2
+        gateway: 10.245.168.1
+        type: static
+      type: physical
+    - id: eth1
+      mac_address: d4:be:d9:a8:49:15
+      mtu: 1500
+      name: eth1
+      subnets:
+      - address: 10.245.188.2/24
+        dns_nameservers: []
+        type: static
+      type: physical
+    - id: eth1.2667
+      mtu: 1500
+      name: eth1.2667
+      subnets:
+      - address: 10.245.184.2/24
+        dns_nameservers: []
+        type: static
+      type: vlan
+      vlan_id: 2667
+      vlan_link: eth1
+    - id: eth1.2668
+      mtu: 1500
+      name: eth1.2668
+      subnets:
+      - address: 10.245.185.1/24
+        dns_nameservers: []
+        type: static
+      type: vlan
+      vlan_id: 2668
+      vlan_link: eth1
+    - id: eth1.2669
+      mtu: 1500
+      name: eth1.2669
+      subnets:
+      - address: 10.245.186.1/24
+        dns_nameservers: []
+        type: static
+      type: vlan
+      vlan_id: 2669
+      vlan_link: eth1
+    - id: eth1.2670
+      mtu: 1500
+      name: eth1.2670
+      subnets:
+      - address: 10.245.187.2/24
+        dns_nameservers: []
+        type: static
+      type: vlan
+      vlan_id: 2670
+      vlan_link: eth1
+    - address: 10.245.168.2
+      search:
+      - dellstack
+      type: nameserver
+
 
=== modified file 'doc/topics/overview.rst'
--- doc/topics/overview.rst	2016-06-02 08:26:30 +0000
+++ doc/topics/overview.rst	2016-08-29 23:08:04 +0000
@@ -20,7 +20,7 @@
 ~~~~~~~~~~~~~~~~~~~~~~~~
 At the moment, curtin doesn't address how the system that it is running on is booted.  It could be booted from a live-cd or from a pxe boot environment.  It could even be booted off a disk in the system (although installation to that disk would probably break things).
 
-Curtin's assumption is that a fairly rich linux (Ubuntu) environment is booted.
+Curtin's assumption is that a fairly rich Linux (Ubuntu) environment is booted.
 
 Early Commands
 ~~~~~~~~~~~~~~
@@ -38,13 +38,13 @@
 
 Partitioning
 ~~~~~~~~~~~~
-Partitioning covers setting up filesystems on the system.  A series of commands are run serially in order.  At the end, a fstab formated file must be populated in ``OUTPUT_FSTAB`` that contains mount information, and the filesystems are expected to be mounted at the ``TARGET_MOUNT_POINT``.
+Partitioning covers setting up filesystems on the system.  A series of commands are run serially in order.  At the end, a fstab formatted file must be populated in ``OUTPUT_FSTAB`` that contains mount information, and the filesystems are expected to be mounted at the ``TARGET_MOUNT_POINT``.
 
 Any commands can be used to create this filesystem, but curtin contains some tools to facilitate with this process.
 
 **Config Example**::
 
- paritioning_commands:
+ partitioning_commands:
   10_wipe_filesystems: curtin wipe --quick --all-unused-disks
   50_setup_raid: curtin disk-setup --all-disks raid0 /
 
@@ -53,7 +53,7 @@
 Partitioning commands have the following environment variables available to them:
 
 - ``WORKING_DIR``: This is simply for some sort of inter-command state.  It will be the same directory for each command run and will only be deleted at the end of all partitioning_commands.
-- ``OUTPUT_FSTAB``: This is the target path for a fstab file.  After all partitioning commands have been run, a file should exist, formated per fstab(5) that describes how the filesystems should be mounted.
+- ``OUTPUT_FSTAB``: This is the target path for a fstab file.  After all partitioning commands have been run, a file should exist, formatted per fstab(5) that describes how the filesystems should be mounted.
 - ``TARGET_MOUNT_POINT``:
 
 
@@ -61,7 +61,7 @@
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 Networking is done in a similar fashion to partitioning.  A series of commands, specified in the config are run.  At the end of these commands, a interfaces(5) style file is expected to be written to ``OUTPUT_INTERFACES``.
 
-Note, that as with fstab, this file is not copied verbatum to the target filesystem, but rather made availble to the OS customization stage.  That stage may just copy the file verbatum, but may also parse it, and use that as input.
+Note, that as with fstab, this file is not copied verbatim to the target filesystem, but rather made available to the OS customization stage.  That stage may just copy the file verbatim, but may also parse it, and use that as input.
 
 **Config Example**::
 
@@ -73,7 +73,7 @@
 Networking commands have the following environment variables available to them:
 
 - ``WORKING_DIR``: This is simply for some sort of inter-command state.  It will be the same directory for each command run and will only be deleted at the end of all network_commands.
-- ``OUTPUT_INTERFACES``: This is the target path for an interfaces style file. After all commands have been run, a file should exist, formated per interfaces(5) that describes the systems network setup.
+- ``OUTPUT_INTERFACES``: This is the target path for an interfaces style file. After all commands have been run, a file should exist, formatted per interfaces(5) that describes the systems network setup.
 
 Extraction of sources
 ~~~~~~~~~~~~~~~~~~~~~
@@ -84,7 +84,7 @@
  sources:
   05_primary: http://cloud-images.ubuntu.com/releases/precise/release/ubuntu-12.04-server-cloudimg-amd64-root.tar.gz
 
-Given the source above, curtin will essentiall do a::
+Given the source above, curtin will essentially do a::
 
  wget $URL | tar -Sxvzf 
 
 
=== modified file 'doc/topics/reporting.rst'
--- doc/topics/reporting.rst	2016-06-02 08:26:30 +0000
+++ doc/topics/reporting.rst	2016-08-29 23:08:04 +0000
@@ -7,7 +7,7 @@
 
 Events
 ------
-Reporting consists of notifcation of a series of 'events.  Each event has:
+Reporting consists of notification of a series of 'events.  Each event has:
  - **event_type**: 'start' or 'finish'
  - **description**: human readable text
  - **name**: and id for this event
@@ -48,7 +48,7 @@
 Each entry must have a 'type'.  The currently supported values are:
  - **log**: logs via python logger
  - **print**: prints messages to stdout (for debugging)
- - **webhook**: posts json formated data to a remote url.  Supports Oauth.
+ - **webhook**: posts json formatted data to a remote url.  Supports Oauth.
 
 
 Additionally, the webhook reporter will post files on finish of curtin.  The user can declare which files should be posted in the ``install`` item via ``post_files`` as shown above.  If post_files is not present, it will default to the value of log_file.
@@ -154,7 +154,7 @@
 Legacy Reporter
 ---------------
 The legacy 'reporter' config entry is still supported.  This was utilized by
-MAAS for start/end and posting of the install log at the end of isntallation.
+MAAS for start/end and posting of the install log at the end of installation.
 
 Its configuration looks like this:
 
 
=== added file 'doc/topics/storage.rst'
--- doc/topics/storage.rst	1970-01-01 00:00:00 +0000
+++ doc/topics/storage.rst	2016-08-29 23:08:04 +0000
@@ -0,0 +1,894 @@
+=======
+Storage
+=======
+
+Curtin supports a user-configurable storage layout.  This format lets users
+(including via MAAS) to customize their machines' storage configuration by
+creating partitions, RAIDs, LVMs, formatting with file systems and setting
+mount points.
+
+Custom storage configuration is handled by the ``block-meta custom`` command
+in curtin. Partitioning layout is read as a list of in-order modifications to
+make to achieve the desired configuration. The top level configuration key
+containing this configuration is ``storage``. This key should contain a
+dictionary with at least a version number and the configuration list. The
+current config specification is ``version: 1``.
+
+**Config Example**::
+
+ storage:
+   version: 1
+   config:
+     - id: sda
+       type: disk
+       ptable: gpt
+       serial: QM00002
+       model: QEMU_HARDDISK
+
+Configuration Types
+-------------------
+Each entry in the config list is a dictionary with several keys which vary
+between commands. The two dictionary keys that every entry in the list needs
+to have are ``id: <id>`` and ``type: <type>``.
+
+An entry's ``id`` allows other entries in the config to refer to a specific
+entry. It can be any string other than one with a special meaning in yaml, such
+as ``true`` or ``none``.
+
+An entry's ``type`` tells curtin how to handle a particular entry. Available
+commands include:
+
+- Disk Command (``disk``)
+- Partition Command (``partition``)
+- Format Command (``format``)
+- Mount Command  (``mount``)
+- LVM_VolGroup Command (``lvm_volgroup``)
+- LVM_Partition Command (``lvm_partition``)
+- DM_Crypt Command (``dm_crypt``)
+- RAID Command (``raid``)
+- Bcache Command (``bcache``)
+
+Disk Command
+~~~~~~~~~~~~
+The disk command sets up disks for use by curtin. It can wipe the disks, create
+partition tables, or just verify that the disks exist with an existing partition
+table. A disk command may contain all or some of the following keys:
+
+**ptable**: *msdos, gpt*
+
+If the ``ptable`` key is present and a valid type of partition table, curtin
+will create an empty partition table of that type on the disk.  At the moment,
+msdos and gpt partition tables are supported.
+
+**serial**: *<serial number>*
+
+In order to uniquely identify a disk on the system its serial number should be
+specified. This ensures that even if additional storage devices
+are added to the system during installation, or udev rules cause the path to a
+disk to change curtin will still be able to correctly identify the disk it
+should be operating on using ``/dev/disk/by-id``.
+
+This is the preferred way to identify a disk and should be used in all
+production environments as it is less likely to point to an incorrect device.
+
+**path**: *<path to device with leading /dev*
+
+The ``path`` key can be used to identify the disk.  If both ``serial`` and
+``path`` are specified, curtin will use the serial number and ignore the path
+that was specified.
+
+**model**: *<disk model>*
+
+This can specify the manufacturer or model of the disk. It is not currently
+used by curtin, but can be useful for a human reading a config file. Future
+versions of curtin may make use of this information.
+
+**wipe**: *superblock, superblock-recursive, zero, random*
+
+If wipe is specified, **the disk contents will be destroyed**.  In the case that
+a disk is a part of virtual block device, like bcache, RAID array, or LVM, then
+curtin will attempt to tear down the virtual device to allow access to the disk
+for resetting the disk.
+
+The most common option for clearing a disk is  ``wipe: superblock``.  In some
+cases use of ``wipe: superblock-recursive`` is useful to ensure that embedded
+superblocks on a disk aren't rediscovered during probing.  For example, LVM,
+bcache and RAID on a partition would have metadata outside of the range of a
+superblock wipe of the start and end sections of the disk.
+
+The ``wipe: zero`` option will write zeros to each sector of the disk.
+Depending on the size and speed of the disk; it may take a long time to
+complete.
+
+The ``wipe: random`` option will write pseudo-random data from /dev/urandom
+Depending on the size and speed of the disk; it may take a long time to
+complete.
+
+**preserve**: *true, false*
+
+When the preserve key is present and set to ``true`` curtin will attempt
+to use the disk without damaging data present on it. If ``preserve`` is set and
+``ptable`` is also set, then curtin will validate that the partition table
+specified by ``ptable`` exists on the disk and will raise an error if it does
+not. If ``preserve`` is set and ``ptable`` is not, then curtin will be able to
+use the disk in later commands, but will not check if the disk has a valid
+partition table, and will only verify that the disk exists.
+
+It can be dangerous to try to move or re-size filesystems and partitions
+containing data that needs to be preserved. Therefor curtin does not support
+preserving a disk without also preserving the partitions on it. If a disk is
+set to be preserved and curtin is told to move a partition on that disk,
+installation will stop. It is still possible to reformat partitions that do
+not need to be preserved.
+
+**name**: *<name>*
+
+If the ``name`` key is present, curtin will create a udev rule that makes a
+symbolic link to the disk with the given name value. This makes it easy to find
+disks on an installed system. The links are created in
+``/dev/disk/by-dname/<name>``.
+A link to each partition on the disk will also be created at
+``/dev/disk/by-dname/<name>-part<number>``, so if ``name: maindisk`` is set,
+the disk will be at ``/dev/disk/by-dname/maindisk`` and the first partition on
+it will be at ``/dev/disk/by-dname/maindisk-part1``.
+
+**grub_device**: *true, false*
+
+If the ``grub_device`` key is present and set to true, then when post
+installation hooks are run grub will be installed onto this disk. In most
+situations it is not necessary to specify this value as curtin will detect
+and determine which device to use as a boot disk.  In cases where the boot
+device is on a special volume, such as a RAID array or a LVM Logical Volume,
+it may be necessary to specify the device that will hold the grub bootloader.
+
+**Config Example**::
+
+ - id: disk0
+   type: disk
+   ptable: gpt
+   serial: QM00002
+   model: QEMU_HARDDISK
+   name: maindisk
+   wipe: superblock
+
+Partition Command
+~~~~~~~~~~~~~~~~~
+The partition command creates a single partition on a disk. Curtin only needs
+to be told which disk to use and the size of the partition.  Additional options
+are available.
+
+**number**: *<number>*
+
+The partition number can be specified using ``number``. However, numbers must
+be in order and some situations, such as extended/logical partitions on msdos
+partition tables will require special numbering, so it maybe better to omit 
+the partition number. If the ``number`` key is not present, curtin will attempt
+determine the right number to use.
+
+**size**: *<size>*
+
+The partition size can be specified with the ``size`` key. Sizes must be
+given with an appropriate SI unit, such as *B, kB, MB, GB, TB*, or using just
+the appropriate SI prefix, i.e. *B, k, M, G, T...*
+
+.. note::
+
+  Curtin does not adjust size values.  If you specific a size that exceeds the 
+  capacity of a device then installation will fail.
+
+**device**: *<device id>*
+
+The ``device`` key refers to the ``id`` of a disk in the storage configuration.
+The disk entry must already be defined in the list of commands to ensure that
+it has already been processed.
+
+**wipe**: *superblock, pvremove, zero, random*
+
+After the partition is added to the disk's partition table, curtin can run a
+wipe command on the partition. The wipe command values are the sames as for
+disks.
+
+**flag**: *logical, extended, boot, bios_grub, swap, lvm, raid, home, prep*
+
+If the ``flag`` key is present, curtin will set the specified flag on the
+partition. Note that some flags only apply to msdos partition tables, and some
+only apply to gpt partition tables.
+
+The *logical/extended* partition flags can be used to create logical partitions
+on a msdos table. An extended partition should be created containing all of the
+empty space on the drive, and logical partitions can be created within it. A
+extended partition must already be present to create logical partitions. If the
+``number`` flag is set for an extended partition it must be set to 4, and
+each logical partition should be numbered starting from 5.
+
+On msdos partition tables, the *boot* flag sets the boot parameter to that
+partition. On gpt partition tables, the boot flag sets the esp flag on the
+partition.
+
+If the host system for curtin has been booted using UEFI then curtin will
+install grub to the esp partition. If the system installation media
+has been booted using an MBR, grub will be installed onto the disk's MBR.
+However, on a disk with a gpt partition table, there is not enough space after
+the MBR for grub to store its second stage core.img, so a small un-formatted
+partition with the *bios_grub* flag is needed. This partition should be placed
+at the beginning of the disk and should be 1MB in size. It should not contain a
+filesystem or be mounted anywhere on the system.
+
+**preserve**: *true, false*
+
+If the preserve flag is set to true, curtin will verify that the partition
+exists and will not modify the partition.
+
+**Config Example**::
+
+ - id: disk0-part1
+   type: partition
+   number: 1
+   size: 8GB
+   device: disk0
+   flag: boot
+
+Format Command
+~~~~~~~~~~~~~~
+The format command makes filesystems on a volume. The filesystem type and
+target volume can be specified, as well as a few other options.
+
+**fstype**: ext4, ext3, fat32, fat16, swap, xfs
+
+The ``fstype`` key specifies what type of filesystem format curtin should use
+for this volume. Curtin knows about common Linux filesystems such as ext4/3 and
+fat filesystems and makes use of additional parameters and flags to optimize the
+filesystem.  If the ``fstype`` value is not known to curtin, that is not fatal.
+Curtin will check if ``mkfs.<fstype>`` exists and if so,  will use that tool to
+format the target volume.
+
+For fat filesystems, the size of the fat table can be specified by entering
+*fat64*, *fat32*, *fat16*, or *fat12* instead of just entering *fat*.
+If *fat* is used, then ``mkfs.fat`` will automatically determine the best
+size fat table to use, probably *fat32*.
+
+If ``fstype: swap`` is set, curtin will create a swap partition on the target
+volume.
+
+**volume**: *<volume id>*
+
+The ``volume`` key refers to the ``id`` of the target volume in the storage
+config.  The target volume must already exist and be accessible. Any type
+of target volume can be used as long as it has a block device that curtin
+can locate.
+
+**label**: *<volume name>*
+
+The ``label`` key tells curtin to create a filesystem LABEL when formatting a
+volume. Note that not all filesystem types support names and that there are
+length limits for names. For fat filesystems, names are limited to 11
+characters. For ext4/3 filesystems, names are limited to 16 characters.
+
+If curtin does not know about the filesystem type it is using, then the
+``label`` key will be ignored, because curtin will not know the correct flags
+to set the label value in the filesystem metadata.
+
+**uuid**: *<uuid>*
+
+If the ``uuid`` key is set and ``fstype`` is set to *ext4* or *ext3*, then
+curtin will set the uuid of the new filesystem to the specified value.
+
+**preserve**: *true, false*
+
+If the ``preserve`` key is set to true, curtin will not format the partition.
+
+**Config Example**::
+
+ - id: disk0-part1-fs1
+   type: format
+   fstype: ext4
+   label: cloud-image
+   volume: disk0-part1
+
+Mount Command
+~~~~~~~~~~~~~
+The mount command mounts the target filesystem and creates an entry for it in
+the newly installed system's ``/etc/fstab``. The path to the target mountpoint
+must be specified as well as the target filesystem.
+
+**path**: *<path>*
+
+The ``path`` key tells curtin where the filesystem should be mounted on the
+target system. An entry in the target system's ``/etc/fstab`` will be created
+for the target device which will mount it in the correct place once the
+installed system boots.
+
+If the device specified is formatted as swap space, then an entry will be added
+to the target system's ``/etc/fstab`` to make use of this swap space.
+
+When entries are created in ``/etc/fstab``, curtin will use the most reliable
+method available to identify each device. For regular partitions, curtin will
+use the UUID of the filesystem present on the partition. For special devices,
+such as RAID arrays, or LVM logical volumes, curtin will use their normal path
+in ``/dev``.
+
+**device**: *<device id>*
+
+The ``device`` key refers to the ``id`` of the target device in the storage
+config. The target device must already contain a valid filesystem and be
+accessible.
+
+**Config Example**::
+
+ - id: disk0-part1-fs1-mount0
+   type: mount
+   path: /home
+   device: disk0-part1-fs1
+
+Lvm Volgroup Command
+~~~~~~~~~~~~~~~~~~~~
+The lvm_volgroup command creates LVM Physical Volumes (PV) and connects them in
+a LVM Volume Group (vg). The command requires a name for the volgroup and a
+list of the devices that should be used as physical volumes.
+
+**name**: *<name>*
+
+The ``name`` key specifies the name of the volume group.  It anything can be
+used except words with special meanings in YAML, such as *true*, or *none*.
+
+**devices**: *[]*
+
+The ``devices`` key gives a list of devices to use as physical volumes. Each
+device is specified using the ``id`` of existing devices in the storage config.
+Almost anything can be used as a device such as partitions, whole disks, RAID.
+
+**Config Example**::
+
+ - id: volgroup1
+   type: lvm_volgroup
+   name: vg1
+   devices:
+     - disk0-part2
+     - disk1
+
+Lvm Partition Command
+~~~~~~~~~~~~~~~~~~~~~
+The lvm_partition command creates a lvm logical volume on the specified
+volgroup with the specified size. It also assigns it the specified name.
+
+**name**: *<name>*
+
+The ``name`` key specifies the name of the Logical Volume (LV) to be created.
+
+Curtin creates udev rules for Logical Volumes to give them consistently named 
+symbolic links in the target system under ``/dev/disk/by-dname/``. The naming
+scheme for Logical Volumes follows the pattern
+``<volgroup name>-<logical volume name>``.  For example a ``lvm_partition``
+with ``name`` *lv1* on a ``lvm_volgroup`` named *vg1* would have the path
+``/dev/disk/by-dname/vg1-lv1``.
+
+**volgroup**: *<volgroup id>*
+
+The ``volgroup`` key specifies the ``id`` of the Volume Group in which to
+create the logical volume. The volgroup must already have been created and must
+have enough free space on it to create the logical volume.  The volgroup should
+be specified using the ``id`` key of the volgroup in the storage config, not the
+name of the volgroup.
+
+**size**: *<size>*
+
+The ``size`` key tells curtin what size to make the logical volume. The size
+can be entered in any format that can be processed by the lvm2 tools, so a
+number followed by a SI unit should work, i.e. *B, kB, MB, GB, TB*.
+
+If the ``size`` key is omitted then all remaining space on the volgroup will be
+used for the logical volume.
+
+.. note::
+
+  Curtin does not adjust size values.  If you specific a size that exceeds the 
+  capacity of a device then installation will fail.
+
+
+**Config Example**::
+
+ - id: lvm_partition_1
+   type: lvm_partition
+   name: lv1
+   volgroup: volgroup1
+   size: 10G
+
+
+**Combined Example**::
+
+ - id: volgroup1
+   type: lvm_volgroup
+   name: vg1
+   devices:
+     - disk0-part2
+     - disk1
+ - id: lvm_partition_1
+   type: lvm_partition
+   name: lv1
+   volgroup: volgroup1
+   size: 10G
+
+
+
+Dm-Crypt Command
+~~~~~~~~~~~~~~~~
+The dm_crypt command creates encrypted volumes using ``cryptsetup``. It
+requires a name for the encrypted volume, the volume to be encrypted and a key.
+Note that this should not be used for systems where security is a requirement.
+The key is stored in plain-text in the storage configuration and it could be
+possible for the storage configuration to be intercepted between the utility
+that generates it and curtin.
+
+**volume**: *<volume id>*
+
+The ``volume`` key gives the volume that is to be encrypted.
+
+**dm_name**: *<name>*
+
+The ``name`` key specifies the name of the encrypted volume.
+
+**key**: *<key>*
+
+The ``key`` key specifies the password of the encryption key.  The target
+system will prompt for this password in order to mount the disk.
+
+.. note::
+
+  Encrypted disks and partitions are tracked in ``/etc/crypttab`` and will  be
+  mounted at boot time.
+
+**Config Example**::
+
+ - id: lvm_partition_1
+   type: dm_crypt
+   dm_name: crypto
+   volume: sdb1
+   key: testkey
+
+RAID Command
+~~~~~~~~~~~~
+The RAID command configures Linux Software RAID using mdadm. It needs to be given
+a name for the md device, a list of volumes for to compose the md device, an
+optional list of devices to be used as spare volumes, and RAID level.
+
+**name**: *<name>*
+
+The ``name`` key specifies the name of the md device.
+
+.. note::
+
+  Curtin creates a udev rule to create a link to the md device in
+  ``/dev/disk/by-dname/<name>`` using the specified name.
+
+**raidlevel**: *0, 1, 5, 6, 10*
+
+The ``raidlevel`` key specifies the raid level of the array.
+
+**devices**: *[]*
+
+The ``devices`` key specifies a list of the devices that will be used for the
+raid array. Each device must be referenced by ``id`` and the device must be
+previously defined in the storage configuration.  Must not be empty.
+
+Devices can either be full disks or partition.
+
+
+**spare_devices**: *[]*
+
+The ``spare_devices`` key specifies a list of the devices that will be used for
+spares in the raid array. Each device must be referenced by ``id`` and the
+device must be previously defined in the storage configuration.  May be empty.
+
+
+**Config Example**::
+
+ - id: raid_array
+   type: raid
+   name: md0
+   raidlevel: 1
+   devices:
+     - sdb
+     - sdc
+   spare_devices:
+     - sdd
+
+Bcache Command
+~~~~~~~~~~~~~~
+The bcache command will configure a block-cache device using the Linux kernel
+bcache module.  Bcache allows users to use a typically small, but fast SSD or
+NVME device as a cache for larger, slower spinning disks.
+
+The bcache command needs to be told which device to use hold the data and which
+device to use as its cache device.  A cache device may be reused with multiple
+backing devices.
+
+
+**backing_device**: *<device id>*
+
+The ``backing_device`` key specifies the item in storage configuration to use
+as the backing device. This can be any device that would normally be used with
+a filesystem on it, such as a partition or a raid array.
+
+**cache_device**: *<device id>*
+
+The ``cache_device`` key specifies the item in the storage configuration to use
+as the cache device. This can be a partition or a whole disk. It should be on a
+ssd in most cases, as bcache is designed around the performance characteristics
+of a ssd.
+
+**cache_mode**: *writethrough, writeback, writearound, none*
+
+The ``cache_mode`` key specifies the mode in which bcache operates.  The
+default mode is writethrough which ensures data hits the backing device
+before completing the operation.  writeback mode will have higher performance
+but exposes dataloss if the cache device fails.  writearound will avoid using
+the cache for large sequential writes; useful for not evicting smaller
+reads/writes from the cache.  None effectively disables bcache.
+
+**name**: *<name>*
+
+If the ``name`` key is present, curtin will create a link to the device at
+``/dev/disk/by-dname/<name>``.
+
+**Config Example**::
+
+ - id: bcache0
+   type: bcache
+   name: cached_raid
+   backing_device: raid_array
+   cache_device: sdb
+
+
+Additional Examples
+-------------------
+
+Learn by examples.
+
+- Basic
+- LVM
+- Bcache
+- RAID Boot
+- RAID5 + Bcache
+
+Basic Layout
+~~~~~~~~~~~~
+
+::
+
+  storage:
+    version: 1
+    config:
+      - id: disk0
+        type: disk
+        ptable: msdos
+        model: QEMU HARDDISK
+        path: /dev/vdb
+        name: main_disk
+        wipe: superblock
+        grub_device: true
+      - id: disk0-part1
+        type: partition
+        number: 1
+        size: 3GB
+        device: disk0
+        flag: boot
+      - id: disk0-part2
+        type: partition
+        number: 2
+        size: 1GB
+        device: disk0
+      - id: disk0-part1-format-root
+        type: format
+        fstype: ext4
+        volume: disk0-part1
+      - id: disk0-part2-format-home
+        type: format
+        fstype: ext4
+        volume: disk0-part2
+      - id: disk0-part1-mount-root
+        type: mount
+        path: /
+        device: disk0-part1-format-root
+      - id: disk0-part2-mount-home
+        type: mount
+        path: /home
+        device: disk0-part2-format-home
+
+LVM
+~~~
+
+::
+
+  storage:
+    version: 1
+    config:
+      - id: sda
+        type: disk
+        ptable: msdos
+        model: QEMU HARDDISK
+        path: /dev/vdb
+        name: main_disk
+      - id: sda1
+        type: partition
+        size: 3GB
+        device: sda
+        flag: boot
+      - id: sda_extended
+        type: partition
+        size: 5G
+        flag: extended
+        device: sda
+      - id: sda2
+        type: partition
+        size: 2G
+        flag: logical
+        device: sda
+      - id: sda3
+        type: partition
+        size: 3G
+        flag: logical
+        device: sda
+      - id: volgroup1
+        name: vg1
+        type: lvm_volgroup
+        devices:
+            - sda2
+            - sda3
+      - id: lvmpart1
+        name: lv1
+        size: 1G
+        type: lvm_partition
+        volgroup: volgroup1
+      - id: lvmpart2
+        name: lv2
+        type: lvm_partition
+        volgroup: volgroup1
+      - id: sda1_root
+        type: format
+        fstype: ext4
+        volume: sda1
+      - id: lv1_fs
+        name: storage
+        type: format
+        fstype: fat32
+        volume: lvmpart1
+      - id: lv2_fs
+        name: storage
+        type: format
+        fstype: ext3
+        volume: lvmpart2
+      - id: sda1_mount
+        type: mount
+        path: /
+        device: sda1_root
+      - id: lv1_mount
+        type: mount
+        path: /srv/data
+        device: lv1_fs
+      - id: lv2_mount
+        type: mount
+        path: /srv/backup
+        device: lv2_fs
+
+Bcache
+~~~~~~
+
+::
+
+  storage:
+    version: 1
+    config:
+      - id: id_rotary0
+        type: disk
+        name: rotary0
+        path: /dev/vdb
+        ptable: msdos
+        wipe: superblock
+        grub_device: true
+      - id: id_ssd0
+        type: disk
+        name: ssd0
+        path: /dev/vdc
+        wipe: superblock
+      - id: id_rotary0_part1
+        type: partition
+        name: rotary0-part1
+        device: id_rotary0
+        number: 1
+        size: 999M
+        wipe: superblock
+      - id: id_rotary0_part2
+        type: partition
+        name: rotary0-part2
+        device: id_rotary0
+        number: 2
+        size: 9G
+        wipe: superblock
+      - id: id_bcache0
+        type: bcache
+        name: bcache0
+        backing_device: id_rotary0_part2
+        cache_device: id_ssd0
+        cache_mode: writeback
+      - id: bootfs
+        type: format
+        label: boot-fs
+        volume: id_rotary0_part1
+        fstype: ext4
+      - id: rootfs
+        type: format
+        label: root-fs
+        volume: id_bcache0
+        fstype: ext4
+      - id: rootfs_mount
+        type: mount
+        path: /
+        device: rootfs
+      - id: bootfs_mount
+        type: mount
+        path: /boot
+        device: bootfs
+
+RAID Boot
+~~~~~~~~~
+
+::
+
+  storage:
+    version: 1
+    config:
+       - id: sda
+         type: disk
+         ptable: gpt
+         model: QEMU HARDDISK
+         path: /dev/vdb
+         name: main_disk
+         grub_device: 1
+       - id: bios_boot_partition
+         type: partition
+         size: 1MB
+         device: sda
+         flag: bios_grub
+       - id: sda1
+         type: partition
+         size: 3GB
+         device: sda
+       - id: sdb
+         type: disk
+         ptable: gpt
+         model: QEMU HARDDISK
+         path: /dev/vdc
+         name: second_disk
+       - id: sdb1
+         type: partition
+         size: 3GB
+         device: sdb
+       - id: sdc
+         type: disk
+         ptable: gpt
+         model: QEMU HARDDISK
+         path: /dev/vdd
+         name: third_disk
+       - id: sdc1
+         type: partition
+         size: 3GB
+         device: sdc
+       - id: mddevice
+         name: md0
+         type: raid
+         raidlevel: 5
+         devices:
+           - sda1
+           - sdb1
+           - sdc1
+       - id: md_root
+         type: format
+         fstype: ext4
+         volume: mddevice
+       - id: md_mount
+         type: mount
+         path: /
+         device: md_root
+
+
+RAID5 + Bcache
+~~~~~~~~~~~~~~
+
+::
+
+  storage:
+    config:
+    - grub_device: true
+      id: sda
+      model: QEMU HARDDISK
+      name: sda
+      ptable: msdos
+      path: /dev/vdb
+      type: disk
+      wipe: superblock
+    - id: sdb
+      model: QEMU HARDDISK
+      name: sdb
+      path: /dev/vdc
+      type: disk
+      wipe: superblock
+    - id: sdc
+      model: QEMU HARDDISK
+      name: sdc
+      path: /dev/vdd
+      type: disk
+      wipe: superblock
+    - id: sdd
+      model: QEMU HARDDISK
+      name: sdd
+      path: /dev/vde
+      type: disk
+      wipe: superblock
+    - id: sde
+      model: QEMU HARDDISK
+      name: sde
+      path: /dev/vdf
+      type: disk
+      wipe: superblock
+    - devices:
+      - sdc
+      - sdd
+      - sde
+      id: md0
+      name: md0
+      raidlevel: 5
+      spare_devices: []
+      type: raid
+    - device: sda
+      id: sda-part1
+      name: sda-part1
+      number: 1
+      size: 1000001536B
+      type: partition
+      uuid: 3a38820c-d675-4069-b060-509a3d9d13cc
+      wipe: superblock
+    - device: sda
+      id: sda-part2
+      name: sda-part2
+      number: 2
+      size: 7586787328B
+      type: partition
+      uuid: 17747faa-4b9e-4411-97e5-12fd3d199fb8
+      wipe: superblock
+    - backing_device: sda-part2
+      cache_device: sdb
+      cache_mode: writeback
+      id: bcache0
+      name: bcache0
+      type: bcache
+    - fstype: ext4
+      id: sda-part1_format
+      label: ''
+      type: format
+      uuid: 71b1ef6f-5cab-4a77-b4c8-5a209ec11d7c
+      volume: sda-part1
+    - fstype: ext4
+      id: md0_format
+      label: ''
+      type: format
+      uuid: b031f0a0-adb3-43be-bb43-ce0fc8a224a4
+      volume: md0
+    - fstype: ext4
+      id: bcache0_format
+      label: ''
+      type: format
+      uuid: ce45bbaf-5a44-4487-b89e-035c2dd40657
+      volume: bcache0
+    - device: bcache0_format
+      id: bcache0_mount
+      path: /
+      type: mount
+    - device: sda-part1_format
+      id: sda-part1_mount
+      path: /boot
+      type: mount
+    - device: md0_format
+      id: md0_mount
+      path: /srv/data
+      type: mount
+    version: 1
 
=== modified file 'setup.py'
--- setup.py	2015-08-31 14:19:25 +0000
+++ setup.py	2016-08-29 23:08:04 +0000
@@ -2,7 +2,7 @@
 from glob import glob
 import os
 
-VERSION = '0.1.0'
+import curtin
 
 
 def is_f(p):
@@ -11,7 +11,7 @@
 setup(
     name="curtin",
     description='The curtin installer',
-    version=VERSION,
+    version=curtin.__version__,
     author='Scott Moser',
     author_email='[e-mail]',
     license="AGPL",
 
=== modified file 'tox.ini'
--- tox.ini	2016-07-27 19:46:21 +0000
+++ tox.ini	2016-08-29 23:08:04 +0000
@@ -57,6 +57,7 @@
 [testenv:docs]
 deps = {[testenv]deps}
     sphinx
+    sphinx-rtd-theme
 commands =
     sphinx-build -b html -d doc/_build/doctrees doc/ doc/_build/html