Merge lp:~smoser/curtin/trunk.doc-interface into lp:~curtin-dev/curtin/trunk

[Scott Moser]

doc: update documentation on curtin-hooks and non-ubuntu installation.

Add the section 'curthooks' to the documentation.
Generalize the environment variables in overview.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:536
https://jenkins.ubuntu.com/server/job/curtin-ci/637/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/637
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/637
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/637
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/637

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/637/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

Minor spelling nits inline. Approved afterward. Good content (and good education/reference for me).

review: Approve

[Scott Moser (smoser)]

ok. so comments in line. i took your obvious feedback and we can work on the others subsequently (mainly whether or not to document CURTIN_REPORTSTACK).

=== modified file 'doc/index.rst'
--- doc/index.rst	2016-08-18 16:17:20 +0000
+++ doc/index.rst	2017-09-29 22:00:32 +0000
@@ -17,6 +17,7 @@
    topics/apt_source
    topics/networking
    topics/storage
+   topics/curthooks
    topics/reporting
    topics/development
    topics/integration-testing

=== added file 'doc/topics/curthooks.rst'
--- doc/topics/curthooks.rst	1970-01-01 00:00:00 +0000
+++ doc/topics/curthooks.rst	2017-09-29 22:00:32 +0000
@@ -0,0 +1,108 @@
+========================================
+Curthooks / New OS Support 
+========================================
+Curtin has built-in support for installation of Ubuntu.
+Other operating systems are supported through a mechanism called
+'curthooks' or 'curtin-hooks'.
+
+A curtin install runs through different stages.  See the 
+:ref:`Stages <stages>`
+documentation for function of each stage.
+The stages communicate with each other via data in a working directory and
+environment variables as described in
+:ref:`Command Environment`.
+
+Curtin handles partitioning, filesystem creation and target filesystem
+population for all operating systems. Curthooks are the mechanism provided
+so that the operating system can customize itself before reboot. This
+customization typically would need to include:
+
+ - ensuring that appropriate device drivers are loaded on first boot
+ - consuming the network interfaces file and applying its declarations.
+ - ensuring that necessary packages are installed to utilize storage
+   configuration or networking configuration.
+ - making the system boot (running grub-install or equivalent).
+
+Image provided curtin-hooks
+---------------------------
+An image provides curtin hooks support by containing a file
+``/curtin/curtin-hooks``.
+
+If an Ubuntu image image contains this path it will override the builtin
+curtin support.
+
+The ``curtin-hooks`` program should be executable in the filesystem and
+will be executed without any arguments.  It will be executed in the install
+environment, *not* the target environment.  A change of root to the
+target environment can be done with ``curtin in-target``.
+
+The hook is provided with some environment variables that can be used
+to find more information.  See the :ref:`Command Environment` doc for
+details.  Specifically interesting to this stage are:.
+
+ - ``OUTPUT_NETWORK_CONFIG``: This is a path to the file created during
+   network discovery stage. 
+ - ``FSTAB``: This is a path to the file created during partitioning stage
+ - ``CONFIG``: This is a path to the curtin config file.  It is provided so
+   that additional configuration could be provided through to the OS
+   customization.
+
+**TODO**: We should add 'PYTHON' or 'CURTIN_PYTHON' to this environment
+so that the hook can easily run a python program with the same python
+that curtin ran with (ie, python2 or python3).
+
+
+Networking configuration
+------------------------
+Access to the network configuration that is desired is inside the config
+and is in the format described in :ref:`networking`.
+
+.. TODO: We should guarantee that the presense
+         of network config v1 in the file OUTPUT_NETWORK_CONFIG.
+
+The curtin-hooks program must read the configuration from the
+path contained in ``OUTPUT_NETWORK_CONFIG`` and then set up
+the installed system to use it.
+
+If the installed system has cloud-init at version 17.1 or higher, it may
+be possible to simply copy this section into the target in
+``/etc/cloud/cloud.cfg.d/`` and let cloud-init render the correct
+networking on first boot.
+
+Storage configuration
+---------------------
+Access to the storage configuration that was set up is inside the config
+and is in the format described in :ref:`storage`.
+
+.. TODO: We should guarantee that the presense
+         of storage config v1 in the file OUTPUT_STORAGE_CONFIG.
+         This would mean the user would not have to pull it out
+         of CONFIG.  We should guarantee its presense and format
+         even in the 'simple' path.
+
+To apply this storage configuration, the curthooks may need to:
+
+ * update /etc/fstab to add the expected mounts entries.  The environment
+   variable ``FSTAB`` contains a path to a file that may be suitable
+   for use.
+
+ * install any packages that are not already installed that are required
+   to boot with the provided storage config.  For example, if the storage
+   layout includes raid you may need to install the mdadm package.
+
+ * update or create an initramfs.
+
+
+System boot
+-----------
+In Ubuntu, curtin will run 'grub-setup' and to install grub.  This covers
+putting the bootloader onto the disk(s) that are marked as
+``grub_device``.  The provided hook will need to do the equivalent
+operation.
+
+finalize hook
+-------------
+There is one other hook that curtin will invoke in an install, called
+``finalize``.  This program is invoked in the same environment as
+``curtin-hooks`` above.  It is intended to give the OS a final opportunity
+make updates before reboot.  It is called before ``late_commands``.

=== modified file 'doc/topics/networking.rst'
--- doc/topics/networking.rst	2016-08-18 15:39:15 +0000
+++ doc/topics/networking.rst	2017-09-29 22:00:32 +0000
@@ -1,3 +1,5 @@
+.. _networking:
+
 ==========
 Networking
 ==========

=== modified file 'doc/topics/overview.rst'
--- doc/topics/overview.rst	2016-08-18 16:02:27 +0000
+++ doc/topics/overview.rst	2017-09-29 22:00:32 +0000
@@ -4,6 +4,8 @@
 
 Curtin is intended to be a bare bones "installer".   Its goal is to take data from a source, and get it onto disk as quick as possible and then boot it.  The key difference from traditional package based installers is that curtin assumes the thing its installing is intelligent and will do the right thing.
 
+.. _Stages:
+
 Stages
 ------
 A usage of curtin will go through the following stages:
@@ -22,6 +24,32 @@
 
 Curtin's assumption is that a fairly rich Linux (Ubuntu) environment is booted.
 
+.. _Command Environment:
+
+Command Environment
+~~~~~~~~~~~~~~~~~~~
+Stages and commands invoked by curtin always have the following environment
+variables defined.
+
+- ``WORKING_DIR``: This is for inter-command state.  It will be the same
+  directory for each command run and will only be deleted at the end of the
+  install. Files referenced in other environment variables will be in
+  this directory.
+
+- ``TARGET_MOUNT_POINT``: The path in the filesystem where the target
+  filesystem will be mounted.
+
+- ``OUTPUT_NETWORK_CONFIG``: After the network discovery stage, this file
+  should contain networking config information that should then be written
+  to the target.
+
+- ``OUTPUT_FSTAB``: After partitioning and filesystem creation, this file
+  will contain fstab(5) style content representing mounts.
+
+- ``CONFIG``: This variable contains a path to a yaml formatted file with
+  the fully rendered config.
+
+
 Early Commands
 ~~~~~~~~~~~~~~
 Early commands are executed on the system, and non-zero exit status will terminate the installation process.  These commands are intended to be used for things like
@@ -48,32 +76,23 @@
   10_wipe_filesystems: curtin wipe --quick --all-unused-disks
   50_setup_raid: curtin disk-setup --all-disks raid0 /
 
-**Command environment**
-
-Partitioning commands have the following environment variables available to them:
-
-- ``WORKING_DIR``: This is simply for some sort of inter-command state.  It will be the same directory for each command run and will only be deleted at the end of all partitioning_commands.
-- ``OUTPUT_FSTAB``: This is the target path for a fstab file.  After all partitioning commands have been run, a file should exist, formatted per fstab(5) that describes how the filesystems should be mounted.
-- ``TARGET_MOUNT_POINT``:
-
-
-Network Discovery and Setup
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-Networking is done in a similar fashion to partitioning.  A series of commands, specified in the config are run.  At the end of these commands, a interfaces(5) style file is expected to be written to ``OUTPUT_INTERFACES``.
-
-Note, that as with fstab, this file is not copied verbatim to the target filesystem, but rather made available to the OS customization stage.  That stage may just copy the file verbatim, but may also parse it, and use that as input.
-
-**Config Example**::
-
- network_commands:
-  10_netconf: curtin network copy-existing
-
-**Command environment**
-
-Networking commands have the following environment variables available to them:
-
-- ``WORKING_DIR``: This is simply for some sort of inter-command state.  It will be the same directory for each command run and will only be deleted at the end of all network_commands.
-- ``OUTPUT_INTERFACES``: This is the target path for an interfaces style file. After all commands have been run, a file should exist, formatted per interfaces(5) that describes the systems network setup.
+
+Network Discovery
+~~~~~~~~~~~~~~~~~
+Networking configuration is *discovered* in the 'network' stage.
+The default command run at this stage is ``curtin net-meta auto``.  After
+execution, it will write the discovered networking to the file specified
+in the environment variable ``OUTPUT_NETWORK_CONFIG``.  The format of this
+file is as described in :ref:`networking`.
+
+If curtin's config has a network section, the net-meta will simply parrot the
+data to the output file.  If there is no network section, then its default
+behavior is to copy existing config from the running environment.
+
+Note, that as with fstab, this file is not copied verbatim to the target
+filesystem, but rather made available to the OS customization stage.  That
+stage may just copy the file verbatim, but may also parse it, and apply the
+settings.
 
 Extraction of sources
 ~~~~~~~~~~~~~~~~~~~~~
@@ -88,27 +107,6 @@
 
  wget $URL | tar -Sxvzf 
 
-Hook for installed OS to customize itself
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-After extraction of sources, the source that was extracted is then given a chance to customize itself for the system.  This customization may include:
- - ensuring that appropriate device drivers are loaded on first boot
- - consuming the network interfaces file and applying its declarations.
- - ensuring that necessary packages 
-
-**Config Example**::
-
- config_hook: {{TARGET_MP}}/opt/curtin/config-hook
-
-**Command environment**
- - ``INTERFACES``: This is a path to the file created during networking stage
- - ``FSTAB``: This is a path to the file created during partitioning stage
- - ``CONFIG``: This is a path to the curtin config file.  It is provided so that additional configuration could be provided through to the OS customization.
-
-**Helpers**
-
-Curtin provides some helpers to make the OS customization easier.
- - `curtin in-target`: run the command while chrooted into the target.
-
 Final Commands
 ~~~~~~~~~~~~~~
 

=== modified file 'doc/topics/storage.rst'
--- doc/topics/storage.rst	2017-04-21 17:36:27 +0000
+++ doc/topics/storage.rst	2017-09-29 22:00:32 +0000
@@ -1,3 +1,5 @@
+.. _storage:
+
 =======
 Storage
 =======