Merge ~raharper/cloud-init:fix/merge-how-docs into cloud-init:master

[Ryan Harper]

doc: update merging doc with fixes and some additional details/examples

Update config merging documentation with cloud-config syntax fix. Add an
example showing how to merge two files with runcmd.

[Scott Moser (smoser)]

A doc reference to the tools/ccfg-merge-debug might be useful (probably that could use some improvements too)

[Ryan Harper (raharper)]

I'll look at the tool/ccfg-merge-debug; but it didn't look immediately helpful, so I ended up launching an lxd container with two files in /etc/cloud/cloud.cfg.d/ and using cloud-init clean --logs --reboot instead.

I'll take a second look at the tool.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:8429b81e62c2fce1bd9dfae1420e261c02e6f9e5
https://jenkins.ubuntu.com/server/job/cloud-init-ci/528/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/528/rebuild

review: Approve (continuous-integration)

8b53811... by Ryan Harper on 2019-01-22

doc: add docs on built-in mergers and their options.

[Ryan Harper (raharper)]

% PYTHONPATH=`pwd` ./tools/ccfg-merge-debug wark/10-runcmd.cfg wark/20-runcmd.cfg --verbose -o merged.cfg
2019-01-22 19:00:48,677 - helpers.py[WARNING]: No per instance data available, is there an datasource/iid set?
2019-01-22 19:00:48,677 - __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'runcmd:...'
2019-01-22 19:00:48,677 - __init__.py[WARNING]: Unhandled non-multipart (text/x-not-multipart) userdata: 'runcmd:...'
2019-01-22 19:00:48,677 - util.py[WARNING]: Failed calling handler CloudConfigPartHandler: [['text/cloud-config', 'text/cloud-config-jsonp']] (__end__, None, 3) with frequency once-per-instance

It needs some fixing.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:8b5381195ce394c0a6b99121207816e29ee6bb72
https://jenkins.ubuntu.com/server/job/cloud-init-ci/529/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/529/rebuild

review: Approve (continuous-integration)

[Dan Watkins (oddbloke)]

review: Approve

diff --git a/doc/rtd/topics/merging.rst b/doc/rtd/topics/merging.rst
index c75ca59..5f7ca18 100644
--- a/doc/rtd/topics/merging.rst
+++ b/doc/rtd/topics/merging.rst
@@ -21,12 +21,12 @@ For example.
 .. code-block:: yaml
 
    #cloud-config (1)
-   run_cmd:
+   runcmd:
      - bash1
      - bash2
 
    #cloud-config (2)
-   run_cmd:
+   runcmd:
      - bash3
      - bash4
 
@@ -36,7 +36,7 @@ cloud-config object that contains the following.
 .. code-block:: yaml
 
    #cloud-config (merged)
-   run_cmd:
+   runcmd:
      - bash3
      - bash4
 
@@ -45,7 +45,7 @@ Typically this is not what users want; instead they would likely prefer:
 .. code-block:: yaml
 
    #cloud-config (merged)
-   run_cmd:
+   runcmd:
      - bash1
      - bash2
      - bash3
@@ -55,6 +55,45 @@ This way makes it easier to combine the various cloud-config objects you have
 into a more useful list, thus reducing duplication necessary to accomplish the
 same result with the previous method.
 
+
+Built-in Mergers
+================
+
+Cloud-init provides merging for the following built-in types:
+
+- Dict
+- List
+- String
+
+The ``Dict`` merger has the following options which control what is done with
+values contained within the config.
+
+- ``allow_delete``: Existing values not present in the new value can be deleted, defaults to False
+- ``no_replace``: Do not replace an existing value if one is already present, enabled by default.
+- ``replace``: Overwrite existing values with new ones.
+
+The ``List`` merger has the following options which control what is done with
+the values contained within the config.
+
+- ``append``:  Add new value to the end of the list, defaults to False.
+- ``prepend``:  Add new values to the start of the list, defaults to False.
+- ``no_replace``: Do not replace an existing value if one is already present, enabled by default.
+- ``replace``: Overwrite existing values with new ones.
+
+The ``Str`` merger has the following options which control what is done with
+the values contained within the config.
+
+- ``append``:  Add new value to the end of the string, defaults to False.
+
+Common options for all merge types which control how recursive merging is
+done on other types.
+
+- ``recurse_dict``: If True merge the new values of the dictionary, defaults to True.
+- ``recurse_list``: If True merge the new values of the list, defaults to False.
+- ``recurse_array``: Alias for ``recurse_list``.
+- ``recurse_str``: If True merge the new values of the string, defaults to False.
+
+
 Customizability
 ===============
 
@@ -164,8 +203,8 @@ string format (i.e. the second option above), for example:
 
 .. code-block:: python
 
-   {'merge_how': [{'name': 'list', 'settings': ['extend']},
-                  {'name': 'dict', 'settings': []},
+   {'merge_how': [{'name': 'list', 'settings': ['append']},
+                  {'name': 'dict', 'settings': ['no_replace', 'recurse_list']},
                   {'name': 'str', 'settings': ['append']}]}
 
 This would be the equivalent format for default string format but in dictionary
@@ -201,4 +240,43 @@ Note, however, that merge algorithms are not used *across* types of
 configuration.  As was the case before merging was implemented,
 user-data will overwrite conf.d configuration without merging.
 
+Example cloud-config
+====================
+
+A common request is to include multiple ``runcmd`` directives in different
+files and merge all of the commands together.  To achieve this, we must modify
+the default merging to allow for dictionaries to join list values.
+
+
+The first config
+
+.. code-block:: yaml
+
+   #cloud-config
+   merge_how:
+    - name: list
+      settings: [append]
+    - name: dict
+      settings: [no_replace, recurse_list]
+
+   runcmd:
+     - bash1
+     - bash2
+
+The second config
+
+.. code-block:: yaml
+
+   #cloud-config
+   merge_how:
+    - name: list
+      settings: [append]
+    - name: dict
+      settings: [no_replace, recurse_list]
+
+   runcmd:
+     - bash3
+     - bash4
+
+
 .. vi: textwidth=78