Merge ~farcaller/cloud-init:nocloud-dmi into cloud-init:master

[Vladimir Pouzanov]

NoCloud: support seed of nocloud from smbios information

This allows the user to seed NoCloud in a trivial way from qemu/libvirt, by
using a stock image and passing a single command line flag.
No custom command line, no filesystem modification, no bootstrap disk image.

This is particularly handy now that Ec2 backend is discouraged from use:
https://bugs.launchpad.net/cloud-init/+bug/1660385

LP: #1691772

provide a way to seed NoCloud from network without image modification (bug 1691772).

[Scott Moser (smoser)]

Vladimir,
This looks really good, thank you.

I'd like to update doc/rtd/topics/datasources/nocloud.rst with an example on how to use this directly with a qemu cmdline.

Also I think the kernel cmdline should override the smbios information.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:ff8b5683b0db8f0c899111fddf8d2e52737fdef1
https://jenkins.ubuntu.com/server/job/cloud-init-ci/352/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/352/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/352/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/352/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/352/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/352/rebuild

review: Needs Fixing (continuous-integration)

30e6420... by Vladimir Pouzanov <email address hidden> on 2017-05-18

Added kernel command line / smbios docs to nocloud

d7db02d... by Vladimir Pouzanov <email address hidden> on 2017-05-18

Prefer command line over DMI

4e6d0db... by Vladimir Pouzanov <email address hidden> on 2017-05-18

Fixed flake8 linting issues

[Vladimir Pouzanov (farcaller)]

Updated the docs, ordering, and fixed linter issue.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:4e6d0db439a755a1d1f46cbc084c5fe6c313f75b
https://jenkins.ubuntu.com/server/job/cloud-init-ci/354/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/354
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/354
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/354
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/354

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/354/rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

review: Needs Fixing

[Scott Moser (smoser)]

On Fri, 19 May 2017, Ryan Harper wrote:

> > try:
> > + # Parse the system serial label from dmi. If not empty, try parsing
> > + # like the commandline
> > + md = {}
> > + serial = util.read_dmi_data('system-serial-number')
> > + if serial is not None:
>
> Your comment "if not empty" doesn't appear to match the check here.
>
> read_dmi_data may return None, or "" (see _call_dmicode which has a return "" for empty fields)
>
> serial = ""
> That returns True when checking if it is 'not None'; I think just:
>
> if serial
>
> is what we want here instead.

The comment is not right, but the function i think is fine. because
load_cmdline_data will return False on "". We do want to specifically
not pass it None. On None as the cmdline, it will call get_cmdline().

So either way is really ok, but make sure we do not pass None.

[Ryan Harper (raharper)]

On Fri, May 19, 2017 at 12:27 PM, Scott Moser <email address hidden> wrote:

> On Fri, 19 May 2017, Ryan Harper wrote:
>
> > > try:
> > > + # Parse the system serial label from dmi. If not empty,
> try parsing
> > > + # like the commandline
> > > + md = {}
> > > + serial = util.read_dmi_data('system-serial-number')
> > > + if serial is not None:
> >
> > Your comment "if not empty" doesn't appear to match the check here.
> >
> > read_dmi_data may return None, or "" (see _call_dmicode which has a
> return "" for empty fields)
> >
> > serial = ""
> > That returns True when checking if it is 'not None'; I think just:
> >
> > if serial
> >
> > is what we want here instead.
>
> The comment is not right, but the function i think is fine. because
> load_cmdline_data will return False on "". We do want to specifically
>

Why bother checking the command line for a serial of "" ? That's not going
to do anything productive AFAICT. Surely not calling a function with it
isn't needed
is better than calling it when we know ahead of time that it's not a valid
serial number
we can expect on the command line.

Ryan

9c021fd... by Vladimir Pouzanov <email address hidden> on 2017-05-20

Simplified the condition that verifies correct data in serial field

[Vladimir Pouzanov (farcaller)]

Fixed.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:9c021fd4eb270a8456786e67d8e817c177a53623
https://jenkins.ubuntu.com/server/job/cloud-init-ci/364/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/364
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/364
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/364
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/364
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/364

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/364/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

For reference, from the DMTF smbios spec DSP0134_2.7.1, the serial field is a string.
With regard to string length:

  http://www.dmtf.org/sites/default/files/standards/documents/DSP0134_2.7.1.pdf

 NOTE: There is no limit on the length of each individual text string.
 However, the length of the entire structure table (including all strings)
 must be reported in the Structure Table Length field of the SMBIOS
 Structure Table Entry Point which is a WORD field limited to 65,535 bytes.

[Ryan Harper (raharper)]

Looks good!

review: Approve

diff --git a/cloudinit/sources/DataSourceNoCloud.py b/cloudinit/sources/DataSourceNoCloud.py
index c68f6b8..e641244 100644
--- a/cloudinit/sources/DataSourceNoCloud.py
+++ b/cloudinit/sources/DataSourceNoCloud.py
@@ -43,6 +43,18 @@ class DataSourceNoCloud(sources.DataSource):
                   'network-config': None}
 
         try:
+            # Parse the system serial label from dmi. If not empty, try parsing
+            # like the commandline
+            md = {}
+            serial = util.read_dmi_data('system-serial-number')
+            if serial and load_cmdline_data(md, serial):
+                found.append("dmi")
+                mydata = _merge_new_seed(mydata, {'meta-data': md})
+        except Exception:
+            util.logexc(LOG, "Unable to parse dmi data")
+            return False
+
+        try:
             # Parse the kernel command line, getting data passed in
             md = {}
             if load_cmdline_data(md):
diff --git a/doc/rtd/topics/datasources/nocloud.rst b/doc/rtd/topics/datasources/nocloud.rst
index 0159e85..665057f 100644
--- a/doc/rtd/topics/datasources/nocloud.rst
+++ b/doc/rtd/topics/datasources/nocloud.rst
@@ -11,6 +11,28 @@ You can provide meta-data and user-data to a local vm boot via files on a
 `vfat`_ or `iso9660`_ filesystem. The filesystem volume label must be
 ``cidata``.
 
+Alternatively, you can provide meta-data via kernel command line or SMBIOS
+"serial number" option. The data must be passed in the form of a string:
+
+::
+
+  ds=nocloud[;key=val;key=val]
+
+or
+
+::
+
+  ds=nocloud-net[;key=val;key=val]
+
+e.g. you can pass this option to QEMU:
+
+::
+
+  -smbios type=1,serial=ds=nocloud-net;s=http://10.10.0.1:8000/
+
+to cause NoCloud to fetch the full meta-data from http://10.10.0.1:8000/meta-data
+after the network initialization is complete.
+
 These user-data and meta-data files are expected to be in the following format.
 
 ::
diff --git a/tools/ds-identify b/tools/ds-identify
index aff26eb..b272d18 100755
--- a/tools/ds-identify
+++ b/tools/ds-identify
@@ -544,6 +544,9 @@ dscheck_NoCloud() {
     case " ${DI_KERNEL_CMDLINE} " in
         *\ ds=nocloud*) return ${DS_FOUND};;
     esac
+    case " ${DI_DMI_PRODUCT_SERIAL} " in
+        *\ ds=nocloud*) return ${DS_FOUND};;
+    esac
     for d in nocloud nocloud-net; do
         check_seed_dir "$d" meta-data user-data && return ${DS_FOUND}
     done