Merge lp:~harlowja/cloud-init/cloud-init-seedy into lp:~cloud-init-dev/cloud-init/trunk

[Joshua Harlow]

  Add config drive support for random_seed

  A new field in the metadata has emerged, one
  that provides a way to seed the linux random
  generator. Add support for writing the seed
  and rewrite parts of the on_boot code to use
  a little helper class.

[lp:~harlowja/cloud-init/cloud-init-seedy]

866. By Joshua Harlow on 2013-09-03

Raise when no seed filename provided

[Scott Moser (smoser)]

What about just a cc_dev_random_seed that read from datasource.random_seed() if available ?

The only reason I'd lean that way is that azure also provides random seed data.

We'd just want the cc_dev_random_seed to early in the init_modules.

[Joshua Harlow (harlowja)]

Sounds good, also do u know if the file paths are right. The ones I found look right for ubunutu/rhel but second verification wouldn't hurt.

[Scott Moser (smoser)]

Josh,
  I actually think its simpler if you just write to /dev/urandom.
  You should probably make that configurable via cloud-config. but just putting it into /dev/urandom correctly seeds the kernel entropy pool.

[lp:~harlowja/cloud-init/cloud-init-seedy]

867. By Joshua Harlow on 2013-09-04

Review adjustments.

[Joshua Harlow (harlowja)]

Ok dokie, adjustments made :) I should probably add a test though.

[Scott Moser (smoser)]

It looks nice now. Some other thoughts:

 * I don't know if I prefer cloud.distro.set_random_seed() over cfg.get('random_seed_file', '/dev/urandom')
   You have thoughts on that? The latter allows me to change it easily from user-data.
 * I wouldn't prefer the metadata from user-data, but rather just append them. the way /dev/urandom works is that "more is better".
 * I guess we're expecting binary data to be in cfg.get('random_seed'). Thats fine, as yaml actually supports binary data, but might be useful to allow the user to specify that the data is base64 and decode it. I dont know.
 * I *think* I'm moving towards getting stuff out of the top level of cloud_config and into specific dictionaries. Ie:
   random_seed:
    file: /dev/urandom
    encoding: base64
    data: Zm9vYmFyCg==

   As opposed to:
    random_seed_file: /dev/urandom
    random_seed_encoding: /dev/urandom
    random_seed: Zm9vYmFyCg==

 * test would be good. :)

Thanks for doing this.

[lp:~harlowja/cloud-init/cloud-init-seedy]

868. By Joshua Harlow on 2013-09-07

Add jsonschema for namespaced and verifiable module
configuration checking as well as make most of the
module logic happen in the module itself instead of
interacting with the distro object.

[Joshua Harlow (harlowja)]

Ok I updated it with some new goodies. No tests yet, but let me know what u think :-)

[lp:~harlowja/cloud-init/cloud-init-seedy]

869. By Joshua Harlow on 2013-09-07

Ensure validate checks key existence.

[Scott Moser (smoser)]

Josh,
  I like the direction here. I like the jsonschema, but I'm targetting 0.7.3 for later this month, and i'd rather not pick up the new dependency there. Could we drop the jsonschema stuff for now, and add it post 0.7.3 ?

  I really do like the 'validate' idea though.

  I think for now just take the 'validate' use out, and drop the dependency and we're probably good.
  Of course a test would be nice :)
Thanks.

[Joshua Harlow (harlowja)]

I, I captain, will do that (later tonight I hope).

[lp:~harlowja/cloud-init/cloud-init-seedy]

870. By Joshua Harlow on 2013-09-09

Add test + remove jsonschema (for now)

=== added file 'cloudinit/config/cc_seed_random.py'
--- cloudinit/config/cc_seed_random.py	1970-01-01 00:00:00 +0000
+++ cloudinit/config/cc_seed_random.py	2013-09-09 05:48:24 +0000
@@ -0,0 +1,61 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2013 Yahoo! Inc.
+#
+#    Author: Joshua Harlow <[e-mail]>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import base64
+from StringIO import StringIO
+
+from cloudinit.settings import PER_INSTANCE
+from cloudinit import util
+
+frequency = PER_INSTANCE
+
+
+def _decode(data, encoding=None):
+    if not data:
+        return ''
+    if not encoding or encoding.lower() in ['raw']:
+        return data
+    elif encoding.lower() in ['base64', 'b64']:
+        return base64.b64decode(data)
+    elif encoding.lower() in ['gzip', 'gz']:
+        return util.decomp_gzip(data, quiet=False)
+    else:
+        raise IOError("Unknown random_seed encoding: %s" % (encoding))
+
+
+def handle(name, cfg, cloud, log, _args):
+    if not cfg or "random_seed" not in cfg:
+        log.debug(("Skipping module named %s, "
+                   "no 'random_seed' configuration found"), name)
+        return
+
+    my_cfg = cfg['random_seed']
+    seed_path = my_cfg.get('file', '/dev/urandom')
+    seed_buf = StringIO()
+    seed_buf.write(_decode(my_cfg.get('data', ''),
+                           encoding=my_cfg.get('encoding')))
+
+    metadata = cloud.datasource.metadata
+    if metadata and 'random_seed' in metadata:
+        seed_buf.write(metadata['random_seed'])
+
+    seed_data = seed_buf.getvalue()
+    if len(seed_data):
+        log.debug("%s: adding %s bytes of random seed entrophy to %s", name,
+                  len(seed_data), seed_path)
+        util.append_file(seed_path, seed_data)

=== added file 'cloudinit/exceptions.py'
--- cloudinit/exceptions.py	1970-01-01 00:00:00 +0000
+++ cloudinit/exceptions.py	2013-09-09 05:48:24 +0000
@@ -0,0 +1,21 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2013 Yahoo! Inc.
+#
+#    Author: Joshua Harlow <[e-mail]>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+class FormatValidationError(Exception):
+    pass

=== modified file 'cloudinit/sources/DataSourceConfigDrive.py'
--- cloudinit/sources/DataSourceConfigDrive.py	2013-06-05 00:42:55 +0000
+++ cloudinit/sources/DataSourceConfigDrive.py	2013-09-09 05:48:24 +0000
@@ -18,6 +18,7 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+import base64
 import json
 import os
 
@@ -41,6 +42,25 @@
 VALID_DSMODES = ("local", "net", "pass", "disabled")
 
 
+class ConfigDriveHelper(object):
+    def __init__(self, distro):
+        self.distro = distro
+
+    def on_first_boot(self, data):
+        if not data:
+            data = {}
+        if 'network_config' in data:
+            LOG.debug("Updating network interfaces from config drive")
+            self.distro.apply_network(data['network_config'])
+        files = data.get('files')
+        if files:
+            LOG.debug("Writing %s injected files", len(files))
+            try:
+                write_files(files)
+            except IOError:
+                util.logexc(LOG, "Failed writing files")
+
+
 class DataSourceConfigDrive(sources.DataSource):
     def __init__(self, sys_cfg, distro, paths):
         sources.DataSource.__init__(self, sys_cfg, distro, paths)
@@ -49,6 +69,7 @@
         self.seed_dir = os.path.join(paths.seed_dir, 'config_drive')
         self.version = None
         self.ec2_metadata = None
+        self.helper = ConfigDriveHelper(distro)
 
     def __str__(self):
         root = sources.DataSource.__str__(self)
@@ -187,20 +208,8 @@
         # instance-id
         prev_iid = get_previous_iid(self.paths)
         cur_iid = md['instance-id']
-
-        if ('network_config' in results and self.dsmode == "local" and
-            prev_iid != cur_iid):
-            LOG.debug("Updating network interfaces from config drive (%s)",
-                      dsmode)
-            self.distro.apply_network(results['network_config'])
-
-        # file writing occurs in local mode (to be as early as possible)
-        if self.dsmode == "local" and prev_iid != cur_iid and results['files']:
-            LOG.debug("writing injected files")
-            try:
-                write_files(results['files'])
-            except:
-                util.logexc(LOG, "Failed writing files")
+        if prev_iid != cur_iid and self.dsmode == "local":
+            self.helper.on_first_boot(results)
 
         # dsmode != self.dsmode here if:
         #  * dsmode = "pass",  pass means it should only copy files and then
@@ -338,6 +347,13 @@
         except KeyError:
             raise BrokenConfigDriveDir("No uuid entry in metadata")
 
+    if 'random_seed' in results['metadata']:
+        random_seed = results['metadata']['random_seed']
+        try:
+            results['metadata']['random_seed'] = base64.b64decode(random_seed)
+        except (ValueError, TypeError) as exc:
+            raise BrokenConfigDriveDir("Badly formatted random_seed: %s" % exc)
+
     def read_content_path(item):
         # do not use os.path.join here, as content_path starts with /
         cpath = os.path.sep.join((source_dir, "openstack",

=== modified file 'config/cloud.cfg'
--- config/cloud.cfg	2013-03-01 06:43:06 +0000
+++ config/cloud.cfg	2013-09-09 05:48:24 +0000
@@ -24,6 +24,7 @@
 # The modules that run in the 'init' stage
 cloud_init_modules:
  - migrator
+ - seed_random
  - bootcmd
  - write-files
  - growpart

=== added file 'tests/unittests/test_handler/test_handler_seed_random.py'
--- tests/unittests/test_handler/test_handler_seed_random.py	1970-01-01 00:00:00 +0000
+++ tests/unittests/test_handler/test_handler_seed_random.py	2013-09-09 05:48:24 +0000
@@ -0,0 +1,150 @@
+            #    Copyright (C) 2013 Hewlett-Packard Development Company, L.P.
+#
+#    Author: Juerg Haefliger <[e-mail]>
+#
+#    Based on test_handler_set_hostname.py
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+from cloudinit.config import cc_seed_random
+
+import base64
+import tempfile
+import gzip
+
+from StringIO import StringIO
+
+from cloudinit import cloud
+from cloudinit import distros
+from cloudinit import helpers
+from cloudinit import util
+
+from cloudinit.sources import DataSourceNone
+                                   
+from tests.unittests import helpers as t_help
+
+import logging
+
+LOG = logging.getLogger(__name__)
+
+
+class TestRandomSeed(t_help.TestCase):
+    def setUp(self):
+        super(TestRandomSeed, self).setUp()
+        self._seed_file = tempfile.mktemp()
+
+    def tearDown(self):
+        util.del_file(self._seed_file)
+
+    def _compress(self, text):
+        contents = StringIO()
+        gz_fh = gzip.GzipFile(mode='wb', fileobj=contents)
+        gz_fh.write(text)
+        gz_fh.close()
+        return contents.getvalue()
+
+    def _get_cloud(self, distro, metadata=None):
+        paths = helpers.Paths({})
+        cls = distros.fetch(distro)
+        ubuntu_distro = cls(distro, {}, paths)
+        ds = DataSourceNone.DataSourceNone({}, ubuntu_distro, paths)
+        if metadata:
+            ds.metadata = metadata
+        return cloud.Cloud(ds, paths, {}, ubuntu_distro, None)
+
+    def test_append_random(self):
+        cfg = {
+            'random_seed': {
+                'file': self._seed_file,
+                'data': 'tiny-tim-was-here',
+            }
+        }
+        cc_seed_random.handle('test', cfg, self._get_cloud('ubuntu'), LOG, [])
+        contents = util.load_file(self._seed_file)
+        self.assertEquals("tiny-tim-was-here", contents)
+
+    def test_append_random_unknown_encoding(self):
+        data = self._compress("tiny-toe")
+        cfg = {
+            'random_seed': {
+                'file': self._seed_file,
+                'data': data,
+                'encoding': 'special_encoding',
+            }
+        }
+        self.assertRaises(IOError, cc_seed_random.handle, 'test', cfg,
+                          self._get_cloud('ubuntu'), LOG, [])
+
+    def test_append_random_gzip(self):
+        data = self._compress("tiny-toe")
+        cfg = {
+            'random_seed': {
+                'file': self._seed_file,
+                'data': data,
+                'encoding': 'gzip',
+            }
+        }
+        cc_seed_random.handle('test', cfg, self._get_cloud('ubuntu'), LOG, [])
+        contents = util.load_file(self._seed_file)
+        self.assertEquals("tiny-toe", contents)
+
+    def test_append_random_gz(self):
+        data = self._compress("big-toe")
+        cfg = {
+            'random_seed': {
+                'file': self._seed_file,
+                'data': data,
+                'encoding': 'gz',
+            }
+        }
+        cc_seed_random.handle('test', cfg, self._get_cloud('ubuntu'), LOG, [])
+        contents = util.load_file(self._seed_file)
+        self.assertEquals("big-toe", contents)
+
+    def test_append_random_base64(self):
+        data = base64.b64encode('bubbles')
+        cfg = {
+            'random_seed': {
+                'file': self._seed_file,
+                'data': data,
+                'encoding': 'base64',
+            }
+        }
+        cc_seed_random.handle('test', cfg, self._get_cloud('ubuntu'), LOG, [])
+        contents = util.load_file(self._seed_file)
+        self.assertEquals("bubbles", contents)
+
+    def test_append_random_b64(self):
+        data = base64.b64encode('kit-kat')
+        cfg = {
+            'random_seed': {
+                'file': self._seed_file,
+                'data': data,
+                'encoding': 'b64',
+            }
+        }
+        cc_seed_random.handle('test', cfg, self._get_cloud('ubuntu'), LOG, [])
+        contents = util.load_file(self._seed_file)
+        self.assertEquals("kit-kat", contents)
+
+    def test_append_random_metadata(self):
+        cfg = {
+            'random_seed': {
+                'file': self._seed_file,
+                'data': 'tiny-tim-was-here',
+            }
+        }
+        c = self._get_cloud('ubuntu', {'random_seed': '-so-was-josh'})
+        cc_seed_random.handle('test', cfg, c, LOG, [])
+        contents = util.load_file(self._seed_file)
+        self.assertEquals('tiny-tim-was-here-so-was-josh', contents)