Merge lp:~harlowja/cloud-init/query-tool-is-back into lp:~cloud-init-dev/cloud-init/trunk

[Scott Moser]

[lp:~harlowja/cloud-init/query-tool-is-back]

648. By Joshua Harlow on 2012-09-08

Updated with no encryption and
clearing out of the userdata/raw fields
to prevent access when querying.

649. By Joshua Harlow on 2012-09-10

Fix some pylint issues and update the datasource query
to just return a map describing the database and use the
borrowed pprint code to show this map in a nice CLI format.

Also allow for printing of the init configuration as well
as the datasource via the query entrypoint.

[Scott Moser (smoser)]

I'd like to have the query tool back. Some comments:
 * I'd like some cmdline mechanism to request a single variable 'cloud-init query --instance-id' or something like that. Saving that, at very least well formatted data needs to be output for '--what=ds'. As it exists right now, I dont think its machine parsable output really. I kind of liked hte way it was in revno 649 in that respect.
    Basically, I want people to be able to use this as a replacement for 'ec2metadata --instance-id' or 'ec2metadata --local-hostname'. I dont think doing this consistently across data sources is easy, but I'd like to try.

 * 'self._write_to_cache(safe=False)' is more readable than below, which just looks odd.
    + self._write_to_cache(False)
    + self._write_to_cache(True)

 * seems like unused change for 'hash_blob' creeped in.
 * random newline in 'Requires' was added.
 * cloudinit/pprint.py has 'vim: not consistent with other 'vi:'. I'm not opposed to that change, but I'd prefer it done once all over if we wanted that.
   # vim: tabstop=4 shiftwidth=4 softtabstop=4

[Joshua Harlow (harlowja)]

Will try to fix these adjustments over the holidays and or do some stuff differently.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:649
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~harlowja/cloud-init/query-tool-is-back/+merge/123394/+edit-commit-message

https://server-team-jenkins.canonical.com/job/cloud-init-ci/33/
Executed test runs:
    None: https://server-team-jenkins.canonical.com/job/lp-vote-on-merge/7/console

Click here to trigger a rebuild:
https://server-team-jenkins.canonical.com/job/cloud-init-ci/33/rebuild

review: Needs Fixing (continuous-integration)

[Chad Smith (chad.smith)]

Hello,

Thank you for taking the time to contribute to cloud-init. Cloud-init has moved its revision control system to git. As a result, we are marking all bzr merge proposals as 'rejected'. If you would like to re-submit this proposal for review, please do so by following the current HACKING documentation at http://cloudinit.readthedocs.io/en/latest/topics/hacking.html.

This branch will no longer apply against master, the cloud-init utility has moved to cloudinit/cmd/main.py and I think there are some review comments that would need to be addressed here to make this solution a bit more generic.

Unmerged revisions

649. By Joshua Harlow on 2012-09-10

Fix some pylint issues and update the datasource query
to just return a map describing the database and use the
borrowed pprint code to show this map in a nice CLI format.

Also allow for printing of the init configuration as well
as the datasource via the query entrypoint.

648. By Joshua Harlow on 2012-09-08

Updated with no encryption and
clearing out of the userdata/raw fields
to prevent access when querying.

647. By Joshua Harlow on 2012-09-08

Start adding a query entrypoint with encryption using
aes of the userdata (raw and not raw) if possible using
the provided users (currently root) private ssh key sha256
hash as the secret (openssl was tried, didn't work due to
key file formats being all different).

=== modified file 'Requires'
--- Requires	2012-07-09 20:41:45 +0000
+++ Requires	2012-09-10 21:48:19 +0000
@@ -26,3 +26,4 @@
 
 # The new main entrypoint uses argparse instead of optparse
 argparse
+
 
=== modified file 'bin/cloud-init'
--- bin/cloud-init	2012-08-10 03:48:01 +0000
+++ bin/cloud-init	2012-09-10 21:48:19 +0000
@@ -35,6 +35,7 @@
 
 from cloudinit import log as logging
 from cloudinit import netinfo
+from cloudinit import pprint as cp
 from cloudinit import sources
 from cloudinit import stages
 from cloudinit import templater
@@ -52,11 +53,10 @@
 # Module section template
 MOD_SECTION_TPL = "cloud_%s_modules"
 
-# Things u can query on
-QUERY_DATA_TYPES = [
-    'data',
-    'data_raw',
-    'instance_id',
+# Things u can query on...
+QUERY_NAMES = [
+    'ds',
+    'cfg',
 ]
 
 # Frequency shortname to full name
@@ -342,9 +342,35 @@
     return run_module_section(mods, name, name)
 
 
-def main_query(name, _args):
-    raise NotImplementedError(("Action '%s' is not"
-                               " currently implemented") % (name))
+def main_query(name, args):
+    w_msg = welcome_format(name)
+    welcome(name, msg=w_msg)
+    items = args.what
+    if not items:
+        return 1
+    init = stages.Init()
+    ds = None
+    try:
+        # Try the 'privileged' datasource first
+        ds = init.fetch(attempt_find=False)
+    except Exception:
+        pass
+    if not ds:
+        # Use the safer version (if its there)
+        ds = init.fetch(safe=True)
+    if not ds:
+        print("No datasource available for querying!")
+        return 1
+    for i in items:
+        if i == 'ds':
+            print("Datasource details")
+            print("-" * len("Datasource details"))
+            cp.pprint(ds.describe())
+        elif i == 'cfg':
+            print("Configuration details")
+            print("-" * len("Configuration details"))
+            cp.pprint(init.cfg)
+    return 0
 
 
 def main_single(name, args):
@@ -464,10 +490,10 @@
     parser_query = subparsers.add_parser('query',
                                          help=('query information stored '
                                                'in cloud-init'))
-    parser_query.add_argument("--name", '-n', action="store",
+    parser_query.add_argument("--what", '-w', action="store",
                               help="item name to query on",
                               required=True,
-                              choices=QUERY_DATA_TYPES)
+                              choices=sorted(QUERY_NAMES))
     parser_query.set_defaults(action=('query', main_query))
 
     # This subcommand allows you to run a single module
 
=== added file 'cloudinit/pprint.py'
--- cloudinit/pprint.py	1970-01-01 00:00:00 +0000
+++ cloudinit/pprint.py	2012-09-10 21:48:19 +0000
@@ -0,0 +1,107 @@
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+#    Copyright (C) 2012 Yahoo! Inc. All Rights Reserved.
+#
+#    Licensed under the Apache License, Version 2.0 (the "License"); you may
+#    not use this file except in compliance with the License. You may obtain
+#    a copy of the License at
+#
+#         http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+#    License for the specific language governing permissions and limitations
+#    under the License.
+
+
+def center_text(text, fill, max_len):
+    return '{0:{fill}{align}{size}}'.format(text,
+                                            fill=fill, align="^", size=max_len)
+
+
+def _pformat_list(lst, item_max_len):
+    lines = []
+    if not lst:
+        lines.append("+------+")
+        lines.append("'------'")
+        return "\n".join(lines)
+    entries = []
+    max_len = 0
+    for i in lst:
+        e = pformat(i, item_max_len)
+        for v in e.split("\n"):
+            max_len = max(max_len, len(v) + 2)
+        entries.append(e)
+    lines.append("+%s+" % ("-" * (max_len)))
+    for e in entries:
+        for line in e.split("\n"):
+            lines.append("|%s|" % (center_text(line, ' ', max_len)))
+    lines.append("'%s'" % ("-" * (max_len)))
+    return "\n".join(lines)
+
+
+
+def _pformat_hash(hsh, item_max_len):
+    lines = []
+    if not hsh:
+        lines.append("+-----+-----+")
+        lines.append("'-----+-----'")
+        return "\n".join(lines)
+    # Figure out the lengths to place items in...
+    max_key_len = 0
+    max_value_len = 0
+    entries = []
+    for (k, v) in hsh.items():
+        entry = ("%s" % (_pformat_escape(k, item_max_len)),
+                         "%s" % (pformat(v, item_max_len)))
+        max_key_len = max(max_key_len, len(entry[0]) + 2)
+        for v in entry[1].split("\n"):
+            max_value_len = max(max_value_len, len(v) + 2)
+        entries.append(entry)
+    # Now actually do the placement since we have the lengths
+    lines.append("+%s+%s+" % ("-" * max_key_len, "-" * max_value_len))
+    for (key, value) in entries:
+        value_lines = value.split("\n")
+        lines.append("|%s|%s|" % (center_text(key, ' ', max_key_len),
+                                  center_text(value_lines[0], ' ',
+                                              max_value_len)))
+        if len(value_lines) > 1:
+            for j in range(1, len(value_lines)):
+                lines.append("|%s|%s|" % (center_text("-", ' ', max_key_len),
+                                          center_text(value_lines[j], ' ',
+                                                      max_value_len)))
+    lines.append("'%s+%s'" % ("-" * max_key_len, "-" * max_value_len))
+    return "\n".join(lines)
+
+
+def _pformat_escape(item, item_max_len):
+    item = _pformat_simple(item, item_max_len)
+    item = item.replace("\n", "\\n")
+    item = item.replace("\t", "\\t")
+    return item
+
+
+def _pformat_simple(item, item_max_len):
+    if item_max_len is None or item_max_len < 0:
+        return  "%s" % (item)
+    if item_max_len == 0:
+        return ''
+    item_str = "%s" % (item)
+    if len(item_str) > item_max_len:
+        # TODO(harlowja) use utf8 ellipse or '...'??
+        item_str = item_str[0:item_max_len] + '...'
+    return item_str
+
+
+def pformat(item, item_max_len=None):
+    if isinstance(item, (list, set, tuple)):
+        return _pformat_list(item, item_max_len)
+    elif isinstance(item, (dict)):
+        return _pformat_hash(item, item_max_len)
+    else:
+        return _pformat_simple(item, item_max_len)
+
+
+def pprint(item, item_max_len=None):
+    print("%s" % (pformat(item, item_max_len)))
 
=== modified file 'cloudinit/sources/__init__.py'
--- cloudinit/sources/__init__.py	2012-08-28 03:51:00 +0000
+++ cloudinit/sources/__init__.py	2012-09-10 21:48:19 +0000
@@ -23,6 +23,7 @@
 from email.mime.multipart import MIMEMultipart
 
 import abc
+import copy
 
 from cloudinit import importer
 from cloudinit import log as logging
@@ -56,6 +57,7 @@
         name = util.obj_name(self)
         if name.startswith(DS_PREFIX):
             name = name[len(DS_PREFIX):]
+        self.name = name
         self.ds_cfg = util.get_cfg_by_path(self.sys_cfg,
                                           ("datasource", name), {})
         if not ud_proc:
@@ -63,6 +65,16 @@
         else:
             self.ud_proc = ud_proc
 
+    def copy(self, safe=False):
+        nds = copy.deepcopy(self)
+        if not safe:
+            return nds
+        # Clear it out, nothing to see here...
+        nds.ud_proc = None
+        nds.userdata = None
+        nds.userdata_raw = None
+        return nds
+
     def get_userdata(self, apply_filter=False):
         if self.userdata is None:
             self.userdata = self.ud_proc.process(self.get_userdata_raw())
@@ -78,6 +90,24 @@
             return self.metadata['launch-index']
         return None
 
+    # describes the datasource as a nice map
+    # with basics such as userdata, raw userdata,
+    # anything else that a subclass can provide...
+    def describe(self):
+        return {
+            'hostname': self.get_hostname(),
+            'instance-id': self.get_instance_id(),
+            'launch-index': self.launch_index,
+            'locale': self.get_locale(),
+            'metadata': self.metadata,
+            'name': self.name,
+            'package-mirror-info': self.get_package_mirror_info(),
+            'public-ssh-keys': self.get_public_ssh_keys(),
+            'user-data': self.userdata,
+            'user-data-raw': self.userdata_raw,
+            'configuration': self.ds_cfg,
+        }
+
     def _filter_userdata(self, processed_ud):
         filters = [
             launch_index.Filter(util.safe_int(self.launch_index)),
 
=== modified file 'cloudinit/stages.py'
--- cloudinit/stages.py	2012-08-26 22:04:06 +0000
+++ cloudinit/stages.py	2012-09-10 21:48:19 +0000
@@ -60,6 +60,8 @@
         self._distro = None
         # Created only when a fetch occurs
         self.datasource = None
+        # Only created if asked and available
+        self.safe_datasource = None
 
     @property
     def distro(self):
@@ -170,11 +172,11 @@
                                       base_cfg=self._read_base_cfg())
         return merger.cfg
 
-    def _restore_from_cache(self):
+    def _restore_from_cache(self, name):
         # We try to restore from a current link and static path
         # by using the instance link, if purge_cache was called
         # the file wont exist.
-        pickled_fn = self.paths.get_ipath_cur('obj_pkl')
+        pickled_fn = self.paths.get_ipath_cur(name)
         pickle_contents = None
         try:
             pickle_contents = util.load_file(pickled_fn)
@@ -190,21 +192,18 @@
             util.logexc(LOG, "Failed loading pickled blob from %s", pickled_fn)
             return None
 
-    def _write_to_cache(self):
-        if not self.datasource:
-            return False
-        pickled_fn = self.paths.get_ipath_cur("obj_pkl")
-        try:
-            pk_contents = pickle.dumps(self.datasource)
-        except Exception:
-            util.logexc(LOG, "Failed pickling datasource %s", self.datasource)
-            return False
-        try:
-            util.write_file(pickled_fn, pk_contents, mode=0400)
+    def _write_to_cache(self, safe):
+        if not safe:
+            fmode = 0400
+            pickled_fn = self.paths.get_ipath_cur("obj_pkl")
+        else:
+            fmode = 0644
+            pickled_fn = self.paths.get_ipath_cur("safe_obj_pkl")
+        try:
+            pk_contents = pickle.dumps(self.datasource.copy(safe))
+            util.write_file(pickled_fn, pk_contents, mode=fmode)
         except Exception:
             util.logexc(LOG, "Failed pickling datasource to %s", pickled_fn)
-            return False
-        return True
 
     def _get_datasources(self):
         # Any config provided???
@@ -216,12 +215,23 @@
         cfg_list = self.cfg.get('datasource_list') or []
         return (cfg_list, pkg_list)
 
-    def _get_data_source(self):
+    def _get_safe_datasource(self):
+        if self.safe_datasource:
+            return self.safe_datasource
+        ds = self._restore_from_cache('safe_obj_pkl')
+        if ds:
+            LOG.debug("Restored from cache, safe datasource: %s", ds)
+            self.safe_datasource = ds
+        return ds
+
+    def _get_data_source(self, attempt_find):
         if self.datasource:
             return self.datasource
-        ds = self._restore_from_cache()
+        ds = self._restore_from_cache('obj_pkl')
         if ds:
             LOG.debug("Restored from cache, datasource: %s", ds)
+        if not ds and not attempt_find:
+            return None
         if not ds:
             (cfg_list, pkg_list) = self._get_datasources()
             # Deep copy so that user-data handlers can not modify
@@ -298,8 +308,11 @@
                         "%s\n" % (previous_iid))
         return iid
 
-    def fetch(self):
-        return self._get_data_source()
+    def fetch(self, safe=False, attempt_find=True):
+        if not safe:
+            return self._get_data_source(attempt_find)
+        else:
+            return self._get_safe_datasource()
 
     def instancify(self):
         return self._reflect_cur_instance()
@@ -311,8 +324,11 @@
                            self.distro, helpers.Runners(self.paths))
 
     def update(self):
-        if not self._write_to_cache():
-            return
+        if not self.datasource:
+            raise RuntimeError(("Unable to update with the given datasource, "
+                                "no datasource fetched!"))
+        self._write_to_cache(False)
+        self._write_to_cache(True)
         self._store_userdata()
 
     def _store_userdata(self):
 
=== modified file 'cloudinit/util.py'
--- cloudinit/util.py	2012-08-28 03:51:00 +0000
+++ cloudinit/util.py	2012-09-10 21:48:19 +0000
@@ -1093,10 +1093,13 @@
     log.debug(msg, exc_info=1, *args)
 
 
-def hash_blob(blob, routine, mlen=None):
+def hash_blob(blob, routine, mlen=None, give_hex=True):
     hasher = hashlib.new(routine)
     hasher.update(blob)
-    digest = hasher.hexdigest()
+    if give_hex:
+        digest = hasher.hexdigest()
+    else:
+        digest = hasher.digest()
     # Don't get to long now
     if mlen is not None:
         return digest[0:mlen]