Merge lp:~harlowja/cloud-init/yum-repo-conf-write into lp:~cloud-init-dev/cloud-init/trunk

[Joshua Harlow]

[lp:~harlowja/cloud-init/yum-repo-conf-write]

690. By Joshua Harlow on 2012-10-16

Fix copyright.

691. By Joshua Harlow on 2012-10-16

Update comments as to why format is the way it is.

692. By Joshua Harlow on 2012-10-17

Add in a created by cloud-init header.

693. By Joshua Harlow on 2012-10-17

Make the yum value 'string' translation a function.

694. By Joshua Harlow on 2012-10-17

More cleanups, fix header function.

[Scott Moser (smoser)]

Pep8:
cloudinit/distros/__init__.py:442:45: W291 trailing whitespace
tests/unittests/test_datasource/test_configdrive.py:10:1: N306: imports not in alphabetical order (cloudinit.sources.datasourceconfigdrive, cloudinit.settings)
tests/unittests/test_datasource/test_configdrive.py:16:1: E303 too many blank lines (3)

Also, needs documentation in doc/examples somewhere.

review: Needs Fixing

[lp:~harlowja/cloud-init/yum-repo-conf-write]

695. By Joshua Harlow on 2012-11-08

1. Rebased with trunk
2. Added example cloud-config
3. Added functioning test for yum config

[Joshua Harlow (harlowja)]

> Pep8:
> cloudinit/distros/__init__.py:442:45: W291 trailing whitespace
> tests/unittests/test_datasource/test_configdrive.py:10:1: N306: imports not in
> alphabetical order (cloudinit.sources.datasourceconfigdrive,
> cloudinit.settings)
> tests/unittests/test_datasource/test_configdrive.py:16:1: E303 too many blank
> lines (3)
>
> Also, needs documentation in doc/examples somewhere.

All fixed.

[Scott Moser (smoser)]

It looks good now, except for some probably unintended whitespace cleanups / unrelated changes.

I went ahead and un-applied those, merged, and then re-applied them as a separate commit.

[Joshua Harlow (harlowja)]

Thanks much, making sure I run pep8 more often now that it doesn't warn about everything under the sun.

=== added file 'cloudinit/config/cc_yum_add_repo.py'
--- cloudinit/config/cc_yum_add_repo.py	1970-01-01 00:00:00 +0000
+++ cloudinit/config/cc_yum_add_repo.py	2012-11-08 04:57:21 +0000
@@ -0,0 +1,106 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Joshua Harlow <[e-mail]>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+from cloudinit import util
+
+import configobj
+
+
+def _canonicalize_id(repo_id):
+    repo_id = repo_id.lower().replace("-", "_")
+    repo_id = repo_id.replace(" ", "_")
+    return repo_id
+
+
+def _format_repo_value(val):
+    if isinstance(val, (bool)):
+        # Seems like yum prefers 1/0
+        return str(int(val))
+    if isinstance(val, (list, tuple)):
+        # Can handle 'lists' in certain cases
+        # See: http://bit.ly/Qqrf1t
+        return "\n    ".join([_format_repo_value(v) for v in val])
+    if not isinstance(val, (basestring, str)):
+        return str(val)
+    return val
+
+
+## TODO(harlowja): move to distro?
+# See man yum.conf
+def _format_repository_config(repo_id, repo_config):
+    to_be = configobj.ConfigObj()
+    to_be[repo_id] = {}
+    # Do basic translation of the items -> values
+    for (k, v) in repo_config.items():
+        # For now assume that people using this know
+        # the format of yum and don't verify keys/values further
+        to_be[repo_id][k] = _format_repo_value(v)
+    lines = to_be.write()
+    lines.insert(0, "# Created by cloud-init on %s" % (util.time_rfc2822()))
+    return "\n".join(lines)
+
+
+def handle(name, cfg, _cloud, log, _args):
+    repos = cfg.get('yum_repos')
+    if not repos:
+        log.debug(("Skipping module named %s,"
+                   " no 'yum_repos' configuration found"), name)
+        return
+    repo_base_path = util.get_cfg_option_str(cfg, 'yum_repo_dir',
+                                             '/etc/yum.repos.d/')
+    repo_locations = {}
+    repo_configs = {}
+    for (repo_id, repo_config) in repos.items():
+        canon_repo_id = _canonicalize_id(repo_id)
+        repo_fn_pth = os.path.join(repo_base_path, "%s.repo" % (canon_repo_id))
+        if os.path.exists(repo_fn_pth):
+            log.info("Skipping repo %s, file %s already exists!",
+                     repo_id, repo_fn_pth)
+            continue
+        elif canon_repo_id in repo_locations:
+            log.info("Skipping repo %s, file %s already pending!",
+                     repo_id, repo_fn_pth)
+            continue
+        if not repo_config:
+            repo_config = {}
+        # Do some basic sanity checks/cleaning
+        n_repo_config = {}
+        for (k, v) in repo_config.items():
+            k = k.lower().strip().replace("-", "_")
+            if k:
+                n_repo_config[k] = v
+        repo_config = n_repo_config
+        missing_required = 0
+        for req_field in ['baseurl']:
+            if not req_field in repo_config:
+                log.warn(("Repository %s does not contain a %s"
+                           " configuration 'required' entry"),
+                         repo_id, req_field)
+                missing_required += 1
+        if not missing_required:
+            repo_configs[canon_repo_id] = repo_config
+            repo_locations[canon_repo_id] = repo_fn_pth
+        else:
+            log.warn("Repository %s is missing %s required fields, skipping!",
+                     repo_id, missing_required)
+    for (c_repo_id, path) in repo_locations.items():
+        repo_blob = _format_repository_config(c_repo_id,
+                                              repo_configs.get(c_repo_id))
+        util.write_file(path, repo_blob)

=== modified file 'cloudinit/distros/__init__.py'
--- cloudinit/distros/__init__.py	2012-11-07 15:42:54 +0000
+++ cloudinit/distros/__init__.py	2012-11-08 04:57:21 +0000
@@ -81,7 +81,7 @@
 
     def _get_arch_package_mirror_info(self, arch=None):
         mirror_info = self.get_option("package_mirrors", [])
-        if arch == None:
+        if not arch:
             arch = self.get_primary_arch()
         return _get_arch_package_mirror_info(mirror_info, arch)
 
@@ -187,23 +187,23 @@
         # inputs. If something goes wrong, we can end up with a system
         # that nobody can login to.
         adduser_opts = {
-                "gecos": '--comment',
-                "homedir": '--home',
-                "primary_group": '--gid',
-                "groups": '--groups',
-                "passwd": '--password',
-                "shell": '--shell',
-                "expiredate": '--expiredate',
-                "inactive": '--inactive',
-                "selinux_user": '--selinux-user',
-                }
+            "gecos": '--comment',
+            "homedir": '--home',
+            "primary_group": '--gid',
+            "groups": '--groups',
+            "passwd": '--password',
+            "shell": '--shell',
+            "expiredate": '--expiredate',
+            "inactive": '--inactive',
+            "selinux_user": '--selinux-user',
+        }
 
         adduser_opts_flags = {
-                "no_user_group": '--no-user-group',
-                "system": '--system',
-                "no_log_init": '--no-log-init',
-                "no_create_home": "-M",
-                }
+            "no_user_group": '--no-user-group',
+            "system": '--system',
+            "no_log_init": '--no-log-init',
+            "no_create_home": "-M",
+        }
 
         # Now check the value and create the command
         for option in kwargs:
@@ -312,11 +312,9 @@
                 raise e
         util.ensure_dir(path, 0755)
 
-    def write_sudo_rules(self,
-        user,
-        rules,
-        sudo_file="/etc/sudoers.d/90-cloud-init-users",
-        ):
+    def write_sudo_rules(self, user, rules, sudo_file=None):
+        if not sudo_file:
+            sudo_file = "/etc/sudoers.d/90-cloud-init-users"
 
         content_header = "# user rules for %s" % user
         content = "%s\n%s %s\n\n" % (content_header, user, rules)

=== modified file 'cloudinit/sources/DataSourceAltCloud.py'
--- cloudinit/sources/DataSourceAltCloud.py	2012-09-21 00:56:22 +0000
+++ cloudinit/sources/DataSourceAltCloud.py	2012-11-08 04:57:21 +0000
@@ -47,7 +47,7 @@
     'instance-id': 455,
     'local-hostname': 'localhost',
     'placement': {},
-    }
+}
 
 
 def read_user_data_callback(mount_dir):

=== modified file 'cloudinit/util.py'
--- cloudinit/util.py	2012-10-23 16:58:32 +0000
+++ cloudinit/util.py	2012-11-08 04:57:21 +0000
@@ -1193,8 +1193,7 @@
                     indent=4,
                     explicit_start=True,
                     explicit_end=True,
-                    default_flow_style=False,
-                    )
+                    default_flow_style=False)
     return formatted
 
 

=== added file 'doc/examples/cloud-config-yum-repo.txt'
--- doc/examples/cloud-config-yum-repo.txt	1970-01-01 00:00:00 +0000
+++ doc/examples/cloud-config-yum-repo.txt	2012-11-08 04:57:21 +0000
@@ -0,0 +1,20 @@
+#cloud-config
+# vim: syntax=yaml
+#
+# Add yum repository configuration to the system
+#
+# The following example adds the file /etc/yum.repos.d/epel_testing.repo
+# which can then subsequently be used by yum for later operations.
+yum_repos:
+    # The name of the repository
+    epel-testing:
+        # Any repository configuration options
+        # See: man yum.conf
+        #
+        # This one is required!
+        baseurl: http://download.fedoraproject.org/pub/epel/testing/5/$basearch
+        enabled: false
+        failovermethod: priority
+        gpgcheck: true
+        gpgkey: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
+        name: Extra Packages for Enterprise Linux 5 - Testing

=== modified file 'tests/configs/sample1.yaml'
--- tests/configs/sample1.yaml	2012-07-03 01:43:54 +0000
+++ tests/configs/sample1.yaml	2012-11-08 04:57:21 +0000
@@ -50,4 +50,3 @@
  
 
 byobu_by_default: user
-output: {all: '| tee -a /var/log/cloud-init-output.log'}

=== added file 'tests/unittests/test_handler/test_handler_yum_add_repo.py'
--- tests/unittests/test_handler/test_handler_yum_add_repo.py	1970-01-01 00:00:00 +0000
+++ tests/unittests/test_handler/test_handler_yum_add_repo.py	2012-11-08 04:57:21 +0000
@@ -0,0 +1,68 @@
+from cloudinit import helpers
+from cloudinit import util
+
+from cloudinit.config import cc_yum_add_repo
+
+from tests.unittests import helpers
+
+import logging
+
+from StringIO import StringIO
+
+import configobj
+
+LOG = logging.getLogger(__name__)
+
+
+class TestConfig(helpers.FilesystemMockingTestCase):
+    def setUp(self):
+        super(TestConfig, self).setUp()
+        self.tmp = self.makeDir(prefix="unittest_")
+
+    def test_bad_config(self):
+        cfg = {
+            'yum_repos': {
+                'epel-testing': {
+                    'name': 'Extra Packages for Enterprise Linux 5 - Testing',
+                    # Missing this should cause the repo not to be written
+                    #'baseurl': 'http://blah.org/pub/epel/testing/5/$basearch',
+                    'enabled': False,
+                    'gpgcheck': True,
+                    'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL',
+                    'failovermethod': 'priority',
+                },
+            },
+        }
+        self.patchUtils(self.tmp)
+        cc_yum_add_repo.handle('yum_add_repo', cfg, None, LOG, [])
+        with self.assertRaises(IOError):
+            util.load_file("/etc/yum.repos.d/epel_testing.repo")
+
+    def test_write_config(self):
+        cfg = {
+            'yum_repos': {
+                'epel-testing': {
+                    'name': 'Extra Packages for Enterprise Linux 5 - Testing',
+                    'baseurl': 'http://blah.org/pub/epel/testing/5/$basearch',
+                    'enabled': False,
+                    'gpgcheck': True,
+                    'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL',
+                    'failovermethod': 'priority',
+                },
+            },
+        }
+        self.patchUtils(self.tmp)
+        cc_yum_add_repo.handle('yum_add_repo', cfg, None, LOG, [])
+        contents = util.load_file("/etc/yum.repos.d/epel_testing.repo")
+        contents = configobj.ConfigObj(StringIO(contents))
+        expected = {
+            'epel_testing': {
+                'name': 'Extra Packages for Enterprise Linux 5 - Testing',
+                'failovermethod': 'priority',
+                'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL',
+                'enabled': '0',
+                'baseurl': 'http://blah.org/pub/epel/testing/5/$basearch',
+                'gpgcheck': '1',
+            }
+        }
+        self.assertEquals(expected, dict(contents))

=== modified file 'tests/unittests/test_runs/test_simple_run.py'
--- tests/unittests/test_runs/test_simple_run.py	2012-10-28 02:25:48 +0000
+++ tests/unittests/test_runs/test_simple_run.py	2012-11-08 04:57:21 +0000
@@ -37,11 +37,13 @@
         self.replicateTestRoot('simple_ubuntu', new_root)
         cfg = {
             'datasource_list': ['None'],
-            'write_files': [{
+            'write_files': [
+                {
                     'path': '/etc/blah.ini',
                     'content': 'blah',
                     'permissions': 0755,
-            }],
+                },
+            ],
             'cloud_init_modules': ['write-files'],
         }
         cloud_cfg = util.yaml_dumps(cfg)

=== modified file 'tools/hacking.py'
--- tools/hacking.py	2012-08-22 18:12:32 +0000
+++ tools/hacking.py	2012-11-08 04:57:21 +0000
@@ -66,8 +66,8 @@
     # handle import x
     # use .lower since capitalization shouldn't dictate order
     split_line = import_normalize(physical_line.strip()).lower().split()
-    split_previous = import_normalize(lines[line_number - 2]
-            ).strip().lower().split()
+    split_previous = import_normalize(lines[line_number - 2])
+    split_previous = split_previous.strip().lower().split()
     # with or without "as y"
     length = [2, 4]
     if (len(split_line) in length and len(split_previous) in length and