Merge lp:~kaihuan-pkh/cloud-init/cloud-init into lp:~cloud-init-dev/cloud-init/trunk

[lawrence peng]

add the support for the AliCloud

[Scott Moser (smoser)]

Lawrence,
Sorry for just seeing this now.

Over all comment, is that you will need to sign the contributors agreement at http://www.ubuntu.com/legal/contributors . Please feel free to ping me in IRC (freenode 'smoser') or in email if you have further questions.

Second, how does this differe from the EC2 MD ?
Is it only in the base_url ?

[lawrence peng (kaihuan-pkh)]

Hi, Scott :
I signed the contributors agreement.
the previous merging request (284627)  is just add one file ( cloudinit/sources/DataSourceAliYun.py )，which we copied from ec2 datasource and change the DEF_MD_URL to our testing environment.
from the day i push that commit to now, we have more changes on cloud-init （but have not yet push to launchpad site for now ) , the main changes including:
1.  add some new config modules in cloudinit/config/2.  add some distro support in cloudinit/distros/3.  change the aliyun datasource DEF_MD_URL again4.  some bug fix
now, our local devel branch is almostly complete, and we are in process of testing.  we will push a new merge request in later when it passed test.
by the way, we are design cloud-init self-upgrade plan, do you have some suggest ?
thx.

[Scott Moser (smoser)]

if the DataSourceAliYun datasource in fact is just a Ec2DataSource except with the DEF_MD_URL changed then can we just inherit from the Ec2 and change the things that are needed ?

[lawrence peng (kaihuan-pkh)]

Yes, I agree with you, so on the next commit i will change the code construction of the DataSourceAliYun.py,  it won't be too long.

[lp:~kaihuan-pkh/cloud-init/cloud-init]

1159. By kaihuan-pkh <email address hidden> on 2016-07-16

bug fix for user_data.py

in _do_include:

when one url requests timeout, it will raise a exception and break the execution of cloudinit

1160. By kaihuan-pkh <email address hidden> on 2016-07-17

bug fix for distro/__init__.py

in get_primary_arch:
 os.uname is a function

1161. By kaihuan-pkh <email address hidden> on 2016-07-17

1. change default metaserver address
2. make DataSourceAliYun inherit from EC2.DataSourceEc2

[lp:~kaihuan-pkh/cloud-init/cloud-init]

1162. By kaihuan-pkh <email address hidden> on 2016-07-23

simplify get_public_ssh_keys

[Scott Moser (smoser)]

hi,

I put some comments inline. I'd like to get this in.

We have moved development to git now. Information is available
https://git.launchpad.net/cloud-init/tree/HACKING.rst

If you want to continue on here, we can, but ultimately we'll have to get it into the git branch.

[Scott Moser (smoser)]

Aliyun support is added in trunk via MP at
https://code.launchpad.net/~kaihuan-pkh/cloud-init/+git/cloud-init/+merge/308483

=== modified file 'cloudinit/distros/__init__.py'
--- cloudinit/distros/__init__.py	2016-07-14 02:10:22 +0000
+++ cloudinit/distros/__init__.py	2016-07-25 04:31:42 +0000
@@ -112,7 +112,7 @@
         raise NotImplementedError()
 
     def get_primary_arch(self):
-        arch = os.uname[4]
+        arch = os.uname()[4]
         if arch in ("i386", "i486", "i586", "i686"):
             return "i386"
         return arch

=== added file 'cloudinit/sources/DataSourceAliYun.py'
--- cloudinit/sources/DataSourceAliYun.py	1970-01-01 00:00:00 +0000
+++ cloudinit/sources/DataSourceAliYun.py	2016-07-25 04:31:42 +0000
@@ -0,0 +1,112 @@
+# vi: ts=4 expandtab
+#
+#    Copyright (C) 2009-2010 Canonical Ltd.
+#    Copyright (C) 2012 Hewlett-Packard Development Company, L.P.
+#    Copyright (C) 2012 Yahoo! Inc.
+#
+#    Author: Scott Moser <[e-mail]>
+#    Author: Juerg Hafliger <[e-mail]>
+#    Author: Joshua Harlow <[e-mail]>
+#
+#    This program is free software: you can redistribute it and/or modify
+#    it under the terms of the GNU General Public License version 3, as
+#    published by the Free Software Foundation.
+#
+#    This program is distributed in the hope that it will be useful,
+#    but WITHOUT ANY WARRANTY; without even the implied warranty of
+#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#    GNU General Public License for more details.
+#
+#    You should have received a copy of the GNU General Public License
+#    along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import time
+from copy import deepcopy
+
+from cloudinit import ec2_utils as ec2
+from cloudinit import log as logging
+from cloudinit import sources
+from cloudinit import url_helper as uhelp
+from cloudinit import util
+from cloudinit.sources import DataSourceEc2 as EC2
+
+LOG = logging.getLogger(__name__)
+
+DEF_MD_URL = "http://100.100.100.200"
+
+DEF_MD_VERSION = 'latest'
+
+# Default metadata urls that will be used if none are provided
+# They will be checked for 'resolveability' and some of the
+# following may be discarded if they do not resolve
+DEF_MD_URLS = [DEF_MD_URL,]
+
+
+class DataSourceAliYun(EC2.DataSourceEc2):
+    EC2.DEF_MD_URL = DEF_MD_URL
+    EC2.DEF_MD_URLS = DEF_MD_URLS
+
+    def __init__(self, sys_cfg, distro, paths):
+        super(DataSourceAliYun, self).__init__(sys_cfg, distro, paths)
+        self.metadata_address = DEF_MD_URL
+        self.seed_dir = os.path.join(paths.seed_dir, "AliYun")
+        self.api_ver = DEF_MD_VERSION
+
+    def get_instance_id(self):
+        if not self.metadata:
+            url = uhelp.combine_url(self.metadata_address, self.api_ver, 'meta-data/', 'instance-id')
+            try:
+                out = util.read_file_or_url(url=url, ssl_details=None, timeout=5, retries=5)
+            except Exception as e:
+                out = ''
+            finally:
+                return str(out).strip()
+        else:
+            return self.metadata.get('instance-id', '')
+
+    def get_hostname(self, fqdn=False, _resolve_ip=False):
+        return self.metadata.get('hostname', 'localhost.localdomain')
+
+    def get_public_ssh_keys(self):
+        if not self.metadata:
+            return []
+        public_keys = self.metadata.get('public-keys', {})
+
+        keys = []
+        for key_id, key_body in public_keys.items():
+            if isinstance(key_body, (str, basestring)):
+                keys.append( key_body.strip() )
+            elif isinstance(key_body, list):
+                keys.extend( key_body )
+            elif isinstance(key_body, dict):
+                key = key_body.get('openssh-key', [])
+                if isinstance(key, str):
+                    keys.append( key.strip() )
+                elif isinstance(key, list):
+                    keys.extend(key)
+        return keys
+
+    def get_ntp_conf(self):
+        if not self.metadata:
+            return {}
+        return self.metadata.get('ntp-conf')
+
+    def get_source_address(self):
+        if not self.metadata or not self.metadata.get('source-address'):
+            return []
+        source_address = self.metadata.get('source-address')
+        if isinstance(source_address, str):
+            source_address = [source_address]
+        return source_address
+
+
+# Used to match classes to dependencies
+datasources = [
+  (DataSourceAliYun, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
+]
+
+
+# Return a list of data sources that match this set of dependencies
+def get_datasource_list(depends):
+    return sources.list_from_depends(depends, datasources)

=== modified file 'cloudinit/user_data.py'
--- cloudinit/user_data.py	2016-06-21 20:27:02 +0000
+++ cloudinit/user_data.py	2016-07-25 04:31:42 +0000
@@ -233,8 +233,12 @@
             if include_once_on and os.path.isfile(include_once_fn):
                 content = util.load_file(include_once_fn)
             else:
-                resp = util.read_file_or_url(include_url,
-                                             ssl_details=self.ssl_details)
+                try:
+                    resp = util.read_file_or_url(include_url,
+                                                ssl_details=self.ssl_details)
+                except Exception as e:
+                    LOG.warn("Fetching from %s resulted in exception happend" % include_url)
+                    continue
                 if include_once_on and resp.ok():
                     util.write_file(include_once_fn, resp.contents, mode=0o600)
                 if resp.ok():