cloud-init

Merge ~chad.smith/cloud-init:aws-local-dhcp into cloud-init:master

  1. Git
  2. lp:~chad.smith/cloud-init
  3. aws-local-dhcp
  4. Merge into master
Proposed by Chad Smith on 2017-07-28
Status: Merged
Merged at revision: d5f855dd96ccbea77f61b0515b574ad2c43d116d
Proposed branch: ~chad.smith/cloud-init:aws-local-dhcp
Merge into: cloud-init:master
Prerequisite: ~chad.smith/cloud-init:unittests-in-cloudinit-package
Diff against target: 901 lines (+511/-91)
11 files modified
cloudinit/net/__init__.py (+20/-33)
cloudinit/net/dhcp.py (+119/-0)
cloudinit/net/tests/test_dhcp.py (+144/-0)
cloudinit/net/tests/test_init.py (+1/-1)
cloudinit/sources/DataSourceAliYun.py (+6/-3)
cloudinit/sources/DataSourceEc2.py (+99/-22)
tests/unittests/helpers.py (+1/-1)
tests/unittests/test_datasource/test_aliyun.py (+6/-5)
tests/unittests/test_datasource/test_common.py (+1/-0)
tests/unittests/test_datasource/test_ec2.py (+112/-24)
tox.ini (+2/-2)
Reviewer Review Type Date Requested Status
Andrew Jorgensen (community) 2017-08-04 Approve on 2017-08-07
Server Team CI bot continuous-integration Approve on 2017-08-07
Scott Moser 2017-07-28 Approve on 2017-08-04
Review via email: mp+328241@code.launchpad.net

Commit Message

ec2: Allow Ec2 to run in init-local using dhclient in a sandbox.

This branch is a prerequisite for IPv6 support in AWS by allowing Ec2 datasource to query the metadata source version 2016-09-02 about whether or not it needs to configure IPv6 on interfaces. If version 2016-09-02 is not present, fallback to the min_metadata_version of 2009-04-04. The DataSourceEc2Local not run on FreeBSD because dhclient in doesn't support the -sf flag allowing us to run dhclient without filesystem side-effects.

To query AWS' metadata address @ 169.254.169.254, the instance must have a dhcp-allocated address configured. Configuring IPv4 link-local addresses result in timeouts from the metadata service. We introduced a DataSourceEc2Local subclass which will perform a sandboxed dhclient discovery which obtains an authorized IP address on eth0 and crawl metadata about full instance network configuration.

Since ec2 IPv6 metadata is not sufficient in itself to tell us all the ipv6 knownledge we need, it only be used as a boolean to tell us which nics need IPv6. Cloud-init will then configure desired interfaces to DHCPv6 versus DHCPv4.

Performance side note: Shifting the dhcp work into init-local for Ec2 actually gets us 1 second faster deployments by skipping init-network phase of alternate datasource checks because Ec2Local is configured in an ealier boot stage. In 3 test runs prior to this change: cloud-init runs were 5.5 seconds, with the change we now average 4.6 seconds.

This efficiency could be even further improved if we avoiding dhcp discovery in order to talk to the metadata service from an AWS authorized dhcp address if there were some way to advertize the dhcp configuration via DMI/SMBIOS or system environment variables.

Inspecting time costs of the dhclient setup/teardown in 3 live runs the time cost for the dhcp setup round trip on AWS is:
test 1: 76 milliseconds
         dhcp discovery + metadata: 0.347 seconds
         metadata alone: 0.271 seconds
test 2: 88 milliseconds
         dhcp discovery + metadata: 0.388 seconds
         metadata alone: 0.300 seconds
test 3: 75 milliseconds
         dhcp discovery + metadata: 0.366 seconds
         metadata alone: 0.291 seconds

LP: #1709772

Description of the Change

ec2: Allow Ec2 to run in init-local using dhclient in a sandbox.

This branch is a prerequisite for IPv6 support in AWS by allowing Ec2 datasource to query the metadata source version 2016-09-02 about whether or not it needs to connfigure IPv6 on interfaces. If version 2016-09-02 is not present, fallback to the min_metadata_version of 2009-04-04. The DataSourceEc2Local will explicitly not run on FreeBSD because dhclient in that environment doesn't support the -sf flag to allow us to run dhclient without side-effects.

To query AWS' metadata address @ 169.254.169.254, the instance must have an AWS-dhcp-allocated address configured. Configuring IPv4 link-local addresses result in timeouts from the metadata service. So we now have a DataSourceEc2Local subclass which will perform a sandboxed dhclient discovery in order to obtain an authorized IP address which is used to set up eth0 and curl metadata about full instance network configuration.

A subsequent branch will inspect IPv6 configuration from the metadata harvested and properly write network IPv4/IPv6 configuration for the instance for all enabled interfaces.

Side note: The only way AWS supports querying ipv6 info from the vm
is via queries of the metadata service. This logic adds an extra dhclient attempt in init-local phase for AWS so there is an additional time cost of around a 10th of a second for boots because of the sandboxed dhclient discovery runs. This timecost would be greater if AWS' dhcp service is slow to respond.

To post a comment you must log in.
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:3b17a848a888090f61a7087eb617e455c905123e
https://jenkins.ubuntu.com/server/job/cloud-init-ci/108/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: CentOS 6 & 7: Build & Test
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/108/rebuild

review: Approve (continuous-integration)
Ryan Harper (raharper) wrote :

This looks really good. Just some questions and clarifications in inline comments.

~chad.smith/cloud-init:aws-local-dhcp updated on 2017-08-03
92e1952... by Chad Smith on 2017-08-03

address review comments: renaming drop _clean from maybe_dhcp_clean_discovery & dhcp_clean_discovery. updated log messages. adapted unit tests to DEF_MD_VERSION

Chad Smith (chad.smith) :
Chad Smith (chad.smith) :
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:f08dc1ea917cbd86a5ee4ab9e343800a6d904104
https://jenkins.ubuntu.com/server/job/cloud-init-ci/113/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/113/rebuild

review: Needs Fixing (continuous-integration)
Scott Moser (smoser) wrote :

theres nothing huge in my comments.
thanks.

Chad Smith (chad.smith) :
~chad.smith/cloud-init:aws-local-dhcp updated on 2017-08-03
e2dc2b7... by Chad Smith on 2017-08-03

update comment to declare mac-address is what we are looking for instead of IP address

Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:a1268952438c26bb949b7799019b1a8fbb1fe6e6
https://jenkins.ubuntu.com/server/job/cloud-init-ci/114/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/114/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:aws-local-dhcp updated on 2017-08-03
222567d... by Chad Smith on 2017-08-03

drop logic duplication. update unit tests for new log messages

Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:137c2881ed2d330f9b2f26a267680bbe2f945149
https://jenkins.ubuntu.com/server/job/cloud-init-ci/116/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/116/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:aws-local-dhcp updated on 2017-08-04
e0fb53a... by Chad Smith on 2017-08-03

retain backward compatibility with 2009-04-04 metadata service for Ec2 lookalike clouds

25b526c... by Chad Smith on 2017-08-04

simplify log message

23bb804... by Chad Smith on 2017-08-04

WIP

Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:d114d715a42a0b4ce7fa3435eb5bff342a2fbe04
https://jenkins.ubuntu.com/server/job/cloud-init-ci/118/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/118/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:aws-local-dhcp updated on 2017-08-04
c61035a... by Chad Smith on 2017-08-04

update aliyun datasource and unit tests for supported_metadata_versions

Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:1a01cae3f83a00cd57bdb9322f58c5b4bfc67686
https://jenkins.ubuntu.com/server/job/cloud-init-ci/119/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/119/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:aws-local-dhcp updated on 2017-08-04
87d53b4... by Chad Smith on 2017-08-04

fix flakes

Chad Smith (chad.smith) wrote :

All comments addressed, and we now handle backward compatibility for old metadata versions.

Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:b43779d06d30ca0877ab320855a8975bb7fb4df7
https://jenkins.ubuntu.com/server/job/cloud-init-ci/120/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: CentOS 6 & 7: Build & Test
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/120/rebuild

review: Approve (continuous-integration)
Scott Moser (smoser) wrote :

I have one fun comment in line.

~chad.smith/cloud-init:aws-local-dhcp updated on 2017-08-04
da6658b... by Chad Smith on 2017-08-04

move dhclient command checks into maybe_perform_dhcp_discovery

Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:e77a177c07a628226055bf034f45f4cde5be1577
https://jenkins.ubuntu.com/server/job/cloud-init-ci/121/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/121/rebuild

review: Needs Fixing (continuous-integration)
Scott Moser (smoser) wrote :

I added Andrew to this merge specifically for his insight on knowing how to determine if a metadata version is available .

~chad.smith/cloud-init:aws-local-dhcp updated on 2017-08-04
4eb786a... by Chad Smith on 2017-08-04

add a min_metadata_version versus extended_metadata_version class attrs and a get_metadadata_version method on Ec2 and AliYun datasources

3bdde87... by Chad Smith on 2017-08-04

flakes

Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:d91194fb65daabbed97b5e6eccdc087669420993
https://jenkins.ubuntu.com/server/job/cloud-init-ci/122/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/122/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:aws-local-dhcp updated on 2017-08-04
22f4f6f... by Chad Smith on 2017-08-04

construct proper versioned url in get_metadata_api_version

Scott Moser (smoser) wrote :

I can approve this as is, assuming you've tested.
you'll need to fix the merge conflicts, rebase, squash...

i think it seems sane.

review: Approve
~chad.smith/cloud-init:aws-local-dhcp updated on 2017-08-04
813f998... by Chad Smith on 2017-08-04

response codes are ints

84bf898... by Chad Smith on 2017-08-04

additional log message when we discover an extended/preferred metadata version

Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:5ace7f3f788a87799709a3bb25cf5b00b79434d1
https://jenkins.ubuntu.com/server/job/cloud-init-ci/123/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: CentOS 6 & 7: Build & Test
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/123/rebuild

review: Approve (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:67a0f861ed927f80ecbdb007faf5e085dc8b3eea
https://jenkins.ubuntu.com/server/job/cloud-init-ci/124/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    FAILED: CentOS 6 & 7: Build & Test

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/124/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:aws-local-dhcp updated on 2017-08-04
b666d7e... by Chad Smith on 2017-08-04

log typo prefered -> prefered

Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:84bf898f49a17184dfeb7de4309821e7ef904a22
https://jenkins.ubuntu.com/server/job/cloud-init-ci/125/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: CentOS 6 & 7: Build & Test
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/125/rebuild

review: Approve (continuous-integration)
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:b666d7e84134970d066c36b003a913b704dedab2
https://jenkins.ubuntu.com/server/job/cloud-init-ci/126/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    FAILED: CentOS 6 & 7: Build & Test

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/126/rebuild

review: Needs Fixing (continuous-integration)
Andrew Jorgensen (ajorgens) wrote :

> I added Andrew to this merge specifically for his insight on knowing how to
> determine if a metadata version is available .

The most explicit way to discover if a version is available is to ask the instance meta-data service to tell you:

$ curl http://169.254.169.254/
1.0
2007-01-19
2007-03-01
2007-08-29
2007-10-10
2007-12-15
2008-02-01
2008-09-01
2009-04-04
2011-01-01
2011-05-01
2012-01-12
2014-02-25
2014-11-05
2015-10-20
2016-04-19
2016-06-30
2016-09-02
latest

I don't know if that is supported on work-alike clouds.

But you will get a 404 from EC2 if you ask for a version that is not supported:

$ curl --fail http://169.254.169.254/2017-08-07/
curl: (22) The requested URL returned error: 404 Not Found

Andrew Jorgensen (ajorgens) wrote :

> This logic adds an extra dhclient attempt in init-local phase for AWS so there is an additional
> time cost of around a 10th of a second for boots because of the sandboxed dhclient discovery
> runs. This timecost would be greater if AWS' dhcp service is slow to respond.

I would want to see some comparisons of launch times in practice, rather than relying on theory here.

review: Needs Information
Andrew Jorgensen (ajorgens) wrote :

A clearer explanation in the commit message might be helpful too: IPv6 information is given via DHCPv6, but the only way to tell the difference between DHCPv6 failing to answer and IPv6 not being configured for the instance is by asking the instance metadata service.

Chad Smith (chad.smith) wrote :

> > I added Andrew to this merge specifically for his insight on knowing how to
> > determine if a metadata version is available .
>
> The most explicit way to discover if a version is available is to ask the
> instance meta-data service to tell you:
>
> $ curl http://169.254.169.254/
> 1.0
> 2007-01-19
...
> 2016-09-02
> latest
>
> I don't know if that is supported on work-alike clouds.

Andrew, Thanks for this lookover. Yes, we were concerned as well about whether work-alike clouds supported this root-level curl opted to avoid the top-level listing as a 404 at root-level is the ~same~ cost (404 status response) as a direct curl against the specific version we hope to query which would give us a discrete/complete understanding about a specific version. I'd prefer only using the top-level listing (http://169.254.169.254/) if our datasources start supporting > 2 metadata versions to avoid discrete round trips of each individual datasource. For the moment, I'd like to avoid adding the top-level query cost to all work-alike clouds as we don't know for sure if they support the version listing.

~chad.smith/cloud-init:aws-local-dhcp updated on 2017-08-07
d462d39... by Chad Smith on 2017-08-07

use util.log_time instead of making direct calls to time.time for crawl_metadata

Andrew Jorgensen (ajorgens) wrote :

I should have been more clear about the "launch times in practice" bit. In the first minutes of an instance's existence there can be delays in DHCPv4, networking, and even instance metadata. The only valid way to test the impact is to bake the changes onto an AMI and compare launch times with the vanilla AMI. Does that make sense? A reboot isn't sufficient because all the early setup of the instance is already done.

Chad Smith (chad.smith) wrote :

Roger Andrew. I'll upload my own AMI with this changeset and compare it to a fresh new instance out of the gate before initial any dhcp has run for the instance.

Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:d462d397d4a6d1823584e21372117d19cabf433d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/127/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: CentOS 6 & 7: Build & Test
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/127/rebuild

review: Approve (continuous-integration)
Chad Smith (chad.smith) wrote :

Andrew:
Ok went through 6 more runs w/ fresh AMIs creations, upstream (without Ec2Local) versus proposed (with Dhcp discovery in Ec2Local) Looks like even in the fresh AMI case we still see that benefit.

cloud-init runtime of upstream without dhcp init-local setup or teardown:

test 1: 6.204 seconds
test 2: 7.179 seconds
test 3: 7.406 seconds

cloud-init runtime the new init-local dhcp discovery setup & teardown from DataSourceEc2Local:
test 1: 6.255 seconds
test 2: 5.152 seconds
test 3: 5.411 seconds

Andrew Jorgensen (ajorgens) wrote :

That does indeed look promising. Cool!

review: Approve
Andrew Jorgensen (ajorgens) :
Andrew Jorgensen (ajorgens) wrote :

Several unit tests hang forever on CentOS 7 in EC2 for me:

test_valid_platform_with_strict_true
test_valid_platform_with_strict_false
test_unknown_platform_with_strict_false
test_ec2_local_performs_dhcp_on_non_bsd

Scott Moser (smoser) wrote :

Andrew,
./tools/run-centos --unittest --keep 7

That ran to completion for me. there definitely are slower tests, and possibly buggily slow, but i don tsee a forever hang.

Andrew Jorgensen (ajorgens) wrote :

Hi Scott, Did you run that in an EC2 instance? I suspect some interaction with the instance meta-data service and the new code is causing the hang.

Andrew Jorgensen (ajorgens) wrote :

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/cloudinit/net/__init__.py b/cloudinit/net/__init__.py
2index 1b87357..a1b0db1 100644
3--- a/cloudinit/net/__init__.py
4+++ b/cloudinit/net/__init__.py
5@@ -175,13 +175,8 @@ def is_disabled_cfg(cfg):
6 return cfg.get('config') == "disabled"
7
8
9-def generate_fallback_config(blacklist_drivers=None, config_driver=None):
10- """Determine which attached net dev is most likely to have a connection and
11- generate network state to run dhcp on that interface"""
12-
13- if not config_driver:
14- config_driver = False
15-
16+def find_fallback_nic(blacklist_drivers=None):
17+ """Return the name of the 'fallback' network device."""
18 if not blacklist_drivers:
19 blacklist_drivers = []
20
21@@ -233,15 +228,24 @@ def generate_fallback_config(blacklist_drivers=None, config_driver=None):
22 if DEFAULT_PRIMARY_INTERFACE in names:
23 names.remove(DEFAULT_PRIMARY_INTERFACE)
24 names.insert(0, DEFAULT_PRIMARY_INTERFACE)
25- target_name = None
26- target_mac = None
27+
28+ # pick the first that has a mac-address
29 for name in names:
30- mac = read_sys_net_safe(name, 'address')
31- if mac:
32- target_name = name
33- target_mac = mac
34- break
35- if target_mac and target_name:
36+ if read_sys_net_safe(name, 'address'):
37+ return name
38+ return None
39+
40+
41+def generate_fallback_config(blacklist_drivers=None, config_driver=None):
42+ """Determine which attached net dev is most likely to have a connection and
43+ generate network state to run dhcp on that interface"""
44+
45+ if not config_driver:
46+ config_driver = False
47+
48+ target_name = find_fallback_nic(blacklist_drivers=blacklist_drivers)
49+ if target_name:
50+ target_mac = read_sys_net_safe(target_name, 'address')
51 nconf = {'config': [], 'version': 1}
52 cfg = {'type': 'physical', 'name': target_name,
53 'mac_address': target_mac, 'subnets': [{'type': 'dhcp'}]}
54@@ -511,25 +515,7 @@ def get_interfaces_by_mac():
55
56 Bridges and any devices that have a 'stolen' mac are excluded."""
57 ret = {}
58-<<<<<<< cloudinit/net/__init__.py
59 for name, mac, _driver, _devid in get_interfaces():
60-=======
61- devs = get_devicelist()
62- empty_mac = '00:00:00:00:00:00'
63- for name in devs:
64- if not interface_has_own_mac(name):
65- continue
66- if is_bridge(name):
67- continue
68- if is_vlan(name):
69- continue
70- mac = get_interface_mac(name)
71- # some devices may not have a mac (tun0)
72- if not mac:
73- continue
74- if mac == empty_mac and name != 'lo':
75- continue
76->>>>>>> cloudinit/net/__init__.py
77 if mac in ret:
78 raise RuntimeError(
79 "duplicate mac found! both '%s' and '%s' have mac '%s'" %
80@@ -603,6 +589,7 @@ class EphemeralIPv4Network(object):
81 self._bringup_router()
82
83 def __exit__(self, excp_type, excp_value, excp_traceback):
84+ """Teardown anything we set up."""
85 for cmd in self.cleanup_cmds:
86 util.subp(cmd, capture=True)
87
88diff --git a/cloudinit/net/dhcp.py b/cloudinit/net/dhcp.py
89new file mode 100644
90index 0000000..c7febc5
91--- /dev/null
92+++ b/cloudinit/net/dhcp.py
93@@ -0,0 +1,119 @@
94+# Copyright (C) 2017 Canonical Ltd.
95+#
96+# Author: Chad Smith <chad.smith@canonical.com>
97+#
98+# This file is part of cloud-init. See LICENSE file for license information.
99+
100+import logging
101+import os
102+import re
103+
104+from cloudinit.net import find_fallback_nic, get_devicelist
105+from cloudinit import util
106+
107+LOG = logging.getLogger(__name__)
108+
109+
110+class InvalidDHCPLeaseFileError(Exception):
111+ """Raised when parsing an empty or invalid dhcp.leases file.
112+
113+ Current uses are DataSourceAzure and DataSourceEc2 during ephemeral
114+ boot to scrape metadata.
115+ """
116+ pass
117+
118+
119+def maybe_perform_dhcp_discovery(nic=None):
120+ """Perform dhcp discovery if nic valid and dhclient command exists.
121+
122+ If the nic is invalid or undiscoverable or dhclient command is not found,
123+ skip dhcp_discovery and return an empty dict.
124+
125+ @param nic: Name of the network interface we want to run dhclient on.
126+ @return: A dict of dhcp options from the dhclient discovery if run,
127+ otherwise an empty dict is returned.
128+ """
129+ if nic is None:
130+ nic = find_fallback_nic()
131+ if nic is None:
132+ LOG.debug(
133+ 'Skip dhcp_discovery: Unable to find fallback nic.')
134+ return {}
135+ elif nic not in get_devicelist():
136+ LOG.debug(
137+ 'Skip dhcp_discovery: nic %s not found in get_devicelist.', nic)
138+ return {}
139+ dhclient_path = util.which('dhclient')
140+ if not dhclient_path:
141+ LOG.debug('Skip dhclient configuration: No dhclient command found.')
142+ return {}
143+ with util.tempdir(prefix='cloud-init-dhcp-') as tmpdir:
144+ return dhcp_discovery(dhclient_path, nic, tmpdir)
145+
146+
147+def parse_dhcp_lease_file(lease_file):
148+ """Parse the given dhcp lease file for the most recent lease.
149+
150+ Return a dict of dhcp options as key value pairs for the most recent lease
151+ block.
152+
153+ @raises: InvalidDHCPLeaseFileError on empty of unparseable leasefile
154+ content.
155+ """
156+ lease_regex = re.compile(r"lease {(?P<lease>[^}]*)}\n")
157+ dhcp_leases = []
158+ lease_content = util.load_file(lease_file)
159+ if len(lease_content) == 0:
160+ raise InvalidDHCPLeaseFileError(
161+ 'Cannot parse empty dhcp lease file {0}'.format(lease_file))
162+ for lease in lease_regex.findall(lease_content):
163+ lease_options = []
164+ for line in lease.split(';'):
165+ # Strip newlines, double-quotes and option prefix
166+ line = line.strip().replace('"', '').replace('option ', '')
167+ if not line:
168+ continue
169+ lease_options.append(line.split(' ', 1))
170+ dhcp_leases.append(dict(lease_options))
171+ if not dhcp_leases:
172+ raise InvalidDHCPLeaseFileError(
173+ 'Cannot parse dhcp lease file {0}. No leases found'.format(
174+ lease_file))
175+ return dhcp_leases
176+
177+
178+def dhcp_discovery(dhclient_cmd_path, interface, cleandir):
179+ """Run dhclient on the interface without scripts or filesystem artifacts.
180+
181+ @param dhclient_cmd_path: Full path to the dhclient used.
182+ @param interface: Name of the network inteface on which to dhclient.
183+ @param cleandir: The directory from which to run dhclient as well as store
184+ dhcp leases.
185+
186+ @return: A dict of dhcp options parsed from the dhcp.leases file or empty
187+ dict.
188+ """
189+ LOG.debug('Performing a dhcp discovery on %s', interface)
190+
191+ # XXX We copy dhclient out of /sbin/dhclient to avoid dealing with strict
192+ # app armor profiles which disallow running dhclient -sf <our-script-file>.
193+ # We want to avoid running /sbin/dhclient-script because of side-effects in
194+ # /etc/resolv.conf any any other vendor specific scripts in
195+ # /etc/dhcp/dhclient*hooks.d.
196+ sandbox_dhclient_cmd = os.path.join(cleandir, 'dhclient')
197+ util.copy(dhclient_cmd_path, sandbox_dhclient_cmd)
198+ pid_file = os.path.join(cleandir, 'dhclient.pid')
199+ lease_file = os.path.join(cleandir, 'dhcp.leases')
200+
201+ # ISC dhclient needs the interface up to send initial discovery packets.
202+ # Generally dhclient relies on dhclient-script PREINIT action to bring the
203+ # link up before attempting discovery. Since we are using -sf /bin/true,
204+ # we need to do that "link up" ourselves first.
205+ util.subp(['ip', 'link', 'set', 'dev', interface, 'up'], capture=True)
206+ cmd = [sandbox_dhclient_cmd, '-1', '-v', '-lf', lease_file,
207+ '-pf', pid_file, interface, '-sf', '/bin/true']
208+ util.subp(cmd, capture=True)
209+ return parse_dhcp_lease_file(lease_file)
210+
211+
212+# vi: ts=4 expandtab
213diff --git a/cloudinit/net/tests/test_dhcp.py b/cloudinit/net/tests/test_dhcp.py
214new file mode 100644
215index 0000000..47d8d46
216--- /dev/null
217+++ b/cloudinit/net/tests/test_dhcp.py
218@@ -0,0 +1,144 @@
219+# This file is part of cloud-init. See LICENSE file for license information.
220+
221+import mock
222+import os
223+from textwrap import dedent
224+
225+from cloudinit.net.dhcp import (
226+ InvalidDHCPLeaseFileError, maybe_perform_dhcp_discovery,
227+ parse_dhcp_lease_file, dhcp_discovery)
228+from cloudinit.util import ensure_file, write_file
229+from tests.unittests.helpers import CiTestCase
230+
231+
232+class TestParseDHCPLeasesFile(CiTestCase):
233+
234+ def test_parse_empty_lease_file_errors(self):
235+ """parse_dhcp_lease_file errors when file content is empty."""
236+ empty_file = self.tmp_path('leases')
237+ ensure_file(empty_file)
238+ with self.assertRaises(InvalidDHCPLeaseFileError) as context_manager:
239+ parse_dhcp_lease_file(empty_file)
240+ error = context_manager.exception
241+ self.assertIn('Cannot parse empty dhcp lease file', str(error))
242+
243+ def test_parse_malformed_lease_file_content_errors(self):
244+ """parse_dhcp_lease_file errors when file content isn't dhcp leases."""
245+ non_lease_file = self.tmp_path('leases')
246+ write_file(non_lease_file, 'hi mom.')
247+ with self.assertRaises(InvalidDHCPLeaseFileError) as context_manager:
248+ parse_dhcp_lease_file(non_lease_file)
249+ error = context_manager.exception
250+ self.assertIn('Cannot parse dhcp lease file', str(error))
251+
252+ def test_parse_multiple_leases(self):
253+ """parse_dhcp_lease_file returns a list of all leases within."""
254+ lease_file = self.tmp_path('leases')
255+ content = dedent("""
256+ lease {
257+ interface "wlp3s0";
258+ fixed-address 192.168.2.74;
259+ option subnet-mask 255.255.255.0;
260+ option routers 192.168.2.1;
261+ renew 4 2017/07/27 18:02:30;
262+ expire 5 2017/07/28 07:08:15;
263+ }
264+ lease {
265+ interface "wlp3s0";
266+ fixed-address 192.168.2.74;
267+ option subnet-mask 255.255.255.0;
268+ option routers 192.168.2.1;
269+ }
270+ """)
271+ expected = [
272+ {'interface': 'wlp3s0', 'fixed-address': '192.168.2.74',
273+ 'subnet-mask': '255.255.255.0', 'routers': '192.168.2.1',
274+ 'renew': '4 2017/07/27 18:02:30',
275+ 'expire': '5 2017/07/28 07:08:15'},
276+ {'interface': 'wlp3s0', 'fixed-address': '192.168.2.74',
277+ 'subnet-mask': '255.255.255.0', 'routers': '192.168.2.1'}]
278+ write_file(lease_file, content)
279+ self.assertItemsEqual(expected, parse_dhcp_lease_file(lease_file))
280+
281+
282+class TestDHCPDiscoveryClean(CiTestCase):
283+ with_logs = True
284+
285+ @mock.patch('cloudinit.net.dhcp.find_fallback_nic')
286+ def test_no_fallback_nic_found(self, m_fallback_nic):
287+ """Log and do nothing when nic is absent and no fallback is found."""
288+ m_fallback_nic.return_value = None # No fallback nic found
289+ self.assertEqual({}, maybe_perform_dhcp_discovery())
290+ self.assertIn(
291+ 'Skip dhcp_discovery: Unable to find fallback nic.',
292+ self.logs.getvalue())
293+
294+ def test_provided_nic_does_not_exist(self):
295+ """When the provided nic doesn't exist, log a message and no-op."""
296+ self.assertEqual({}, maybe_perform_dhcp_discovery('idontexist'))
297+ self.assertIn(
298+ 'Skip dhcp_discovery: nic idontexist not found in get_devicelist.',
299+ self.logs.getvalue())
300+
301+ @mock.patch('cloudinit.net.dhcp.util.which')
302+ @mock.patch('cloudinit.net.dhcp.find_fallback_nic')
303+ def test_absent_dhclient_command(self, m_fallback, m_which):
304+ """When dhclient doesn't exist in the OS, log the issue and no-op."""
305+ m_fallback.return_value = 'eth9'
306+ m_which.return_value = None # dhclient isn't found
307+ self.assertEqual({}, maybe_perform_dhcp_discovery())
308+ self.assertIn(
309+ 'Skip dhclient configuration: No dhclient command found.',
310+ self.logs.getvalue())
311+
312+ @mock.patch('cloudinit.net.dhcp.dhcp_discovery')
313+ @mock.patch('cloudinit.net.dhcp.util.which')
314+ @mock.patch('cloudinit.net.dhcp.find_fallback_nic')
315+ def test_dhclient_run_with_tmpdir(self, m_fallback, m_which, m_dhcp):
316+ """maybe_perform_dhcp_discovery passes tmpdir to dhcp_discovery."""
317+ m_fallback.return_value = 'eth9'
318+ m_which.return_value = '/sbin/dhclient'
319+ m_dhcp.return_value = {'address': '192.168.2.2'}
320+ self.assertEqual(
321+ {'address': '192.168.2.2'}, maybe_perform_dhcp_discovery())
322+ m_dhcp.assert_called_once()
323+ call = m_dhcp.call_args_list[0]
324+ self.assertEqual('/sbin/dhclient', call[0][0])
325+ self.assertEqual('eth9', call[0][1])
326+ self.assertIn('/tmp/cloud-init-dhcp-', call[0][2])
327+
328+ @mock.patch('cloudinit.net.dhcp.util.subp')
329+ def test_dhcp_discovery_run_in_sandbox(self, m_subp):
330+ """dhcp_discovery brings up the interface and runs dhclient.
331+
332+ It also returns the parsed dhcp.leases file generated in the sandbox.
333+ """
334+ tmpdir = self.tmp_dir()
335+ dhclient_script = os.path.join(tmpdir, 'dhclient.orig')
336+ script_content = '#!/bin/bash\necho fake-dhclient'
337+ write_file(dhclient_script, script_content, mode=0o755)
338+ lease_content = dedent("""
339+ lease {
340+ interface "eth9";
341+ fixed-address 192.168.2.74;
342+ option subnet-mask 255.255.255.0;
343+ option routers 192.168.2.1;
344+ }
345+ """)
346+ lease_file = os.path.join(tmpdir, 'dhcp.leases')
347+ write_file(lease_file, lease_content)
348+ self.assertItemsEqual(
349+ [{'interface': 'eth9', 'fixed-address': '192.168.2.74',
350+ 'subnet-mask': '255.255.255.0', 'routers': '192.168.2.1'}],
351+ dhcp_discovery(dhclient_script, 'eth9', tmpdir))
352+ # dhclient script got copied
353+ with open(os.path.join(tmpdir, 'dhclient')) as stream:
354+ self.assertEqual(script_content, stream.read())
355+ # Interface was brought up before dhclient called from sandbox
356+ m_subp.assert_has_calls([
357+ mock.call(
358+ ['ip', 'link', 'set', 'dev', 'eth9', 'up'], capture=True),
359+ mock.call(
360+ [os.path.join(tmpdir, 'dhclient'), '-1', '-v', '-lf',
361+ lease_file, '-pf', os.path.join(tmpdir, 'dhclient.pid'),
362+ 'eth9', '-sf', '/bin/true'], capture=True)])
363diff --git a/cloudinit/net/tests/test_init.py b/cloudinit/net/tests/test_init.py
364index 272a6eb..cc052a7 100644
365--- a/cloudinit/net/tests/test_init.py
366+++ b/cloudinit/net/tests/test_init.py
367@@ -414,7 +414,7 @@ class TestEphemeralIPV4Network(CiTestCase):
368 self.assertIn('Cannot init network on', str(error))
369 self.assertEqual(0, m_subp.call_count)
370
371- def test_ephemeral_ipv4_network_errors_invalid_mask(self, m_subp):
372+ def test_ephemeral_ipv4_network_errors_invalid_mask_prefix(self, m_subp):
373 """Raise an error when prefix_or_mask is not a netmask or prefix."""
374 params = {
375 'interface': 'eth0', 'ip': '192.168.2.2',
376diff --git a/cloudinit/sources/DataSourceAliYun.py b/cloudinit/sources/DataSourceAliYun.py
377index 380e27c..43a7e42 100644
378--- a/cloudinit/sources/DataSourceAliYun.py
379+++ b/cloudinit/sources/DataSourceAliYun.py
380@@ -6,17 +6,20 @@ from cloudinit import sources
381 from cloudinit.sources import DataSourceEc2 as EC2
382 from cloudinit import util
383
384-DEF_MD_VERSION = "2016-01-01"
385 ALIYUN_PRODUCT = "Alibaba Cloud ECS"
386
387
388 class DataSourceAliYun(EC2.DataSourceEc2):
389- metadata_urls = ["http://100.100.100.200"]
390+
391+ metadata_urls = ['http://100.100.100.200']
392+
393+ # The minimum supported metadata_version from the ec2 metadata apis
394+ min_metadata_version = '2016-01-01'
395+ extended_metadata_versions = []
396
397 def __init__(self, sys_cfg, distro, paths):
398 super(DataSourceAliYun, self).__init__(sys_cfg, distro, paths)
399 self.seed_dir = os.path.join(paths.seed_dir, "AliYun")
400- self.api_ver = DEF_MD_VERSION
401
402 def get_hostname(self, fqdn=False, _resolve_ip=False):
403 return self.metadata.get('hostname', 'localhost.localdomain')
404diff --git a/cloudinit/sources/DataSourceEc2.py b/cloudinit/sources/DataSourceEc2.py
405index 4ec9592..8e5f8ee 100644
406--- a/cloudinit/sources/DataSourceEc2.py
407+++ b/cloudinit/sources/DataSourceEc2.py
408@@ -13,6 +13,8 @@ import time
409
410 from cloudinit import ec2_utils as ec2
411 from cloudinit import log as logging
412+from cloudinit import net
413+from cloudinit.net import dhcp
414 from cloudinit import sources
415 from cloudinit import url_helper as uhelp
416 from cloudinit import util
417@@ -20,8 +22,7 @@ from cloudinit import warnings
418
419 LOG = logging.getLogger(__name__)
420
421-# Which version we are requesting of the ec2 metadata apis
422-DEF_MD_VERSION = '2009-04-04'
423+SKIP_METADATA_URL_CODES = frozenset([uhelp.NOT_FOUND])
424
425 STRICT_ID_PATH = ("datasource", "Ec2", "strict_id")
426 STRICT_ID_DEFAULT = "warn"
427@@ -41,17 +42,28 @@ class Platforms(object):
428
429
430 class DataSourceEc2(sources.DataSource):
431+
432 # Default metadata urls that will be used if none are provided
433 # They will be checked for 'resolveability' and some of the
434 # following may be discarded if they do not resolve
435 metadata_urls = ["http://169.254.169.254", "http://instance-data.:8773"]
436+
437+ # The minimum supported metadata_version from the ec2 metadata apis
438+ min_metadata_version = '2009-04-04'
439+
440+ # Priority ordered list of additional metadata versions which will be tried
441+ # for extended metadata content. IPv6 support comes in 2016-09-02
442+ extended_metadata_versions = ['2016-09-02']
443+
444 _cloud_platform = None
445
446+ # Whether we want to get network configuration from the metadata service.
447+ get_network_metadata = False
448+
449 def __init__(self, sys_cfg, distro, paths):
450 sources.DataSource.__init__(self, sys_cfg, distro, paths)
451 self.metadata_address = None
452 self.seed_dir = os.path.join(paths.seed_dir, "ec2")
453- self.api_ver = DEF_MD_VERSION
454
455 def get_data(self):
456 seed_ret = {}
457@@ -73,21 +85,27 @@ class DataSourceEc2(sources.DataSource):
458 elif self.cloud_platform == Platforms.NO_EC2_METADATA:
459 return False
460
461- try:
462- if not self.wait_for_metadata_service():
463+ if self.get_network_metadata: # Setup networking in init-local stage.
464+ if util.is_FreeBSD():
465+ LOG.debug("FreeBSD doesn't support running dhclient with -sf")
466 return False
467- start_time = time.time()
468- self.userdata_raw = \
469- ec2.get_instance_userdata(self.api_ver, self.metadata_address)
470- self.metadata = ec2.get_instance_metadata(self.api_ver,
471- self.metadata_address)
472- LOG.debug("Crawl of metadata service took %.3f seconds",
473- time.time() - start_time)
474- return True
475- except Exception:
476- util.logexc(LOG, "Failed reading from metadata address %s",
477- self.metadata_address)
478- return False
479+ dhcp_leases = dhcp.maybe_perform_dhcp_discovery()
480+ if not dhcp_leases:
481+ # DataSourceEc2Local failed in init-local stage. DataSourceEc2
482+ # will still run in init-network stage.
483+ return False
484+ dhcp_opts = dhcp_leases[-1]
485+ net_params = {'interface': dhcp_opts.get('interface'),
486+ 'ip': dhcp_opts.get('fixed-address'),
487+ 'prefix_or_mask': dhcp_opts.get('subnet-mask'),
488+ 'broadcast': dhcp_opts.get('broadcast-address'),
489+ 'router': dhcp_opts.get('routers')}
490+ with net.EphemeralIPv4Network(**net_params):
491+ return util.log_time(
492+ logfunc=LOG.debug, msg='Crawl of metadata service',
493+ func=self._crawl_metadata)
494+ else:
495+ return self._crawl_metadata()
496
497 @property
498 def launch_index(self):
499@@ -95,6 +113,32 @@ class DataSourceEc2(sources.DataSource):
500 return None
501 return self.metadata.get('ami-launch-index')
502
503+ def get_metadata_api_version(self):
504+ """Get the best supported api version from the metadata service.
505+
506+ Loop through all extended support metadata versions in order and
507+ return the most-fully featured metadata api version discovered.
508+
509+ If extended_metadata_versions aren't present, return the datasource's
510+ min_metadata_version.
511+ """
512+ # Assumes metadata service is already up
513+ for api_ver in self.extended_metadata_versions:
514+ url = '{0}/{1}/meta-data/instance-id'.format(
515+ self.metadata_address, api_ver)
516+ try:
517+ resp = uhelp.readurl(url=url)
518+ except uhelp.UrlError as e:
519+ LOG.debug('url %s raised exception %s', url, e)
520+ else:
521+ if resp.code == 200:
522+ LOG.debug('Found preferred metadata version %s', api_ver)
523+ return api_ver
524+ elif resp.code == 404:
525+ msg = 'Metadata api version %s not present. Headers: %s'
526+ LOG.debug(msg, api_ver, resp.headers)
527+ return self.min_metadata_version
528+
529 def get_instance_id(self):
530 return self.metadata['instance-id']
531
532@@ -138,21 +182,22 @@ class DataSourceEc2(sources.DataSource):
533 urls = []
534 url2base = {}
535 for url in mdurls:
536- cur = "%s/%s/meta-data/instance-id" % (url, self.api_ver)
537+ cur = '{0}/{1}/meta-data/instance-id'.format(
538+ url, self.min_metadata_version)
539 urls.append(cur)
540 url2base[cur] = url
541
542 start_time = time.time()
543- url = uhelp.wait_for_url(urls=urls, max_wait=max_wait,
544- timeout=timeout, status_cb=LOG.warn)
545+ url = uhelp.wait_for_url(
546+ urls=urls, max_wait=max_wait, timeout=timeout, status_cb=LOG.warn)
547
548 if url:
549- LOG.debug("Using metadata source: '%s'", url2base[url])
550+ self.metadata_address = url2base[url]
551+ LOG.debug("Using metadata source: '%s'", self.metadata_address)
552 else:
553 LOG.critical("Giving up on md from %s after %s seconds",
554 urls, int(time.time() - start_time))
555
556- self.metadata_address = url2base.get(url)
557 return bool(url)
558
559 def device_name_to_device(self, name):
560@@ -234,6 +279,37 @@ class DataSourceEc2(sources.DataSource):
561 util.get_cfg_by_path(cfg, STRICT_ID_PATH, STRICT_ID_DEFAULT),
562 cfg)
563
564+ def _crawl_metadata(self):
565+ """Crawl metadata service when available.
566+
567+ @returns: True on success, False otherwise.
568+ """
569+ if not self.wait_for_metadata_service():
570+ return False
571+ api_version = self.get_metadata_api_version()
572+ try:
573+ self.userdata_raw = ec2.get_instance_userdata(
574+ api_version, self.metadata_address)
575+ self.metadata = ec2.get_instance_metadata(
576+ api_version, self.metadata_address)
577+ except Exception:
578+ util.logexc(
579+ LOG, "Failed reading from metadata address %s",
580+ self.metadata_address)
581+ return False
582+ return True
583+
584+
585+class DataSourceEc2Local(DataSourceEc2):
586+ """Datasource run at init-local which sets up network to query metadata.
587+
588+ In init-local, no network is available. This subclass sets up minimal
589+ networking with dhclient on a viable nic so that it can talk to the
590+ metadata service. If the metadata service provides network configuration
591+ then render the network configuration for that instance based on metadata.
592+ """
593+ get_network_metadata = True # Get metadata network config if present
594+
595
596 def read_strict_mode(cfgval, default):
597 try:
598@@ -349,6 +425,7 @@ def _collect_platform_data():
599
600 # Used to match classes to dependencies
601 datasources = [
602+ (DataSourceEc2Local, (sources.DEP_FILESYSTEM,)), # Run at init-local
603 (DataSourceEc2, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
604 ]
605
606diff --git a/tests/unittests/helpers.py b/tests/unittests/helpers.py
607index 08c5c46..bf1dc5d 100644
608--- a/tests/unittests/helpers.py
609+++ b/tests/unittests/helpers.py
610@@ -278,7 +278,7 @@ class FilesystemMockingTestCase(ResourceUsingTestCase):
611 return root
612
613
614-class HttprettyTestCase(TestCase):
615+class HttprettyTestCase(CiTestCase):
616 # necessary as http_proxy gets in the way of httpretty
617 # https://github.com/gabrielfalcao/HTTPretty/issues/122
618 def setUp(self):
619diff --git a/tests/unittests/test_datasource/test_aliyun.py b/tests/unittests/test_datasource/test_aliyun.py
620index 990bff2..996560e 100644
621--- a/tests/unittests/test_datasource/test_aliyun.py
622+++ b/tests/unittests/test_datasource/test_aliyun.py
623@@ -70,7 +70,6 @@ class TestAliYunDatasource(test_helpers.HttprettyTestCase):
624 paths = helpers.Paths({})
625 self.ds = ay.DataSourceAliYun(cfg, distro, paths)
626 self.metadata_address = self.ds.metadata_urls[0]
627- self.api_ver = self.ds.api_ver
628
629 @property
630 def default_metadata(self):
631@@ -82,13 +81,15 @@ class TestAliYunDatasource(test_helpers.HttprettyTestCase):
632
633 @property
634 def metadata_url(self):
635- return os.path.join(self.metadata_address,
636- self.api_ver, 'meta-data') + '/'
637+ return os.path.join(
638+ self.metadata_address,
639+ self.ds.min_metadata_version, 'meta-data') + '/'
640
641 @property
642 def userdata_url(self):
643- return os.path.join(self.metadata_address,
644- self.api_ver, 'user-data')
645+ return os.path.join(
646+ self.metadata_address,
647+ self.ds.min_metadata_version, 'user-data')
648
649 def regist_default_server(self):
650 register_mock_metaserver(self.metadata_url, self.default_metadata)
651diff --git a/tests/unittests/test_datasource/test_common.py b/tests/unittests/test_datasource/test_common.py
652index 413e87a..4802f10 100644
653--- a/tests/unittests/test_datasource/test_common.py
654+++ b/tests/unittests/test_datasource/test_common.py
655@@ -35,6 +35,7 @@ DEFAULT_LOCAL = [
656 OpenNebula.DataSourceOpenNebula,
657 OVF.DataSourceOVF,
658 SmartOS.DataSourceSmartOS,
659+ Ec2.DataSourceEc2Local,
660 ]
661
662 DEFAULT_NETWORK = [
663diff --git a/tests/unittests/test_datasource/test_ec2.py b/tests/unittests/test_datasource/test_ec2.py
664index 12230ae..33d0261 100644
665--- a/tests/unittests/test_datasource/test_ec2.py
666+++ b/tests/unittests/test_datasource/test_ec2.py
667@@ -8,35 +8,67 @@ from cloudinit import helpers
668 from cloudinit.sources import DataSourceEc2 as ec2
669
670
671-# collected from api version 2009-04-04/ with
672+# collected from api version 2016-09-02/ with
673 # python3 -c 'import json
674 # from cloudinit.ec2_utils import get_instance_metadata as gm
675-# print(json.dumps(gm("2009-04-04"), indent=1, sort_keys=True))'
676+# print(json.dumps(gm("2016-09-02"), indent=1, sort_keys=True))'
677 DEFAULT_METADATA = {
678- "ami-id": "ami-80861296",
679+ "ami-id": "ami-8b92b4ee",
680 "ami-launch-index": "0",
681 "ami-manifest-path": "(unknown)",
682 "block-device-mapping": {"ami": "/dev/sda1", "root": "/dev/sda1"},
683- "hostname": "ip-10-0-0-149",
684+ "hostname": "ip-172-31-31-158.us-east-2.compute.internal",
685 "instance-action": "none",
686- "instance-id": "i-0052913950685138c",
687- "instance-type": "t2.micro",
688- "local-hostname": "ip-10-0-0-149",
689- "local-ipv4": "10.0.0.149",
690- "placement": {"availability-zone": "us-east-1b"},
691+ "instance-id": "i-0a33f80f09c96477f",
692+ "instance-type": "t2.small",
693+ "local-hostname": "ip-172-3-3-15.us-east-2.compute.internal",
694+ "local-ipv4": "172.3.3.15",
695+ "mac": "06:17:04:d7:26:09",
696+ "metrics": {"vhostmd": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>"},
697+ "network": {
698+ "interfaces": {
699+ "macs": {
700+ "06:17:04:d7:26:09": {
701+ "device-number": "0",
702+ "interface-id": "eni-e44ef49e",
703+ "ipv4-associations": {"13.59.77.202": "172.3.3.15"},
704+ "ipv6s": "2600:1f16:aeb:b20b:9d87:a4af:5cc9:73dc",
705+ "local-hostname": ("ip-172-3-3-15.us-east-2."
706+ "compute.internal"),
707+ "local-ipv4s": "172.3.3.15",
708+ "mac": "06:17:04:d7:26:09",
709+ "owner-id": "950047163771",
710+ "public-hostname": ("ec2-13-59-77-202.us-east-2."
711+ "compute.amazonaws.com"),
712+ "public-ipv4s": "13.59.77.202",
713+ "security-group-ids": "sg-5a61d333",
714+ "security-groups": "wide-open",
715+ "subnet-id": "subnet-20b8565b",
716+ "subnet-ipv4-cidr-block": "172.31.16.0/20",
717+ "subnet-ipv6-cidr-blocks": "2600:1f16:aeb:b20b::/64",
718+ "vpc-id": "vpc-87e72bee",
719+ "vpc-ipv4-cidr-block": "172.31.0.0/16",
720+ "vpc-ipv4-cidr-blocks": "172.31.0.0/16",
721+ "vpc-ipv6-cidr-blocks": "2600:1f16:aeb:b200::/56"
722+ }
723+ }
724+ }
725+ },
726+ "placement": {"availability-zone": "us-east-2b"},
727 "profile": "default-hvm",
728- "public-hostname": "",
729- "public-ipv4": "107.23.188.247",
730+ "public-hostname": "ec2-13-59-77-202.us-east-2.compute.amazonaws.com",
731+ "public-ipv4": "13.59.77.202",
732 "public-keys": {"brickies": ["ssh-rsa AAAAB3Nz....w== brickies"]},
733- "reservation-id": "r-00a2c173fb5782a08",
734- "security-groups": "wide-open"
735+ "reservation-id": "r-01efbc9996bac1bd6",
736+ "security-groups": "my-wide-open",
737+ "services": {"domain": "amazonaws.com", "partition": "aws"}
738 }
739
740
741 def _register_ssh_keys(rfunc, base_url, keys_data):
742 """handle ssh key inconsistencies.
743
744- public-keys in the ec2 metadata is inconsistently formatted compared
745+ public-keys in the ec2 metadata is inconsistently formated compared
746 to other entries.
747 Given keys_data of {name1: pubkey1, name2: pubkey2}
748
749@@ -115,6 +147,8 @@ def register_mock_metaserver(base_url, data):
750
751
752 class TestEc2(test_helpers.HttprettyTestCase):
753+ with_logs = True
754+
755 valid_platform_data = {
756 'uuid': 'ec212f79-87d1-2f1d-588f-d86dc0fd5412',
757 'uuid_source': 'dmi',
758@@ -123,16 +157,20 @@ class TestEc2(test_helpers.HttprettyTestCase):
759
760 def setUp(self):
761 super(TestEc2, self).setUp()
762- self.metadata_addr = ec2.DataSourceEc2.metadata_urls[0]
763- self.api_ver = '2009-04-04'
764+ self.datasource = ec2.DataSourceEc2
765+ self.metadata_addr = self.datasource.metadata_urls[0]
766
767 @property
768 def metadata_url(self):
769- return '/'.join([self.metadata_addr, self.api_ver, 'meta-data', ''])
770+ return '/'.join([
771+ self.metadata_addr,
772+ self.datasource.min_metadata_version, 'meta-data', ''])
773
774 @property
775 def userdata_url(self):
776- return '/'.join([self.metadata_addr, self.api_ver, 'user-data'])
777+ return '/'.join([
778+ self.metadata_addr,
779+ self.datasource.min_metadata_version, 'user-data'])
780
781 def _patch_add_cleanup(self, mpath, *args, **kwargs):
782 p = mock.patch(mpath, *args, **kwargs)
783@@ -144,7 +182,7 @@ class TestEc2(test_helpers.HttprettyTestCase):
784 paths = helpers.Paths({})
785 if sys_cfg is None:
786 sys_cfg = {}
787- ds = ec2.DataSourceEc2(sys_cfg=sys_cfg, distro=distro, paths=paths)
788+ ds = self.datasource(sys_cfg=sys_cfg, distro=distro, paths=paths)
789 if platform_data is not None:
790 self._patch_add_cleanup(
791 "cloudinit.sources.DataSourceEc2._collect_platform_data",
792@@ -157,14 +195,16 @@ class TestEc2(test_helpers.HttprettyTestCase):
793 return ds
794
795 @httpretty.activate
796- def test_valid_platform_with_strict_true(self):
797+ @mock.patch('cloudinit.net.dhcp.maybe_perform_dhcp_discovery')
798+ def test_valid_platform_with_strict_true(self, m_dhcp):
799 """Valid platform data should return true with strict_id true."""
800 ds = self._setup_ds(
801 platform_data=self.valid_platform_data,
802 sys_cfg={'datasource': {'Ec2': {'strict_id': True}}},
803 md=DEFAULT_METADATA)
804 ret = ds.get_data()
805- self.assertEqual(True, ret)
806+ self.assertTrue(ret)
807+ self.assertEqual(0, m_dhcp.call_count)
808
809 @httpretty.activate
810 def test_valid_platform_with_strict_false(self):
811@@ -174,7 +214,7 @@ class TestEc2(test_helpers.HttprettyTestCase):
812 sys_cfg={'datasource': {'Ec2': {'strict_id': False}}},
813 md=DEFAULT_METADATA)
814 ret = ds.get_data()
815- self.assertEqual(True, ret)
816+ self.assertTrue(ret)
817
818 @httpretty.activate
819 def test_unknown_platform_with_strict_true(self):
820@@ -185,7 +225,7 @@ class TestEc2(test_helpers.HttprettyTestCase):
821 sys_cfg={'datasource': {'Ec2': {'strict_id': True}}},
822 md=DEFAULT_METADATA)
823 ret = ds.get_data()
824- self.assertEqual(False, ret)
825+ self.assertFalse(ret)
826
827 @httpretty.activate
828 def test_unknown_platform_with_strict_false(self):
829@@ -196,7 +236,55 @@ class TestEc2(test_helpers.HttprettyTestCase):
830 sys_cfg={'datasource': {'Ec2': {'strict_id': False}}},
831 md=DEFAULT_METADATA)
832 ret = ds.get_data()
833- self.assertEqual(True, ret)
834+ self.assertTrue(ret)
835+
836+ @httpretty.activate
837+ @mock.patch('cloudinit.sources.DataSourceEc2.util.is_FreeBSD')
838+ def test_ec2_local_returns_false_on_bsd(self, m_is_freebsd):
839+ """DataSourceEc2Local returns False on BSD.
840+
841+ FreeBSD dhclient doesn't support dhclient -sf to run in a sandbox.
842+ """
843+ m_is_freebsd.return_value = True
844+ self.datasource = ec2.DataSourceEc2Local
845+ ds = self._setup_ds(
846+ platform_data=self.valid_platform_data,
847+ sys_cfg={'datasource': {'Ec2': {'strict_id': False}}},
848+ md=DEFAULT_METADATA)
849+ ret = ds.get_data()
850+ self.assertFalse(ret)
851+ self.assertIn(
852+ "FreeBSD doesn't support running dhclient with -sf",
853+ self.logs.getvalue())
854+
855+ @httpretty.activate
856+ @mock.patch('cloudinit.net.EphemeralIPv4Network')
857+ @mock.patch('cloudinit.net.dhcp.maybe_perform_dhcp_discovery')
858+ @mock.patch('cloudinit.sources.DataSourceEc2.util.is_FreeBSD')
859+ def test_ec2_local_performs_dhcp_on_non_bsd(self, m_is_bsd, m_dhcp, m_net):
860+ """Ec2Local returns True for valid platform data on non-BSD with dhcp.
861+
862+ DataSourceEc2Local will setup initial IPv4 network via dhcp discovery.
863+ Then the metadata services is crawled for more network config info.
864+ When the platform data is valid, return True.
865+ """
866+ m_is_bsd.return_value = False
867+ m_dhcp.return_value = [{
868+ 'interface': 'eth9', 'fixed-address': '192.168.2.9',
869+ 'routers': '192.168.2.1', 'subnet-mask': '255.255.255.0',
870+ 'broadcast-address': '192.168.2.255'}]
871+ self.datasource = ec2.DataSourceEc2Local
872+ ds = self._setup_ds(
873+ platform_data=self.valid_platform_data,
874+ sys_cfg={'datasource': {'Ec2': {'strict_id': False}}},
875+ md=DEFAULT_METADATA)
876+ ret = ds.get_data()
877+ self.assertTrue(ret)
878+ m_dhcp.assert_called_once_with()
879+ m_net.assert_called_once_with(
880+ broadcast='192.168.2.255', interface='eth9', ip='192.168.2.9',
881+ prefix_or_mask='255.255.255.0', router='192.168.2.1')
882+ self.assertIn('Crawl of metadata service took', self.logs.getvalue())
883
884
885 # vi: ts=4 expandtab
886diff --git a/tox.ini b/tox.ini
887index ef76884..1e7ca2d 100644
888--- a/tox.ini
889+++ b/tox.ini
890@@ -21,10 +21,10 @@ setenv =
891 LC_ALL = en_US.utf-8
892
893 [testenv:pylint]
894-deps =
895+deps =
896 # requirements
897 pylint==1.7.1
898- # test-requirements because unit tests are now present in cloudinit tree
899+ # test-requirements because unit tests are now present in cloudinit tree
900 -r{toxinidir}/test-requirements.txt
901 commands = {envpython} -m pylint {posargs:cloudinit}
902

Subscribers

People subscribed via source and target branches