Merge ~ma-brothier/cloud-init:interfaces-order into cloud-init:master

[Marc-Aurele Brothier]

Natural sort order of interfaces

The code deciding which interface to choose as the default to request the IP address through DHCP does not sort the interfaces correctly. On Ubuntu Xenial images for example, the interfaces are named ens1, ens2, ens3..., ens11, ... depending on the pci bus address. The python sorting will list 'ens11' before 'ens3' for example despite the fact that 'ens3' should be before 'ens11'.

This patch address this issue and sort the interface names according to a human sorting.

[Scott Moser (smoser)]

Marc,

This looks good. Some comments... they're all minor, and should be explained more in HACKING.rst
http://cloudinit.readthedocs.io/en/latest/topics/hacking.html

a.) change description to be
    Summary
    <blank line>
    Description

b.) lets go ahead and just keep the function in a net/__init__.py rather than putting it in util. And name it _natural_sort_key(). At some point if someone else needs such a function we can move it to util.

c.) a unit test would be good. Just add something to tests/unittests/test_net.py to test your _natural_key_sort. Also realize you can (and do) get interfaces named 'enp0s25' (as I have on this system).

d.) You'll need to sign the contributors agreement (see HACKING.rst). Please feel free to ping me if you have any questions.

e.) Thanks!

d.) Thanks!

[Scott Moser (smoser)]

review: Needs Fixing

[Marc-Aurele Brothier (ma-brothier)]

Thanks Scott for the comments.

a) done

b) change done

c) the tests don't run on my OSX, they got stuck in test_datasource/test_azure.py", line 333, in test_password_given. Can you give me a hint how to run only those unittests?

d) done

Dunno what's e) and f) but glad for the "thanks".
I must say the user experience on launchpad is quite an adventure to figure out things.

About the test: I created a virtualenv with py2.7 and installed the requirements. So far if I run this command I got this error:
python -m nose test_net.py
ERROR: Failure: ImportError (cannot import name renderers)

0f705cc... by Marc-Aurele Brothier on 2017-05-26

Add simple test for natural sorting

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:1ec863c13092ab1791e4f5e03e0c9a42cb1dc9bb
https://jenkins.ubuntu.com/server/job/cloud-init-ci/412/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/412/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/412/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/412/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/412/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/412/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/412/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:0f705cce22f6028431b8593171369f98a98497c5
https://jenkins.ubuntu.com/server/job/cloud-init-ci/414/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/414
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/414
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/414
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/414
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/414

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/414/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

Marc,
to run the unit tests, you need to be in the top level directory (so that it can "import cloudinit").

so:
python -m nose tests/unittests/test_net.py

[Scott Moser (smoser)]

thanks.

review: Approve

diff --git a/cloudinit/net/__init__.py b/cloudinit/net/__init__.py
index 8c6cd05..65accbb 100644
--- a/cloudinit/net/__init__.py
+++ b/cloudinit/net/__init__.py
@@ -17,6 +17,17 @@ SYS_CLASS_NET = "/sys/class/net/"
 DEFAULT_PRIMARY_INTERFACE = 'eth0'
 
 
+def _natural_sort_key(s, _nsre=re.compile('([0-9]+)')):
+    """Sorting for Humans: natural sort order. Can be use as the key to sort
+    functions.
+    This will sort ['eth0', 'ens3', 'ens10', 'ens12', 'ens8', 'ens0'] as
+    ['ens0', 'ens3', 'ens8', 'ens10', 'ens12', 'eth0'] instead of the simple
+    python way which will produce ['ens0', 'ens10', 'ens12', 'ens3', 'ens8',
+    'eth0']."""
+    return [int(text) if text.isdigit() else text.lower()
+            for text in re.split(_nsre, s)]
+
+
 def sys_dev_path(devname, path=""):
     return SYS_CLASS_NET + devname + "/" + path
 
@@ -169,7 +180,7 @@ def generate_fallback_config():
 
     # if eth0 exists use it above anything else, otherwise get the interface
     # that we can read 'first' (using the sorted defintion of first).
-    names = list(sorted(potential_interfaces))
+    names = list(sorted(potential_interfaces, key=_natural_sort_key))
     if DEFAULT_PRIMARY_INTERFACE in names:
         names.remove(DEFAULT_PRIMARY_INTERFACE)
         names.insert(0, DEFAULT_PRIMARY_INTERFACE)
diff --git a/tests/unittests/test_net.py b/tests/unittests/test_net.py
index 5169821..9993241 100644
--- a/tests/unittests/test_net.py
+++ b/tests/unittests/test_net.py
@@ -1,6 +1,7 @@
 # This file is part of cloud-init. See LICENSE file for license information.
 
 from cloudinit import net
+from cloudinit.net import _natural_sort_key
 from cloudinit.net import cmdline
 from cloudinit.net import eni
 from cloudinit.net import netplan
@@ -1627,6 +1628,19 @@ class TestGetInterfacesByMac(CiTestCase):
         self.assertEqual('lo', ret[empty_mac])
 
 
+class TestInterfacesSorting(CiTestCase):
+
+    def test_natural_order(self):
+        data = ['ens5', 'ens6', 'ens3', 'ens20', 'ens13', 'ens2']
+        self.assertEqual(
+            sorted(data, key=_natural_sort_key),
+            ['ens2', 'ens3', 'ens5', 'ens6', 'ens13', 'ens20'])
+        data2 = ['enp2s0', 'enp2s3', 'enp0s3', 'enp0s13', 'enp0s8', 'enp1s2']
+        self.assertEqual(
+            sorted(data2, key=_natural_sort_key),
+            ['enp0s3', 'enp0s8', 'enp0s13', 'enp1s2', 'enp2s0', 'enp2s3'])
+
+
 def _gzip_data(data):
     with io.BytesIO() as iobuf:
         gzfp = gzip.GzipFile(mode="wb", fileobj=iobuf)