Merge ~akaris/cloud-init:bug1679817-c into cloud-init:master

[Andreas Karis]

Fix dual stack IPv4/IPv6 configuration for RHEL

Dual stack IPv4/IPv6 configuration via config drive is broken for RHEL7.
This patch fixes several scenarios for IPv4/IPv6/dual stack with multiple IP assignment
Removes unpopular IPv4 alias files and invalid IPv6 alias files

Also fixes associated unit tests

LP: #1679817
LP: #1685534
LP: #1685532

[Andreas Karis (akaris)]

Fix dual stack IPv4/IPv6 configuration for RHEL

Dual stack IPv4/IPv6 configuration via config drive is broken for RHEL7.
This patch fixes several scenarios for IPv4/IPv6/dual stack with multiple IP assignment
Removes unpopular IPv4 alias files and invalid IPv6 alias files

Also fixes associated unit tests

LP: #1679817
LP: #1685534
LP: #1685532

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:1a401c57978855b5d56ac23976252e2506d206fa
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~akaris/cloud-init/+git/cloud-init/+merge/324195/+edit-commit-message

https://jenkins.ubuntu.com/server/job/cloud-init-ci/345/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/345
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/345
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/345
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/345

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/345/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:1a401c57978855b5d56ac23976252e2506d206fa
https://jenkins.ubuntu.com/server/job/cloud-init-ci/346/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/346
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/346
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/346
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/346

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/346/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

Hi,
I assume this will still work for Centos 6 (5?)

It looks very good, thanks for your work.

Will this work correctly for centos 5 and 6 ?

review: Needs Information

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:48bf21bcb6046f1a2d1d029d286dcbe9eaf4f62e
https://jenkins.ubuntu.com/server/job/cloud-init-ci/348/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/348
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/348
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/348
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/348

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/348/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

Assuming this works with centos 5, 6, 7, I'm happy to pull this.

[Andreas Karis (akaris)]

Hi,

Let me test all of this out once more and give a final ack.

- Andreas

[Andreas Karis (akaris)]

Test procedure for RHEL 7:

### Enable config drive IPv4/IPv6 address injection ###
On the compute nodes and controllers, configure
~~~
crudini --set /etc/nova/nova.conf DEFAULT injected_network_template /usr/lib/python2.7/site-packages/nova/virt/interfaces.template
crudini --set /etc/nova/nova.conf DEFAULT flat_injected true
crudini --set /etc/nova/nova.conf DEFAULT force_config_drive true
crudini --set /etc/nova/nova.conf DEFAULT config_drive_cdrom True
crudini --set /etc/nova/nova.conf DEFAULT debug true
crudini --set /etc/nova/nova.conf DEFAULT use_ipv6 true
crudini --set /etc/nova/nova.conf os_vif_linux_bridge use_ipv6 true
crudini --set /etc/nova/nova.conf libvirt inject_partition -1
~~~

Restart all OpenStack services on computes and controllers:
~~~
systemctl list-units | grep nova | awk '{print $1}' | xargs -I {} systemctl restart {}
~~~

### Use a recent version of cloud-init within the instances ###
~~~
sudo yumdownloader cloud-init pyserial python-jinja2 python-babel python-markupsafe pytz
for i in *.rpm;do virt-customize -a rhel.qcow2 --upload $i:/root/$i ; done
virt-customize -a rhel.qcow2 -v --run-command 'yum -y localinstall /root/*.rpm'
source overcloudrc
~~~

Set password for console login
~~~
virt-customize -a rhel.qcow2 --root-password password:Redhat01
~~~

Create glance image
~~~
glance image-create --name rhel-cloud-init --file rhel.qcow2 --container-format bare --disk-format qcow2 --progress
~~~

### Open all security groups and add keypair ###
~~~
nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
nova secgroup-add-rule default tcp 1 65535 0.0.0.0/0
nova secgroup-add-rule default udp 1 65535 0.0.0.0/0
nova secgroup-add-rule default icmp -1 -1 ::/0
nova secgroup-add-rule default tcp 1 65535 ::/0
nova secgroup-add-rule default udp 1 65535 ::/0
~~~

~~~
nova keypair-add --pub-key ~/.ssh/id_rsa.pub id_rsa
~~~

### Configure networks without DHCP ###
Make sure that none of the subnets has DHCP enabled!
~~~
# access network
neutron net-create provider1 --provider:network_type vlan --provider:physical_network $PROVIDER_PHYSICAL_NETWORK --provider:segmentation_id $PROVIDER_SEGMENTATION_ID --shared --router:external
neutron subnet-create --gateway 10.0.0.1 --allocation-pool start=10.0.0.100,end=10.0.0.150 --dns-nameserver 8.8.8.8 --name provider1-subnet provider1 10.0.0.0/24
neutron subnet-update provider1-subnet --disable-dhcp

# test networks
neutron net-create private-no-dhcp-1
neutron net-delete private-no-dhcp-1
neutron net-create private-no-dhcp-1
neutron net-create private-no-dhcp-2
neutron net-create private-no-dhcp-3
neutron subnet-create --disable-dhcp private-no-dhcp-1 192.168.100.0/24
neutron subnet-create --disable-dhcp private-no-dhcp-1 192.168.101.0/24
neutron subnet-create --disable-dhcp private-no-dhcp-1 192.168.102.0/24
neutron subnet-create --disable-dhcp private-no-dhcp-2 192.168.200.0/24
neutron subnet-create --disable-dhcp --ip-version 6 private-no-dhcp-2 2000:192:168:200::/64
neutron subnet-create --disable-dhcp --ip-version 6 private-no-dhcp-2 2000:192:168:201::/64
neutron subnet-create --disable-dhcp --ip-version 6 --gateway 2000:192:168:202::1 private-no-dhcp-3 2000:192:168:202::/64
neutr...

[Andreas Karis (akaris)]

One issue that I have with this still is the dup default route generation for IPv6:

[root@rhel-cloud-init-patch-test network-scripts]# ip r
default via 192.168.200.1 dev eth2
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.106
169.254.0.0/16 dev eth0 scope link metric 1002
169.254.0.0/16 dev eth1 scope link metric 1003
169.254.0.0/16 dev eth2 scope link metric 1004
169.254.0.0/16 dev eth3 scope link metric 1005
192.168.101.0/24 dev eth1 proto kernel scope link src 192.168.101.7
192.168.200.0/24 dev eth2 proto kernel scope link src 192.168.200.2
[root@rhel-cloud-init-patch-test network-scripts]# ip -6 r
unreachable ::/96 dev lo metric 1024 error -113
unreachable ::ffff:0.0.0.0/96 dev lo metric 1024 error -113
2000:192:168:201::/64 dev eth2 proto kernel metric 256
2000:192:168:204::/64 dev eth3 proto kernel metric 256
unreachable 2002:a00::/24 dev lo metric 1024 error -113
unreachable 2002:7f00::/24 dev lo metric 1024 error -113
unreachable 2002:a9fe::/32 dev lo metric 1024 error -113
unreachable 2002:ac10::/28 dev lo metric 1024 error -113
unreachable 2002:c0a8::/32 dev lo metric 1024 error -113
unreachable 2002:e000::/19 dev lo metric 1024 error -113
unreachable 3ffe:ffff::/32 dev lo metric 1024 error -113
fe80::/64 dev eth0 proto kernel metric 256
fe80::/64 dev eth1 proto kernel metric 256
fe80::/64 dev eth2 proto kernel metric 256
fe80::/64 dev eth3 proto kernel metric 256
default via 2000:192:168:201::1 dev eth2 metric 1
default via 2000:192:168:201::1 dev eth2 metric 1024

This could be easy to work around. However, it was never taken into account before that default routes should not be added in the route- and route6- files before, so I'm wondering if this should go into a different fix instead of overloading this one.

[Scott Moser (smoser)]

Akaris, I'm fine to have another subsequent fix to pick up your issue in the last comment.
I'm interested in knowing though if this works for rhel 5 and rhel 6.
I dont want to regress those platforms.

Were you able to verify that ?
Scott

[Andreas Karis (akaris)]

RHEL 6:
~~~
# make sure to have rhel-guest-image-6.10-23.x86_64.qcow2
# make sure to have python-crypto-2.6.1-1.el6ost.x86_64.rpm
# make sure to have python-oauthlib-0.6.0-3.el6ost.noarch.rpm
virt-customize -a rhel-guest-image-6.10-23.x86_64.qcow2 --root-password password:Redhat01
for i in *el6ost*.rpm;do virt-customize -a rhel6-cloud-init.patch.qcow2 --upload $i:/root/$i ; done
virt-customize -a rhel6-cloud-init.patch.qcow2 -v --run-command 'yum -y localinstall /root/*.rpm'
for i in cloudinit.0.7.9.patch.tar.gz ; do virt-customize -a rhel6-cloud-init.patch.qcow2 --upload $i:/root/$i; done
virt-customize -a rhel6-cloud-init.patch.qcow2 -v --run-command 'rm -Rf /usr/lib/python2.6/site-packages/cloudinit ; tar -xzf /root/cloudinit.0.7.9.patch.tar.gz -C /usr/lib/python2.6/site-packages/'
glance image-create --name rhel6-cloud-init --file rhel6-cloud-init.patch.qcow2 --container-format bare --disk-format qcow2 --progress
nova boot --nic net-id=$NETID1 --nic net-id=$NETID2 --nic net-id=$NETID3 --nic net-id=$NETID4 --image rhel6-cloud-init --flavor m1.small --key-name id_rsa rhel6-cloud-init
~~~

Unfortunately, for RHEL 6, this is not working.

[Andreas Karis (akaris)]

* not working in the sense that: cloud-init is not running for me, my test is flawed, and so far I cannot test this. I'll check with lars about how we can test this.

[Lars Kellogg-Stedman (larsks)]

Just chiming in to say that this seems to work great under RHEL 7 for both plain ipv4 and mixed ipv4/ipv6 environments, with config drive or without. I haven't tested under EL6.

[Scott Moser (smoser)]

Thanks for the input.
We can't break rhel/centos 5 and 6 though.

On May 22, 2017 5:12:13 PM EDT, Lars Kellogg-Stedman <email address hidden> wrote:
>Just chiming in to say that this seems to work great under RHEL 7 for
>both plain ipv4 and mixed ipv4/ipv6 environments, with config drive or
>without. I haven't tested under EL6.
>--
>https://code.launchpad.net/~akaris/cloud-init/+git/cloud-init/+merge/324196
>You are reviewing the proposed merge of ~akaris/cloud-init:bug1679817-c
>into cloud-init:master.

[Andreas Karis (akaris)]

Hi,

I think we can at least forget about CentOS/RHEL 5:
https://wiki.centos.org/FAQ/General#head-fe8a0be91ee3e7dea812e8694491e1dde5b75e6d
https://access.redhat.com/support/policy/updates/errata

They went both end of production phase 3 at March 31, 2017

Remains testing for 6 to do

- Andreas

[Scott Moser (smoser)]

Andreas,
thanks for the RHEL 5 info, II agree.

[Lars Kellogg-Stedman (larsks)]

I have tested this under RHEL 6 and it seems to work just fine. It produces a more correct network configuration than does the current master, and it runs without producing any errors.

Setup for testing:

- booted with a recent rhel-guest-image
- removed cloud-init via 'yum -y remove cloud-init'
- cloned cloud-init repository from https://git.launchpad.net/~akaris/cloud-init and checked out the "bug1679817-c" branch
- installed via "python setup.py install --init-system sysvinit"

Testing:

- rm -rf /etc/sysconfig/network-scripts/ifcfg-eth* /etc/resolv.conf /var/lib/cloud/* /var/log/cloud*
- cloud-init init --local
- cloud-init init

At this point, /etc/resolv.conf and /etc/sysconfig/network-scripts/ifcfg-eth0 are generated and correct, and interfaces are up and configured.

As akaris indicated, I don't think it makes sense to worry about RHEL5 given its current lifecycle stage.

[Scott Moser (smoser)]

Thanks. I'll pull this.

review: Approve

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:48bf21bcb6046f1a2d1d029d286dcbe9eaf4f62e
https://jenkins.ubuntu.com/server/job/cloud-init-ci/438/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/438
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/438
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/438
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/438
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/438

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/438/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

Hi,
I'm marking this as 'merged' based on the fact that the new merge proposal *is* merged.
(https://code.launchpad.net/~akaris/cloud-init/+git/cloud-init/+merge/324196)
Please move back to 'Needs Review' (and explain) if you think otherwise.

diff --git a/cloudinit/net/sysconfig.py b/cloudinit/net/sysconfig.py
index d981277..58c5713 100644
--- a/cloudinit/net/sysconfig.py
+++ b/cloudinit/net/sysconfig.py
@@ -59,6 +59,9 @@ class ConfigMap(object):
     def __setitem__(self, key, value):
         self._conf[key] = value
 
+    def __getitem__(self, key):
+        return self._conf[key]
+
     def drop(self, key):
         self._conf.pop(key, None)
 
@@ -83,7 +86,8 @@ class ConfigMap(object):
 class Route(ConfigMap):
     """Represents a route configuration."""
 
-    route_fn_tpl = '%(base)s/network-scripts/route-%(name)s'
+    route_fn_tpl_ipv4 = '%(base)s/network-scripts/route-%(name)s'
+    route_fn_tpl_ipv6 = '%(base)s/network-scripts/route6-%(name)s'
 
     def __init__(self, route_name, base_sysconf_dir):
         super(Route, self).__init__()
@@ -102,9 +106,58 @@ class Route(ConfigMap):
         return r
 
     @property
-    def path(self):
-        return self.route_fn_tpl % ({'base': self._base_sysconf_dir,
-                                     'name': self._route_name})
+    def path_ipv4(self):
+        return self.route_fn_tpl_ipv4 % ({'base': self._base_sysconf_dir,
+                                          'name': self._route_name})
+
+    @property
+    def path_ipv6(self):
+        return self.route_fn_tpl_ipv6 % ({'base': self._base_sysconf_dir,
+                                          'name': self._route_name})
+
+    def is_ipv6_route(self, address):
+        return ':' in address
+
+    def to_string(self, proto="ipv4"):
+        # only accept ipv4 and ipv6
+        if proto not in ['ipv4', 'ipv6']:
+            raise ValueError("Unknown protocol '%s'" % (str(proto)))
+        buf = six.StringIO()
+        buf.write(_make_header())
+        if self._conf:
+            buf.write("\n")
+        # need to reindex IPv4 addresses
+        # (because Route can contain a mix of IPv4 and IPv6)
+        reindex = -1
+        for key in sorted(self._conf.keys()):
+            if 'ADDRESS' in key:
+                index = key.replace('ADDRESS', '')
+                address_value = str(self._conf[key])
+                # only accept combinations:
+                # if proto ipv6 only display ipv6 routes
+                # if proto ipv4 only display ipv4 routes
+                # do not add ipv6 routes if proto is ipv4
+                # do not add ipv4 routes if proto is ipv6
+                # (this array will contain a mix of ipv4 and ipv6)
+                if proto == "ipv4" and not self.is_ipv6_route(address_value):
+                    netmask_value = str(self._conf['NETMASK' + index])
+                    gateway_value = str(self._conf['GATEWAY' + index])
+                    # increase IPv4 index
+                    reindex = reindex + 1
+                    buf.write("%s=%s\n" % ('ADDRESS' + str(reindex),
+                                           _quote_value(address_value)))
+                    buf.write("%s=%s\n" % ('GATEWAY' + str(reindex),
+                                           _quote_value(gateway_value)))
+                    buf.write("%s=%s\n" % ('NETMASK' + str(reindex),
+                                           _quote_value(netmask_value)))
+                elif proto == "ipv6" and self.is_ipv6_route(address_value):
+                    netmask_value = str(self._conf['NETMASK' + index])
+                    gateway_value = str(self._conf['GATEWAY' + index])
+                    buf.write("%s/%s via %s\n" % (address_value,
+                                                  netmask_value,
+                                                  gateway_value))
+
+        return buf.getvalue()
 
 
 class NetInterface(ConfigMap):
@@ -211,65 +264,119 @@ class Renderer(renderer.Renderer):
                 iface_cfg[new_key] = old_value
 
     @classmethod
-    def _render_subnet(cls, iface_cfg, route_cfg, subnet):
-        subnet_type = subnet.get('type')
-        if subnet_type == 'dhcp6':
-            iface_cfg['DHCPV6C'] = True
-            iface_cfg['IPV6INIT'] = True
-            iface_cfg['BOOTPROTO'] = 'dhcp'
-        elif subnet_type in ['dhcp4', 'dhcp']:
-            iface_cfg['BOOTPROTO'] = 'dhcp'
-        elif subnet_type == 'static':
-            iface_cfg['BOOTPROTO'] = 'static'
-            if subnet_is_ipv6(subnet):
-                iface_cfg['IPV6ADDR'] = subnet['address']
+    def _render_subnets(cls, iface_cfg, subnets):
+        # setting base values
+        iface_cfg['BOOTPROTO'] = 'none'
+
+        # modifying base values according to subnets
+        for i, subnet in enumerate(subnets, start=len(iface_cfg.children)):
+            subnet_type = subnet.get('type')
+            if subnet_type == 'dhcp6':
                 iface_cfg['IPV6INIT'] = True
+                iface_cfg['DHCPV6C'] = True
+                iface_cfg['BOOTPROTO'] = 'dhcp'
+            elif subnet_type in ['dhcp4', 'dhcp']:
+                iface_cfg['BOOTPROTO'] = 'dhcp'
+            elif subnet_type == 'static':
+                # grep BOOTPROTO sysconfig.txt -A2 | head -3
+                # BOOTPROTO=none|bootp|dhcp
+                # 'bootp' or 'dhcp' cause a DHCP client
+                # to run on the device. Any other
+                # value causes any static configuration
+                # in the file to be applied.
+                # ==> the following should not be set to 'static'
+                # but should remain 'none'
+                # if iface_cfg['BOOTPROTO'] == 'none':
+                #    iface_cfg['BOOTPROTO'] = 'static'
+                if subnet_is_ipv6(subnet):
+                    iface_cfg['IPV6INIT'] = True
             else:
-                iface_cfg['IPADDR'] = subnet['address']
-        else:
-            raise ValueError("Unknown subnet type '%s' found"
-                             " for interface '%s'" % (subnet_type,
-                                                      iface_cfg.name))
-        if 'netmask' in subnet:
-            iface_cfg['NETMASK'] = subnet['netmask']
-        is_ipv6 = subnet.get('ipv6')
-        for route in subnet.get('routes', []):
-            if _is_default_route(route):
-                if (
-                        (subnet.get('ipv4') and
-                         route_cfg.has_set_default_ipv4) or
-                        (subnet.get('ipv6') and
-                         route_cfg.has_set_default_ipv6)
-                ):
-                    raise ValueError("Duplicate declaration of default "
-                                     "route found for interface '%s'"
-                                     % (iface_cfg.name))
-                # NOTE(harlowja): ipv6 and ipv4 default gateways
-                gw_key = 'GATEWAY0'
-                nm_key = 'NETMASK0'
-                addr_key = 'ADDRESS0'
-                # The owning interface provides the default route.
-                #
-                # TODO(harlowja): add validation that no other iface has
-                # also provided the default route?
-                iface_cfg['DEFROUTE'] = True
-                if 'gateway' in route:
-                    if is_ipv6:
-                        iface_cfg['IPV6_DEFAULTGW'] = route['gateway']
-                        route_cfg.has_set_default_ipv6 = True
+                raise ValueError("Unknown subnet type '%s' found"
+                                 " for interface '%s'" % (subnet_type,
+                                                          iface_cfg.name))
+
+        # set IPv4 and IPv6 static addresses
+        ipv4_index = -1
+        ipv6_index = -1
+        for i, subnet in enumerate(subnets, start=len(iface_cfg.children)):
+            subnet_type = subnet.get('type')
+            if subnet_type == 'dhcp6':
+                continue
+            elif subnet_type in ['dhcp4', 'dhcp']:
+                continue
+            elif subnet_type == 'static':
+                if subnet_is_ipv6(subnet):
+                    ipv6_index = ipv6_index + 1
+                    if 'netmask' in subnet and str(subnet['netmask']) != "":
+                        ipv6_cidr = (subnet['address'] +
+                                     '/' +
+                                     str(subnet['netmask']))
                     else:
-                        iface_cfg['GATEWAY'] = route['gateway']
-                        route_cfg.has_set_default_ipv4 = True
-            else:
-                gw_key = 'GATEWAY%s' % route_cfg.last_idx
-                nm_key = 'NETMASK%s' % route_cfg.last_idx
-                addr_key = 'ADDRESS%s' % route_cfg.last_idx
-                route_cfg.last_idx += 1
-            for (old_key, new_key) in [('gateway', gw_key),
-                                       ('netmask', nm_key),
-                                       ('network', addr_key)]:
-                if old_key in route:
-                    route_cfg[new_key] = route[old_key]
+                        ipv6_cidr = subnet['address']
+                    if ipv6_index == 0:
+                        iface_cfg['IPV6ADDR'] = ipv6_cidr
+                    elif ipv6_index == 1:
+                        iface_cfg['IPV6ADDR_SECONDARIES'] = ipv6_cidr
+                    else:
+                        iface_cfg['IPV6ADDR_SECONDARIES'] = (
+                            iface_cfg['IPV6ADDR_SECONDARIES'] +
+                            " " + ipv6_cidr)
+                else:
+                    ipv4_index = ipv4_index + 1
+                    if ipv4_index == 0:
+                        iface_cfg['IPADDR'] = subnet['address']
+                        if 'netmask' in subnet:
+                            iface_cfg['NETMASK'] = subnet['netmask']
+                    else:
+                        iface_cfg['IPADDR' + str(ipv4_index)] = \
+                            subnet['address']
+                        if 'netmask' in subnet:
+                            iface_cfg['NETMASK' + str(ipv4_index)] = \
+                                subnet['netmask']
+
+    @classmethod
+    def _render_subnet_routes(cls, iface_cfg, route_cfg, subnets):
+        for i, subnet in enumerate(subnets, start=len(iface_cfg.children)):
+            for route in subnet.get('routes', []):
+                is_ipv6 = subnet.get('ipv6')
+
+                if _is_default_route(route):
+                    if (
+                            (subnet.get('ipv4') and
+                             route_cfg.has_set_default_ipv4) or
+                            (subnet.get('ipv6') and
+                             route_cfg.has_set_default_ipv6)
+                    ):
+                        raise ValueError("Duplicate declaration of default "
+                                         "route found for interface '%s'"
+                                         % (iface_cfg.name))
+                    # NOTE(harlowja): ipv6 and ipv4 default gateways
+                    gw_key = 'GATEWAY0'
+                    nm_key = 'NETMASK0'
+                    addr_key = 'ADDRESS0'
+                    # The owning interface provides the default route.
+                    #
+                    # TODO(harlowja): add validation that no other iface has
+                    # also provided the default route?
+                    iface_cfg['DEFROUTE'] = True
+                    if 'gateway' in route:
+                        if is_ipv6:
+                            iface_cfg['IPV6_DEFAULTGW'] = route['gateway']
+                            route_cfg.has_set_default_ipv6 = True
+                        else:
+                            iface_cfg['GATEWAY'] = route['gateway']
+                            route_cfg.has_set_default_ipv4 = True
+
+                else:
+                    gw_key = 'GATEWAY%s' % route_cfg.last_idx
+                    nm_key = 'NETMASK%s' % route_cfg.last_idx
+                    addr_key = 'ADDRESS%s' % route_cfg.last_idx
+                    route_cfg.last_idx += 1
+                for (old_key, new_key) in [('gateway', gw_key),
+                                           ('netmask', nm_key),
+                                           ('network', addr_key)]:
+                    if old_key in route:
+                        route_cfg[new_key] = route[old_key]
 
     @classmethod
     def _render_bonding_opts(cls, iface_cfg, iface):
@@ -295,15 +402,9 @@ class Renderer(renderer.Renderer):
             iface_subnets = iface.get("subnets", [])
             iface_cfg = iface_contents[iface_name]
             route_cfg = iface_cfg.routes
-            if len(iface_subnets) == 1:
-                cls._render_subnet(iface_cfg, route_cfg, iface_subnets[0])
-            elif len(iface_subnets) > 1:
-                for i, isubnet in enumerate(iface_subnets,
-                                            start=len(iface_cfg.children)):
-                    iface_sub_cfg = iface_cfg.copy()
-                    iface_sub_cfg.name = "%s:%s" % (iface_name, i)
-                    iface_cfg.children.append(iface_sub_cfg)
-                    cls._render_subnet(iface_sub_cfg, route_cfg, isubnet)
+
+            cls._render_subnets(iface_cfg, iface_subnets)
+            cls._render_subnet_routes(iface_cfg, route_cfg, iface_subnets)
 
     @classmethod
     def _render_bond_interfaces(cls, network_state, iface_contents):
@@ -387,7 +488,10 @@ class Renderer(renderer.Renderer):
                     if iface_cfg:
                         contents[iface_cfg.path] = iface_cfg.to_string()
             if iface_cfg.routes:
-                contents[iface_cfg.routes.path] = iface_cfg.routes.to_string()
+                contents[iface_cfg.routes.path_ipv4] = \
+                    iface_cfg.routes.to_string("ipv4")
+                contents[iface_cfg.routes.path_ipv6] = \
+                    iface_cfg.routes.to_string("ipv6")
         return contents
 
     def render_network_state(self, network_state, target=None):
diff --git a/tests/unittests/test_distros/test_netconfig.py b/tests/unittests/test_distros/test_netconfig.py
index 1e10a33..fd7c051 100644
--- a/tests/unittests/test_distros/test_netconfig.py
+++ b/tests/unittests/test_distros/test_netconfig.py
@@ -476,7 +476,7 @@ NETWORKING=yes
             expected_buf = '''
 # Created by cloud-init on instance boot automatically, do not edit.
 #
-BOOTPROTO=static
+BOOTPROTO=none
 DEVICE=eth0
 IPADDR=192.168.1.5
 NETMASK=255.255.255.0
@@ -533,7 +533,6 @@ NETWORKING=yes
                 mock.patch.object(util, 'load_file', return_value=''))
             mocks.enter_context(
                 mock.patch.object(os.path, 'isfile', return_value=False))
-
             rh_distro.apply_network(BASE_NET_CFG_IPV6, False)
 
             self.assertEqual(len(write_bufs), 4)
@@ -626,11 +625,10 @@ IPV6_AUTOCONF=no
             expected_buf = '''
 # Created by cloud-init on instance boot automatically, do not edit.
 #
-BOOTPROTO=static
+BOOTPROTO=none
 DEVICE=eth0
-IPV6ADDR=2607:f0d0:1002:0011::2
+IPV6ADDR=2607:f0d0:1002:0011::2/64
 IPV6INIT=yes
-NETMASK=64
 NM_CONTROLLED=no
 ONBOOT=yes
 TYPE=Ethernet
diff --git a/tests/unittests/test_net.py b/tests/unittests/test_net.py
index d36d0e7..cf4cedc 100644
--- a/tests/unittests/test_net.py
+++ b/tests/unittests/test_net.py
@@ -137,7 +137,7 @@ OS_SAMPLES = [
              """
 # Created by cloud-init on instance boot automatically, do not edit.
 #
-BOOTPROTO=static
+BOOTPROTO=none
 DEFROUTE=yes
 DEVICE=eth0
 GATEWAY=172.19.3.254
@@ -205,38 +205,14 @@ nameserver 172.19.0.12
 # Created by cloud-init on instance boot automatically, do not edit.
 #
 BOOTPROTO=none
-DEVICE=eth0
-HWADDR=fa:16:3e:ed:9a:59
-NM_CONTROLLED=no
-ONBOOT=yes
-TYPE=Ethernet
-USERCTL=no
-""".lstrip()),
-            ('etc/sysconfig/network-scripts/ifcfg-eth0:0',
-             """
-# Created by cloud-init on instance boot automatically, do not edit.
-#
-BOOTPROTO=static
 DEFROUTE=yes
-DEVICE=eth0:0
+DEVICE=eth0
 GATEWAY=172.19.3.254
 HWADDR=fa:16:3e:ed:9a:59
 IPADDR=172.19.1.34
+IPADDR1=10.0.0.10
 NETMASK=255.255.252.0
-NM_CONTROLLED=no
-ONBOOT=yes
-TYPE=Ethernet
-USERCTL=no
-""".lstrip()),
-            ('etc/sysconfig/network-scripts/ifcfg-eth0:1',
-             """
-# Created by cloud-init on instance boot automatically, do not edit.
-#
-BOOTPROTO=static
-DEVICE=eth0:1
-HWADDR=fa:16:3e:ed:9a:59
-IPADDR=10.0.0.10
-NETMASK=255.255.255.0
+NETMASK1=255.255.255.0
 NM_CONTROLLED=no
 ONBOOT=yes
 TYPE=Ethernet
@@ -266,7 +242,7 @@ nameserver 172.19.0.12
                 }],
                 "ip_address": "172.19.1.34", "id": "network0"
             }, {
-                "network_id": "public-ipv6",
+                "network_id": "public-ipv6-a",
                 "type": "ipv6", "netmask": "",
                 "link": "tap1a81968a-79",
                 "routes": [
@@ -277,6 +253,20 @@ nameserver 172.19.0.12
                     }
                 ],
                 "ip_address": "2001:DB8::10", "id": "network1"
+            }, {
+                "network_id": "public-ipv6-b",
+                "type": "ipv6", "netmask": "64",
+                "link": "tap1a81968a-79",
+                "routes": [
+                ],
+                "ip_address": "2001:DB9::10", "id": "network2"
+            }, {
+                "network_id": "public-ipv6-c",
+                "type": "ipv6", "netmask": "64",
+                "link": "tap1a81968a-79",
+                "routes": [
+                ],
+                "ip_address": "2001:DB10::10", "id": "network3"
             }],
             "links": [
                 {
@@ -296,41 +286,16 @@ nameserver 172.19.0.12
 # Created by cloud-init on instance boot automatically, do not edit.
 #
 BOOTPROTO=none
-DEVICE=eth0
-HWADDR=fa:16:3e:ed:9a:59
-NM_CONTROLLED=no
-ONBOOT=yes
-TYPE=Ethernet
-USERCTL=no
-""".lstrip()),
-            ('etc/sysconfig/network-scripts/ifcfg-eth0:0',
-             """
-# Created by cloud-init on instance boot automatically, do not edit.
-#
-BOOTPROTO=static
 DEFROUTE=yes
-DEVICE=eth0:0
+DEVICE=eth0
 GATEWAY=172.19.3.254
 HWADDR=fa:16:3e:ed:9a:59
 IPADDR=172.19.1.34
-NETMASK=255.255.252.0
-NM_CONTROLLED=no
-ONBOOT=yes
-TYPE=Ethernet
-USERCTL=no
-""".lstrip()),
-            ('etc/sysconfig/network-scripts/ifcfg-eth0:1',
-             """
-# Created by cloud-init on instance boot automatically, do not edit.
-#
-BOOTPROTO=static
-DEFROUTE=yes
-DEVICE=eth0:1
-HWADDR=fa:16:3e:ed:9a:59
 IPV6ADDR=2001:DB8::10
+IPV6ADDR_SECONDARIES="2001:DB9::10/64 2001:DB10::10/64"
 IPV6INIT=yes
 IPV6_DEFAULTGW=2001:DB8::1
-NETMASK=
+NETMASK=255.255.252.0
 NM_CONTROLLED=no
 ONBOOT=yes
 TYPE=Ethernet